Trace number 3378234

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2015.98 509.333

General information on the benchmark

Namerandom/large/
unif-k7-r85-v400-c34000-S817927155-033-UNKNOWN.cnf
MD5SUMbdd02c131ae0dd9fa14ba01f07a63379
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables400
Number of clauses34000
Sum of the clauses size238000
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 534000

Solver Data 

0.31/0.67	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.67	c INFO: solving HOME/instance-3378234-1305463979.cnf
0.31/0.67	c INFO: computing task features
0.59/0.90	c INFO: computed variable-clause graph statistics
0.59/0.91	c INFO: computed clause balance statistics
1.80/2.13	c INFO: computed variable balance statistics
2.28/2.61	c INFO: computed small-clause counts
2.28/2.61	c INFO: computed Horn-clause counts
2.28/2.65	c INFO: computed variable graph statistics
2.28/2.65	c INFO: collected features for HOME/instance-3378234-1305463979.cnf in 1.97 s
2.28/2.66	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.38)
2.28/2.67	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
2.28/2.68	c INFO: running adaptg2wsat2009++@0 for 97 with 1197 remaining (b = 0.08)
2.28/2.69	c INFO: running adaptg2wsat2009++@0 for 97 with 1197 remaining (b = 0.07)
401.78/102.98	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.05)
402.17/103.00	c INFO: running gnovelty+2@0 for 97 with 1097 remaining (b = 0.04)
402.17/103.02	c INFO: running FH@0 for 97 with 1097 remaining (b = 0.03)
402.30/103.11	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.03)
800.99/203.30	c INFO: running adaptg2wsat2009++@0 for 97 with 997 remaining (b = 0.02)
800.99/203.38	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
800.99/203.39	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
801.40/203.43	c INFO: running gnovelty+2@0 for 97 with 997 remaining (b = 0.02)
1199.77/303.62	c INFO: running hybridGM3@0 for 194 with 897 remaining (b = 0.03)
1200.04/303.70	c INFO: running adaptg2wsat2009++@0 for 97 with 896 remaining (b = 0.01)
1220.39/308.88	c INFO: running hybridGM3@0 for 194 with 891 remaining (b = 0.03)
1583.18/400.12	c INFO: running hybridGM3@99 for 97 with 800 remaining (b = 0.01)
1598.66/404.04	c INFO: running hybridGM3@0 for 194 with 796 remaining (b = 0.02)
1985.00/501.42	c INFO: running TNM@0 for 699 with 699 remaining (b = 0.08)
1996.47/504.38	c INFO: running hybridGM3@99 for 97 with 696 remaining (b = 0.01)
2013.52/508.65	c INFO: running FH@0 for 97 with 692 remaining (b = 0.01)
2015.98/509.32	c INFO: running hybridGM3@0 for 194 with 691 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378234-1305463979/watcher-3378234-1305463979 -o /tmp/evaluation-result-3378234-1305463979/solver-3378234-1305463979 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378234-1305463979.cnf -seed 1925460585 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.93 9.01 7.61 7/420 28847
/proc/meminfo: memFree=16664176/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=15908 CPUtime=0 cores=0,2,4,6
/proc/28847/stat : 28847 (python) R 28845 28847 28149 0 -1 4202496 944 0 0 0 0 0 0 0 25 0 1 0 657995896 16289792 821 33554432000 4194304 5683644 140733922881808 18446744073709551615 4681245 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/28847/statm: 3977 821 357 364 0 464 0

[startup+0.0439441 s]
/proc/loadavg: 8.93 9.01 7.61 7/420 28847
/proc/meminfo: memFree=16664176/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=18884 CPUtime=0.02 cores=0,2,4,6
/proc/28847/stat : 28847 (python) R 28845 28847 28149 0 -1 4202496 1284 0 0 0 2 0 0 0 25 0 1 0 657995896 19337216 1137 33554432000 4194304 5683644 140733922881808 18446744073709551615 4876492 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/28847/statm: 4721 1137 421 364 0 690 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 18884

[startup+0.100951 s]
/proc/loadavg: 8.93 9.01 7.61 7/420 28847
/proc/meminfo: memFree=16664176/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=89768 CPUtime=0.05 cores=0,2,4,6
/proc/28847/stat : 28847 (python) S 28845 28847 28149 0 -1 4202496 2672 0 0 0 4 1 0 0 25 0 1 0 657995896 91922432 2388 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244346880 0 2147483391 16781312 2 18446744071699199166 0 0 17 2 0 0 0
/proc/28847/statm: 22442 2388 750 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 89768

[startup+0.300946 s]
/proc/loadavg: 8.93 9.01 7.61 7/420 28847
/proc/meminfo: memFree=16664176/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=135748 CPUtime=0.17 cores=0,2,4,6
/proc/28847/stat : 28847 (python) R 28845 28847 28149 0 -1 4202496 5962 934 1 0 13 4 0 0 17 0 1 0 657995896 139005952 4151 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244041424 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/28847/statm: 33937 4151 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135748

[startup+0.700831 s]
/proc/loadavg: 8.93 9.01 7.61 7/420 28847
/proc/meminfo: memFree=16664176/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=279984 CPUtime=0.39 cores=0,2,4,6
/proc/28847/stat : 28847 (python) R 28845 28847 28149 0 -1 4202496 9650 934 1 0 30 9 0 0 16 0 1 0 657995896 286703616 7333 33554432000 4194304 5683644 140733922881808 18446744073709551615 47458158287846 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/28847/statm: 69996 7334 1761 364 0 5920 0
Current children cumulated CPU time (s) 0.39
Current children cumulated vsize (KiB) 279984

[startup+1.50069 s]
/proc/loadavg: 8.93 9.01 7.61 7/421 28852
/proc/meminfo: memFree=16630192/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=289500 CPUtime=1.18 cores=0,2,4,6
/proc/28847/stat : 28847 (python) R 28845 28847 28149 0 -1 4202496 13882 934 1 0 109 9 0 0 17 0 1 0 657995896 296448000 9760 33554432000 4194304 5683644 140733922881808 18446744073709551615 47457987575490 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/28847/statm: 72375 9760 1826 364 0 8299 0
Current children cumulated CPU time (s) 1.18
Current children cumulated vsize (KiB) 289500

[startup+3.10037 s]
/proc/loadavg: 8.93 9.01 7.61 7/421 28852
/proc/meminfo: memFree=16629820/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=322836 CPUtime=2.38 cores=0,2,4,6
/proc/28847/stat : 28847 (python) S 28845 28847 28149 0 -1 4202496 16634 934 1 0 228 10 0 0 22 0 5 0 657995896 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/28847/statm: 80709 7925 1883 364 0 16618 0
Current children cumulated CPU time (s) 2.38
Current children cumulated vsize (KiB) 322836
heavy processes:

[startup+6.30081 s]
/proc/loadavg: 8.94 9.01 7.62 10/433 28864
/proc/meminfo: memFree=16602692/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=322836 CPUtime=2.38 cores=0,2,4,6
/proc/28847/stat : 28847 (python) S 28845 28847 28149 0 -1 4202496 16634 934 1 0 228 10 0 0 22 0 5 0 657995896 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/28847/statm: 80709 7925 1883 364 0 16618 0
[pid=28853] ppid=28847 vsize=282240 CPUtime=0 cores=0,2,4,6
/proc/28853/stat : 28853 (python) S 28847 28847 28149 0 -1 4202560 726 0 0 0 0 0 0 0 18 0 1 0 657996162 289013760 6332 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368911 0 0 16781312 514 14086539733471559938 0 0 17 2 0 0 0
/proc/28853/statm: 70560 6332 355 364 0 6478 0
[pid=28855] ppid=28853 vsize=1020188 CPUtime=3.56 cores=0,2,4,6
/proc/28855/stat : 28855 (gnovelty+2) R 28853 28855 28855 0 -1 4202496 1139 0 0 0 356 0 0 0 25 0 1 0 657996163 1044672512 877 33554432000 134512640 135663704 4292358800 18446744073709551615 134524882 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/28855/statm: 255047 877 113 282 0 254763 0
[pid=28856] ppid=28847 vsize=293172 CPUtime=0 cores=0,2,4,6
/proc/28856/stat : 28856 (python) S 28847 28847 28149 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 657996163 300208128 6380 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 1076513422230651137 0 0 17 4 0 0 0
/proc/28856/statm: 73293 6380 349 364 0 9208 0
[pid=28858] ppid=28856 vsize=1020188 CPUtime=3.61 cores=0,2,4,6
/proc/28858/stat : 28858 (gnovelty+2) R 28856 28858 28858 0 -1 4202496 1139 0 0 0 361 0 0 0 25 0 1 0 657996164 1044672512 877 33554432000 134512640 135663704 4293482368 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/28858/statm: 255047 877 113 282 0 254763 0
[pid=28859] ppid=28847 vsize=302888 CPUtime=0 cores=0,2,4,6
/proc/28859/stat : 28859 (python) S 28847 28847 28149 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 657996164 310157312 6384 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 4168868434873909505 0 0 17 2 0 0 0
/proc/28859/statm: 75722 6384 349 364 0 11634 0
[pid=28861] ppid=28859 vsize=421140 CPUtime=3.6 cores=0,2,4,6
/proc/28861/stat : 28861 (adaptg2wsat2009) R 28859 28861 28861 0 -1 4202496 1804 0 0 0 360 0 0 0 25 0 1 0 657996165 431247360 1540 33554432000 134512640 135107411 4293075104 18446744073709551615 134523285 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/28861/statm: 105285 1540 49 146 0 104161 0
[pid=28862] ppid=28847 vsize=313020 CPUtime=0 cores=0,2,4,6
/proc/28862/stat : 28862 (python) S 28847 28847 28149 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 657996165 320532480 6388 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 17889886518512681220 0 0 17 2 0 0 0
/proc/28862/statm: 78255 6388 349 364 0 14164 0
[pid=28864] ppid=28862 vsize=421140 CPUtime=3.58 cores=0,2,4,6
/proc/28864/stat : 28864 (adaptg2wsat2009) R 28862 28864 28864 0 -1 4202496 1801 0 0 0 358 0 0 0 25 0 1 0 657996166 431247360 1539 33554432000 134512640 135107411 4289897408 18446744073709551615 134523516 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/28864/statm: 105285 1539 49 146 0 104161 0
[pid=28847/tid=28854] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28854/stat : 28854 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 657996162 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28857] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28857/stat : 28857 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 23 0 5 0 657996163 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28860] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28860/stat : 28860 (python) S 28845 28847 28149 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 5 0 657996164 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 2 0 0 0
[pid=28847/tid=28863] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28863/stat : 28863 (python) S 28845 28847 28149 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 5 0 657996166 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 16.73
Current children cumulated vsize (KiB) 4396812

[startup+12.7007 s]
/proc/loadavg: 8.94 9.01 7.63 11/433 28864
/proc/meminfo: memFree=16598512/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=322836 CPUtime=2.38 cores=0,2,4,6
/proc/28847/stat : 28847 (python) S 28845 28847 28149 0 -1 4202496 16634 934 1 0 228 10 0 0 22 0 5 0 657995896 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/28847/statm: 80709 7925 1883 364 0 16618 0
[pid=28853] ppid=28847 vsize=282240 CPUtime=0.01 cores=0,2,4,6
/proc/28853/stat : 28853 (python) S 28847 28847 28149 0 -1 4202560 885 0 0 0 1 0 0 0 18 0 1 0 657996162 289013760 6355 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368911 0 0 16781312 514 14086539733471559938 0 0 17 2 0 0 0
/proc/28853/statm: 70560 6355 378 364 0 6478 0
[pid=28855] ppid=28853 vsize=1020188 CPUtime=9.87 cores=0,2,4,6
/proc/28855/stat : 28855 (gnovelty+2) R 28853 28855 28855 0 -1 4202496 1139 0 0 0 987 0 0 0 25 0 1 0 657996163 1044672512 877 33554432000 134512640 135663704 4292358800 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/28855/statm: 255047 877 113 282 0 254763 0
[pid=28856] ppid=28847 vsize=293172 CPUtime=0.01 cores=0,2,4,6
/proc/28856/stat : 28856 (python) S 28847 28847 28149 0 -1 4202560 901 0 0 0 0 1 0 0 18 0 1 0 657996163 300208128 6403 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 1076513422230651137 0 0 17 4 0 0 0
/proc/28856/statm: 73293 6403 372 364 0 9208 0
[pid=28858] ppid=28856 vsize=1020188 CPUtime=10.01 cores=0,2,4,6
/proc/28858/stat : 28858 (gnovelty+2) R 28856 28858 28858 0 -1 4202496 1139 0 0 0 1001 0 0 0 25 0 1 0 657996164 1044672512 877 33554432000 134512640 135663704 4293482368 18446744073709551615 134524498 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/28858/statm: 255047 877 113 282 0 254763 0
[pid=28859] ppid=28847 vsize=302888 CPUtime=0 cores=0,2,4,6
/proc/28859/stat : 28859 (python) S 28847 28847 28149 0 -1 4202560 898 0 0 0 0 0 0 0 18 0 1 0 657996164 310157312 6407 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 4168868434873909505 0 0 17 2 0 0 0
/proc/28859/statm: 75722 6407 372 364 0 11634 0
[pid=28861] ppid=28859 vsize=421804 CPUtime=9.99 cores=0,2,4,6
/proc/28861/stat : 28861 (adaptg2wsat2009) R 28859 28861 28861 0 -1 4202496 2137 0 0 0 999 0 0 0 25 0 1 0 657996165 431927296 1873 33554432000 134512640 135107411 4293075104 18446744073709551615 134531696 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/28861/statm: 105451 1873 59 146 0 104327 0
[pid=28862] ppid=28847 vsize=313020 CPUtime=0.01 cores=0,2,4,6
/proc/28862/stat : 28862 (python) S 28847 28847 28149 0 -1 4202560 904 0 0 0 1 0 0 0 18 0 1 0 657996165 320532480 6411 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 17889886518512681220 0 0 17 2 0 0 0
/proc/28862/statm: 78255 6411 372 364 0 14164 0
[pid=28864] ppid=28862 vsize=421804 CPUtime=9.92 cores=0,2,4,6
/proc/28864/stat : 28864 (adaptg2wsat2009) R 28862 28864 28864 0 -1 4202496 2134 0 0 0 992 0 0 0 25 0 1 0 657996166 431927296 1872 33554432000 134512640 135107411 4289897408 18446744073709551615 134531656 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/28864/statm: 105451 1872 59 146 0 104327 0
[pid=28847/tid=28854] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28854/stat : 28854 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 657996162 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28857] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28857/stat : 28857 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 23 0 5 0 657996163 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28860] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28860/stat : 28860 (python) S 28845 28847 28149 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 5 0 657996164 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 2 0 0 0
[pid=28847/tid=28863] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28863/stat : 28863 (python) S 28845 28847 28149 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 5 0 657996166 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 42.2
Current children cumulated vsize (KiB) 4398140

[startup+25.5014 s]
/proc/loadavg: 8.96 9.01 7.65 10/433 28865
/proc/meminfo: memFree=16599628/32951124 swapFree=67082152/67111528
[pid=28847] ppid=28845 vsize=322836 CPUtime=2.38 cores=0,2,4,6
/proc/28847/stat : 28847 (python) S 28845 28847 28149 0 -1 4202496 16634 934 1 0 228 10 0 0 22 0 5 0 657995896 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/28847/statm: 80709 7925 1883 364 0 16618 0
[pid=28853] ppid=28847 vsize=282240 CPUtime=0.07 cores=0,2,4,6
/proc/28853/stat : 28853 (python) S 28847 28847 28149 0 -1 4202560 896 0 0 0 4 3 0 0 18 0 1 0 657996162 289013760 6355 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368911 0 0 16781312 514 12209821539754803458 0 0 17 2 0 0 0
/proc/28853/statm: 70560 6355 378 364 0 6478 0
[pid=28855] ppid=28853 vsize=1020188 CPUtime=22.49 cores=0,2,4,6
/proc/28855/stat : 28855 (gnovelty+2) R 28853 28855 28855 0 -1 4202496 1139 0 0 0 2249 0 0 0 25 0 1 0 657996163 1044672512 877 33554432000 134512640 135663704 4292358800 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/28855/statm: 255047 877 113 282 0 254763 0
[pid=28856] ppid=28847 vsize=293172 CPUtime=0.06 cores=0,2,4,6
/proc/28856/stat : 28856 (python) S 28847 28847 28149 0 -1 4202560 910 0 0 0 2 4 0 0 18 0 1 0 657996163 300208128 6403 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 12209821539754803457 0 0 17 4 0 0 0
/proc/28856/statm: 73293 6403 372 364 0 9208 0
[pid=28858] ppid=28856 vsize=1020188 CPUtime=22.82 cores=0,2,4,6
/proc/28858/stat : 28858 (gnovelty+2) R 28856 28858 28858 0 -1 4202496 1139 0 0 0 2282 0 0 0 25 0 1 0 657996164 1044672512 877 33554432000 134512640 135663704 4293482368 18446744073709551615 134525286 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/28858/statm: 255047 877 113 282 0 254763 0
[pid=28859] ppid=28847 vsize=302888 CPUtime=0.06 cores=0,2,4,6
/proc/28859/stat : 28859 (python) S 28847 28847 28149 0 -1 4202560 904 0 0 0 3 3 0 0 18 0 1 0 657996164 310157312 6407 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 12209821539754803457 0 0 17 2 0 0 0
/proc/28859/statm: 75722 6407 372 364 0 11634 0
[pid=28861] ppid=28859 vsize=421804 CPUtime=22.73 cores=0,2,4,6
/proc/28861/stat : 28861 (adaptg2wsat2009) R 28859 28861 28861 0 -1 4202496 2137 0 0 0 2273 0 0 0 25 0 1 0 657996165 431927296 1873 33554432000 134512640 135107411 4293075104 18446744073709551615 134531239 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/28861/statm: 105451 1873 59 146 0 104327 0
[pid=28862] ppid=28847 vsize=313020 CPUtime=0.06 cores=0,2,4,6
/proc/28862/stat : 28862 (python) S 28847 28847 28149 0 -1 4202560 912 0 0 0 3 3 0 0 18 0 1 0 657996165 320532480 6411 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 12209821539754803460 0 0 17 2 0 0 0
/proc/28862/statm: 78255 6411 372 364 0 14164 0
[pid=28864] ppid=28862 vsize=421804 CPUtime=22.56 cores=0,2,4,6
/proc/28864/stat : 28864 (adaptg2wsat2009) R 28862 28864 28864 0 -1 4202496 2134 0 0 0 2256 0 0 0 25 0 1 0 657996166 431927296 1872 33554432000 134512640 135107411 4289897408 18446744073709551615 134531709 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/28864/statm: 105451 1872 59 146 0 104327 0
[pid=28847/tid=28854] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28854/stat : 28854 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 5 0 657996162 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28857] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28857/stat : 28857 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 23 0 5 0 657996163 330584064 7925 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28860] ppid=28845 vsize=322836 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

/proc/28862/stat : 28862 (python) S 28847 28847 28149 0 -1 4202560 1085 0 0 0 14 15 0 0 19 0 2 0 657996165 320532480 6440 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/28862/statm: 78255 6440 400 364 0 14164 0
[pid=28864] ppid=28862 vsize=421804 CPUtime=99.36 cores=0,2,4,6
/proc/28864/stat : 28864 (adaptg2wsat2009) T 28862 28864 28864 0 -1 4202496 2134 0 0 0 9936 0 0 0 25 0 1 0 657996166 431927296 1872 33554432000 134512640 135107411 4289897408 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/28864/statm: 105451 1872 59 146 0 104327 0
[pid=28862/tid=28872] ppid=28847 vsize=313020 CPUtime=0 cores=0,2,4,6
/proc/28862/task/28872/stat : 28872 (python) S 28847 28847 28149 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658006196 320532480 6440 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604444193105152 0 0 -1 4 0 0 0
[pid=28869] ppid=28847 vsize=323956 CPUtime=0.31 cores=0,2,4,6
/proc/28869/stat : 28869 (python) S 28847 28847 28149 0 -1 4202560 1101 0 0 0 16 15 0 0 19 0 2 0 658006195 331730944 6448 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/28869/statm: 80989 6448 400 364 0 16895 0
[pid=28871] ppid=28869 vsize=421804 CPUtime=98.01 cores=0,2,4,6
/proc/28871/stat : 28871 (adaptg2wsat2009) T 28869 28871 28871 0 -1 4202496 2137 0 0 0 9801 0 0 0 25 0 1 0 658006195 431927296 1873 33554432000 134512640 135107411 4294660896 18446744073709551615 134532135 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/28871/statm: 105451 1873 59 146 0 104327 0
[pid=28869/tid=28893] ppid=28847 vsize=323956 CPUtime=0 cores=0,2,4,6
/proc/28869/task/28893/stat : 28893 (python) S 28847 28847 28149 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658016225 331730944 6448 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604444193106048 0 0 -1 2 0 0 0
[pid=28873] ppid=28847 vsize=333728 CPUtime=0.3 cores=0,2,4,6
/proc/28873/stat : 28873 (python) S 28847 28847 28149 0 -1 4202560 1112 0 0 0 16 14 0 0 19 0 2 0 658006197 341737472 6586 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/28873/statm: 83432 6586 399 364 0 19335 0
[pid=28875] ppid=28873 vsize=1020188 CPUtime=99.99 cores=0,2,4,6
/proc/28875/stat : 28875 (gnovelty+2) T 28873 28875 28875 0 -1 4202496 1135 0 0 0 9999 0 0 0 25 0 1 0 658006197 1044672512 876 33554432000 134512640 135663704 4290949184 18446744073709551615 134525255 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/28875/statm: 255047 876 113 282 0 254763 0
[pid=28873/tid=28897] ppid=28847 vsize=333728 CPUtime=0 cores=0,2,4,6
/proc/28873/task/28897/stat : 28897 (python) S 28847 28847 28149 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 658016233 341737472 6586 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604446202030848 0 0 -1 4 0 0 0
[pid=28877] ppid=28847 vsize=344172 CPUtime=0.31 cores=0,2,4,6
/proc/28877/stat : 28877 (python) S 28847 28847 28149 0 -1 4202560 1101 0 0 0 18 13 0 0 19 0 2 0 658006199 352432128 6591 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/28877/statm: 86043 6591 400 364 0 21943 0
[pid=28879] ppid=28877 vsize=251292 CPUtime=100.33 cores=0,2,4,6
/proc/28879/stat : 28879 (FH) T 28877 28879 28879 0 -1 4202496 1953 0 0 0 10033 0 0 0 25 0 1 0 658006199 257323008 1693 33554432000 134512640 134936988 4288678896 18446744073709551615 134554229 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/28879/statm: 62823 1693 41 104 0 62327 0
[pid=28877/tid=28899] ppid=28847 vsize=344172 CPUtime=0 cores=0,2,4,6
/proc/28877/task/28899/stat : 28899 (python) S 28847 28847 28149 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658016234 352432128 6591 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604446202029056 0 0 -1 2 0 0 0
[pid=28881] ppid=28847 vsize=354424 CPUtime=0.31 cores=0,2,4,6
/proc/28881/stat : 28881 (python) S 28847 28847 28149 0 -1 4202560 1141 0 0 0 18 13 0 0 19 0 2 0 658006207 362930176 6601 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/28881/statm: 88606 6601 406 364 0 24503 0
[pid=28883] ppid=28881 vsize=421804 CPUtime=99.41 cores=0,2,4,6
/proc/28883/stat : 28883 (adaptg2wsat2009) T 28881 28883 28883 0 -1 4202496 2144 0 0 0 9941 0 0 0 25 0 1 0 658006207 431927296 1872 33554432000 134512640 135107411 4292249024 18446744073709551615 134531246 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/28883/statm: 105451 1872 59 146 0 104327 0
[pid=28881/tid=28903] ppid=28847 vsize=354424 CPUtime=0 cores=0,2,4,6
/proc/28881/task/28903/stat : 28903 (python) S 28847 28847 28149 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658016237 362930176 6601 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604446202028160 0 0 -1 4 0 0 0
[pid=28894] ppid=28847 vsize=364892 CPUtime=0.32 cores=0,2,4,6
/proc/28894/stat : 28894 (python) S 28847 28847 28149 0 -1 4202560 1154 0 0 0 17 15 0 0 19 0 2 0 658016226 373649408 6605 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/28894/statm: 91223 6605 406 364 0 27117 0
[pid=28896] ppid=28894 vsize=421804 CPUtime=100.24 cores=0,2,4,6
/proc/28896/stat : 28896 (adaptg2wsat2009) T 28894 28896 28896 0 -1 4202496 2141 0 0 0 10024 0 0 0 25 0 1 0 658016226 431927296 1873 33554432000 134512640 135107411 4289623088 18446744073709551615 134531435 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/28896/statm: 105451 1873 59 146 0 104327 0
[pid=28894/tid=28912] ppid=28847 vsize=364892 CPUtime=0 cores=0,2,4,6
/proc/28894/task/28912/stat : 28912 (python) S 28847 28847 28149 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658026257 373649408 6605 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604453915755456 0 0 -1 4 0 0 0
[pid=28898] ppid=28847 vsize=374932 CPUtime=0.32 cores=0,2,4,6
/proc/28898/stat : 28898 (python) S 28847 28847 28149 0 -1 4202560 1123 0 0 0 16 16 0 0 19 0 2 0 658016234 383930368 6603 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/28898/statm: 93733 6603 400 364 0 29624 0
[pid=28901] ppid=28898 vsize=251292 CPUtime=99.98 cores=0,2,4,6
/proc/28901/stat : 28901 (FH) T 28898 28901 28901 0 -1 4202496 1961 0 0 0 9998 0 0 0 25 0 1 0 658016234 257323008 1694 33554432000 134512640 134936988 4289019968 18446744073709551615 134552472 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/28901/statm: 62823 1694 41 104 0 62327 0
[pid=28898/tid=28916] ppid=28847 vsize=374932 CPUtime=0 cores=0,2,4,6
/proc/28898/task/28916/stat : 28916 (python) S 28847 28847 28149 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658026265 383930368 6603 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604441249573888 0 0 -1 0 0 0 0
[pid=28902] ppid=28847 vsize=385428 CPUtime=0.93 cores=0,2,4,6
/proc/28902/stat : 28902 (python) S 28847 28847 28149 0 -1 4202560 1186 0 0 0 48 45 0 0 18 0 2 0 658016235 394678272 6616 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/28902/statm: 96357 6616 409 364 0 32245 0
[pid=28905] ppid=28902 vsize=268428 CPUtime=293.36 cores=0,2,4,6
/proc/28905/stat : 28905 (hybridGM3) T 28902 28905 28905 0 -1 4202496 5645 0 0 0 29334 2 0 0 25 0 1 0 658016238 274870272 5378 33554432000 134512640 135204385 4294198848 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/28905/statm: 67107 5378 63 169 0 66937 0
[pid=28902/tid=28931] ppid=28847 vsize=385428 CPUtime=0 cores=0,2,4,6
/proc/28902/task/28931/stat : 28931 (python) S 28847 28847 28149 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 658035906 394678272 6616 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604453822293120 0 0 -1 4 0 0 0
[pid=28906] ppid=28847 vsize=395540 CPUtime=0.33 cores=0,2,4,6
/proc/28906/stat : 28906 (python) S 28847 28847 28149 0 -1 4202560 1140 0 0 0 16 17 0 0 19 0 2 0 658016239 405032960 6611 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/28906/statm: 98885 6611 400 364 0 34770 0
[pid=28908] ppid=28906 vsize=1020188 CPUtime=102.72 cores=0,2,4,6
/proc/28908/stat : 28908 (gnovelty+2) T 28906 28908 28908 0 -1 4202496 1142 0 0 0 10272 0 0 0 25 0 1 0 658016239 1044672512 877 33554432000 134512640 135663704 4287592656 18446744073709551615 134525684 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/28908/statm: 255047 877 113 282 0 254763 0
[pid=28906/tid=28920] ppid=28847 vsize=395540 CPUtime=0 cores=0,2,4,6
/proc/28906/task/28920/stat : 28920 (python) S 28847 28847 28149 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658026783 405032960 6611 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604448561986496 0 0 -1 4 0 0 0
[pid=28913] ppid=28847 vsize=405932 CPUtime=0.64 cores=0,2,4,6
/proc/28913/stat : 28913 (python) S 28847 28847 28149 0 -1 4202560 1177 0 0 0 33 31 0 0 18 0 2 0 658026259 415674368 6615 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/28913/statm: 101483 6615 400 364 0 37365 0
[pid=28915] ppid=28913 vsize=268420 CPUtime=201.83 cores=0,2,4,6
/proc/28915/stat : 28915 (hybridGM3) R 28913 28915 28915 0 -1 4202496 5643 0 0 0 20181 2 0 0 25 0 1 0 658026259 274862080 5377 33554432000 134512640 135204385 4294763136 18446744073709551615 134537699 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/28915/statm: 67105 5377 63 169 0 66935 0
[pid=28913/tid=29038] ppid=28847 vsize=405932 CPUtime=0 cores=0,2,4,6
/proc/28913/task/29038/stat : 29038 (python) S 28847 28847 28149 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658046333 415674368 6615 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604444155369408 0 0 -1 4 0 0 0
[pid=28917] ppid=28847 vsize=416328 CPUtime=0.31 cores=0,2,4,6
/proc/28917/stat : 28917 (python) S 28847 28847 28149 0 -1 4202560 1164 0 0 0 16 15 0 0 19 0 2 0 658026267 426319872 6619 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/28917/statm: 104082 6619 400 364 0 39961 0
[pid=28919] ppid=28917 vsize=421804 CPUtime=99.83 cores=0,2,4,6
/proc/28919/stat : 28919 (adaptg2wsat2009) T 28917 28919 28919 0 -1 4202496 2148 0 0 0 9983 0 0 0 25 0 1 0 658026267 431927296 1872 33554432000 134512640 135107411 4292002992 18446744073709551615 134531239 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/28919/statm: 105451 1872 59 146 0 104327 0
[pid=28917/tid=28932] ppid=28847 vsize=416328 CPUtime=0 cores=0,2,4,6
/proc/28917/task/28932/stat : 28932 (python) S 28847 28847 28149 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658036299 426319872 6619 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604444155370304 0 0 -1 6 0 0 0
[pid=28921] ppid=28847 vsize=426588 CPUtime=0.64 cores=0,2,4,6
/proc/28921/stat : 28921 (python) S 28847 28847 28149 0 -1 4202560 1232 0 0 0 35 29 0 0 19 0 2 0 658026785 436826112 6647 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/28921/statm: 106647 6647 407 364 0 42523 0
[pid=28923] ppid=28921 vsize=268420 CPUtime=199.74 cores=0,2,4,6
/proc/28923/stat : 28923 (hybridGM3) T 28921 28923 28923 0 -1 4202496 5652 0 0 0 19973 1 0 0 25 0 1 0 658026785 274862080 5377 33554432000 134512640 135204385 4293020688 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/28923/statm: 67105 5377 63 169 0 66935 0
[pid=28921/tid=29039] ppid=28847 vsize=426588 CPUtime=0 cores=0,2,4,6
/proc/28921/task/29039/stat : 29039 (python) S 28847 28847 28149 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658046760 436826112 6647 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 514 18446604449639770048 0 0 -1 0 0 0 0
[pid=28933] ppid=28847 vsize=436844 CPUtime=0.3 cores=0,2,4,6
/proc/28933/stat : 28933 (python) S 28847 28847 28149 0 -1 4202560 1049 0 0 0 16 14 0 0 18 0 1 0 658036300 447328256 6662 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 12209821539754803459 0 0 17 2 0 0 0
/proc/28933/statm: 109211 6662 379 364 0 45084 0
[pid=28935] ppid=28933 vsize=268424 CPUtime=104.8 cores=0,2,4,6
/proc/28935/stat : 28935 (hybridGM3) R 28933 28935 28935 0 -1 4202496 5652 0 0 0 10476 4 0 0 25 0 1 0 658036301 274866176 5378 33554432000 134512640 135204385 4287755408 18446744073709551615 134538047 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/28935/statm: 67106 5378 63 169 0 66936 0
[pid=29035] ppid=28847 vsize=447520 CPUtime=0.01 cores=0,2,4,6
/proc/29035/stat : 29035 (python) S 28847 28847 28149 0 -1 4202560 1018 0 0 0 1 0 0 0 18 0 1 0 658046038 458260480 6667 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 6598816620049563906 0 0 17 6 0 0 0
/proc/29035/statm: 111880 6667 380 364 0 47750 0
[pid=29037] ppid=29035 vsize=459772 CPUtime=7.66 cores=0,2,4,6
/proc/29037/stat : 29037 (TNM) R 29035 29037 29037 0 -1 4202496 1932 0 0 0 766 0 0 0 25 0 1 0 658046039 470806528 1657 33554432000 134512640 135008748 4292847296 18446744073709551615 134529195 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/29037/statm: 114943 1657 46 122 0 113843 0
[pid=29040] ppid=28847 vsize=457728 CPUtime=0 cores=0,2,4,6
/proc/29040/stat : 29040 (python) S 28847 28847 28149 0 -1 4202560 825 0 0 0 0 0 0 0 19 0 1 0 658046762 468713472 6682 33554432000 4194304 5683644 140733922881808 18446744073709551615 228244368966 0 0 16781312 514 8338468040685387775 0 0 17 6 0 0 0
/proc/29040/statm: 114432 6682 346 364 0 50299 0
[pid=29042] ppid=29040 vsize=249144 CPUtime=0.56 cores=0,2,4,6
/proc/29042/stat : 29042 (FH) R 29040 29042 29042 0 -1 4202496 1569 0 0 0 56 0 0 0 22 0 1 0 658046762 255123456 1296 33554432000 134512640 134936988 4291180160 18446744073709551615 134523387 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/29042/statm: 62286 1296 37 104 0 61790 0
[pid=28847/tid=28854] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28854/stat : 28854 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 19 0 657996162 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28857] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28857/stat : 28857 (python) S 28845 28847 28149 0 -1 4202560 24 934 0 0 0 0 0 0 23 0 19 0 657996163 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28860] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28860/stat : 28860 (python) S 28845 28847 28149 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 19 0 657996164 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 2 0 0 0
[pid=28847/tid=28863] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28863/stat : 28863 (python) S 28845 28847 28149 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 19 0 657996166 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 2 0 0 0
[pid=28847/tid=28870] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28870/stat : 28870 (python) S 28845 28847 28149 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 658006195 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 2 0 0 0
[pid=28847/tid=28874] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28874/stat : 28874 (python) S 28845 28847 28149 0 -1 4202560 31 934 0 0 0 0 0 0 21 0 19 0 658006197 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28878] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28878/stat : 28878 (python) S 28845 28847 28149 0 -1 4202560 27 934 0 0 0 0 0 0 22 0 19 0 658006199 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28882] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28882/stat : 28882 (python) S 28845 28847 28149 0 -1 4202560 27 934 0 0 0 0 0 0 23 0 19 0 658006207 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28895] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28895/stat : 28895 (python) S 28845 28847 28149 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 19 0 658016226 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28900] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28900/stat : 28900 (python) S 28845 28847 28149 0 -1 4202560 26 934 0 0 0 0 0 0 21 0 19 0 658016234 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 0 0 0 0
[pid=28847/tid=28904] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28904/stat : 28904 (python) S 28845 28847 28149 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 19 0 658016238 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 4 0 0 0
[pid=28847/tid=28907] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28907/stat : 28907 (python) S 28845 28847 28149 0 -1 4202560 28 934 0 0 0 0 0 0 25 0 19 0 658016239 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 6 0 0 0
[pid=28847/tid=28914] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28914/stat : 28914 (python) S 28845 28847 28149 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 19 0 658026259 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 6 0 0 0
[pid=28847/tid=28918] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28918/stat : 28918 (python) S 28845 28847 28149 0 -1 4202560 29 934 0 0 0 0 0 0 21 0 19 0 658026267 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 6 0 0 0
[pid=28847/tid=28922] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28922/stat : 28922 (python) S 28845 28847 28149 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 19 0 658026785 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 6 0 0 0
[pid=28847/tid=28934] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/28934/stat : 28934 (python) S 28845 28847 28149 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 658036300 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 6 0 0 0
[pid=28847/tid=29036] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/29036/stat : 29036 (python) S 28845 28847 28149 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 658046038 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 6 0 0 0
[pid=28847/tid=29041] ppid=28845 vsize=467452 CPUtime=0 cores=0,2,4,6
/proc/28847/task/29041/stat : 29041 (python) S 28845 28847 28149 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 658046762 478670848 8251 33554432000 4194304 5683644 140733922881808 18446744073709551615 228256172465 0 0 16781312 2 18446604441249572992 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2015.98
Current children cumulated vsize (KiB) 16033796

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 28847 and gives
#  childrusage.ru_utime.tv_sec=119
#  childrusage.ru_utime.tv_usec=44902
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=293499
# CPU time returned by wait4() is 122.338
# while last known CPU time is 2015.98
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 509.333
CPU time (s): 2015.98
CPU user time (s): 2012.79
CPU system time (s): 3.19
CPU usage (%): 395.808
Max. virtual memory (cumulated for all children) (KiB): 16033796

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 119.045
system time used= 3.2935
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 61208
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5926
involuntary context switches= 708

runsolver used 3.70344 second user time and 8.39572 second system time

The end

Launcher Data 

Begin job on node136 at 2011-05-15 14:52:59
IDJOB=3378234
IDBENCH=82643
IDSOLVER=1859
FILE ID=node136/3378234-1305463979
RUNJOBID= node136-1305462567-28167
PBS_JOBID= 13324968
Free space on /tmp= 72996 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v400-c34000-S817927155-033-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378234-1305463979/watcher-3378234-1305463979 -o /tmp/evaluation-result-3378234-1305463979/solver-3378234-1305463979 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378234-1305463979.cnf -seed 1925460585 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= bdd02c131ae0dd9fa14ba01f07a63379
RANDOM SEED=1925460585

node136.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16664340 kB
Buffers:        315312 kB
Cached:        5921796 kB
SwapCached:       2740 kB
Active:       11526740 kB
Inactive:      4398516 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16664340 kB
SwapTotal:    67111528 kB
SwapFree:     67082152 kB
Dirty:           32704 kB
Writeback:           0 kB
AnonPages:     9686380 kB
Mapped:          37284 kB
Slab:           241040 kB
PageTables:      53432 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 96896748 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72964 MiB
End job on node136 at 2011-05-15 15:01:31