Trace number 3317210

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (MO) 2378.7 599.526

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S922811046-076-UNKNOWN.cnf
MD5SUMe99bff721d695892a53f60adc73d403b
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark3694.54
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.31/0.46	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.46	c INFO: solving HOME/instance-3317210-1304268248.cnf
0.31/0.46	c INFO: computing task features
0.50/0.68	c INFO: computed variable-clause graph statistics
0.50/0.69	c INFO: computed clause balance statistics
1.60/1.70	c INFO: computed variable balance statistics
2.11/2.27	c INFO: computed small-clause counts
2.11/2.27	c INFO: computed Horn-clause counts
2.20/2.31	c INFO: computed variable graph statistics
2.20/2.31	c INFO: collected features for HOME/instance-3317210-1304268248.cnf in 1.84 s
2.20/2.34	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.38)
2.20/2.37	c INFO: running gnovelty+2@0 for 97 with 4798 remaining (b = 0.13)
2.29/2.40	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
2.29/2.43	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.07)
401.97/102.58	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.05)
402.34/102.62	c INFO: running gnovelty+2@0 for 97 with 4697 remaining (b = 0.05)
402.34/102.65	c INFO: running FH@0 for 97 with 4697 remaining (b = 0.03)
402.34/102.69	c INFO: running adaptg2wsat2009++@0 for 97 with 4697 remaining (b = 0.03)
800.87/202.78	c INFO: running adaptg2wsat2009++@0 for 97 with 4597 remaining (b = 0.02)
801.25/202.87	c INFO: running FH@0 for 97 with 4597 remaining (b = 0.02)
801.47/202.91	c INFO: running hybridGM3@0 for 194 with 4597 remaining (b = 0.04)
801.47/202.96	c INFO: running gnovelty+2@0 for 97 with 4597 remaining (b = 0.02)
1199.81/303.00	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.03)
1200.21/303.09	c INFO: running adaptg2wsat2009++@0 for 97 with 4497 remaining (b = 0.01)
1200.68/303.23	c INFO: running hybridGM3@0 for 194 with 4497 remaining (b = 0.03)
1599.48/403.31	c INFO: running hybridGM3@0 for 194 with 4397 remaining (b = 0.02)
1603.44/404.39	c INFO: running hybridGM3@98 for 97 with 4396 remaining (b = 0.01)
1981.37/499.44	c INFO: running hybridGM3@99 for 97 with 4301 remaining (b = 0.01)
1981.74/499.58	c INFO: running hybridGM3@294 for 97 with 4300 remaining (b = 0.01)
2005.97/505.67	c INFO: running hybridGM3@98 for 97 with 4294 remaining (b = 0.01)
2359.60/594.63	c INFO: running TNM@0 for 1552 with 4205 remaining (b = 0.19)
2375.88/598.78	c INFO: running FH@0 for 97 with 4201 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317210-1304268248/watcher-3317210-1304268248 -o /tmp/evaluation-result-3317210-1304268248/solver-3317210-1304268248 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317210-1304268248.cnf -seed 1199177234 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.27 7.78 5.14 5/244 22084
/proc/meminfo: memFree=28281304/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=15780 CPUtime=0 cores=1,3,5,7
/proc/22084/stat : 22084 (python) R 22082 22084 21633 0 -1 4202496 860 0 0 0 0 0 0 0 25 0 1 0 538422535 16158720 741 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905730704 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/22084/statm: 3945 741 335 364 0 432 0

[startup+0.050908 s]
/proc/loadavg: 8.27 7.78 5.14 5/244 22084
/proc/meminfo: memFree=28281304/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=29792 CPUtime=0.03 cores=1,3,5,7
/proc/22084/stat : 22084 (python) R 22082 22084 21633 0 -1 4202496 1460 0 0 0 2 1 0 0 25 0 1 0 538422535 30507008 1299 33554432000 4194304 5683644 140734348047696 18446744073709551615 4864842 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/22084/statm: 7448 1299 443 364 0 818 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29792

[startup+0.100908 s]
/proc/loadavg: 8.27 7.78 5.14 5/244 22084
/proc/meminfo: memFree=28281304/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=104232 CPUtime=0.06 cores=1,3,5,7
/proc/22084/stat : 22084 (python) R 22082 22084 21633 0 -1 4202496 3067 0 0 0 4 2 0 0 25 0 1 0 538422535 106733568 2748 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905729024 0 2147483391 16781312 2 0 0 0 17 1 0 0 0
/proc/22084/statm: 26058 2748 847 364 0 1806 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104232

[startup+0.300884 s]
/proc/loadavg: 8.27 7.78 5.14 5/244 22084
/proc/meminfo: memFree=28281304/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=182432 CPUtime=0.22 cores=1,3,5,7
/proc/22084/stat : 22084 (python) S 22082 22084 21633 0 -1 4202496 6753 938 1 0 16 6 0 0 17 0 1 0 538422535 186810368 4817 33554432000 4194304 5683644 140734348047696 18446744073709551615 218900807303 0 2147483391 16781312 2 18446744071699207358 0 0 17 1 0 0 0
/proc/22084/statm: 45608 4817 1340 364 0 3724 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 182432

[startup+0.700792 s]
/proc/loadavg: 8.27 7.78 5.14 5/244 22084
/proc/meminfo: memFree=28281304/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=290804 CPUtime=0.6 cores=1,3,5,7
/proc/22084/stat : 22084 (python) R 22082 22084 21633 0 -1 4202496 14009 938 1 0 51 9 0 0 17 0 1 0 538422535 297783296 10085 33554432000 4194304 5683644 140734348047696 18446744073709551615 4510876 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/22084/statm: 72701 10085 1826 364 0 8625 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 290804

[startup+1.50064 s]
/proc/loadavg: 8.27 7.78 5.14 6/245 22089
/proc/meminfo: memFree=28246700/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=290804 CPUtime=1.4 cores=1,3,5,7
/proc/22084/stat : 22084 (python) R 22082 22084 21633 0 -1 4202496 14009 938 1 0 131 9 0 0 18 0 1 0 538422535 297783296 10085 33554432000 4194304 5683644 140734348047696 18446744073709551615 47417130206645 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/22084/statm: 72701 10085 1826 364 0 8625 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 290804

[startup+3.10033 s]
/proc/loadavg: 8.27 7.78 5.14 6/245 22089
/proc/meminfo: memFree=28246032/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=325512 CPUtime=2.33 cores=1,3,5,7
/proc/22084/stat : 22084 (python) S 22082 22084 21633 0 -1 4202496 16956 938 1 0 223 10 0 0 20 0 5 0 538422535 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/22084/statm: 81378 8576 1883 364 0 17287 0
Current children cumulated CPU time (s) 2.33
Current children cumulated vsize (KiB) 325512
heavy processes:

[startup+6.30075 s]
/proc/loadavg: 8.33 7.80 5.16 10/257 22101
/proc/meminfo: memFree=28212688/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=325512 CPUtime=2.33 cores=1,3,5,7
/proc/22084/stat : 22084 (python) S 22082 22084 21633 0 -1 4202496 16956 938 1 0 223 10 0 0 20 0 5 0 538422535 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/22084/statm: 81378 8576 1883 364 0 17287 0
[pid=22090] ppid=22084 vsize=284984 CPUtime=0 cores=1,3,5,7
/proc/22090/stat : 22090 (python) S 22084 22084 21633 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 538422770 291823616 6898 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751055 0 0 16781312 514 16410295711247073544 0 0 17 3 0 0 0
/proc/22090/statm: 71246 6898 355 364 0 7164 0
[pid=22092] ppid=22090 vsize=1020308 CPUtime=3.94 cores=1,3,5,7
/proc/22092/stat : 22092 (gnovelty+2) R 22090 22092 22092 0 -1 4202496 1152 0 0 0 394 0 0 0 25 0 1 0 538422770 1044795392 891 33554432000 134512640 135663704 4291359856 18446744073709551615 134545644 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22092/statm: 255077 891 113 282 0 254793 0
[pid=22093] ppid=22084 vsize=295004 CPUtime=0 cores=1,3,5,7
/proc/22093/stat : 22093 (python) S 22084 22084 21633 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 538422773 302084096 7031 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 10275134352832561413 0 0 17 3 0 0 0
/proc/22093/statm: 73751 7031 349 364 0 9666 0
[pid=22095] ppid=22093 vsize=1020308 CPUtime=3.88 cores=1,3,5,7
/proc/22095/stat : 22095 (gnovelty+2) R 22093 22095 22095 0 -1 4202496 1154 0 0 0 388 0 0 0 25 0 1 0 538422773 1044795392 891 33554432000 134512640 135663704 4288668080 18446744073709551615 134524882 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22095/statm: 255077 891 113 282 0 254793 0
[pid=22096] ppid=22084 vsize=305160 CPUtime=0 cores=1,3,5,7
/proc/22096/stat : 22096 (python) S 22084 22084 21633 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 538422775 312483840 7035 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 10275130504541864200 0 0 17 1 0 0 0
/proc/22096/statm: 76290 7035 349 364 0 12202 0
[pid=22098] ppid=22096 vsize=423400 CPUtime=3.89 cores=1,3,5,7
/proc/22098/stat : 22098 (adaptg2wsat2009) R 22096 22098 22098 0 -1 4202496 2625 0 0 0 389 0 0 0 25 0 1 0 538422776 433561600 2358 33554432000 134512640 135107411 4294165536 18446744073709551615 134532313 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22098/statm: 105850 2358 59 146 0 104726 0
[pid=22099] ppid=22084 vsize=316020 CPUtime=0 cores=1,3,5,7
/proc/22099/stat : 22099 (python) S 22084 22084 21633 0 -1 4202560 749 0 0 0 0 0 0 0 18 0 1 0 538422778 323604480 7039 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 4599715364752818440 0 0 17 3 0 0 0
/proc/22099/statm: 79005 7039 349 364 0 14914 0
[pid=22101] ppid=22099 vsize=423400 CPUtime=3.86 cores=1,3,5,7
/proc/22101/stat : 22101 (adaptg2wsat2009) R 22099 22101 22101 0 -1 4202496 2623 0 0 0 386 0 0 0 25 0 1 0 538422779 433561600 2358 33554432000 134512640 135107411 4292229232 18446744073709551615 134548427 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22101/statm: 105850 2358 59 146 0 104726 0
[pid=22084/tid=22091] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22091/stat : 22091 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 22 0 5 0 538422770 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 1 0 0 0
[pid=22084/tid=22094] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22094/stat : 22094 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 24 0 5 0 538422773 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22097] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22097/stat : 22097 (python) S 22082 22084 21633 0 -1 4202560 23 938 0 0 0 0 0 0 20 0 5 0 538422776 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 1 0 0 0
[pid=22084/tid=22100] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22100/stat : 22100 (python) S 22082 22084 21633 0 -1 4202560 31 938 0 0 0 0 0 0 20 0 5 0 538422779 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 17.9
Current children cumulated vsize (KiB) 4414096

[startup+12.7006 s]
/proc/loadavg: 8.30 7.80 5.18 9/265 22109
/proc/meminfo: memFree=28160068/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=325512 CPUtime=2.33 cores=1,3,5,7
/proc/22084/stat : 22084 (python) S 22082 22084 21633 0 -1 4202496 16956 938 1 0 223 10 0 0 20 0 5 0 538422535 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/22084/statm: 81378 8576 1883 364 0 17287 0
[pid=22090] ppid=22084 vsize=284984 CPUtime=0.01 cores=1,3,5,7
/proc/22090/stat : 22090 (python) S 22084 22084 21633 0 -1 4202560 895 0 0 0 0 1 0 0 18 0 1 0 538422770 291823616 6921 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751055 0 0 16781312 514 2737748768790118664 0 0 17 3 0 0 0
/proc/22090/statm: 71246 6921 378 364 0 7164 0
[pid=22092] ppid=22090 vsize=1020308 CPUtime=10.28 cores=1,3,5,7
/proc/22092/stat : 22092 (gnovelty+2) R 22090 22092 22092 0 -1 4202496 1152 0 0 0 1028 0 0 0 25 0 1 0 538422770 1044795392 891 33554432000 134512640 135663704 4291359856 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22092/statm: 255077 891 113 282 0 254793 0
[pid=22093] ppid=22084 vsize=295004 CPUtime=0.01 cores=1,3,5,7
/proc/22093/stat : 22093 (python) S 22084 22084 21633 0 -1 4202560 906 0 0 0 0 1 0 0 18 0 1 0 538422773 302084096 7054 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 2737748768790118661 0 0 17 3 0 0 0
/proc/22093/statm: 73751 7054 372 364 0 9666 0
[pid=22095] ppid=22093 vsize=1020308 CPUtime=10.23 cores=1,3,5,7
/proc/22095/stat : 22095 (gnovelty+2) R 22093 22095 22095 0 -1 4202496 1154 0 0 0 1023 0 0 0 25 0 1 0 538422773 1044795392 891 33554432000 134512640 135663704 4288668080 18446744073709551615 134547031 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22095/statm: 255077 891 113 282 0 254793 0
[pid=22096] ppid=22084 vsize=305160 CPUtime=0.01 cores=1,3,5,7
/proc/22096/stat : 22096 (python) S 22084 22084 21633 0 -1 4202560 901 0 0 0 0 1 0 0 18 0 1 0 538422775 312483840 7058 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 2737748768790118664 0 0 17 1 0 0 0
/proc/22096/statm: 76290 7058 372 364 0 12202 0
[pid=22098] ppid=22096 vsize=423400 CPUtime=10.27 cores=1,3,5,7
/proc/22098/stat : 22098 (adaptg2wsat2009) R 22096 22098 22098 0 -1 4202496 2625 0 0 0 1027 0 0 0 25 0 1 0 538422776 433561600 2358 33554432000 134512640 135107411 4294165536 18446744073709551615 134531615 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22098/statm: 105850 2358 59 146 0 104726 0
[pid=22099] ppid=22084 vsize=316020 CPUtime=0.01 cores=1,3,5,7
/proc/22099/stat : 22099 (python) S 22084 22084 21633 0 -1 4202560 940 0 0 0 0 1 0 0 18 0 1 0 538422778 323604480 7062 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 2737748768790118664 0 0 17 3 0 0 0
/proc/22099/statm: 79005 7062 372 364 0 14914 0
[pid=22101] ppid=22099 vsize=423400 CPUtime=10.25 cores=1,3,5,7
/proc/22101/stat : 22101 (adaptg2wsat2009) R 22099 22101 22101 0 -1 4202496 2623 0 0 0 1025 0 0 0 25 0 1 0 538422779 433561600 2358 33554432000 134512640 135107411 4292229232 18446744073709551615 134532264 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22101/statm: 105850 2358 59 146 0 104726 0
[pid=22084/tid=22091] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22091/stat : 22091 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 22 0 5 0 538422770 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 1 0 0 0
[pid=22084/tid=22094] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22094/stat : 22094 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 24 0 5 0 538422773 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22097] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22097/stat : 22097 (python) S 22082 22084 21633 0 -1 4202560 23 938 0 0 0 0 0 0 20 0 5 0 538422776 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 1 0 0 0
[pid=22084/tid=22100] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22100/stat : 22100 (python) S 22082 22084 21633 0 -1 4202560 31 938 0 0 0 0 0 0 20 0 5 0 538422779 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 43.4
Current children cumulated vsize (KiB) 4414096

[startup+25.5013 s]
/proc/loadavg: 8.24 7.81 5.22 9/265 22109
/proc/meminfo: memFree=28160072/32951124 swapFree=67111356/67111528
[pid=22084] ppid=22082 vsize=325512 CPUtime=2.33 cores=1,3,5,7
/proc/22084/stat : 22084 (python) S 22082 22084 21633 0 -1 4202496 16956 938 1 0 223 10 0 0 20 0 5 0 538422535 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/22084/statm: 81378 8576 1883 364 0 17287 0
[pid=22090] ppid=22084 vsize=284984 CPUtime=0.03 cores=1,3,5,7
/proc/22090/stat : 22090 (python) S 22084 22084 21633 0 -1 4202560 900 0 0 0 2 1 0 0 18 0 1 0 538422770 291823616 6921 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751055 0 0 16781312 514 2737748768790118664 0 0 17 3 0 0 0
/proc/22090/statm: 71246 6921 378 364 0 7164 0
[pid=22092] ppid=22090 vsize=1020308 CPUtime=23.03 cores=1,3,5,7
/proc/22092/stat : 22092 (gnovelty+2) R 22090 22092 22092 0 -1 4202496 1152 0 0 0 2303 0 0 0 25 0 1 0 538422770 1044795392 891 33554432000 134512640 135663704 4291359856 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22092/statm: 255077 891 113 282 0 254793 0
[pid=22093] ppid=22084 vsize=295004 CPUtime=0.02 cores=1,3,5,7
/proc/22093/stat : 22093 (python) S 22084 22084 21633 0 -1 4202560 911 0 0 0 1 1 0 0 18 0 1 0 538422773 302084096 7054 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 2737748768790118661 0 0 17 3 0 0 0
/proc/22093/statm: 73751 7054 372 364 0 9666 0
[pid=22095] ppid=22093 vsize=1020308 CPUtime=22.94 cores=1,3,5,7
/proc/22095/stat : 22095 (gnovelty+2) R 22093 22095 22095 0 -1 4202496 1154 0 0 0 2294 0 0 0 25 0 1 0 538422773 1044795392 891 33554432000 134512640 135663704 4288668080 18446744073709551615 134525309 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22095/statm: 255077 891 113 282 0 254793 0
[pid=22096] ppid=22084 vsize=305160 CPUtime=0.03 cores=1,3,5,7
/proc/22096/stat : 22096 (python) S 22084 22084 21633 0 -1 4202560 904 0 0 0 1 2 0 0 18 0 1 0 538422775 312483840 7058 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 2737748768790118664 0 0 17 1 0 0 0
/proc/22096/statm: 76290 7058 372 364 0 12202 0
[pid=22098] ppid=22096 vsize=423400 CPUtime=23.05 cores=1,3,5,7
/proc/22098/stat : 22098 (adaptg2wsat2009) R 22096 22098 22098 0 -1 4202496 2625 0 0 0 2305 0 0 0 25 0 1 0 538422776 433561600 2358 33554432000 134512640 135107411 4294165536 18446744073709551615 134539082 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22098/statm: 105850 2358 59 146 0 104726 0
[pid=22099] ppid=22084 vsize=316020 CPUtime=0.03 cores=1,3,5,7
/proc/22099/stat : 22099 (python) S 22084 22084 21633 0 -1 4202560 945 0 0 0 1 2 0 0 18 0 1 0 538422778 323604480 7062 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 2737748768790118664 0 0 17 3 0 0 0
/proc/22099/statm: 79005 7062 372 364 0 14914 0
[pid=22101] ppid=22099 vsize=423400 CPUtime=23.05 cores=1,3,5,7
/proc/22101/stat : 22101 (adaptg2wsat2009) R 22099 22101 22101 0 -1 4202496 2623 0 0 0 2305 0 0 0 25 0 1 0 538422779 433561600 2358 33554432000 134512640 135107411 4292229232 18446744073709551615 134532379 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22101/statm: 105850 2358 59 146 0 104726 0
[pid=22084/tid=22091] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22091/stat : 22091 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 22 0 5 0 538422770 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 1 0 0 0
[pid=22084/tid=22094] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22094/stat : 22094 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 24 0 5 0 538422773 333324288 8576 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22097] ppid=22082 vsize=325512 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

[pid=22101] ppid=22099 vsize=423400 CPUtime=100.15 cores=1,3,5,7
/proc/22101/stat : 22101 (adaptg2wsat2009) T 22099 22101 22101 0 -1 4202496 2623 0 0 0 10015 0 0 0 25 0 1 0 538422779 433561600 2358 33554432000 134512640 135107411 4292229232 18446744073709551615 134531561 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22101/statm: 105850 2358 59 146 0 104726 0
[pid=22099/tid=22117] ppid=22084 vsize=316020 CPUtime=0 cores=1,3,5,7
/proc/22099/task/22117/stat : 22117 (python) S 22084 22084 21633 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 538432794 323604480 7091 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604471163517120 0 0 -1 1 0 0 0
[pid=22115] ppid=22084 vsize=325888 CPUtime=0.16 cores=1,3,5,7
/proc/22115/stat : 22115 (python) S 22084 22084 21633 0 -1 4202560 1134 0 0 0 9 7 0 0 19 0 2 0 538432794 333709312 7095 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/22115/statm: 81472 7095 400 364 0 17378 0
[pid=22118] ppid=22115 vsize=423400 CPUtime=99.83 cores=1,3,5,7
/proc/22118/stat : 22118 (adaptg2wsat2009) T 22115 22118 22118 0 -1 4202496 2626 0 0 0 9983 0 0 0 25 0 1 0 538432794 433561600 2357 33554432000 134512640 135107411 4293802960 18446744073709551615 134531600 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22118/statm: 105850 2357 59 146 0 104726 0
[pid=22115/tid=22143] ppid=22084 vsize=325888 CPUtime=0 cores=1,3,5,7
/proc/22115/task/22143/stat : 22143 (python) S 22084 22084 21633 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 538442810 333709312 7095 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604470147889216 0 0 -1 1 0 0 0
[pid=22120] ppid=22084 vsize=336228 CPUtime=0.16 cores=1,3,5,7
/proc/22120/stat : 22120 (python) S 22084 22084 21633 0 -1 4202560 1141 0 0 0 7 9 0 0 18 0 2 0 538432797 344297472 7099 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22120/statm: 84057 7099 400 364 0 19960 0
[pid=22122] ppid=22120 vsize=1020308 CPUtime=100.22 cores=1,3,5,7
/proc/22122/stat : 22122 (gnovelty+2) T 22120 22122 22122 0 -1 4202496 1160 0 0 0 10022 0 0 0 25 0 1 0 538432798 1044795392 892 33554432000 134512640 135663704 4291184560 18446744073709551615 134524897 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/22122/statm: 255077 892 113 282 0 254793 0
[pid=22120/tid=22148] ppid=22084 vsize=336228 CPUtime=0 cores=1,3,5,7
/proc/22120/task/22148/stat : 22148 (python) S 22084 22084 21633 0 -1 4202560 53 0 0 0 0 0 0 0 19 0 2 0 538442820 344297472 7099 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604457098910784 0 0 -1 5 0 0 0
[pid=22123] ppid=22084 vsize=346316 CPUtime=0.16 cores=1,3,5,7
/proc/22123/stat : 22123 (python) S 22084 22084 21633 0 -1 4202560 1196 0 0 0 10 6 0 0 19 0 2 0 538432801 354627584 7109 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22123/statm: 86579 7109 406 364 0 22479 0
[pid=22125] ppid=22123 vsize=253036 CPUtime=99.99 cores=1,3,5,7
/proc/22125/stat : 22125 (FH) T 22123 22125 22125 0 -1 4202496 2499 0 0 0 9999 0 0 0 25 0 1 0 538432801 259108864 2234 33554432000 134512640 134936988 4290385408 18446744073709551615 134545122 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22125/statm: 63259 2234 41 104 0 62763 0
[pid=22123/tid=22147] ppid=22084 vsize=346316 CPUtime=0 cores=1,3,5,7
/proc/22123/task/22147/stat : 22147 (python) S 22084 22084 21633 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538442818 354627584 7109 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604464399051456 0 0 -1 7 0 0 0
[pid=22126] ppid=22084 vsize=357180 CPUtime=0.16 cores=1,3,5,7
/proc/22126/stat : 22126 (python) S 22084 22084 21633 0 -1 4202560 1177 0 0 0 8 8 0 0 19 0 2 0 538432805 365752320 7113 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22126/statm: 89295 7113 406 364 0 25192 0
[pid=22128] ppid=22126 vsize=423400 CPUtime=98.26 cores=1,3,5,7
/proc/22128/stat : 22128 (adaptg2wsat2009) T 22126 22128 22128 0 -1 4202496 2628 0 0 0 9826 0 0 0 25 0 1 0 538432805 433561600 2357 33554432000 134512640 135107411 4293061008 18446744073709551615 134531239 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22128/statm: 105850 2357 59 146 0 104726 0
[pid=22126/tid=22149] ppid=22084 vsize=357180 CPUtime=0 cores=1,3,5,7
/proc/22126/task/22149/stat : 22149 (python) S 22084 22084 21633 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538442821 365752320 7113 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604457098908992 0 0 -1 3 0 0 0
[pid=22144] ppid=22084 vsize=366952 CPUtime=0.17 cores=1,3,5,7
/proc/22144/stat : 22144 (python) S 22084 22084 21633 0 -1 4202560 1151 0 0 0 9 8 0 0 19 0 2 0 538442814 375758848 7211 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22144/statm: 91738 7211 399 364 0 27632 0
[pid=22146] ppid=22144 vsize=423400 CPUtime=100.16 cores=1,3,5,7
/proc/22146/stat : 22146 (adaptg2wsat2009) T 22144 22146 22146 0 -1 4202496 2630 0 0 0 10015 1 0 0 25 0 1 0 538442814 433561600 2358 33554432000 134512640 135107411 4293420592 18446744073709551615 134548400 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22146/statm: 105850 2358 59 146 0 104726 0
[pid=22144/tid=22206] ppid=22084 vsize=366952 CPUtime=0 cores=1,3,5,7
/proc/22144/task/22206/stat : 22206 (python) S 22084 22084 21633 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538452831 375758848 7211 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604457098908096 0 0 -1 5 0 0 0
[pid=22150] ppid=22084 vsize=377432 CPUtime=0.17 cores=1,3,5,7
/proc/22150/stat : 22150 (python) S 22084 22084 21633 0 -1 4202560 1159 0 0 0 9 8 0 0 19 0 2 0 538442823 386490368 7216 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/22150/statm: 94358 7216 400 364 0 30249 0
[pid=22152] ppid=22150 vsize=253028 CPUtime=99.97 cores=1,3,5,7
/proc/22152/stat : 22152 (FH) T 22150 22152 22152 0 -1 4202496 2505 0 0 0 9997 0 0 0 25 0 1 0 538442823 259100672 2233 33554432000 134512640 134936988 4294733232 18446744073709551615 134554594 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22152/statm: 63257 2233 41 104 0 62761 0
[pid=22150/tid=22210] ppid=22084 vsize=377432 CPUtime=0 cores=1,3,5,7
/proc/22150/task/22210/stat : 22210 (python) S 22084 22084 21633 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538452840 386490368 7216 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604457098909888 0 0 -1 5 0 0 0
[pid=22153] ppid=22084 vsize=387668 CPUtime=0.65 cores=1,3,5,7
/proc/22153/stat : 22153 (python) S 22084 22084 21633 0 -1 4202560 1197 0 0 0 34 31 0 0 18 0 2 0 538442827 396972032 7220 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/22153/statm: 96917 7220 400 364 0 32805 0
[pid=22155] ppid=22153 vsize=268424 CPUtime=391.34 cores=1,3,5,7
/proc/22155/stat : 22155 (hybridGM3) R 22153 22155 22155 0 -1 4202496 6190 0 0 0 39134 0 0 0 25 0 1 0 538442827 274866176 5919 33554432000 134512640 135204385 4293699392 18446744073709551615 134538284 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/22155/statm: 67106 5919 63 169 0 66936 0
[pid=22153/tid=22229] ppid=22084 vsize=387668 CPUtime=0 cores=1,3,5,7
/proc/22153/task/22229/stat : 22229 (python) S 22084 22084 21633 0 -1 4202560 45 0 0 0 0 0 0 0 18 0 2 0 538462969 396972032 7220 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604470455331648 0 0 -1 7 0 0 0
[pid=22156] ppid=22084 vsize=398052 CPUtime=0.17 cores=1,3,5,7
/proc/22156/stat : 22156 (python) S 22084 22084 21633 0 -1 4202560 1183 0 0 0 8 9 0 0 19 0 2 0 538442831 407605248 7224 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22156/statm: 99513 7224 400 364 0 35398 0
[pid=22158] ppid=22156 vsize=1020308 CPUtime=99.86 cores=1,3,5,7
/proc/22158/stat : 22158 (gnovelty+2) T 22156 22158 22158 0 -1 4202496 1165 0 0 0 9986 0 0 0 25 0 1 0 538442832 1044795392 892 33554432000 134512640 135663704 4290430896 18446744073709551615 134524888 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22158/statm: 255077 892 113 282 0 254793 0
[pid=22156/tid=22214] ppid=22084 vsize=398052 CPUtime=0 cores=1,3,5,7
/proc/22156/task/22214/stat : 22214 (python) S 22084 22084 21633 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538452854 407605248 7224 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604470455330752 0 0 -1 3 0 0 0
[pid=22207] ppid=22084 vsize=408444 CPUtime=0.47 cores=1,3,5,7
/proc/22207/stat : 22207 (python) S 22084 22084 21633 0 -1 4202560 1197 0 0 0 25 22 0 0 18 0 2 0 538452835 418246656 7228 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22207/statm: 102111 7228 400 364 0 37993 0
[pid=22209] ppid=22207 vsize=268428 CPUtime=291.53 cores=1,3,5,7
/proc/22209/stat : 22209 (hybridGM3) T 22207 22209 22209 0 -1 4202496 6193 0 0 0 29153 0 0 0 25 0 1 0 538452835 274870272 5920 33554432000 134512640 135204385 4291786032 18446744073709551615 134545396 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/22209/statm: 67107 5920 63 169 0 66937 0
[pid=22207/tid=22237] ppid=22084 vsize=408444 CPUtime=0 cores=1,3,5,7
/proc/22207/task/22237/stat : 22237 (python) S 22084 22084 21633 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 538472475 418246656 7228 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604471149064576 0 0 -1 5 0 0 0
[pid=22211] ppid=22084 vsize=418844 CPUtime=0.15 cores=1,3,5,7
/proc/22211/stat : 22211 (python) S 22084 22084 21633 0 -1 4202560 1178 0 0 0 8 7 0 0 19 0 2 0 538452844 428896256 7232 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22211/statm: 104711 7232 400 364 0 40590 0
[pid=22213] ppid=22211 vsize=423400 CPUtime=98.15 cores=1,3,5,7
/proc/22213/stat : 22213 (adaptg2wsat2009) T 22211 22213 22213 0 -1 4202496 2634 0 0 0 9814 1 0 0 25 0 1 0 538452844 433561600 2358 33554432000 134512640 135107411 4287415040 18446744073709551615 134531600 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/22213/statm: 105850 2358 59 146 0 104726 0
[pid=22211/tid=22225] ppid=22084 vsize=418844 CPUtime=0 cores=1,3,5,7
/proc/22211/task/22225/stat : 22225 (python) S 22084 22084 21633 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538462862 428896256 7232 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604471149066368 0 0 -1 3 0 0 0
[pid=22215] ppid=22084 vsize=429248 CPUtime=0.48 cores=1,3,5,7
/proc/22215/stat : 22215 (python) S 22084 22084 21633 0 -1 4202560 1242 0 0 0 25 23 0 0 18 0 2 0 538452858 439549952 7276 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22215/statm: 107312 7276 408 364 0 43188 0
[pid=22217] ppid=22215 vsize=268424 CPUtime=292.34 cores=1,3,5,7
/proc/22217/stat : 22217 (hybridGM3) R 22215 22217 22217 0 -1 4202496 6197 0 0 0 29234 0 0 0 25 0 1 0 538452859 274866176 5920 33554432000 134512640 135204385 4293447904 18446744073709551615 134538375 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/22217/statm: 67106 5920 63 169 0 66936 0
[pid=22215/tid=22238] ppid=22084 vsize=429248 CPUtime=0 cores=1,3,5,7
/proc/22215/task/22238/stat : 22238 (python) S 22084 22084 21633 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538473098 439549952 7276 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604466271666304 0 0 -1 7 0 0 0
[pid=22226] ppid=22084 vsize=439380 CPUtime=0.32 cores=1,3,5,7
/proc/22226/stat : 22226 (python) S 22084 22084 21633 0 -1 4202560 1234 0 0 0 17 15 0 0 19 0 2 0 538462867 449925120 7317 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/22226/statm: 109845 7317 408 364 0 45718 0
[pid=22228] ppid=22226 vsize=268420 CPUtime=195.4 cores=1,3,5,7
/proc/22228/stat : 22228 (hybridGM3) T 22226 22228 22228 0 -1 4202496 6195 0 0 0 19540 0 0 0 25 0 1 0 538462867 274862080 5918 33554432000 134512640 135204385 4292591568 18446744073709551615 134538284 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/22228/statm: 67105 5918 63 169 0 66935 0
[pid=22226/tid=22244] ppid=22084 vsize=439380 CPUtime=0 cores=1,3,5,7
/proc/22226/task/22244/stat : 22244 (python) S 22084 22084 21633 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 538482409 449925120 7317 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 514 18446604454222996416 0 0 -1 1 0 0 0
[pid=22241] ppid=22084 vsize=449740 CPUtime=0 cores=1,3,5,7
/proc/22241/stat : 22241 (python) S 22084 22084 21633 0 -1 4202560 1026 0 0 0 0 0 0 0 18 0 1 0 538481999 460533760 7296 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 4445365097811902728 0 0 17 3 0 0 0
/proc/22241/statm: 112435 7296 374 364 0 48305 0
[pid=22243] ppid=22241 vsize=461504 CPUtime=4.73 cores=1,3,5,7
/proc/22243/stat : 22243 (TNM) R 22241 22243 22243 0 -1 4202496 2472 0 0 0 473 0 0 0 25 0 1 0 538481999 472580096 2194 33554432000 134512640 135008748 4287210720 18446744073709551615 134527524 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/22243/statm: 115376 2194 46 122 0 114276 0
[pid=22245] ppid=22084 vsize=460340 CPUtime=0 cores=1,3,5,7
/proc/22245/stat : 22245 (python) S 22084 22084 21633 0 -1 4202560 830 0 0 0 0 0 0 0 19 0 1 0 538482414 471388160 7272 33554432000 4194304 5683644 140734348047696 18446744073709551615 218905751110 0 0 16781312 514 1558074226429132799 0 0 17 3 0 0 0
/proc/22245/statm: 115085 7272 346 364 0 50952 0
[pid=22247] ppid=22245 vsize=248616 CPUtime=0.62 cores=1,3,5,7
/proc/22247/stat : 22247 (FH) R 22245 22247 22247 0 -1 4202496 1486 0 0 0 62 0 0 0 22 0 1 0 538482414 254582784 1214 33554432000 134512640 134936988 4287161664 18446744073709551615 134523456 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/22247/statm: 62154 1214 37 104 0 61658 0
[pid=22084/tid=22091] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22091/stat : 22091 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 22 0 19 0 538422770 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 1 0 0 0
[pid=22084/tid=22094] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22094/stat : 22094 (python) S 22082 22084 21633 0 -1 4202560 24 938 0 0 0 0 0 0 24 0 19 0 538422773 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22097] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22097/stat : 22097 (python) S 22082 22084 21633 0 -1 4202560 23 938 0 0 0 0 0 0 20 0 19 0 538422776 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 1 0 0 0
[pid=22084/tid=22100] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22100/stat : 22100 (python) S 22082 22084 21633 0 -1 4202560 31 938 0 0 0 0 0 0 20 0 19 0 538422779 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22116] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22116/stat : 22116 (python) S 22082 22084 21633 0 -1 4202560 27 938 0 0 0 0 0 0 20 0 19 0 538432794 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22121] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22121/stat : 22121 (python) S 22082 22084 21633 0 -1 4202560 25 938 0 0 0 0 0 0 22 0 19 0 538432798 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22124] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22124/stat : 22124 (python) S 22082 22084 21633 0 -1 4202560 23 938 0 0 0 0 0 0 20 0 19 0 538432801 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 7 0 0 0
[pid=22084/tid=22127] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22127/stat : 22127 (python) S 22082 22084 21633 0 -1 4202560 29 938 0 0 0 0 0 0 20 0 19 0 538432805 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 5 0 0 0
[pid=22084/tid=22145] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22145/stat : 22145 (python) S 22082 22084 21633 0 -1 4202560 29 938 0 0 0 0 0 0 20 0 19 0 538442814 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 5 0 0 0
[pid=22084/tid=22151] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22151/stat : 22151 (python) S 22082 22084 21633 0 -1 4202560 22 938 0 0 0 0 0 0 21 0 19 0 538442823 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 5 0 0 0
[pid=22084/tid=22154] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22154/stat : 22154 (python) S 22082 22084 21633 0 -1 4202560 26 938 0 0 0 0 0 0 18 0 19 0 538442827 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 7 0 0 0
[pid=22084/tid=22157] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22157/stat : 22157 (python) S 22082 22084 21633 0 -1 4202560 33 938 0 0 0 0 0 0 20 0 19 0 538442832 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22208] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22208/stat : 22208 (python) S 22082 22084 21633 0 -1 4202560 33 938 0 0 0 0 0 0 18 0 19 0 538452835 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22212] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22212/stat : 22212 (python) S 22082 22084 21633 0 -1 4202560 31 938 0 0 0 0 0 0 21 0 19 0 538452844 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22216] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22216/stat : 22216 (python) S 22082 22084 21633 0 -1 4202560 32 938 0 0 0 0 0 0 18 0 19 0 538452859 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22227] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22227/stat : 22227 (python) S 22082 22084 21633 0 -1 4202560 22 938 0 0 0 0 0 0 20 0 19 0 538462867 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22242] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22242/stat : 22242 (python) S 22082 22084 21633 0 -1 4202560 29 938 0 0 0 0 0 0 20 0 19 0 538481999 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
[pid=22084/tid=22246] ppid=22082 vsize=469776 CPUtime=0 cores=1,3,5,7
/proc/22084/task/22246/stat : 22246 (python) S 22082 22084 21633 0 -1 4202560 27 938 0 0 0 0 0 0 20 0 19 0 538482414 481050624 8841 33554432000 4194304 5683644 140734348047696 18446744073709551615 218917554609 0 0 16781312 2 18446604464054124800 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 2378.7
Current children cumulated vsize (KiB) 16094412

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 22084 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=997544
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=152976
# CPU time returned by wait4() is 3.15052
# while last known CPU time is 2378.7
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 599.526
CPU time (s): 2378.7
CPU user time (s): 2376.7
CPU system time (s): 2
CPU usage (%): 396.764
Max. virtual memory (cumulated for all children) (KiB): 16094412

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.99754
system time used= 0.152976
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 32169
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1145
involuntary context switches= 70

runsolver used 3.18152 second user time and 8.05578 second system time

The end

Launcher Data 

Begin job on node137 at 2011-05-01 18:44:08
IDJOB=3317210
IDBENCH=82675
IDSOLVER=1705
FILE ID=node137/3317210-1304268248
RUNJOBID= node137-1304267281-21652
PBS_JOBID= 13173942
Free space on /tmp= 73384 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S922811046-076-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317210-1304268248/watcher-3317210-1304268248 -o /tmp/evaluation-result-3317210-1304268248/solver-3317210-1304268248 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317210-1304268248.cnf -seed 1199177234 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e99bff721d695892a53f60adc73d403b
RANDOM SEED=1199177234

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28281424 kB
Buffers:        936540 kB
Cached:        2604364 kB
SwapCached:         48 kB
Active:         966036 kB
Inactive:      3225516 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28281424 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:           33388 kB
Writeback:           0 kB
AnonPages:      650436 kB
Mapped:          22200 kB
Slab:           399496 kB
PageTables:      18508 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 14252504 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73356 MiB
End job on node137 at 2011-05-01 18:54:10