Trace number 3378228

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2380.71 600.648

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S1993071608-084-UNKNOWN.cnf
MD5SUM6a416a8eca39558555cbe083cf0c9cff
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark591.01
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.31/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.47	c INFO: solving HOME/instance-3378228-1305463918.cnf
0.31/0.47	c INFO: computing task features
0.50/0.67	c INFO: computed variable-clause graph statistics
0.50/0.68	c INFO: computed clause balance statistics
1.60/1.72	c INFO: computed variable balance statistics
2.11/2.27	c INFO: computed small-clause counts
2.11/2.28	c INFO: computed Horn-clause counts
2.20/2.32	c INFO: computed variable graph statistics
2.20/2.32	c INFO: collected features for HOME/instance-3378228-1305463918.cnf in 1.84 s
2.20/2.33	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.38)
2.20/2.34	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
2.20/2.35	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
2.20/2.36	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.07)
398.04/101.62	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.05)
402.01/102.61	c INFO: running gnovelty+2@0 for 97 with 1097 remaining (b = 0.05)
402.01/102.67	c INFO: running FH@0 for 97 with 1097 remaining (b = 0.03)
402.29/102.73	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.03)
797.09/201.90	c INFO: running adaptg2wsat2009++@0 for 97 with 998 remaining (b = 0.02)
801.06/202.93	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
801.39/203.01	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
816.93/206.98	c INFO: running gnovelty+2@0 for 97 with 993 remaining (b = 0.02)
1195.63/302.18	c INFO: running hybridGM3@0 for 194 with 898 remaining (b = 0.03)
1196.00/302.22	c INFO: running adaptg2wsat2009++@0 for 97 with 898 remaining (b = 0.01)
1216.26/307.38	c INFO: running hybridGM3@0 for 194 with 893 remaining (b = 0.03)
1590.87/401.51	c INFO: running hybridGM3@0 for 194 with 798 remaining (b = 0.02)
1591.13/401.68	c INFO: running hybridGM3@101 for 97 with 798 remaining (b = 0.01)
1984.98/500.87	c INFO: running hybridGM3@99 for 97 with 699 remaining (b = 0.01)
1989.35/501.97	c INFO: running hybridGM3@300 for 97 with 698 remaining (b = 0.01)
2005.68/506.05	c INFO: running hybridGM3@99 for 97 with 694 remaining (b = 0.01)
2375.12/599.17	c INFO: running TNM@0 for 601 with 601 remaining (b = 0.07)
2379.07/600.19	c INFO: running FH@0 for 97 with 600 remaining (b = 0.01)
2379.46/600.27	c INFO: running adaptg2wsat2009++@97 for 97 with 600 remaining (b = 0.01)
2380.71/600.64	c INFO: running hybridGM3@0 for 194 with 599 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378228-1305463918/watcher-3378228-1305463918 -o /tmp/evaluation-result-3378228-1305463918/solver-3378228-1305463918 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378228-1305463918.cnf -seed 1373343021 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.32 8.23 6.91 5/361 31507
/proc/meminfo: memFree=11105020/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=16244 CPUtime=0 cores=1,3,5,7
/proc/31507/stat : 31507 (python) R 31505 31507 30970 0 -1 4202496 997 0 0 0 0 0 0 0 25 0 1 0 657989657 16633856 870 33554432000 4194304 5683644 140735479138560 18446744073709551615 255376962245 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/31507/statm: 4061 870 368 364 0 548 0

[startup+0.047815 s]
/proc/loadavg: 8.32 8.23 6.91 5/361 31507
/proc/meminfo: memFree=11105020/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=29792 CPUtime=0.03 cores=1,3,5,7
/proc/31507/stat : 31507 (python) S 31505 31507 30970 0 -1 4202496 1414 0 0 0 3 0 0 0 25 0 1 0 657989657 30507008 1258 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377299456 0 2147483391 16781312 2 18446744071699453118 0 0 17 1 0 0 0
/proc/31507/statm: 7448 1258 443 364 0 818 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29792

[startup+0.100803 s]
/proc/loadavg: 8.32 8.23 6.91 5/361 31507
/proc/meminfo: memFree=11105020/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=104220 CPUtime=0.06 cores=1,3,5,7
/proc/31507/stat : 31507 (python) S 31505 31507 30970 0 -1 4202496 3046 0 0 0 5 1 0 0 17 0 1 0 657989657 106721280 2731 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377299456 0 2147483391 16781312 2 18446744071699453118 0 0 17 1 0 0 0
/proc/31507/statm: 26055 2731 847 364 0 1803 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104220

[startup+0.300768 s]
/proc/loadavg: 8.32 8.23 6.91 5/361 31507
/proc/meminfo: memFree=11105020/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=168676 CPUtime=0.22 cores=1,3,5,7
/proc/31507/stat : 31507 (python) R 31505 31507 30970 0 -1 4202496 6714 934 1 0 17 5 0 0 18 0 1 0 657989657 172724224 4794 33554432000 4194304 5683644 140735479138560 18446744073709551615 4643462 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/31507/statm: 42169 4794 1331 364 0 3454 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 168676

[startup+0.700706 s]
/proc/loadavg: 8.32 8.23 6.91 5/361 31507
/proc/meminfo: memFree=11105020/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=290800 CPUtime=0.6 cores=1,3,5,7
/proc/31507/stat : 31507 (python) R 31505 31507 30970 0 -1 4202496 14007 934 1 0 52 8 0 0 17 0 1 0 657989657 297779200 10084 33554432000 4194304 5683644 140735479138560 18446744073709551615 5003296 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/31507/statm: 72700 10084 1826 364 0 8624 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 290800

[startup+1.50055 s]
/proc/loadavg: 8.05 8.18 6.90 6/362 31512
/proc/meminfo: memFree=11070412/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=290800 CPUtime=1.4 cores=1,3,5,7
/proc/31507/stat : 31507 (python) R 31505 31507 30970 0 -1 4202496 14007 934 1 0 132 8 0 0 19 0 1 0 657989657 297779200 10084 33554432000 4194304 5683644 140735479138560 18446744073709551615 47340552313561 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/31507/statm: 72700 10084 1826 364 0 8624 0
Current children cumulated CPU time (s) 1.4
Current children cumulated vsize (KiB) 290800

[startup+3.10022 s]
/proc/loadavg: 8.05 8.18 6.90 6/362 31512
/proc/meminfo: memFree=11069916/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=325512 CPUtime=2.26 cores=1,3,5,7
/proc/31507/stat : 31507 (python) S 31505 31507 30970 0 -1 4202496 16795 934 1 0 218 8 0 0 22 0 5 0 657989657 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/31507/statm: 81378 8568 1875 364 0 17287 0
Current children cumulated CPU time (s) 2.26
Current children cumulated vsize (KiB) 325512
heavy processes:

[startup+6.30063 s]
/proc/loadavg: 8.05 8.18 6.90 9/374 31524
/proc/meminfo: memFree=11036340/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=325512 CPUtime=2.26 cores=1,3,5,7
/proc/31507/stat : 31507 (python) S 31505 31507 30970 0 -1 4202496 16795 934 1 0 218 8 0 0 22 0 5 0 657989657 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/31507/statm: 81378 8568 1875 364 0 17287 0
[pid=31513] ppid=31507 vsize=284984 CPUtime=0 cores=1,3,5,7
/proc/31513/stat : 31513 (python) S 31507 31507 30970 0 -1 4202560 724 0 0 0 0 0 0 0 18 0 1 0 657989891 291823616 6898 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321487 0 0 16781312 514 17786038469902958855 0 0 17 3 0 0 0
/proc/31513/statm: 71246 6898 355 364 0 7164 0
[pid=31515] ppid=31513 vsize=1020308 CPUtime=3.95 cores=1,3,5,7
/proc/31515/stat : 31515 (gnovelty+2) R 31513 31515 31515 0 -1 4202496 1154 0 0 0 395 0 0 0 25 0 1 0 657989891 1044795392 891 33554432000 134512640 135663704 4287761808 18446744073709551615 134525286 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31515/statm: 255077 891 113 282 0 254793 0
[pid=31516] ppid=31507 vsize=294904 CPUtime=0 cores=1,3,5,7
/proc/31516/stat : 31516 (python) S 31507 31507 30970 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 657989892 301981696 7031 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 17786046166484353288 0 0 17 7 0 0 0
/proc/31516/statm: 73726 7031 349 364 0 9641 0
[pid=31518] ppid=31516 vsize=1020308 CPUtime=3.94 cores=1,3,5,7
/proc/31518/stat : 31518 (gnovelty+2) R 31516 31518 31518 0 -1 4202496 1154 0 0 0 394 0 0 0 25 0 1 0 657989892 1044795392 892 33554432000 134512640 135663704 4289831760 18446744073709551615 134545682 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/31518/statm: 255077 892 113 282 0 254793 0
[pid=31519] ppid=31507 vsize=305160 CPUtime=0 cores=1,3,5,7
/proc/31519/stat : 31519 (python) S 31507 31507 30970 0 -1 4202560 734 0 0 0 0 0 0 0 18 0 1 0 657989893 312483840 7035 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 12168699534293041413 0 0 17 1 0 0 0
/proc/31519/statm: 76290 7035 349 364 0 12202 0
[pid=31521] ppid=31519 vsize=423400 CPUtime=3.93 cores=1,3,5,7
/proc/31521/stat : 31521 (adaptg2wsat2009) R 31519 31521 31521 0 -1 4202496 2622 0 0 0 393 0 0 0 25 0 1 0 657989893 433561600 2358 33554432000 134512640 135107411 4292449600 18446744073709551615 134532371 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/31521/statm: 105850 2358 59 146 0 104726 0
[pid=31522] ppid=31507 vsize=316024 CPUtime=0 cores=1,3,5,7
/proc/31522/stat : 31522 (python) S 31507 31507 30970 0 -1 4202560 744 0 0 0 0 0 0 0 18 0 1 0 657989894 323608576 7039 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 11513358993179246853 0 0 17 7 0 0 0
/proc/31522/statm: 79006 7039 349 364 0 14915 0
[pid=31524] ppid=31522 vsize=423400 CPUtime=3.87 cores=1,3,5,7
/proc/31524/stat : 31524 (adaptg2wsat2009) R 31522 31524 31524 0 -1 4202496 2620 0 0 0 387 0 0 0 25 0 1 0 657989894 433561600 2358 33554432000 134512640 135107411 4291380736 18446744073709551615 134531613 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/31524/statm: 105850 2358 59 146 0 104726 0
[pid=31507/tid=31514] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31514/stat : 31514 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 5 0 657989891 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 5 0 0 0
[pid=31507/tid=31517] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31517/stat : 31517 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 657989892 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 3 0 0 0
[pid=31507/tid=31520] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31520/stat : 31520 (python) S 31505 31507 30970 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 5 0 657989893 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31523] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31523/stat : 31523 (python) S 31505 31507 30970 0 -1 4202560 27 934 0 0 0 0 0 0 22 0 5 0 657989894 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 17.95
Current children cumulated vsize (KiB) 4414000

[startup+12.7014 s]
/proc/loadavg: 8.19 8.20 6.92 9/374 31524
/proc/meminfo: memFree=11034760/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=325512 CPUtime=2.26 cores=1,3,5,7
/proc/31507/stat : 31507 (python) S 31505 31507 30970 0 -1 4202496 16795 934 1 0 218 8 0 0 22 0 5 0 657989657 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/31507/statm: 81378 8568 1875 364 0 17287 0
[pid=31513] ppid=31507 vsize=284984 CPUtime=0.02 cores=1,3,5,7
/proc/31513/stat : 31513 (python) S 31507 31507 30970 0 -1 4202560 888 0 0 0 1 1 0 0 18 0 1 0 657989891 291823616 6921 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321487 0 0 16781312 514 5751694858477601031 0 0 17 3 0 0 0
/proc/31513/statm: 71246 6921 378 364 0 7164 0
[pid=31515] ppid=31513 vsize=1020308 CPUtime=10.36 cores=1,3,5,7
/proc/31515/stat : 31515 (gnovelty+2) R 31513 31515 31515 0 -1 4202496 1154 0 0 0 1036 0 0 0 25 0 1 0 657989891 1044795392 891 33554432000 134512640 135663704 4287761808 18446744073709551615 134525544 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31515/statm: 255077 891 113 282 0 254793 0
[pid=31516] ppid=31507 vsize=294904 CPUtime=0.02 cores=1,3,5,7
/proc/31516/stat : 31516 (python) S 31507 31507 30970 0 -1 4202560 906 0 0 0 1 1 0 0 18 0 1 0 657989892 301981696 7054 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 5751694858477601032 0 0 17 7 0 0 0
/proc/31516/statm: 73726 7054 372 364 0 9641 0
[pid=31518] ppid=31516 vsize=1020308 CPUtime=10.31 cores=1,3,5,7
/proc/31518/stat : 31518 (gnovelty+2) R 31516 31518 31518 0 -1 4202496 1154 0 0 0 1031 0 0 0 25 0 1 0 657989892 1044795392 892 33554432000 134512640 135663704 4289831760 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/31518/statm: 255077 892 113 282 0 254793 0
[pid=31519] ppid=31507 vsize=305160 CPUtime=0.02 cores=1,3,5,7
/proc/31519/stat : 31519 (python) S 31507 31507 30970 0 -1 4202560 900 0 0 0 0 2 0 0 18 0 1 0 657989893 312483840 7057 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 5751694858477601029 0 0 17 1 0 0 0
/proc/31519/statm: 76290 7057 371 364 0 12202 0
[pid=31521] ppid=31519 vsize=423400 CPUtime=10.3 cores=1,3,5,7
/proc/31521/stat : 31521 (adaptg2wsat2009) R 31519 31521 31521 0 -1 4202496 2622 0 0 0 1030 0 0 0 25 0 1 0 657989893 433561600 2358 33554432000 134512640 135107411 4292449600 18446744073709551615 134539260 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/31521/statm: 105850 2358 59 146 0 104726 0
[pid=31522] ppid=31507 vsize=316024 CPUtime=0.02 cores=1,3,5,7
/proc/31522/stat : 31522 (python) S 31507 31507 30970 0 -1 4202560 916 0 0 0 1 1 0 0 18 0 1 0 657989894 323608576 7062 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 5751694858477601029 0 0 17 7 0 0 0
/proc/31522/statm: 79006 7062 372 364 0 14915 0
[pid=31524] ppid=31522 vsize=423400 CPUtime=10.14 cores=1,3,5,7
/proc/31524/stat : 31524 (adaptg2wsat2009) R 31522 31524 31524 0 -1 4202496 2620 0 0 0 1014 0 0 0 25 0 1 0 657989894 433561600 2358 33554432000 134512640 135107411 4291380736 18446744073709551615 134539354 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/31524/statm: 105850 2358 59 146 0 104726 0
[pid=31507/tid=31514] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31514/stat : 31514 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 5 0 657989891 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 5 0 0 0
[pid=31507/tid=31517] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31517/stat : 31517 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 657989892 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 3 0 0 0
[pid=31507/tid=31520] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31520/stat : 31520 (python) S 31505 31507 30970 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 5 0 657989893 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31523] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31523/stat : 31523 (python) S 31505 31507 30970 0 -1 4202560 27 934 0 0 0 0 0 0 22 0 5 0 657989894 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 43.45
Current children cumulated vsize (KiB) 4414000

[startup+25.5011 s]
/proc/loadavg: 8.16 8.20 6.93 9/374 31524
/proc/meminfo: memFree=11035268/32951124 swapFree=66050672/67111528
[pid=31507] ppid=31505 vsize=325512 CPUtime=2.26 cores=1,3,5,7
/proc/31507/stat : 31507 (python) S 31505 31507 30970 0 -1 4202496 16795 934 1 0 218 8 0 0 22 0 5 0 657989657 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/31507/statm: 81378 8568 1875 364 0 17287 0
[pid=31513] ppid=31507 vsize=284984 CPUtime=0.05 cores=1,3,5,7
/proc/31513/stat : 31513 (python) S 31507 31507 30970 0 -1 4202560 894 0 0 0 3 2 0 0 18 0 1 0 657989891 291823616 6921 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321487 0 0 16781312 514 5751694858477601031 0 0 17 3 0 0 0
/proc/31513/statm: 71246 6921 378 364 0 7164 0
[pid=31515] ppid=31513 vsize=1020308 CPUtime=23.16 cores=1,3,5,7
/proc/31515/stat : 31515 (gnovelty+2) R 31513 31515 31515 0 -1 4202496 1154 0 0 0 2316 0 0 0 25 0 1 0 657989891 1044795392 891 33554432000 134512640 135663704 4287761808 18446744073709551615 134525172 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31515/statm: 255077 891 113 282 0 254793 0
[pid=31516] ppid=31507 vsize=294904 CPUtime=0.05 cores=1,3,5,7
/proc/31516/stat : 31516 (python) S 31507 31507 30970 0 -1 4202560 912 0 0 0 3 2 0 0 18 0 1 0 657989892 301981696 7054 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 5751694858477601032 0 0 17 7 0 0 0
/proc/31516/statm: 73726 7054 372 364 0 9641 0
[pid=31518] ppid=31516 vsize=1020308 CPUtime=23.08 cores=1,3,5,7
/proc/31518/stat : 31518 (gnovelty+2) R 31516 31518 31518 0 -1 4202496 1154 0 0 0 2308 0 0 0 25 0 1 0 657989892 1044795392 892 33554432000 134512640 135663704 4289831760 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/31518/statm: 255077 892 113 282 0 254793 0
[pid=31519] ppid=31507 vsize=305160 CPUtime=0.05 cores=1,3,5,7
/proc/31519/stat : 31519 (python) S 31507 31507 30970 0 -1 4202560 904 0 0 0 2 3 0 0 18 0 1 0 657989893 312483840 7057 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 5751694858477601029 0 0 17 1 0 0 0
/proc/31519/statm: 76290 7057 371 364 0 12202 0
[pid=31521] ppid=31519 vsize=423400 CPUtime=23.07 cores=1,3,5,7
/proc/31521/stat : 31521 (adaptg2wsat2009) R 31519 31521 31521 0 -1 4202496 2622 0 0 0 2307 0 0 0 25 0 1 0 657989893 433561600 2358 33554432000 134512640 135107411 4292449600 18446744073709551615 134531246 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/31521/statm: 105850 2358 59 146 0 104726 0
[pid=31522] ppid=31507 vsize=316024 CPUtime=0.05 cores=1,3,5,7
/proc/31522/stat : 31522 (python) S 31507 31507 30970 0 -1 4202560 920 0 0 0 3 2 0 0 18 0 1 0 657989894 323608576 7062 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 5751694858477601029 0 0 17 7 0 0 0
/proc/31522/statm: 79006 7062 372 364 0 14915 0
[pid=31524] ppid=31522 vsize=423400 CPUtime=22.74 cores=1,3,5,7
/proc/31524/stat : 31524 (adaptg2wsat2009) R 31522 31524 31524 0 -1 4202496 2620 0 0 0 2274 0 0 0 25 0 1 0 657989894 433561600 2358 33554432000 134512640 135107411 4291380736 18446744073709551615 134531781 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/31524/statm: 105850 2358 59 146 0 104726 0
[pid=31507/tid=31514] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31514/stat : 31514 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 5 0 657989891 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 5 0 0 0
[pid=31507/tid=31517] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31517/stat : 31517 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 5 0 657989892 333324288 8568 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 3 0 0 0
[pid=31507/tid=31520] ppid=31505 vsize=325512 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

[pid=31519/tid=31536] ppid=31507 vsize=305160 CPUtime=0 cores=1,3,5,7
/proc/31519/task/31536/stat : 31536 (python) S 31507 31507 30970 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 657999917 312483840 7087 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604468630841280 0 0 -1 1 0 0 0
[pid=31522] ppid=31507 vsize=316024 CPUtime=0.25 cores=1,3,5,7
/proc/31522/stat : 31522 (python) S 31507 31507 30970 0 -1 4202560 1102 0 0 0 12 13 0 0 18 0 2 0 657989894 323608576 7091 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31522/statm: 79006 7091 400 364 0 14915 0
[pid=31524] ppid=31522 vsize=423400 CPUtime=97.83 cores=1,3,5,7
/proc/31524/stat : 31524 (adaptg2wsat2009) R 31522 31524 31524 0 -1 4202496 2620 0 0 0 9783 0 0 0 18 0 1 0 657989894 433561600 2358 33554432000 134512640 135107411 4291380736 18446744073709551615 134548336 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/31524/statm: 105850 2358 59 146 0 104726 0
[pid=31522/tid=31532] ppid=31507 vsize=316024 CPUtime=0 cores=1,3,5,7
/proc/31522/task/31532/stat : 31532 (python) S 31507 31507 30970 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657999819 323608576 7091 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604464183290624 0 0 -1 7 0 0 0
[pid=31533] ppid=31507 vsize=326172 CPUtime=0.27 cores=1,3,5,7
/proc/31533/stat : 31533 (python) S 31507 31507 30970 0 -1 4202560 1077 0 0 0 12 15 0 0 19 0 2 0 657999820 334000128 7095 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31533/statm: 81543 7095 400 364 0 17449 0
[pid=31535] ppid=31533 vsize=423400 CPUtime=100.23 cores=1,3,5,7
/proc/31535/stat : 31535 (adaptg2wsat2009) T 31533 31535 31535 0 -1 4202496 2621 0 0 0 10023 0 0 0 25 0 1 0 657999820 433561600 2357 33554432000 134512640 135107411 4286610304 18446744073709551615 134539089 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31535/statm: 105850 2357 59 146 0 104726 0
[pid=31533/tid=31558] ppid=31507 vsize=326172 CPUtime=0 cores=1,3,5,7
/proc/31533/task/31558/stat : 31558 (python) S 31507 31507 30970 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 658009847 334000128 7095 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604471145538752 0 0 -1 5 0 0 0
[pid=31537] ppid=31507 vsize=336136 CPUtime=0.26 cores=1,3,5,7
/proc/31537/stat : 31537 (python) S 31507 31507 30970 0 -1 4202560 1090 0 0 0 13 13 0 0 19 0 2 0 657999918 344203264 7140 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/31537/statm: 84034 7140 400 364 0 19937 0
[pid=31539] ppid=31537 vsize=1020308 CPUtime=100.3 cores=1,3,5,7
/proc/31539/stat : 31539 (gnovelty+2) T 31537 31539 31539 0 -1 4202496 1153 0 0 0 10030 0 0 0 25 0 1 0 657999919 1044795392 891 33554432000 134512640 135663704 4291799440 18446744073709551615 134524897 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31539/statm: 255077 891 113 282 0 254793 0
[pid=31537/tid=31562] ppid=31507 vsize=336136 CPUtime=0 cores=1,3,5,7
/proc/31537/task/31562/stat : 31562 (python) S 31507 31507 30970 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658009949 344203264 7140 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604471145540544 0 0 -1 1 0 0 0
[pid=31541] ppid=31507 vsize=346200 CPUtime=0.28 cores=1,3,5,7
/proc/31541/stat : 31541 (python) S 31507 31507 30970 0 -1 4202560 1105 0 0 0 14 14 0 0 19 0 2 0 657999925 354508800 7143 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31541/statm: 86550 7143 399 364 0 22450 0
[pid=31543] ppid=31541 vsize=253036 CPUtime=101.42 cores=1,3,5,7
/proc/31543/stat : 31543 (FH) T 31541 31543 31543 0 -1 4202496 2496 0 0 0 10142 0 0 0 25 0 1 0 657999925 259108864 2234 33554432000 134512640 134936988 4294781920 18446744073709551615 134552481 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31543/statm: 63259 2234 41 104 0 62763 0
[pid=31541/tid=31570] ppid=31507 vsize=346200 CPUtime=0 cores=1,3,5,7
/proc/31541/task/31570/stat : 31570 (python) S 31507 31507 30970 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658010355 354508800 7143 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604470122894272 0 0 -1 7 0 0 0
[pid=31545] ppid=31507 vsize=357128 CPUtime=0.26 cores=1,3,5,7
/proc/31545/stat : 31545 (python) S 31507 31507 30970 0 -1 4202560 1127 0 0 0 14 12 0 0 19 0 2 0 657999930 365699072 7154 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31545/statm: 89282 7154 406 364 0 25179 0
[pid=31547] ppid=31545 vsize=423400 CPUtime=99.99 cores=1,3,5,7
/proc/31547/stat : 31547 (adaptg2wsat2009) T 31545 31547 31547 0 -1 4202496 2626 0 0 0 9999 0 0 0 25 0 1 0 657999931 433561600 2357 33554432000 134512640 135107411 4286814576 18446744073709551615 134531709 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/31547/statm: 105850 2357 59 146 0 104726 0
[pid=31545/tid=31566] ppid=31507 vsize=357128 CPUtime=0 cores=1,3,5,7
/proc/31545/task/31566/stat : 31566 (python) S 31507 31507 30970 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658009958 365699072 7154 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604471126734144 0 0 -1 7 0 0 0
[pid=31559] ppid=31507 vsize=367044 CPUtime=0.26 cores=1,3,5,7
/proc/31559/stat : 31559 (python) S 31507 31507 30970 0 -1 4202560 1138 0 0 0 12 14 0 0 19 0 2 0 658009848 375853056 7205 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31559/statm: 91761 7205 406 364 0 27655 0
[pid=31561] ppid=31559 vsize=423400 CPUtime=100.07 cores=1,3,5,7
/proc/31561/stat : 31561 (adaptg2wsat2009) T 31559 31561 31561 0 -1 4202496 2625 0 0 0 10007 0 0 0 25 0 1 0 658009848 433561600 2357 33554432000 134512640 135107411 4291266784 18446744073709551615 134531942 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31561/statm: 105850 2357 59 146 0 104726 0
[pid=31559/tid=31581] ppid=31507 vsize=367044 CPUtime=0 cores=1,3,5,7
/proc/31559/task/31581/stat : 31581 (python) S 31507 31507 30970 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 658019875 375853056 7205 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604469892553536 0 0 -1 7 0 0 0
[pid=31563] ppid=31507 vsize=377528 CPUtime=0.27 cores=1,3,5,7
/proc/31563/stat : 31563 (python) S 31507 31507 30970 0 -1 4202560 1117 0 0 0 14 13 0 0 19 0 2 0 658009951 386588672 7203 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31563/statm: 94382 7203 400 364 0 30273 0
[pid=31565] ppid=31563 vsize=253032 CPUtime=99.27 cores=1,3,5,7
/proc/31565/stat : 31565 (FH) T 31563 31565 31565 0 -1 4202496 2503 0 0 0 9927 0 0 0 25 0 1 0 658009951 259104768 2234 33554432000 134512640 134936988 4288649776 18446744073709551615 134545269 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31565/statm: 63258 2234 41 104 0 62762 0
[pid=31563/tid=31585] ppid=31507 vsize=377528 CPUtime=0 cores=1,3,5,7
/proc/31563/task/31585/stat : 31585 (python) S 31507 31507 30970 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658019879 386588672 7203 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604471138828480 0 0 -1 7 0 0 0
[pid=31567] ppid=31507 vsize=387412 CPUtime=1.09 cores=1,3,5,7
/proc/31567/stat : 31567 (python) S 31507 31507 30970 0 -1 4202560 1176 0 0 0 52 57 0 0 18 0 2 0 658009959 396709888 7206 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31567/statm: 96853 7206 399 364 0 32741 0
[pid=31569] ppid=31567 vsize=268424 CPUtime=389.39 cores=1,3,5,7
/proc/31569/stat : 31569 (hybridGM3) T 31567 31569 31569 0 -1 4202496 6185 0 0 0 38938 1 0 0 25 0 1 0 658009959 274866176 5919 33554432000 134512640 135204385 4287695888 18446744073709551615 134538432 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/31569/statm: 67106 5919 63 169 0 66936 0
[pid=31567/tid=31688] ppid=31507 vsize=387412 CPUtime=0 cores=1,3,5,7
/proc/31567/task/31688/stat : 31688 (python) S 31507 31507 30970 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 658029825 396709888 7206 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604471125181696 0 0 -1 5 0 0 0
[pid=31571] ppid=31507 vsize=398320 CPUtime=0.27 cores=1,3,5,7
/proc/31571/stat : 31571 (python) S 31507 31507 30970 0 -1 4202560 1130 0 0 0 13 14 0 0 19 0 2 0 658010356 407879680 7211 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31571/statm: 99580 7211 400 364 0 35465 0
[pid=31573] ppid=31571 vsize=1020308 CPUtime=97.28 cores=1,3,5,7
/proc/31573/stat : 31573 (gnovelty+2) T 31571 31573 31573 0 -1 4202496 1160 0 0 0 9728 0 0 0 25 0 1 0 658010356 1044795392 891 33554432000 134512640 135663704 4290628720 18446744073709551615 134525255 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31573/statm: 255077 891 113 282 0 254793 0
[pid=31571/tid=31589] ppid=31507 vsize=398320 CPUtime=0 cores=1,3,5,7
/proc/31571/task/31589/stat : 31589 (python) S 31507 31507 30970 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 658020394 407879680 7211 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604468630839488 0 0 -1 5 0 0 0
[pid=31582] ppid=31507 vsize=408332 CPUtime=0.82 cores=1,3,5,7
/proc/31582/stat : 31582 (python) S 31507 31507 30970 0 -1 4202560 1176 0 0 0 40 42 0 0 18 0 2 0 658019876 418131968 7257 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31582/statm: 102083 7257 400 364 0 37965 0
[pid=31584] ppid=31582 vsize=268420 CPUtime=296.88 cores=1,3,5,7
/proc/31584/stat : 31584 (hybridGM3) T 31582 31584 31584 0 -1 4202496 6189 0 0 0 29688 0 0 0 25 0 1 0 658019876 274862080 5919 33554432000 134512640 135204385 4293180912 18446744073709551615 134538520 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/31584/statm: 67105 5919 63 169 0 66935 0
[pid=31582/tid=31708] ppid=31507 vsize=408332 CPUtime=0 cores=1,3,5,7
/proc/31582/task/31708/stat : 31708 (python) S 31507 31507 30970 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 658039744 418131968 7257 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604461348796224 0 0 -1 3 0 0 0
[pid=31586] ppid=31507 vsize=418732 CPUtime=0.28 cores=1,3,5,7
/proc/31586/stat : 31586 (python) S 31507 31507 30970 0 -1 4202560 1153 0 0 0 13 15 0 0 19 0 2 0 658019880 428781568 7261 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31586/statm: 104683 7261 400 364 0 40562 0
[pid=31588] ppid=31586 vsize=423400 CPUtime=99.08 cores=1,3,5,7
/proc/31588/stat : 31588 (adaptg2wsat2009) T 31586 31588 31588 0 -1 4202496 2625 0 0 0 9908 0 0 0 25 0 1 0 658019880 433561600 2357 33554432000 134512640 135107411 4292541456 18446744073709551615 134531340 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31588/statm: 105850 2357 59 146 0 104726 0
[pid=31586/tid=31684] ppid=31507 vsize=418732 CPUtime=0 cores=1,3,5,7
/proc/31586/task/31684/stat : 31684 (python) S 31507 31507 30970 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 658029807 428781568 7261 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604460305888384 0 0 -1 7 0 0 0
[pid=31590] ppid=31507 vsize=428844 CPUtime=0.78 cores=1,3,5,7
/proc/31590/stat : 31590 (python) S 31507 31507 30970 0 -1 4202560 1234 0 0 0 40 38 0 0 18 0 2 0 658020396 439136256 7272 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/31590/statm: 107211 7272 407 364 0 43087 0
[pid=31592] ppid=31590 vsize=268420 CPUtime=289.46 cores=1,3,5,7
/proc/31592/stat : 31592 (hybridGM3) R 31590 31592 31592 0 -1 4202496 6198 0 0 0 28946 0 0 0 25 0 1 0 658020396 274862080 5920 33554432000 134512640 135204385 4287639680 18446744073709551615 134537726 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/31592/statm: 67105 5920 63 169 0 66935 0
[pid=31590/tid=31709] ppid=31507 vsize=428844 CPUtime=0 cores=1,3,5,7
/proc/31590/task/31709/stat : 31709 (python) S 31507 31507 30970 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 658040262 439136256 7272 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604457680737344 0 0 -1 1 0 0 0
[pid=31685] ppid=31507 vsize=439256 CPUtime=0.54 cores=1,3,5,7
/proc/31685/stat : 31685 (python) S 31507 31507 30970 0 -1 4202560 1214 0 0 0 26 28 0 0 19 0 2 0 658029809 449798144 7277 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389128139 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/31685/statm: 109814 7277 408 364 0 45687 0
[pid=31687] ppid=31685 vsize=268420 CPUtime=198.63 cores=1,3,5,7
/proc/31687/stat : 31687 (hybridGM3) T 31685 31687 31687 0 -1 4202496 6194 0 0 0 19863 0 0 0 25 0 1 0 658029809 274862080 5920 33554432000 134512640 135204385 4290147264 18446744073709551615 134538368 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/31687/statm: 67105 5920 63 169 0 66935 0
[pid=31685/tid=31741] ppid=31507 vsize=439256 CPUtime=0 cores=1,3,5,7
/proc/31685/task/31741/stat : 31741 (python) S 31507 31507 30970 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 658049675 449798144 7277 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 514 18446604471126733248 0 0 -1 7 0 0 0
[pid=31738] ppid=31507 vsize=449776 CPUtime=0 cores=1,3,5,7
/proc/31738/stat : 31738 (python) S 31507 31507 30970 0 -1 4202560 858 0 0 0 0 0 0 0 18 0 1 0 658049575 460570624 7290 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 11513347448307187711 0 0 17 7 0 0 0
/proc/31738/statm: 112444 7290 357 364 0 48314 0
[pid=31740] ppid=31738 vsize=461504 CPUtime=1.29 cores=1,3,5,7
/proc/31740/stat : 31740 (TNM) R 31738 31740 31740 0 -1 4202496 2471 0 0 0 129 0 0 0 25 0 1 0 658049575 472580096 2195 33554432000 134512640 135008748 4293775872 18446744073709551615 134527299 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/31740/statm: 115376 2195 46 122 0 114276 0
[pid=31742] ppid=31507 vsize=460204 CPUtime=0 cores=1,3,5,7
/proc/31742/stat : 31742 (python) S 31507 31507 30970 0 -1 4202560 816 0 0 0 0 0 0 0 19 0 1 0 658049677 471248896 7283 33554432000 4194304 5683644 140735479138560 18446744073709551615 255377321542 0 0 16781312 514 11513351296597884927 0 0 17 7 0 0 0
/proc/31742/statm: 115051 7283 346 364 0 50918 0
[pid=31744] ppid=31742 vsize=248616 CPUtime=0.32 cores=1,3,5,7
/proc/31744/stat : 31744 (FH) R 31742 31744 31744 0 -1 4202496 1484 0 0 0 32 0 0 0 21 0 1 0 658049677 254582784 1213 33554432000 134512640 134936988 4290087136 18446744073709551615 134523665 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/31744/statm: 62154 1213 37 104 0 61658 0
[pid=31507/tid=31514] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31514/stat : 31514 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 22 0 19 0 657989891 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 5 0 0 0
[pid=31507/tid=31517] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31517/stat : 31517 (python) S 31505 31507 30970 0 -1 4202560 24 934 0 0 0 0 0 0 20 0 19 0 657989892 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 3 0 0 0
[pid=31507/tid=31520] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31520/stat : 31520 (python) S 31505 31507 30970 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 19 0 657989893 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31523] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31523/stat : 31523 (python) S 31505 31507 30970 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 19 0 657989894 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31534] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31534/stat : 31534 (python) S 31505 31507 30970 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 19 0 657999820 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31538] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31538/stat : 31538 (python) S 31505 31507 30970 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 657999919 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31542] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31542/stat : 31542 (python) S 31505 31507 30970 0 -1 4202560 28 934 0 0 0 0 0 0 21 0 19 0 657999925 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31546] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31546/stat : 31546 (python) S 31505 31507 30970 0 -1 4202560 27 934 0 0 0 0 0 0 22 0 19 0 657999931 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31560] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31560/stat : 31560 (python) S 31505 31507 30970 0 -1 4202560 23 934 0 0 0 0 0 0 20 0 19 0 658009848 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31564] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31564/stat : 31564 (python) S 31505 31507 30970 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 658009951 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31568] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31568/stat : 31568 (python) S 31505 31507 30970 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 19 0 658009959 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31572] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31572/stat : 31572 (python) S 31505 31507 30970 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 19 0 658010356 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31583] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31583/stat : 31583 (python) S 31505 31507 30970 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 19 0 658019876 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31587] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31587/stat : 31587 (python) S 31505 31507 30970 0 -1 4202560 27 934 0 0 0 0 0 0 21 0 19 0 658019880 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31591] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31591/stat : 31591 (python) S 31505 31507 30970 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 19 0 658020396 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31686] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31686/stat : 31686 (python) S 31505 31507 30970 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 658029809 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31739] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31739/stat : 31739 (python) S 31505 31507 30970 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 658049575 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
[pid=31507/tid=31743] ppid=31505 vsize=469860 CPUtime=0 cores=1,3,5,7
/proc/31507/task/31743/stat : 31743 (python) S 31505 31507 30970 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 658049677 481136640 8844 33554432000 4194304 5683644 140735479138560 18446744073709551615 255389125041 0 0 16781312 2 18446604464367268992 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2380.71
Current children cumulated vsize (KiB) 16093764

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31507 and gives
#  childrusage.ru_utime.tv_sec=3
#  childrusage.ru_utime.tv_usec=617450
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=405786
# CPU time returned by wait4() is 5.02324
# while last known CPU time is 2380.71
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 600.648
CPU time (s): 2380.71
CPU user time (s): 2377.32
CPU system time (s): 3.39
CPU usage (%): 396.357
Max. virtual memory (cumulated for all children) (KiB): 16093764

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3.61745
system time used= 1.40579
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 39107
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2256
involuntary context switches= 108

runsolver used 3.83342 second user time and 10.0865 second system time

The end

Launcher Data 

Begin job on node135 at 2011-05-15 14:51:58
IDJOB=3378228
IDBENCH=82637
IDSOLVER=1859
FILE ID=node135/3378228-1305463918
RUNJOBID= node135-1305462647-30989
PBS_JOBID= 13324970
Free space on /tmp= 73328 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S1993071608-084-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378228-1305463918/watcher-3378228-1305463918 -o /tmp/evaluation-result-3378228-1305463918/solver-3378228-1305463918 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378228-1305463918.cnf -seed 1373343021 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 6a416a8eca39558555cbe083cf0c9cff
RANDOM SEED=1373343021

node135.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6640.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      11105060 kB
Buffers:        297476 kB
Cached:        6342848 kB
SwapCached:      50684 kB
Active:       10722376 kB
Inactive:     10857044 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      11105060 kB
SwapTotal:    67111528 kB
SwapFree:     66050672 kB
Dirty:           33400 kB
Writeback:           0 kB
AnonPages:    14894024 kB
Mapped:          26020 kB
Slab:           144172 kB
PageTables:      58832 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 69749928 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73296 MiB
End job on node135 at 2011-05-15 15:02:01