Trace number 3378623

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 3091.48 1002.13

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v90-c8010-S741717046-057.UNKNOWN.cnf
MD5SUM1f00de1fd8adbfbf4cc573ef44af531f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables90
Number of clauses8010
Sum of the clauses size56070
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58010

Solver Data 

0.34/0.63	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.34/0.63	c INFO: solving HOME/instance-3378623-1305466511.cnf
0.34/0.63	c INFO: computing task features
0.34/0.68	c INFO: computed variable-clause graph statistics
0.34/0.69	c INFO: computed clause balance statistics
0.62/0.96	c INFO: computed variable balance statistics
0.73/1.07	c INFO: computed small-clause counts
0.73/1.07	c INFO: computed Horn-clause counts
0.73/1.08	c INFO: computed variable graph statistics
0.73/1.08	c INFO: collected features for HOME/instance-3378623-1305466511.cnf in 0.45 s
0.73/1.09	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.47)
0.73/1.10	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.11)
0.83/1.11	c INFO: running adaptg2wsat2009++@0 for 97 with 1199 remaining (b = 0.06)
0.83/1.12	c INFO: running adaptg2wsat2009++@0 for 97 with 1199 remaining (b = 0.05)
301.74/100.40	c INFO: running adaptg2wsat2009++@0 for 97 with 1100 remaining (b = 0.04)
316.68/105.45	c INFO: running gnovelty+2@0 for 97 with 1095 remaining (b = 0.03)
448.63/149.52	c INFO: running FH@0 for 97 with 1051 remaining (b = 0.03)
577.36/192.68	c INFO: running adaptg2wsat2009++@0 for 97 with 1007 remaining (b = 0.02)
619.15/206.69	c INFO: running adaptg2wsat2009++@0 for 97 with 993 remaining (b = 0.02)
691.68/228.82	c INFO: running FH@0 for 97 with 971 remaining (b = 0.02)
903.90/295.06	c INFO: running hybridGM3@0 for 194 with 905 remaining (b = 0.03)
980.54/317.04	c INFO: running gnovelty+2@0 for 97 with 883 remaining (b = 0.01)
1028.97/333.14	c INFO: running hybridGM3@0 for 194 with 867 remaining (b = 0.02)
1052.99/341.19	c INFO: running adaptg2wsat2009++@0 for 97 with 859 remaining (b = 0.01)
1368.90/442.51	c INFO: running adaptg2wsat2009++@99 for 97 with 758 remaining (b = 0.01)
1574.87/511.62	c INFO: running hybridGM3@0 for 194 with 689 remaining (b = 0.02)
1655.92/538.81	c INFO: running march_hi@0 for 388 with 661 remaining (b = 0.04)
1689.01/549.97	c INFO: running hybridGM3@100 for 97 with 650 remaining (b = 0.01)
1766.17/575.85	c INFO: running hybridGM3@101 for 97 with 624 remaining (b = 0.01)
2049.44/657.27	c INFO: running hybridGM3@296 for 97 with 543 remaining (b = 0.01)
2180.27/701.20	c INFO: running FH@0 for 97 with 499 remaining (b = 0.01)
2440.98/788.63	c INFO: running adaptg2wsat2009++@99 for 97 with 412 remaining (b = 0.01)
2524.01/816.58	c INFO: running hybridGM3@96 for 97 with 384 remaining (b = 0.01)
2604.39/843.58	c INFO: running hybridGM3@393 for 97 with 357 remaining (b = 0.01)
2748.58/891.93	c INFO: running gnovelty+2@0 for 97 with 308 remaining (b = 0.01)
2877.15/929.91	c INFO: running hybridGM3@193 for 97 with 270 remaining (b = 0.01)
3072.42/995.67	c INFO: running march_hi@199 for 204 with 204 remaining (b = 0.03)
3088.51/1001.02	c INFO: running hybridGM3@0 for 199 with 199 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378623-1305466511/watcher-3378623-1305466511 -o /tmp/evaluation-result-3378623-1305466511/solver-3378623-1305466511 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378623-1305466511.cnf -seed 1381436808 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.15 9.16 8.84 6/382 2456
/proc/meminfo: memFree=22551492/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=15384 CPUtime=0 cores=0,2,4,6
/proc/2456/stat : 2456 (python) S 2454 2456 1220 0 -1 4202496 795 0 0 0 0 0 0 0 25 0 1 0 639066480 15753216 682 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261614592 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/2456/statm: 3846 682 322 364 0 333 0

[startup+0.0195021 s]
/proc/loadavg: 9.15 9.16 8.84 6/382 2456
/proc/meminfo: memFree=22551492/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=16240 CPUtime=0 cores=0,2,4,6
/proc/2456/stat : 2456 (python) R 2454 2456 1220 0 -1 4202496 1050 0 0 0 0 0 0 0 25 0 1 0 639066480 16629760 922 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261310415 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/2456/statm: 4060 922 381 364 0 547 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16240

[startup+0.100539 s]
/proc/loadavg: 9.15 9.16 8.84 6/382 2456
/proc/meminfo: memFree=22551492/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=93960 CPUtime=0.05 cores=0,2,4,6
/proc/2456/stat : 2456 (python) S 2454 2456 1220 0 -1 4202496 2693 0 0 0 4 1 0 0 25 0 1 0 639066480 96215040 2407 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261614592 0 2147483391 16781312 2 18446744071699207358 0 0 17 2 0 0 0
/proc/2456/statm: 23490 2407 764 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 93960

[startup+0.300452 s]
/proc/loadavg: 9.15 9.16 8.84 6/382 2456
/proc/meminfo: memFree=22551492/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=135752 CPUtime=0.16 cores=0,2,4,6
/proc/2456/stat : 2456 (python) R 2454 2456 1220 0 -1 4202496 5964 932 1 0 12 4 0 0 18 0 1 0 639066480 139010048 4152 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261612549 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/2456/statm: 33938 4152 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.16
Current children cumulated vsize (KiB) 135752

[startup+0.700449 s]
/proc/loadavg: 9.15 9.16 8.84 6/382 2456
/proc/meminfo: memFree=22551492/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=280236 CPUtime=0.42 cores=0,2,4,6
/proc/2456/stat : 2456 (python) R 2454 2456 1220 0 -1 4202496 9994 932 1 0 34 8 0 0 17 0 1 0 639066480 286961664 7444 33554432000 4194304 5683644 140734273052512 18446744073709551615 4303406 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/2456/statm: 70059 7444 1826 364 0 5983 0
Current children cumulated CPU time (s) 0.42
Current children cumulated vsize (KiB) 280236

[startup+1.50024 s]
/proc/loadavg: 9.15 9.16 8.84 7/383 2461
/proc/meminfo: memFree=22526624/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=320748 CPUtime=0.84 cores=0,2,4,6
/proc/2456/stat : 2456 (python) S 2454 2456 1220 0 -1 4202496 12915 932 1 0 75 9 0 0 23 0 5 0 639066480 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/2456/statm: 80187 7386 1883 364 0 16096 0
Current children cumulated CPU time (s) 0.84
Current children cumulated vsize (KiB) 320748

[startup+3.10095 s]
/proc/loadavg: 9.13 9.16 8.84 10/395 2473
/proc/meminfo: memFree=22507796/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=320748 CPUtime=0.84 cores=0,2,4,6
/proc/2456/stat : 2456 (python) S 2454 2456 1220 0 -1 4202496 12915 932 1 0 75 9 0 0 23 0 5 0 639066480 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/2456/statm: 80187 7386 1883 364 0 16096 0
[pid=2462] ppid=2456 vsize=279416 CPUtime=0 cores=0,2,4,6
/proc/2462/stat : 2462 (python) S 2456 2456 1220 0 -1 4202560 725 0 0 0 0 0 0 0 18 0 1 0 639066590 286121984 5793 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636623 0 0 16781312 514 10740214577242276097 0 0 17 6 0 0 0
/proc/2462/statm: 69854 5793 354 364 0 5772 0
[pid=2464] ppid=2462 vsize=1018088 CPUtime=2 cores=0,2,4,6
/proc/2464/stat : 2464 (gnovelty+2) R 2462 2464 2464 0 -1 4202496 566 0 0 0 200 0 0 0 25 0 1 0 639066590 1042522112 307 33554432000 134512640 135663704 4290192432 18446744073709551615 134525309 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2464/statm: 254522 307 113 282 0 254238 0
[pid=2465] ppid=2456 vsize=289992 CPUtime=0 cores=0,2,4,6
/proc/2465/stat : 2465 (python) S 2456 2456 1220 0 -1 4202560 729 0 0 0 0 0 0 0 18 0 1 0 639066591 296951808 5839 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 4428969455296413953 0 0 17 2 0 0 0
/proc/2465/statm: 72498 5839 348 364 0 8413 0
[pid=2467] ppid=2465 vsize=1018088 CPUtime=1.99 cores=0,2,4,6
/proc/2467/stat : 2467 (gnovelty+2) R 2465 2467 2467 0 -1 4202496 569 0 0 0 199 0 0 0 25 0 1 0 639066591 1042522112 306 33554432000 134512640 135663704 4287496704 18446744073709551615 134546940 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2467/statm: 254522 306 113 282 0 254238 0
[pid=2468] ppid=2456 vsize=300248 CPUtime=0 cores=0,2,4,6
/proc/2468/stat : 2468 (python) S 2456 2456 1220 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 639066592 307453952 5843 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 15935750615867588607 0 0 17 6 0 0 0
/proc/2468/statm: 75062 5843 348 364 0 10974 0
[pid=2472] ppid=2468 vsize=417192 CPUtime=1.95 cores=0,2,4,6
/proc/2472/stat : 2472 (adaptg2wsat2009) R 2468 2472 2472 0 -1 4202496 759 0 0 0 195 0 0 0 25 0 1 0 639066593 427204608 496 33554432000 134512640 135107411 4289782656 18446744073709551615 134532288 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2472/statm: 104298 496 59 146 0 103174 0
[pid=2470] ppid=2456 vsize=310504 CPUtime=0 cores=0,2,4,6
/proc/2470/stat : 2470 (python) S 2456 2456 1220 0 -1 4202560 739 0 0 0 0 0 0 0 18 0 1 0 639066593 317956096 5847 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 15935746767576891391 0 0 17 6 0 0 0
/proc/2470/statm: 77626 5847 348 364 0 13535 0
[pid=2473] ppid=2470 vsize=417048 CPUtime=1.02 cores=0,2,4,6
/proc/2473/stat : 2473 (adaptg2wsat2009) R 2470 2473 2473 0 -1 4202496 680 0 0 0 102 0 0 0 25 0 1 0 639066594 427057152 419 33554432000 134512640 135107411 4293223856 18446744073709551615 134523430 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2473/statm: 104262 419 49 146 0 103138 0
[pid=2456/tid=2463] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2463/stat : 2463 (python) S 2454 2456 1220 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 5 0 639066590 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2466] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2466/stat : 2466 (python) S 2454 2456 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 639066591 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2469] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2469/stat : 2469 (python) S 2454 2456 1220 0 -1 4202560 35 932 0 0 0 0 0 0 22 0 5 0 639066592 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2471] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2471/stat : 2471 (python) S 2454 2456 1220 0 -1 4202560 17 932 0 0 0 0 0 0 24 0 5 0 639066593 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 7.8
Current children cumulated vsize (KiB) 4371324

[startup+6.30139 s]
/proc/loadavg: 9.13 9.16 8.84 10/395 2473
/proc/meminfo: memFree=22502168/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=320748 CPUtime=0.84 cores=0,2,4,6
/proc/2456/stat : 2456 (python) S 2454 2456 1220 0 -1 4202496 12915 932 1 0 75 9 0 0 23 0 5 0 639066480 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/2456/statm: 80187 7386 1883 364 0 16096 0
[pid=2462] ppid=2456 vsize=279416 CPUtime=0.01 cores=0,2,4,6
/proc/2462/stat : 2462 (python) S 2456 2456 1220 0 -1 4202560 889 0 0 0 0 1 0 0 18 0 1 0 639066590 286121984 5816 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636623 0 0 16781312 514 10740214577242276097 0 0 17 6 0 0 0
/proc/2462/statm: 69854 5816 377 364 0 5772 0
[pid=2464] ppid=2462 vsize=1018088 CPUtime=5.2 cores=0,2,4,6
/proc/2464/stat : 2464 (gnovelty+2) R 2462 2464 2464 0 -1 4202496 566 0 0 0 520 0 0 0 25 0 1 0 639066590 1042522112 307 33554432000 134512640 135663704 4290192432 18446744073709551615 134525260 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2464/statm: 254522 307 113 282 0 254238 0
[pid=2465] ppid=2456 vsize=289992 CPUtime=0.01 cores=0,2,4,6
/proc/2465/stat : 2465 (python) S 2456 2456 1220 0 -1 4202560 889 0 0 0 0 1 0 0 18 0 1 0 639066591 296951808 5862 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 4428969455296413953 0 0 17 2 0 0 0
/proc/2465/statm: 72498 5862 371 364 0 8413 0
[pid=2467] ppid=2465 vsize=1018088 CPUtime=5.18 cores=0,2,4,6
/proc/2467/stat : 2467 (gnovelty+2) R 2465 2467 2467 0 -1 4202496 569 0 0 0 518 0 0 0 25 0 1 0 639066591 1042522112 306 33554432000 134512640 135663704 4287496704 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2467/statm: 254522 306 113 282 0 254238 0
[pid=2468] ppid=2456 vsize=300248 CPUtime=0.01 cores=0,2,4,6
/proc/2468/stat : 2468 (python) S 2456 2456 1220 0 -1 4202560 895 0 0 0 1 0 0 0 18 0 1 0 639066592 307453952 5866 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 15935750615867556100 0 0 17 6 0 0 0
/proc/2468/statm: 75062 5866 371 364 0 10974 0
[pid=2472] ppid=2468 vsize=417192 CPUtime=5.06 cores=0,2,4,6
/proc/2472/stat : 2472 (adaptg2wsat2009) R 2468 2472 2472 0 -1 4202496 759 0 0 0 506 0 0 0 25 0 1 0 639066593 427204608 496 33554432000 134512640 135107411 4289782656 18446744073709551615 134531232 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2472/statm: 104298 496 59 146 0 103174 0
[pid=2470] ppid=2456 vsize=310504 CPUtime=0.01 cores=0,2,4,6
/proc/2470/stat : 2470 (python) S 2456 2456 1220 0 -1 4202560 903 0 0 0 1 0 0 0 18 0 1 0 639066593 317956096 5870 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 15935746767576858882 0 0 17 6 0 0 0
/proc/2470/statm: 77626 5870 371 364 0 13535 0
[pid=2473] ppid=2470 vsize=417192 CPUtime=2.61 cores=0,2,4,6
/proc/2473/stat : 2473 (adaptg2wsat2009) R 2470 2473 2473 0 -1 4202496 757 0 0 0 261 0 0 0 25 0 1 0 639066594 427204608 496 33554432000 134512640 135107411 4293223856 18446744073709551615 134532107 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2473/statm: 104298 496 59 146 0 103174 0
[pid=2456/tid=2463] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2463/stat : 2463 (python) S 2454 2456 1220 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 5 0 639066590 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2466] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2466/stat : 2466 (python) S 2454 2456 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 639066591 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2469] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2469/stat : 2469 (python) S 2454 2456 1220 0 -1 4202560 35 932 0 0 0 0 0 0 22 0 5 0 639066592 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2471] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2471/stat : 2471 (python) S 2454 2456 1220 0 -1 4202560 17 932 0 0 0 0 0 0 24 0 5 0 639066593 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 18.93
Current children cumulated vsize (KiB) 4371468

[startup+12.7013 s]
/proc/loadavg: 9.11 9.15 8.84 10/395 2474
/proc/meminfo: memFree=22498464/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=320748 CPUtime=0.84 cores=0,2,4,6
/proc/2456/stat : 2456 (python) S 2454 2456 1220 0 -1 4202496 12915 932 1 0 75 9 0 0 23 0 5 0 639066480 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/2456/statm: 80187 7386 1883 364 0 16096 0
[pid=2462] ppid=2456 vsize=279416 CPUtime=0.03 cores=0,2,4,6
/proc/2462/stat : 2462 (python) S 2456 2456 1220 0 -1 4202560 892 0 0 0 1 2 0 0 18 0 1 0 639066590 286121984 5816 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636623 0 0 16781312 514 1417279284296253697 0 0 17 6 0 0 0
/proc/2462/statm: 69854 5816 377 364 0 5772 0
[pid=2464] ppid=2462 vsize=1018088 CPUtime=11.6 cores=0,2,4,6
/proc/2464/stat : 2464 (gnovelty+2) R 2462 2464 2464 0 -1 4202496 566 0 0 0 1160 0 0 0 25 0 1 0 639066590 1042522112 307 33554432000 134512640 135663704 4290192432 18446744073709551615 134525309 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/2464/statm: 254522 307 113 282 0 254238 0
[pid=2465] ppid=2456 vsize=289992 CPUtime=0.02 cores=0,2,4,6
/proc/2465/stat : 2465 (python) S 2456 2456 1220 0 -1 4202560 892 0 0 0 1 1 0 0 18 0 1 0 639066591 296951808 5862 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 1417279284296253697 0 0 17 2 0 0 0
/proc/2465/statm: 72498 5862 371 364 0 8413 0
[pid=2467] ppid=2465 vsize=1018088 CPUtime=9.81 cores=0,2,4,6
/proc/2467/stat : 2467 (gnovelty+2) R 2465 2467 2467 0 -1 4202496 569 0 0 0 981 0 0 0 25 0 1 0 639066591 1042522112 306 33554432000 134512640 135663704 4287496704 18446744073709551615 134524503 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2467/statm: 254522 306 113 282 0 254238 0
[pid=2468] ppid=2456 vsize=300248 CPUtime=0.02 cores=0,2,4,6
/proc/2468/stat : 2468 (python) S 2456 2456 1220 0 -1 4202560 900 0 0 0 1 1 0 0 18 0 1 0 639066592 307453952 5866 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 1417279284296253700 0 0 17 6 0 0 0
/proc/2468/statm: 75062 5866 371 364 0 10974 0
[pid=2472] ppid=2468 vsize=417192 CPUtime=11.33 cores=0,2,4,6
/proc/2472/stat : 2472 (adaptg2wsat2009) R 2468 2472 2472 0 -1 4202496 759 0 0 0 1133 0 0 0 25 0 1 0 639066593 427204608 496 33554432000 134512640 135107411 4289782656 18446744073709551615 134532349 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2472/statm: 104298 496 59 146 0 103174 0
[pid=2470] ppid=2456 vsize=310504 CPUtime=0.02 cores=0,2,4,6
/proc/2470/stat : 2470 (python) S 2456 2456 1220 0 -1 4202560 905 0 0 0 1 1 0 0 18 0 1 0 639066593 317956096 5870 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 1417279284296253698 0 0 17 6 0 0 0
/proc/2470/statm: 77626 5870 371 364 0 13535 0
[pid=2473] ppid=2470 vsize=417192 CPUtime=5.79 cores=0,2,4,6
/proc/2473/stat : 2473 (adaptg2wsat2009) R 2470 2473 2473 0 -1 4202496 757 0 0 0 579 0 0 0 25 0 1 0 639066594 427204608 496 33554432000 134512640 135107411 4293223856 18446744073709551615 134532326 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2473/statm: 104298 496 59 146 0 103174 0
[pid=2456/tid=2463] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2463/stat : 2463 (python) S 2454 2456 1220 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 5 0 639066590 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2466] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2466/stat : 2466 (python) S 2454 2456 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 5 0 639066591 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2469] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2469/stat : 2469 (python) S 2454 2456 1220 0 -1 4202560 35 932 0 0 0 0 0 0 22 0 5 0 639066592 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2471] ppid=2454 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2471/stat : 2471 (python) S 2454 2456 1220 0 -1 4202560 17 932 0 0 0 0 0 0 24 0 5 0 639066593 328445952 7386 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39.46
Current children cumulated vsize (KiB) 4371468

[startup+25.501 s]
/proc/loadavg: 9.18 9.16 8.85 10/395 2474
/proc/meminfo: memFree=22492144/32951124 swapFree=67057896/67111528
[pid=2456] ppid=2454 vsize=320748 CPUtime=0.84 cores=0,2,4,6

################
# More data... #
################

[pid=2473] ppid=2470 vsize=417192 CPUtime=194.35 cores=0,2,4,6
/proc/2473/stat : 2473 (adaptg2wsat2009) T 2470 2473 2473 0 -1 4202496 757 0 0 0 19435 0 0 0 25 0 1 0 639066594 427204608 496 33554432000 134512640 135107411 4293223856 18446744073709551615 134531830 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/2473/statm: 104298 496 59 146 0 103174 0
[pid=2470/tid=2491] ppid=2456 vsize=310504 CPUtime=0 cores=0,2,4,6
/proc/2470/task/2491/stat : 2491 (python) S 2456 2456 1220 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 639085748 317956096 5900 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604443738121536 0 0 -1 6 0 0 0
[pid=2478] ppid=2456 vsize=320760 CPUtime=0.59 cores=0,2,4,6
/proc/2478/stat : 2478 (python) S 2456 2456 1220 0 -1 4202560 1152 0 0 0 31 28 0 0 18 0 2 0 639076521 328458240 5904 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/2478/statm: 80190 5904 400 364 0 16096 0
[pid=2480] ppid=2478 vsize=417192 CPUtime=199.38 cores=0,2,4,6
/proc/2480/stat : 2480 (adaptg2wsat2009) T 2478 2480 2480 0 -1 4202496 760 0 0 0 19938 0 0 0 25 0 1 0 639076521 427204608 496 33554432000 134512640 135107411 4288855840 18446744073709551615 134531942 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/2480/statm: 104298 496 59 146 0 103174 0
[pid=2478/tid=2496] ppid=2456 vsize=320760 CPUtime=0 cores=0,2,4,6
/proc/2478/task/2496/stat : 2496 (python) S 2456 2456 1220 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 639087149 328458240 5904 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604453961272064 0 0 -1 4 0 0 0
[pid=2482] ppid=2456 vsize=331016 CPUtime=0.34 cores=0,2,4,6
/proc/2482/stat : 2482 (python) S 2456 2456 1220 0 -1 4202560 1099 0 0 0 18 16 0 0 19 0 2 0 639077026 338960384 5908 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2482/statm: 82754 5908 400 364 0 18657 0
[pid=2484] ppid=2482 vsize=1018088 CPUtime=101.06 cores=0,2,4,6
/proc/2484/stat : 2484 (gnovelty+2) T 2482 2484 2484 0 -1 4202496 566 0 0 0 10106 0 0 0 25 0 1 0 639077026 1042522112 307 33554432000 134512640 135663704 4294275488 18446744073709551615 134546906 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/2484/statm: 254522 307 113 282 0 254238 0
[pid=2482/tid=2500] ppid=2456 vsize=331016 CPUtime=0 cores=0,2,4,6
/proc/2482/task/2500/stat : 2500 (python) S 2456 2456 1220 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 639089362 338960384 5908 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604443738123328 0 0 -1 4 0 0 0
[pid=2487] ppid=2456 vsize=341272 CPUtime=0.41 cores=0,2,4,6
/proc/2487/stat : 2487 (python) S 2456 2456 1220 0 -1 4202560 1109 0 0 0 22 19 0 0 19 0 2 0 639081433 349462528 5912 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2487/statm: 85318 5912 400 364 0 21218 0
[pid=2489] ppid=2487 vsize=247344 CPUtime=97.08 cores=0,2,4,6
/proc/2489/stat : 2489 (FH) T 2487 2489 2489 0 -1 4202496 705 0 0 0 9708 0 0 0 25 0 1 0 639081433 253280256 443 33554432000 134512640 134936988 4294291696 18446744073709551615 134553498 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/2489/statm: 61836 443 41 104 0 61340 0
[pid=2487/tid=2506] ppid=2456 vsize=341272 CPUtime=0 cores=0,2,4,6
/proc/2487/task/2506/stat : 2506 (python) S 2456 2456 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 639095985 349462528 5912 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604442335199424 0 0 -1 6 0 0 0
[pid=2492] ppid=2456 vsize=351528 CPUtime=0.35 cores=0,2,4,6
/proc/2492/stat : 2492 (python) S 2456 2456 1220 0 -1 4202560 1152 0 0 0 18 17 0 0 19 0 2 0 639085749 359964672 5922 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/2492/statm: 87882 5922 406 364 0 23779 0
[pid=2494] ppid=2492 vsize=417192 CPUtime=101.53 cores=0,2,4,6
/proc/2494/stat : 2494 (adaptg2wsat2009) T 2492 2494 2494 0 -1 4202496 764 0 0 0 10153 0 0 0 25 0 1 0 639085750 427204608 495 33554432000 134512640 135107411 4293905104 18446744073709551615 134532229 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/2494/statm: 104298 495 59 146 0 103174 0
[pid=2492/tid=2516] ppid=2456 vsize=351528 CPUtime=0 cores=0,2,4,6
/proc/2492/task/2516/stat : 2516 (python) S 2456 2456 1220 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 639098183 359964672 5922 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604449606382720 0 0 -1 4 0 0 0
[pid=2497] ppid=2456 vsize=361784 CPUtime=0.38 cores=0,2,4,6
/proc/2497/stat : 2497 (python) S 2456 2456 1220 0 -1 4202560 1151 0 0 0 18 20 0 0 19 0 2 0 639087150 370466816 5926 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2497/statm: 90446 5926 406 364 0 26340 0
[pid=2499] ppid=2497 vsize=417192 CPUtime=100.92 cores=0,2,4,6
/proc/2499/stat : 2499 (adaptg2wsat2009) T 2497 2499 2499 0 -1 4202496 763 0 0 0 10092 0 0 0 25 0 1 0 639087152 427204608 496 33554432000 134512640 135107411 4287033296 18446744073709551615 134531428 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/2499/statm: 104298 496 59 146 0 103174 0
[pid=2497/tid=2524] ppid=2456 vsize=361784 CPUtime=0 cores=0,2,4,6
/proc/2497/task/2524/stat : 2524 (python) S 2456 2456 1220 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 639100598 370466816 5926 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604453956682944 0 0 -1 4 0 0 0
[pid=2501] ppid=2456 vsize=372040 CPUtime=0.29 cores=0,2,4,6
/proc/2501/stat : 2501 (python) S 2456 2456 1220 0 -1 4202560 1117 0 0 0 16 13 0 0 19 0 2 0 639089363 380968960 5924 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/2501/statm: 93010 5924 400 364 0 28901 0
[pid=2503] ppid=2501 vsize=247348 CPUtime=97.48 cores=0,2,4,6
/proc/2503/stat : 2503 (FH) T 2501 2503 2503 0 -1 4202496 709 0 0 0 9748 0 0 0 25 0 1 0 639089364 253284352 443 33554432000 134512640 134936988 4287400416 18446744073709551615 134553579 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/2503/statm: 61837 443 41 104 0 61341 0
[pid=2501/tid=2520] ppid=2456 vsize=372040 CPUtime=0 cores=0,2,4,6
/proc/2501/task/2520/stat : 2520 (python) S 2456 2456 1220 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 639099794 380968960 5924 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604446823407936 0 0 -1 0 0 0 0
[pid=2507] ppid=2456 vsize=382296 CPUtime=1.15 cores=0,2,4,6
/proc/2507/stat : 2507 (python) S 2456 2456 1220 0 -1 4202560 1167 0 0 0 59 56 0 0 18 0 2 0 639095986 391471104 5928 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/2507/statm: 95574 5928 400 364 0 31462 0
[pid=2509] ppid=2507 vsize=268420 CPUtime=292.27 cores=0,2,4,6
/proc/2509/stat : 2509 (hybridGM3) T 2507 2509 2509 0 -1 4202496 4937 0 0 0 29226 1 0 0 25 0 1 0 639095987 274862080 4671 33554432000 134512640 135204385 4289225424 18446744073709551615 134536749 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/2509/statm: 67105 4671 64 169 0 66935 0
[pid=2507/tid=2543] ppid=2456 vsize=382296 CPUtime=0 cores=0,2,4,6
/proc/2507/task/2543/stat : 2543 (python) S 2456 2456 1220 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 639124065 391471104 5928 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604446399117120 0 0 -1 4 0 0 0
[pid=2517] ppid=2456 vsize=392552 CPUtime=0.56 cores=0,2,4,6
/proc/2517/stat : 2517 (python) S 2456 2456 1220 0 -1 4202560 1162 0 0 0 29 27 0 0 19 0 2 0 639098185 401973248 5962 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2517/statm: 98138 5962 400 364 0 34023 0
[pid=2519] ppid=2517 vsize=1018088 CPUtime=97.1 cores=0,2,4,6
/proc/2519/stat : 2519 (gnovelty+2) T 2517 2519 2519 0 -1 4202496 575 0 0 0 9710 0 0 0 25 0 1 0 639098185 1042522112 307 33554432000 134512640 135663704 4292149712 18446744073709551615 134525278 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/2519/statm: 254522 307 113 282 0 254238 0
[pid=2517/tid=2531] ppid=2456 vsize=392552 CPUtime=0 cores=0,2,4,6
/proc/2517/task/2531/stat : 2531 (python) S 2456 2456 1220 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 639117642 401973248 5962 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604453952734528 0 0 -1 6 0 0 0
[pid=2521] ppid=2456 vsize=402808 CPUtime=1.77 cores=0,2,4,6
/proc/2521/stat : 2521 (python) S 2456 2456 1220 0 -1 4202560 1174 0 0 0 91 86 0 0 18 0 2 0 639099795 412475392 5966 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/2521/statm: 100702 5966 400 364 0 36584 0
[pid=2523] ppid=2521 vsize=268420 CPUtime=489.43 cores=0,2,4,6
/proc/2523/stat : 2523 (hybridGM3) T 2521 2523 2523 0 -1 4202496 4945 0 0 0 48941 2 0 0 25 0 1 0 639099795 274862080 4671 33554432000 134512640 135204385 4291980160 18446744073709551615 134536977 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/2523/statm: 67105 4671 64 169 0 66935 0
[pid=2521/tid=2540] ppid=2456 vsize=402808 CPUtime=0 cores=0,2,4,6
/proc/2521/task/2540/stat : 2540 (python) S 2456 2456 1220 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 639121477 412475392 5966 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604450940387712 0 0 -1 0 0 0 0
[pid=2532] ppid=2456 vsize=413200 CPUtime=1.35 cores=0,2,4,6
/proc/2532/stat : 2532 (python) S 2456 2456 1220 0 -1 4202560 1182 0 0 0 71 64 0 0 18 0 2 0 639117643 423116800 5976 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/2532/statm: 103300 5976 400 364 0 39179 0
[pid=2534] ppid=2532 vsize=268420 CPUtime=331.99 cores=0,2,4,6
/proc/2534/stat : 2534 (hybridGM3) R 2532 2534 2534 0 -1 4202496 4939 0 0 0 33199 0 0 0 25 0 1 0 639117643 274862080 4671 33554432000 134512640 135204385 4292825072 18446744073709551615 134536952 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2534/statm: 67105 4671 64 169 0 66935 0
[pid=2532/tid=2570] ppid=2456 vsize=413200 CPUtime=0 cores=0,2,4,6
/proc/2532/task/2570/stat : 2570 (python) S 2456 2456 1220 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 639148138 423116800 5976 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604453903593600 0 0 -1 6 0 0 0
[pid=2537] ppid=2456 vsize=423456 CPUtime=1.34 cores=0,2,4,6
/proc/2537/stat : 2537 (python) S 2456 2456 1220 0 -1 4202560 1241 0 0 0 72 62 0 0 15 0 2 0 639120362 433618944 5987 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2537/statm: 105864 5987 407 364 0 41740 0
[pid=2539] ppid=2537 vsize=63560 CPUtime=392.27 cores=0,2,4,6
/proc/2539/stat : 2539 (march_hi) R 2537 2539 2539 0 -1 4202496 13561 0 0 0 39224 3 0 0 25 0 1 0 639120368 65085440 12003 33554432000 4194304 4955518 140734678262512 18446744073709551615 4237810 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/2539/statm: 15890 12003 106 186 0 15703 0
[pid=2537/tid=2580] ppid=2456 vsize=423456 CPUtime=0 cores=0,2,4,6
/proc/2537/task/2580/stat : 2580 (python) S 2456 2456 1220 0 -1 4202560 45 0 0 0 0 0 0 0 16 0 2 0 639166047 433618944 5987 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604453903595392 0 0 -1 2 0 0 0
[pid=2547] ppid=2456 vsize=434040 CPUtime=0.39 cores=0,2,4,6
/proc/2547/stat : 2547 (python) S 2456 2456 1220 0 -1 4202560 1197 0 0 0 20 19 0 0 19 0 2 0 639136600 444456960 6071 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273443275 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/2547/statm: 108510 6071 407 364 0 44383 0
[pid=2549] ppid=2547 vsize=247348 CPUtime=97.39 cores=0,2,4,6
/proc/2549/stat : 2549 (FH) T 2547 2549 2549 0 -1 4202496 720 0 0 0 9739 0 0 0 25 0 1 0 639136601 253284352 444 33554432000 134512640 134936988 4291612512 18446744073709551615 134553311 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/2549/statm: 61837 444 41 104 0 61341 0
[pid=2547/tid=2572] ppid=2456 vsize=434040 CPUtime=0 cores=0,2,4,6
/proc/2547/task/2572/stat : 2572 (python) S 2456 2456 1220 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 639150838 444456960 6071 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 514 18446604452784521408 0 0 -1 2 0 0 0
[pid=2574] ppid=2456 vsize=444296 CPUtime=0.3 cores=0,2,4,6
/proc/2574/stat : 2574 (python) S 2456 2456 1220 0 -1 4202560 1033 0 0 0 16 14 0 0 18 0 1 0 639155674 454959104 6049 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 1417279284296253699 0 0 17 6 0 0 0
/proc/2574/statm: 111074 6049 378 364 0 46944 0
[pid=2576] ppid=2574 vsize=1018088 CPUtime=88.32 cores=0,2,4,6
/proc/2576/stat : 2576 (gnovelty+2) R 2574 2576 2576 0 -1 4202496 583 0 0 0 8832 0 0 0 25 0 1 0 639155674 1042522112 307 33554432000 134512640 135663704 4287391744 18446744073709551615 134524897 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/2576/statm: 254522 307 113 282 0 254238 0
[pid=2581] ppid=2456 vsize=454552 CPUtime=0 cores=0,2,4,6
/proc/2581/stat : 2581 (python) S 2456 2456 1220 0 -1 4202560 851 0 0 0 0 0 0 0 19 0 1 0 639166583 465461248 6030 33554432000 4194304 5683644 140734273052512 18446744073709551615 254261636678 0 0 16781312 514 9509746991429582847 0 0 17 6 0 0 0
/proc/2581/statm: 113638 6030 349 364 0 49505 0
[pid=2583] ppid=2581 vsize=268420 CPUtime=0.94 cores=0,2,4,6
/proc/2583/stat : 2583 (hybridGM3) R 2581 2583 2583 0 -1 4202496 4949 0 0 0 94 0 0 0 23 0 1 0 639166584 274862080 4672 33554432000 134512640 135204385 4294067424 18446744073709551615 134537462 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/2583/statm: 67105 4672 64 169 0 66935 0
[pid=2456/tid=2463] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2463/stat : 2463 (python) S 2454 2456 1220 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 19 0 639066590 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2466] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2466/stat : 2466 (python) S 2454 2456 1220 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 639066591 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2469] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2469/stat : 2469 (python) S 2454 2456 1220 0 -1 4202560 39 932 0 0 0 0 0 0 18 0 19 0 639066592 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2471] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2471/stat : 2471 (python) S 2454 2456 1220 0 -1 4202560 21 932 0 0 0 0 0 0 18 0 19 0 639066593 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2479] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2479/stat : 2479 (python) S 2454 2456 1220 0 -1 4202560 33 932 0 0 0 0 0 0 18 0 19 0 639076521 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2483] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2483/stat : 2483 (python) S 2454 2456 1220 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 19 0 639077026 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2488] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2488/stat : 2488 (python) S 2454 2456 1220 0 -1 4202560 29 932 0 0 0 0 0 0 20 0 19 0 639081433 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2493] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2493/stat : 2493 (python) S 2454 2456 1220 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 19 0 639085749 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2498] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2498/stat : 2498 (python) S 2454 2456 1220 0 -1 4202560 22 932 0 0 0 0 0 0 20 0 19 0 639087150 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2502] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2502/stat : 2502 (python) S 2454 2456 1220 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 19 0 639089364 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2508] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2508/stat : 2508 (python) S 2454 2456 1220 0 -1 4202560 29 932 0 0 0 0 0 0 18 0 19 0 639095986 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2518] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2518/stat : 2518 (python) S 2454 2456 1220 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 19 0 639098185 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2522] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2522/stat : 2522 (python) S 2454 2456 1220 0 -1 4202560 35 932 0 0 0 0 0 0 18 0 19 0 639099795 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2533] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2533/stat : 2533 (python) S 2454 2456 1220 0 -1 4202560 35 932 0 0 0 0 0 0 18 0 19 0 639117643 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2538] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2538/stat : 2538 (python) S 2454 2456 1220 0 -1 4202560 30 932 0 0 0 0 0 0 18 0 19 0 639120362 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2548] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2548/stat : 2548 (python) S 2454 2456 1220 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 639136601 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2575] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2575/stat : 2575 (python) S 2454 2456 1220 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 639155674 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
[pid=2456/tid=2582] ppid=2454 vsize=464796 CPUtime=0 cores=0,2,4,6
/proc/2456/task/2582/stat : 2582 (python) S 2454 2456 1220 0 -1 4202560 26 932 0 0 0 0 0 0 20 0 19 0 639166583 475951104 7597 33554432000 4194304 5683644 140734273052512 18446744073709551615 254273440177 0 0 16781312 2 18446604453905600640 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3091.48
Current children cumulated vsize (KiB) 16136480

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2456 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=19844
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=134979
# CPU time returned by wait4() is 1.15482
# while last known CPU time is 3091.48
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1002.13
CPU time (s): 3091.48
CPU user time (s): 3085.91
CPU system time (s): 5.57
CPU usage (%): 308.491
Max. virtual memory (cumulated for all children) (KiB): 16136480

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.01984
system time used= 0.134979
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 27275
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2956
involuntary context switches= 78

runsolver used 7.13891 second user time and 14.0939 second system time

The end

Launcher Data 

Begin job on node140 at 2011-05-15 15:35:11
IDJOB=3378623
IDBENCH=83032
IDSOLVER=1859
FILE ID=node140/3378623-1305466511
RUNJOBID= node140-1305463229-1238
PBS_JOBID= 13324987
Free space on /tmp= 73436 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k7-r89-v90-c8010-S741717046-057.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378623-1305466511/watcher-3378623-1305466511 -o /tmp/evaluation-result-3378623-1305466511/solver-3378623-1305466511 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378623-1305466511.cnf -seed 1381436808 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1f00de1fd8adbfbf4cc573ef44af531f
RANDOM SEED=1381436808

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      22551524 kB
Buffers:        322084 kB
Cached:        4426564 kB
SwapCached:       8088 kB
Active:        6889920 kB
Inactive:      3175668 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      22551524 kB
SwapTotal:    67111528 kB
SwapFree:     67057896 kB
Dirty:           32312 kB
Writeback:           0 kB
AnonPages:     5314020 kB
Mapped:          35352 kB
Slab:           229020 kB
PageTables:      39516 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 87644776 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73436 MiB
End job on node140 at 2011-05-15 15:51:56