Trace number 3317113

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-random? (TO) 4800.1 1212.12

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v40000-c168000-S471691933-060-UNKNOWN.cnf
MD5SUM91da78a39fd68b6c63602d278464956f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables40000
Number of clauses168000
Sum of the clauses size504000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3168000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.36/0.66	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.66	c INFO: solving HOME/instance-3317113-1304267344.cnf
0.36/0.66	c INFO: computing task features
0.94/1.25	c INFO: computed variable-clause graph statistics
0.94/1.27	c INFO: computed clause balance statistics
3.93/4.24	c INFO: computed variable balance statistics
6.34/6.64	c INFO: computed small-clause counts
6.34/6.65	c INFO: computed Horn-clause counts
8.33/8.67	c INFO: computed variable graph statistics
8.33/8.67	c INFO: collected features for HOME/instance-3317113-1304267344.cnf in 7.99 s
8.33/8.70	c INFO: running hybridGM3@0 for 97 with 4791 remaining (b = 0.35)
8.44/8.73	c INFO: running hybridGM3@0 for 97 with 4791 remaining (b = 0.21)
8.44/8.76	c INFO: running hybridGM3@0 for 97 with 4791 remaining (b = 0.14)
8.44/8.79	c INFO: running hybridGM3@0 for 97 with 4791 remaining (b = 0.10)
409.66/109.24	c INFO: running hybridGM3@0 for 97 with 4691 remaining (b = 0.08)
409.66/109.29	c INFO: running hybridGM3@0 for 97 with 4691 remaining (b = 0.08)
410.06/109.32	c INFO: running hybridGM3@0 for 97 with 4691 remaining (b = 0.08)
410.06/109.35	c INFO: running hybridGM3@100 for 97 with 4691 remaining (b = 0.08)
789.45/204.38	c INFO: running hybridGM3@99 for 97 with 4596 remaining (b = 0.08)
789.84/204.43	c INFO: running hybridGM3@100 for 97 with 4596 remaining (b = 0.08)
789.84/204.45	c INFO: running TNM@0 for 97 with 4596 remaining (b = 0.10)
789.84/204.49	c INFO: running hybridGM3@100 for 97 with 4596 remaining (b = 0.07)
1190.01/304.63	c INFO: running hybridGM3@0 for 194 with 4496 remaining (b = 0.13)
1192.70/305.54	c INFO: running hybridGM3@0 for 194 with 4495 remaining (b = 0.10)
1192.70/305.58	c INFO: running hybridGM3@0 for 194 with 4495 remaining (b = 0.07)
1192.93/305.65	c INFO: running TNM@0 for 97 with 4495 remaining (b = 0.03)
1589.07/404.84	c INFO: running hybridGM3@0 for 194 with 4395 remaining (b = 0.05)
1965.93/499.43	c INFO: running hybridGM3@101 for 97 with 4301 remaining (b = 0.02)
1969.08/500.29	c INFO: running hybridGM3@100 for 97 with 4300 remaining (b = 0.02)
1997.34/507.36	c INFO: running hybridGM3@100 for 97 with 4293 remaining (b = 0.02)
2364.73/599.60	c INFO: running hybridGM3@295 for 97 with 4201 remaining (b = 0.02)
2368.31/600.45	c INFO: running hybridGM3@300 for 97 with 4200 remaining (b = 0.02)
2376.66/602.51	c INFO: running TNM@0 for 97 with 4198 remaining (b = 0.02)
2385.03/604.68	c INFO: running TNM@0 for 97 with 4196 remaining (b = 0.02)
2747.51/695.62	c INFO: running hybridGM3@0 for 194 with 4105 remaining (b = 0.04)
2763.81/699.78	c INFO: running hybridGM3@393 for 97 with 4100 remaining (b = 0.01)
2775.76/702.70	c INFO: running hybridGM3@395 for 97 with 4097 remaining (b = 0.01)
2780.14/703.87	c INFO: running TNM@0 for 97 with 4096 remaining (b = 0.01)
3166.84/800.90	c INFO: running TNM@0 for 97 with 3999 remaining (b = 0.01)
3166.84/800.99	c INFO: running TNM@0 for 97 with 3999 remaining (b = 0.01)
3171.20/802.09	c INFO: running TNM@197 for 1843 with 3998 remaining (b = 0.21)
3550.27/897.48	c INFO: running hybridGM3@98 for 97 with 3903 remaining (b = 0.02)
3564.97/901.12	c INFO: running hybridGM3@0 for 194 with 3899 remaining (b = 0.03)
3565.31/901.21	c INFO: running hybridGM3@0 for 194 with 3899 remaining (b = 0.02)
3920.54/990.68	c INFO: running hybridGM3@0 for 194 with 3810 remaining (b = 0.02)
4339.23/1096.01	c INFO: running hybridGM3@93 for 97 with 3704 remaining (b = 0.01)
4370.62/1103.98	c INFO: running hybridGM3@100 for 97 with 3696 remaining (b = 0.01)
4718.63/1191.58	c INFO: running adaptg2wsat2009++@0 for 97 with 3609 remaining (b = 0.01)
4737.28/1196.23	c INFO: running hybridGM3@292 for 97 with 3604 remaining (b = 0.01)
4769.07/1204.21	c INFO: running hybridGM3@297 for 97 with 3596 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3317113-1304267344/watcher-3317113-1304267344 -o /tmp/evaluation-result-3317113-1304267344/solver-3317113-1304267344 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317113-1304267344.cnf -seed 1317240943 -budget 4800 -cores 4 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.27 1.15 0.39 6/189 15319
/proc/meminfo: memFree=30327712/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=15184 CPUtime=0 cores=1,3,5,7
/proc/15319/stat : 15319 (python) S 15317 15319 15176 0 -1 4202496 733 0 0 0 0 0 0 0 17 0 1 0 538331095 15548416 622 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500689920 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/15319/statm: 3796 622 322 364 0 283 0

[startup+0.0290151 s]
/proc/loadavg: 4.27 1.15 0.39 6/189 15319
/proc/meminfo: memFree=30327712/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=16400 CPUtime=0.01 cores=1,3,5,7
/proc/15319/stat : 15319 (python) S 15317 15319 15176 0 -1 4202496 1165 0 0 0 1 0 0 0 17 0 1 0 538331095 16793600 1030 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500689920 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/15319/statm: 4100 1030 414 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+0.101002 s]
/proc/loadavg: 4.27 1.15 0.39 6/189 15319
/proc/meminfo: memFree=30327712/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=93964 CPUtime=0.06 cores=1,3,5,7
/proc/15319/stat : 15319 (python) R 15317 15319 15176 0 -1 4202496 2692 0 0 0 5 1 0 0 17 0 1 0 538331095 96219136 2407 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500689920 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/15319/statm: 23491 2407 764 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 93964

[startup+0.300965 s]
/proc/loadavg: 4.27 1.15 0.39 6/189 15319
/proc/meminfo: memFree=30327712/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=135752 CPUtime=0.18 cores=1,3,5,7
/proc/15319/stat : 15319 (python) S 15317 15319 15176 0 -1 4202496 5966 935 1 0 14 4 0 0 17 0 1 0 538331095 139010048 4151 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500687877 0 2147483391 16781312 2 18446744071699199166 0 0 17 3 0 0 0
/proc/15319/statm: 33938 4151 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.18
Current children cumulated vsize (KiB) 135752

[startup+0.700886 s]
/proc/loadavg: 4.27 1.15 0.39 6/189 15319
/proc/meminfo: memFree=30327712/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=283960 CPUtime=0.43 cores=1,3,5,7
/proc/15319/stat : 15319 (python) R 15317 15319 15176 0 -1 4202496 10646 935 1 0 33 10 0 0 15 0 1 0 538331095 290775040 8324 33554432000 4194304 5683644 140734886922048 18446744073709551615 47361272541376 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/15319/statm: 70990 8324 1761 364 0 6914 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 283960

[startup+1.50072 s]
/proc/loadavg: 4.27 1.15 0.39 6/190 15324
/proc/meminfo: memFree=30281328/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=317988 CPUtime=1.23 cores=1,3,5,7
/proc/15319/stat : 15319 (python) R 15317 15319 15176 0 -1 4202496 23802 935 1 0 111 12 0 0 16 0 1 0 538331095 325619712 16881 33554432000 4194304 5683644 140734886922048 18446744073709551615 47361101827785 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/15319/statm: 79497 16881 1826 364 0 15421 0
Current children cumulated CPU time (s) 1.23
Current children cumulated vsize (KiB) 317988

[startup+3.1004 s]
/proc/loadavg: 4.27 1.15 0.39 6/190 15324
/proc/meminfo: memFree=30265828/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=317988 CPUtime=2.83 cores=1,3,5,7
/proc/15319/stat : 15319 (python) R 15317 15319 15176 0 -1 4202496 23802 935 1 0 271 12 0 0 18 0 1 0 538331095 325619712 16881 33554432000 4194304 5683644 140734886922048 18446744073709551615 4457138 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/15319/statm: 79497 16881 1826 364 0 15421 0
Current children cumulated CPU time (s) 2.83
Current children cumulated vsize (KiB) 317988

[startup+6.30076 s]
/proc/loadavg: 4.33 1.22 0.41 7/190 15325
/proc/meminfo: memFree=30265588/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=317988 CPUtime=6.03 cores=1,3,5,7
/proc/15319/stat : 15319 (python) R 15317 15319 15176 0 -1 4202496 23803 935 1 0 591 12 0 0 25 0 1 0 538331095 325619712 16882 33554432000 4194304 5683644 140734886922048 18446744073709551615 4862088 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/15319/statm: 79497 16882 1827 364 0 15421 0
Current children cumulated CPU time (s) 6.03
Current children cumulated vsize (KiB) 317988

[startup+12.7014 s]
/proc/loadavg: 4.71 1.35 0.46 9/202 15337
/proc/meminfo: memFree=30120872/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=355896 CPUtime=8.53 cores=1,3,5,7
/proc/15319/stat : 15319 (python) S 15317 15319 15176 0 -1 4202496 26754 935 1 0 839 14 0 0 22 0 5 0 538331095 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/15319/statm: 88974 16170 1881 364 0 24883 0
[pid=15326] ppid=15319 vsize=314884 CPUtime=0 cores=1,3,5,7
/proc/15326/stat : 15326 (python) S 15319 15319 15176 0 -1 4202560 735 0 0 0 0 0 0 0 15 0 1 0 538331965 322441216 14626 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500711951 0 0 16781312 514 4582883216121102600 0 0 17 3 0 0 0
/proc/15326/statm: 78721 14626 353 364 0 14639 0
[pid=15328] ppid=15326 vsize=276244 CPUtime=3.99 cores=1,3,5,7
/proc/15328/stat : 15328 (hybridGM3) R 15326 15328 15328 0 -1 4202496 8563 0 0 0 397 2 0 0 25 0 1 0 538331966 282873856 8299 33554432000 134512640 135204385 4287011440 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/15328/statm: 69061 8299 64 169 0 68891 0
[pid=15329] ppid=15319 vsize=325140 CPUtime=0 cores=1,3,5,7
/proc/15329/stat : 15329 (python) S 15319 15319 15176 0 -1 4202560 744 0 0 0 0 0 0 0 15 0 1 0 538331968 332943360 14624 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 10613801251495706888 0 0 17 5 0 0 0
/proc/15329/statm: 81285 14624 347 364 0 17200 0
[pid=15331] ppid=15329 vsize=276236 CPUtime=3.93 cores=1,3,5,7
/proc/15331/stat : 15331 (hybridGM3) R 15329 15331 15331 0 -1 4202496 8562 0 0 0 393 0 0 0 25 0 1 0 538331969 282865664 8298 33554432000 134512640 135204385 4292198144 18446744073709551615 134536605 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/15331/statm: 69059 8298 64 169 0 68889 0
[pid=15332] ppid=15319 vsize=335396 CPUtime=0 cores=1,3,5,7
/proc/15332/stat : 15332 (python) S 15319 15319 15176 0 -1 4202560 745 0 0 0 0 0 0 0 15 0 1 0 538331971 343445504 14628 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 5631368708275273992 0 0 17 3 0 0 0
/proc/15332/statm: 83849 14628 347 364 0 19761 0
[pid=15334] ppid=15332 vsize=276244 CPUtime=3.93 cores=1,3,5,7
/proc/15334/stat : 15334 (hybridGM3) R 15332 15334 15334 0 -1 4202496 8562 0 0 0 392 1 0 0 25 0 1 0 538331972 282873856 8299 33554432000 134512640 135204385 4292171616 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/15334/statm: 69061 8299 64 169 0 68891 0
[pid=15335] ppid=15319 vsize=345652 CPUtime=0 cores=1,3,5,7
/proc/15335/stat : 15335 (python) S 15319 15319 15176 0 -1 4202560 758 0 0 0 0 0 0 0 15 0 1 0 538331974 353947648 14632 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 5631465465298518280 0 0 17 3 0 0 0
/proc/15335/statm: 86413 14632 347 364 0 22322 0
[pid=15337] ppid=15335 vsize=276236 CPUtime=3.9 cores=1,3,5,7
/proc/15337/stat : 15337 (hybridGM3) R 15335 15337 15337 0 -1 4202496 8564 0 0 0 389 1 0 0 25 0 1 0 538331975 282865664 8301 33554432000 134512640 135204385 4290372896 18446744073709551615 134537435 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/15337/statm: 69059 8301 64 169 0 68889 0
[pid=15319/tid=15327] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15327/stat : 15327 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 25 0 5 0 538331965 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 5 0 0 0
[pid=15319/tid=15330] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15330/stat : 15330 (python) S 15317 15319 15176 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538331969 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15333] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15333/stat : 15333 (python) S 15317 15319 15176 0 -1 4202560 21 935 0 0 0 0 0 0 20 0 5 0 538331972 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 3 0 0 0
[pid=15319/tid=15336] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15336/stat : 15336 (python) S 15317 15319 15176 0 -1 4202560 30 935 0 0 0 0 0 0 22 0 5 0 538331975 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.28
Current children cumulated vsize (KiB) 2781928

[startup+25.5008 s]
/proc/loadavg: 5.51 1.69 0.59 9/202 15337
/proc/meminfo: memFree=30119852/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=355896 CPUtime=8.53 cores=1,3,5,7
/proc/15319/stat : 15319 (python) S 15317 15319 15176 0 -1 4202496 26754 935 1 0 839 14 0 0 22 0 5 0 538331095 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/15319/statm: 88974 16170 1881 364 0 24883 0
[pid=15326] ppid=15319 vsize=314884 CPUtime=0.02 cores=1,3,5,7
/proc/15326/stat : 15326 (python) S 15319 15319 15176 0 -1 4202560 900 0 0 0 1 1 0 0 15 0 1 0 538331965 322441216 14649 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500711951 0 0 16781312 514 1917073289818112264 0 0 17 3 0 0 0
/proc/15326/statm: 78721 14649 376 364 0 14639 0
[pid=15328] ppid=15326 vsize=276244 CPUtime=16.79 cores=1,3,5,7
/proc/15328/stat : 15328 (hybridGM3) R 15326 15328 15328 0 -1 4202496 8563 0 0 0 1677 2 0 0 25 0 1 0 538331966 282873856 8299 33554432000 134512640 135204385 4287011440 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/15328/statm: 69061 8299 64 169 0 68891 0
[pid=15329] ppid=15319 vsize=325140 CPUtime=0.02 cores=1,3,5,7
/proc/15329/stat : 15329 (python) S 15319 15319 15176 0 -1 4202560 916 0 0 0 1 1 0 0 15 0 1 0 538331968 332943360 14647 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 1917073289818112264 0 0 17 5 0 0 0
/proc/15329/statm: 81285 14647 370 364 0 17200 0
[pid=15331] ppid=15329 vsize=276236 CPUtime=16.66 cores=1,3,5,7
/proc/15331/stat : 15331 (hybridGM3) R 15329 15331 15331 0 -1 4202496 8562 0 0 0 1666 0 0 0 25 0 1 0 538331969 282865664 8298 33554432000 134512640 135204385 4292198144 18446744073709551615 134537412 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/15331/statm: 69059 8298 64 169 0 68889 0
[pid=15332] ppid=15319 vsize=335396 CPUtime=0.02 cores=1,3,5,7
/proc/15332/stat : 15332 (python) S 15319 15319 15176 0 -1 4202560 918 0 0 0 1 1 0 0 15 0 1 0 538331971 343445504 14651 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 1917073289818112264 0 0 17 3 0 0 0
/proc/15332/statm: 83849 14651 370 364 0 19761 0
[pid=15334] ppid=15332 vsize=276244 CPUtime=16.7 cores=1,3,5,7
/proc/15334/stat : 15334 (hybridGM3) R 15332 15334 15334 0 -1 4202496 8563 0 0 0 1669 1 0 0 25 0 1 0 538331972 282873856 8300 33554432000 134512640 135204385 4292171616 18446744073709551615 134545975 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/15334/statm: 69061 8300 64 169 0 68891 0
[pid=15335] ppid=15319 vsize=345652 CPUtime=0.02 cores=1,3,5,7
/proc/15335/stat : 15335 (python) S 15319 15319 15176 0 -1 4202560 934 0 0 0 1 1 0 0 15 0 1 0 538331974 353947648 14655 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 1917073289818112264 0 0 17 3 0 0 0
/proc/15335/statm: 86413 14655 370 364 0 22322 0
[pid=15337] ppid=15335 vsize=276236 CPUtime=16.63 cores=1,3,5,7
/proc/15337/stat : 15337 (hybridGM3) R 15335 15337 15337 0 -1 4202496 8564 0 0 0 1662 1 0 0 25 0 1 0 538331975 282865664 8301 33554432000 134512640 135204385 4290372896 18446744073709551615 134537565 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/15337/statm: 69059 8301 64 169 0 68889 0
[pid=15319/tid=15327] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15327/stat : 15327 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 25 0 5 0 538331965 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 5 0 0 0
[pid=15319/tid=15330] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15330/stat : 15330 (python) S 15317 15319 15176 0 -1 4202560 25 935 0 0 0 0 0 0 25 0 5 0 538331969 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15333] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15333/stat : 15333 (python) S 15317 15319 15176 0 -1 4202560 21 935 0 0 0 0 0 0 20 0 5 0 538331972 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 3 0 0 0
[pid=15319/tid=15336] ppid=15317 vsize=355896 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15336/stat : 15336 (python) S 15317 15319 15176 0 -1 4202560 30 935 0 0 0 0 0 0 22 0 5 0 538331975 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 75.39
Current children cumulated vsize (KiB) 2781928

[startup+51.1007 s]
/proc/loadavg: 6.36 2.19 0.78 9/202 15338
/proc/meminfo: memFree=30119344/32950928 swapFree=67111464/67111528
[pid=15319] ppid=15317 vsize=355896 CPUtime=8.53 cores=1,3,5,7
/proc/15319/stat : 15319 (python) S 15317 15319 15176 0 -1 4202496 26754 935 1 0 839 14 0 0 22 0 5 0 538331095 364437504 16170 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/15319/statm: 88974 16170 1881 364 0 24883 0
[pid=15326] ppid=15319 vsize=314884 CPUtime=0.05 cores=1,3,5,7
/proc/15326/stat : 15326 (python) S 15319 15319 15176 0 -1 4202560 908 0 0 0 3 2 0 0 15 0 1 0 538331965 322441216 14649 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500711951 0 0 16781312 514 1917073289818112264 0 0 17 3 0 0 0
/proc/15326/statm: 78721 14649 376 364 0 14639 0
[pid=15328] ppid=15326 vsize=276244 CPUtime=42.39 cores=1,3,5,7
/proc/15328/stat : 15328 (hybridGM3) R 15326 15328 15328 0 -1 4202496 8563 0 0 0 4237 2 0 0 25 0 1 0 538331966 282873856 8299 33554432000 134512640 135204385 4287011440 18446744073709551615 134536895 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/15328/statm: 69061 8299 64 169 0 68891 0
[pid=15329] ppid=15319 vsize=325140 CPUtime=0.04 cores=1,3,5,7
/proc/15329/stat : 15329 (python) S 15319 15319 15176 0 -1 4202560 925 0 0 0 2 2 0 0 15 0 1 0 538331968 332943360 14647 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 1917073289818112264 0 0 17 5 0 0 0
/proc/15329/statm: 81285 14647 370 364 0 17200 0
[pid=15331] ppid=15329 vsize=276236 CPUtime=42.12 cores=1,3,5,7
/proc/15331/stat : 15331 (hybridGM3) R 15329 15331 15331 0 -1 4202496 8564 0 0 0 4212 0 0 0 25 0 1 0 538331969 282865664 8300 33554432000 134512640 135204385 4292198144 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/15331/statm: 69059 8300 64 169 0 68889 0
[pid=15332] ppid=15319 vsize=335396 CPUtime=0.05 cores=1,3,5,7
/proc/15332/stat : 15332 (python) S 15319 15319 15176 0 -1 4202560 926 0 0 0 2 3 0 0 15 0 1 0 538331971 343445504 14651 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 1917073289818112264 0 0 17 3 0 0 0
/proc/15332/statm: 83849 14651 370 364 0 19761 0
[pid=15334] ppid=15332 vsize=276244 CPUtime=42.27 cores=1,3,5,7
/proc/15334/stat : 15334 (hybridGM3) R 15332 15334 15334 0 -1 4202496 8563 0 0 0 4226 1 0 0 25 0 1 0 538331972 282873856 8300 33554432000 134512640 135204385 4292171616 18446744073709551615 134536993 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/15334/statm: 69061 8300 64 169 0 68891 0
[pid=15335] ppid=15319 vsize=345652 CPUtime=0.04 cores=1,3,5,7
/proc/15335/stat : 15335 (python) S 15319 15319 15176 0 -1 4202560 945 0 0 0 2 2 0 0 15 0 1 0 538331974 353947648 14655 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 1917073289818112264 0 0 17 3 0 0 0
/proc/15335/statm: 86413 14655 370 364 0 22322 0

################
# More data... #
################

/proc/15393/stat : 15393 (python) S 15319 15319 15176 0 -1 4202560 1235 0 0 0 42 35 0 0 15 0 2 0 538361649 385454080 14703 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15393/statm: 94105 14703 405 364 0 30005 0
[pid=15395] ppid=15393 vsize=276236 CPUtime=488.52 cores=1,3,5,7
/proc/15395/stat : 15395 (hybridGM3) T 15393 15395 15395 0 -1 4202496 8570 0 0 0 48851 1 0 0 25 0 1 0 538361650 282865664 8303 33554432000 134512640 135204385 4288064896 18446744073709551615 134536676 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/15395/statm: 69059 8303 64 169 0 68889 0
[pid=15393/tid=15429] ppid=15319 vsize=376420 CPUtime=0 cores=1,3,5,7
/proc/15393/task/15429/stat : 15429 (python) S 15319 15319 15176 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538381121 385454080 14703 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604471137291136 0 0 -1 7 0 0 0
[pid=15396] ppid=15319 vsize=386676 CPUtime=0.45 cores=1,3,5,7
/proc/15396/stat : 15396 (python) S 15319 15319 15176 0 -1 4202560 1228 0 0 0 23 22 0 0 15 0 2 0 538361654 395956224 14707 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15396/statm: 96669 14707 405 364 0 32566 0
[pid=15398] ppid=15396 vsize=276240 CPUtime=292.68 cores=1,3,5,7
/proc/15398/stat : 15398 (hybridGM3) T 15396 15398 15398 0 -1 4202496 8581 0 0 0 29267 1 0 0 25 0 1 0 538361654 282869760 8308 33554432000 134512640 135204385 4288209696 18446744073709551615 134537483 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/15398/statm: 69060 8308 64 169 0 68890 0
[pid=15396/tid=15430] ppid=15319 vsize=386676 CPUtime=0 cores=1,3,5,7
/proc/15396/task/15430/stat : 15430 (python) S 15319 15319 15176 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538381828 395956224 14707 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604471143856000 0 0 -1 7 0 0 0
[pid=15399] ppid=15319 vsize=396932 CPUtime=0.15 cores=1,3,5,7
/proc/15399/stat : 15399 (python) S 15319 15319 15176 0 -1 4202560 1157 0 0 0 7 8 0 0 19 0 2 0 538361661 406458368 14705 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15399/statm: 99233 14705 399 364 0 35127 0
[pid=15401] ppid=15399 vsize=467024 CPUtime=99.14 cores=1,3,5,7
/proc/15401/stat : 15401 (TNM) T 15399 15401 15401 0 -1 4202496 6826 0 0 0 9914 0 0 0 25 0 1 0 538361661 478232576 6558 33554432000 134512640 135008748 4292928720 18446744073709551615 134528951 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15401/statm: 116756 6558 46 122 0 115656 0
[pid=15399/tid=15416] ppid=15319 vsize=396932 CPUtime=0 cores=1,3,5,7
/proc/15399/task/15416/stat : 15416 (python) S 15319 15319 15176 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538371576 406458368 14705 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604456459772096 0 0 -1 1 0 0 0
[pid=15417] ppid=15319 vsize=407188 CPUtime=0.31 cores=1,3,5,7
/proc/15417/stat : 15417 (python) S 15319 15319 15176 0 -1 4202560 1201 0 0 0 16 15 0 0 16 0 2 0 538371580 416960512 14709 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15417/statm: 101797 14709 399 364 0 37688 0
[pid=15419] ppid=15417 vsize=276240 CPUtime=197.57 cores=1,3,5,7
/proc/15419/stat : 15419 (hybridGM3) T 15417 15419 15419 0 -1 4202496 8580 0 0 0 19755 2 0 0 25 0 1 0 538371580 282869760 8309 33554432000 134512640 135204385 4291933536 18446744073709551615 134536744 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/15419/statm: 69060 8309 64 169 0 68890 0
[pid=15417/tid=15481] ppid=15319 vsize=407188 CPUtime=0 cores=1,3,5,7
/proc/15417/task/15481/stat : 15481 (python) S 15319 15319 15176 0 -1 4202560 50 0 0 0 0 0 0 0 16 0 2 0 538391560 416960512 14709 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604471111403456 0 0 -1 7 0 0 0
[pid=15477] ppid=15319 vsize=417444 CPUtime=0.87 cores=1,3,5,7
/proc/15477/stat : 15477 (python) S 15319 15319 15176 0 -1 4202560 1213 0 0 0 42 45 0 0 18 0 2 0 538391347 427462656 14713 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15477/statm: 104361 14713 399 364 0 40249 0
[pid=15479] ppid=15477 vsize=467024 CPUtime=509.95 cores=1,3,5,7
/proc/15479/stat : 15479 (TNM) R 15477 15479 15479 0 -1 4202496 6829 0 0 0 50994 1 0 0 25 0 1 0 538391347 478232576 6558 33554432000 134512640 135008748 4292393952 18446744073709551615 134528967 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/15479/statm: 116756 6558 46 122 0 115656 0
[pid=15477/tid=15506] ppid=15319 vsize=417444 CPUtime=0 cores=1,3,5,7
/proc/15477/task/15506/stat : 15506 (python) S 15319 15319 15176 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538401362 427462656 14713 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604471144305728 0 0 -1 7 0 0 0
[pid=15482] ppid=15319 vsize=427700 CPUtime=0.15 cores=1,3,5,7
/proc/15482/stat : 15482 (python) S 15319 15319 15176 0 -1 4202560 1176 0 0 0 9 6 0 0 19 0 2 0 538391564 437964800 14717 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15482/statm: 106925 14717 399 364 0 42810 0
[pid=15484] ppid=15482 vsize=467024 CPUtime=99.14 cores=1,3,5,7
/proc/15484/stat : 15484 (TNM) T 15482 15484 15484 0 -1 4202496 6830 0 0 0 9913 1 0 0 25 0 1 0 538391564 478232576 6559 33554432000 134512640 135008748 4292248272 18446744073709551615 134527477 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15484/statm: 116756 6559 46 122 0 115656 0
[pid=15482/tid=15507] ppid=15319 vsize=427700 CPUtime=0 cores=1,3,5,7
/proc/15482/task/15507/stat : 15507 (python) S 15319 15319 15176 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 538401479 437964800 14717 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604471144307520 0 0 -1 7 0 0 0
[pid=15503] ppid=15319 vsize=437956 CPUtime=0.5 cores=1,3,5,7
/proc/15503/stat : 15503 (python) S 15319 15319 15176 0 -1 4202560 1208 0 0 0 26 24 0 0 15 0 2 0 538400658 448466944 14721 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15503/statm: 109489 14721 399 364 0 45371 0
[pid=15505] ppid=15503 vsize=276240 CPUtime=291.5 cores=1,3,5,7
/proc/15505/stat : 15505 (hybridGM3) T 15503 15505 15505 0 -1 4202496 8578 0 0 0 29149 1 0 0 25 0 1 0 538400658 282869760 8307 33554432000 134512640 135204385 4288140528 18446744073709551615 134537403 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/15505/statm: 69060 8307 64 169 0 68890 0
[pid=15503/tid=15551] ppid=15319 vsize=437956 CPUtime=0 cores=1,3,5,7
/proc/15503/task/15551/stat : 15551 (python) S 15319 15319 15176 0 -1 4202560 50 0 0 0 0 0 0 0 15 0 2 0 538420839 448466944 14721 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604466689427328 0 0 -1 7 0 0 0
[pid=15508] ppid=15319 vsize=448212 CPUtime=0.16 cores=1,3,5,7
/proc/15508/stat : 15508 (python) S 15319 15319 15176 0 -1 4202560 1191 0 0 0 9 7 0 0 19 0 2 0 538401483 458969088 14725 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15508/statm: 112053 14725 399 364 0 47932 0
[pid=15510] ppid=15508 vsize=467024 CPUtime=98.13 cores=1,3,5,7
/proc/15510/stat : 15510 (TNM) T 15508 15510 15510 0 -1 4202496 6833 0 0 0 9811 2 0 0 25 0 1 0 538401483 478232576 6558 33554432000 134512640 135008748 4288094096 18446744073709551615 134527320 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/15510/statm: 116756 6558 46 122 0 115656 0
[pid=15508/tid=15536] ppid=15319 vsize=448212 CPUtime=0 cores=1,3,5,7
/proc/15508/task/15536/stat : 15536 (python) S 15319 15319 15176 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 538411300 458969088 14725 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604471149941760 0 0 -1 3 0 0 0
[pid=15530] ppid=15319 vsize=458468 CPUtime=0.16 cores=1,3,5,7
/proc/15530/stat : 15530 (python) S 15319 15319 15176 0 -1 4202560 1240 0 0 0 7 9 0 0 19 0 2 0 538411186 469471232 14737 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15530/statm: 114617 14737 407 364 0 50493 0
[pid=15532] ppid=15530 vsize=467024 CPUtime=100.15 cores=1,3,5,7
/proc/15532/stat : 15532 (TNM) T 15530 15532 15532 0 -1 4202496 6832 0 0 0 10014 1 0 0 25 0 1 0 538411187 478232576 6558 33554432000 134512640 135008748 4290916832 18446744073709551615 134529485 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/15532/statm: 116756 6558 46 122 0 115656 0
[pid=15530/tid=15552] ppid=15319 vsize=458468 CPUtime=0 cores=1,3,5,7
/proc/15530/task/15552/stat : 15552 (python) S 15319 15319 15176 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 538421203 469471232 14737 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604456459772992 0 0 -1 7 0 0 0
[pid=15533] ppid=15319 vsize=468724 CPUtime=0.17 cores=1,3,5,7
/proc/15533/stat : 15533 (python) S 15319 15319 15176 0 -1 4202560 1234 0 0 0 7 10 0 0 19 0 2 0 538411194 479973376 14741 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15533/statm: 117181 14741 407 364 0 53054 0
[pid=15535] ppid=15533 vsize=467024 CPUtime=100.16 cores=1,3,5,7
/proc/15535/stat : 15535 (TNM) T 15533 15535 15535 0 -1 4202496 6834 0 0 0 10015 1 0 0 25 0 1 0 538411195 478232576 6558 33554432000 134512640 135008748 4290331280 18446744073709551615 134527118 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15535/statm: 116756 6558 46 122 0 115656 0
[pid=15533/tid=15556] ppid=15319 vsize=468724 CPUtime=0 cores=1,3,5,7
/proc/15533/task/15556/stat : 15556 (python) S 15319 15319 15176 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 538421211 479973376 14741 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604471152842048 0 0 -1 7 0 0 0
[pid=15553] ppid=15319 vsize=478980 CPUtime=0.54 cores=1,3,5,7
/proc/15553/stat : 15553 (python) S 15319 15319 15176 0 -1 4202560 1247 0 0 0 23 31 0 0 15 0 2 0 538421208 490475520 14739 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15553/statm: 119745 14739 401 364 0 55615 0
[pid=15555] ppid=15553 vsize=276240 CPUtime=304.46 cores=1,3,5,7
/proc/15555/stat : 15555 (hybridGM3) R 15553 15555 15555 0 -1 4202496 8589 0 0 0 30445 1 0 0 25 0 1 0 538421208 282869760 8303 33554432000 134512640 135204385 4290879120 18446744073709551615 134537403 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/15555/statm: 69060 8303 64 169 0 68890 0
[pid=15553/tid=15573] ppid=15319 vsize=478980 CPUtime=0 cores=1,3,5,7
/proc/15553/task/15573/stat : 15573 (python) S 15319 15319 15176 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 538441489 490475520 14739 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604462103552256 0 0 -1 7 0 0 0
[pid=15557] ppid=15319 vsize=489236 CPUtime=0.54 cores=1,3,5,7
/proc/15557/stat : 15557 (python) S 15319 15319 15176 0 -1 4202560 1228 0 0 0 23 31 0 0 15 0 2 0 538421216 500977664 14743 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15557/statm: 122309 14743 401 364 0 58176 0
[pid=15559] ppid=15557 vsize=276236 CPUtime=310.14 cores=1,3,5,7
/proc/15559/stat : 15559 (hybridGM3) R 15557 15559 15559 0 -1 4202496 8576 0 0 0 31014 0 0 0 25 0 1 0 538421217 282865664 8303 33554432000 134512640 135204385 4289372224 18446744073709551615 134545864 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/15559/statm: 69059 8303 64 169 0 68889 0
[pid=15557/tid=15572] ppid=15319 vsize=489236 CPUtime=0 cores=1,3,5,7
/proc/15557/task/15572/stat : 15572 (python) S 15319 15319 15176 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 538440692 500977664 14743 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604461083283776 0 0 -1 7 0 0 0
[pid=15564] ppid=15319 vsize=499492 CPUtime=0.35 cores=1,3,5,7
/proc/15564/stat : 15564 (python) S 15319 15319 15176 0 -1 4202560 1229 0 0 0 17 18 0 0 16 0 2 0 538430164 511479808 14746 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512518603 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/15564/statm: 124873 14746 400 364 0 60737 0
[pid=15566] ppid=15564 vsize=276240 CPUtime=197.26 cores=1,3,5,7
/proc/15566/stat : 15566 (hybridGM3) T 15564 15566 15566 0 -1 4202496 8592 0 0 0 19724 2 0 0 25 0 1 0 538430164 282869760 8313 33554432000 134512640 135204385 4288736192 18446744073709551615 134545851 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/15566/statm: 69060 8313 64 169 0 68890 0
[pid=15564/tid=15582] ppid=15319 vsize=499492 CPUtime=0 cores=1,3,5,7
/proc/15564/task/15582/stat : 15582 (python) S 15319 15319 15176 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 538450249 511479808 14746 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 514 18446604461083285568 0 0 -1 7 0 0 0
[pid=15583] ppid=15319 vsize=509748 CPUtime=0.02 cores=1,3,5,7
/proc/15583/stat : 15583 (python) S 15319 15319 15176 0 -1 4202560 1031 0 0 0 1 1 0 0 18 0 1 0 538450254 521981952 14720 33554432000 4194304 5683644 140734886922048 18446744073709551615 254500712006 0 0 16781312 514 1917073289818112264 0 0 17 7 0 0 0
/proc/15583/statm: 127437 14720 371 364 0 63298 0
[pid=15585] ppid=15583 vsize=434080 CPUtime=20.25 cores=1,3,5,7
/proc/15585/stat : 15585 (adaptg2wsat2009) R 15583 15585 15585 0 -1 4202496 7096 0 0 0 2024 1 0 0 25 0 1 0 538450254 444497920 6819 33554432000 134512640 135107411 4288699328 18446744073709551615 134532074 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/15585/statm: 108520 6819 59 146 0 107396 0
[pid=15319/tid=15327] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15327/stat : 15327 (python) S 15317 15319 15176 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 21 0 538331965 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 3 0 0 0
[pid=15319/tid=15330] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15330/stat : 15330 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 18 0 21 0 538331969 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15333] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15333/stat : 15333 (python) S 15317 15319 15176 0 -1 4202560 28 935 0 0 0 0 0 0 15 0 21 0 538331972 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 3 0 0 0
[pid=15319/tid=15336] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15336/stat : 15336 (python) S 15317 15319 15176 0 -1 4202560 30 935 0 0 0 0 0 0 18 0 21 0 538331975 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 3 0 0 0
[pid=15319/tid=15371] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15371/stat : 15371 (python) S 15317 15319 15176 0 -1 4202560 28 935 0 0 0 0 0 0 20 0 21 0 538351541 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 5 0 0 0
[pid=15319/tid=15391] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15391/stat : 15391 (python) S 15317 15319 15176 0 -1 4202560 35 935 0 0 0 0 0 0 18 0 21 0 538361559 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15394] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15394/stat : 15394 (python) S 15317 15319 15176 0 -1 4202560 37 935 0 0 0 0 0 0 18 0 21 0 538361650 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15397] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15397/stat : 15397 (python) S 15317 15319 15176 0 -1 4202560 87 935 0 0 0 0 0 0 18 0 21 0 538361654 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15400] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15400/stat : 15400 (python) S 15317 15319 15176 0 -1 4202560 34 935 0 0 0 0 0 0 22 0 21 0 538361661 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15418] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15418/stat : 15418 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 21 0 538371580 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15478] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15478/stat : 15478 (python) S 15317 15319 15176 0 -1 4202560 32 935 0 0 0 0 0 0 18 0 21 0 538391347 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15483] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15483/stat : 15483 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 21 0 538391564 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15504] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15504/stat : 15504 (python) S 15317 15319 15176 0 -1 4202560 33 935 0 0 0 0 0 0 18 0 21 0 538400658 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15509] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15509/stat : 15509 (python) S 15317 15319 15176 0 -1 4202560 31 935 0 0 0 0 0 0 20 0 21 0 538401483 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15531] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15531/stat : 15531 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 21 0 538411186 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15534] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15534/stat : 15534 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 21 0 21 0 538411195 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15554] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15554/stat : 15554 (python) S 15317 15319 15176 0 -1 4202560 38 935 0 0 0 0 0 0 18 0 21 0 538421208 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15558] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15558/stat : 15558 (python) S 15317 15319 15176 0 -1 4202560 35 935 0 0 0 0 0 0 18 0 21 0 538421217 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15565] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15565/stat : 15565 (python) S 15317 15319 15176 0 -1 4202560 29 935 0 0 0 0 0 0 20 0 21 0 538430164 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
[pid=15319/tid=15584] ppid=15317 vsize=519992 CPUtime=0 cores=1,3,5,7
/proc/15319/task/15584/stat : 15584 (python) S 15317 15319 15176 0 -1 4202560 27 935 0 0 0 0 0 0 20 0 21 0 538450254 532471808 16267 33554432000 4194304 5683644 140734886922048 18446744073709551615 254512515505 0 0 16781312 2 18446604462103554048 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 15794676

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 15319 and gives
#  childrusage.ru_utime.tv_sec=9
#  childrusage.ru_utime.tv_usec=753517
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=214967
# CPU time returned by wait4() is 9.96848
# while last known CPU time is 4800.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1212.12
CPU time (s): 4800.1
CPU user time (s): 4795.92
CPU system time (s): 4.18
CPU usage (%): 396.007
Max. virtual memory (cumulated for all children) (KiB): 15794676

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 9.75352
system time used= 0.214967
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 45695
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2868
involuntary context switches= 102

runsolver used 6.45802 second user time and 14.6708 second system time

The end

Launcher Data 

Begin job on node149 at 2011-05-01 18:29:04
IDJOB=3317113
IDBENCH=82578
IDSOLVER=1705
FILE ID=node149/3317113-1304267344
RUNJOBID= node149-1304267279-15195
PBS_JOBID= 13173930
Free space on /tmp= 73308 MiB

SOLVER NAME= borg-sat 11.03.02-random
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v40000-c168000-S471691933-060-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3317113-1304267344/watcher-3317113-1304267344 -o /tmp/evaluation-result-3317113-1304267344/solver-3317113-1304267344 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3317113-1304267344.cnf -seed 1317240943 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 91da78a39fd68b6c63602d278464956f
RANDOM SEED=1317240943

node149.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30327744 kB
Buffers:        536716 kB
Cached:        1584452 kB
SwapCached:         60 kB
Active:         389104 kB
Inactive:      1848664 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30327744 kB
SwapTotal:    67111528 kB
SwapFree:     67111464 kB
Dirty:           37748 kB
Writeback:           0 kB
AnonPages:      115872 kB
Mapped:          20976 kB
Slab:           318816 kB
PageTables:       7924 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  3314932 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73276 MiB
End job on node149 at 2011-05-01 18:49:18