Trace number 3377287

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (TO) 1300.3 1300.06

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v20000-c84000-S357264423-067-UNKNOWN.cnf
MD5SUM425b4c5e5535f7d6b9d17ca8feb0ac3e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark78.815
Satisfiable
(Un)Satisfiability was proved
Number of variables20000
Number of clauses84000
Sum of the clauses size252000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 384000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.36/0.50	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.36/0.50	c INFO: solving HOME/instance-3377287-1305452081.cnf
0.36/0.50	c INFO: computing task features
0.56/0.75	c INFO: computed variable-clause graph statistics
0.56/0.76	c INFO: computed clause balance statistics
2.07/2.25	c INFO: computed variable balance statistics
3.26/3.42	c INFO: computed small-clause counts
3.26/3.43	c INFO: computed Horn-clause counts
3.76/3.98	c INFO: computed variable graph statistics
3.76/3.99	c INFO: collected features for HOME/instance-3377287-1305452081.cnf in 3.51 s
3.86/4.00	c INFO: running hybridGM3@0 for 97 with 2396 remaining (b = 0.43)
104.46/104.65	c INFO: running hybridGM3@0 for 97 with 2296 remaining (b = 0.26)
205.18/205.30	c INFO: running hybridGM3@0 for 97 with 2195 remaining (b = 0.19)
306.00/305.96	c INFO: running hybridGM3@100 for 97 with 2095 remaining (b = 0.15)
403.29/403.21	c INFO: running hybridGM3@100 for 97 with 1998 remaining (b = 0.15)
500.50/500.46	c INFO: running TNM@0 for 97 with 1901 remaining (b = 0.16)
600.76/600.72	c INFO: running hybridGM3@100 for 97 with 1801 remaining (b = 0.14)
697.21/696.99	c INFO: running hybridGM3@0 for 194 with 1704 remaining (b = 0.26)
894.24/893.91	c INFO: running hybridGM3@0 for 194 with 1508 remaining (b = 0.20)
1091.12/1090.82	c INFO: running hybridGM3@0 for 194 with 1312 remaining (b = 0.15)
1288.02/1287.71	c INFO: running hybridGM3@0 for 194 with 1115 remaining (b = 0.12)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3377287-1305452081/watcher-3377287-1305452081 -o /tmp/evaluation-result-3377287-1305452081/solver-3377287-1305452081 -C 2400 -W 1300 -M 7680 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3377287-1305452081.cnf -seed 506314655 -budget 2400 

running on 2 cores: 4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 2430 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 7864320 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 7915520 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.00 4.00 3.82 4/355 15423
/proc/meminfo: memFree=15291360/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=16240 CPUtime=0 cores=4,6
/proc/15423/stat : 15423 (python) R 15421 15423 14075 0 -1 4202496 997 0 0 0 0 0 0 0 17 0 1 0 112817531 16629760 870 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880225989 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15423/statm: 4060 870 368 364 0 547 0

[startup+0.051835 s]
/proc/loadavg: 4.00 4.00 3.82 4/355 15423
/proc/meminfo: memFree=15291360/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=29788 CPUtime=0.03 cores=4,6
/proc/15423/stat : 15423 (python) R 15421 15423 14075 0 -1 4202496 1459 0 0 0 3 0 0 0 17 0 1 0 112817531 30502912 1299 33554432000 4194304 5683644 140733870149904 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15423/statm: 7447 1299 443 364 0 817 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 29788

[startup+0.100829 s]
/proc/loadavg: 4.00 4.00 3.82 4/355 15423
/proc/meminfo: memFree=15291360/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=104220 CPUtime=0.05 cores=4,6
/proc/15423/stat : 15423 (python) R 15421 15423 14075 0 -1 4202496 3047 0 0 0 4 1 0 0 17 0 1 0 112817531 106721280 2732 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880560997 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15423/statm: 26055 2732 847 364 0 1803 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 104220

[startup+0.300794 s]
/proc/loadavg: 4.00 4.00 3.82 4/355 15423
/proc/meminfo: memFree=15291360/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=168676 CPUtime=0.2 cores=4,6
/proc/15423/stat : 15423 (python) R 15421 15423 14075 0 -1 4202496 6730 931 1 0 15 5 0 0 17 0 1 0 112817531 172724224 4797 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880608391 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15423/statm: 42168 4798 1331 364 0 3453 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 168676

[startup+0.700727 s]
/proc/loadavg: 4.00 4.00 3.82 4/355 15423
/proc/meminfo: memFree=15291360/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=295084 CPUtime=0.56 cores=4,6
/proc/15423/stat : 15423 (python) R 15421 15423 14075 0 -1 4202496 13334 931 1 0 50 6 0 0 17 0 1 0 112817531 302166016 11008 33554432000 4194304 5683644 140733870149904 18446744073709551615 47508394744517 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15423/statm: 73771 11009 1761 364 0 9695 0
Current children cumulated CPU time (s) 0.56
Current children cumulated vsize (KiB) 295084

[startup+1.50058 s]
/proc/loadavg: 4.00 4.00 3.82 5/356 15432
/proc/meminfo: memFree=15227864/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=297692 CPUtime=1.36 cores=4,6
/proc/15423/stat : 15423 (python) R 15421 15423 14075 0 -1 4202496 16364 931 1 0 130 6 0 0 19 0 1 0 112817531 304836608 11808 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880221347 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15423/statm: 74423 11808 1826 364 0 10347 0
Current children cumulated CPU time (s) 1.36
Current children cumulated vsize (KiB) 297692

[startup+3.1003 s]
/proc/loadavg: 4.00 4.00 3.82 5/359 15435
/proc/meminfo: memFree=15227096/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=297692 CPUtime=2.96 cores=4,6
/proc/15423/stat : 15423 (python) R 15421 15423 14075 0 -1 4202496 16365 931 1 0 290 6 0 0 22 0 1 0 112817531 304836608 11809 33554432000 4194304 5683644 140733870149904 18446744073709551615 4272792 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15423/statm: 74423 11809 1827 364 0 10347 0
Current children cumulated CPU time (s) 2.96
Current children cumulated vsize (KiB) 297692

[startup+6.30074 s]
/proc/loadavg: 4.00 4.00 3.82 5/365 15441
/proc/meminfo: memFree=15181132/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=296980 CPUtime=3.87 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 16698 931 1 0 380 7 0 0 25 0 2 0 112817531 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 74245 9125 1867 364 0 10163 0
[pid=15439] ppid=15423 vsize=287828 CPUtime=0 cores=4,6
/proc/15439/stat : 15439 (python) S 15423 15423 14075 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 112817931 294735872 7610 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585231 0 0 16781312 514 224165136431022338 0 0 17 4 0 0 0
/proc/15439/statm: 71957 7610 355 364 0 7875 0
[pid=15441] ppid=15439 vsize=268424 CPUtime=2.28 cores=4,6
/proc/15441/stat : 15441 (hybridGM3) R 15439 15441 15441 0 -1 4202496 6663 0 0 0 227 1 0 0 25 0 1 0 112817931 274866176 6397 33554432000 134512640 135204385 4286780864 18446744073709551615 134549096 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15441/statm: 67106 6397 64 169 0 66936 0
[pid=15423/tid=15440] ppid=15421 vsize=296980 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 2 0 112817931 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 6.15
Current children cumulated vsize (KiB) 853232

[startup+12.7006 s]
/proc/loadavg: 4.00 4.00 3.82 5/365 15441
/proc/meminfo: memFree=15179284/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=296980 CPUtime=3.87 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 16698 931 1 0 380 7 0 0 25 0 2 0 112817531 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 74245 9125 1867 364 0 10163 0
[pid=15439] ppid=15423 vsize=287828 CPUtime=0 cores=4,6
/proc/15439/stat : 15439 (python) S 15423 15423 14075 0 -1 4202560 985 0 0 0 0 0 0 0 18 0 1 0 112817931 294735872 7638 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585231 0 0 16781312 514 224165136431022338 0 0 17 4 0 0 0
/proc/15439/statm: 71957 7638 383 364 0 7875 0
[pid=15441] ppid=15439 vsize=268424 CPUtime=8.66 cores=4,6
/proc/15441/stat : 15441 (hybridGM3) R 15439 15441 15441 0 -1 4202496 6663 0 0 0 865 1 0 0 25 0 1 0 112817931 274866176 6397 33554432000 134512640 135204385 4286780864 18446744073709551615 134537322 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15441/statm: 67106 6397 64 169 0 66936 0
[pid=15423/tid=15440] ppid=15421 vsize=296980 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 2 0 112817931 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 12.53
Current children cumulated vsize (KiB) 853232

[startup+25.5004 s]
/proc/loadavg: 4.00 4.00 3.82 5/365 15441
/proc/meminfo: memFree=15180032/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=296980 CPUtime=3.87 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 16698 931 1 0 380 7 0 0 25 0 2 0 112817531 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 74245 9125 1867 364 0 10163 0
[pid=15439] ppid=15423 vsize=287828 CPUtime=0.04 cores=4,6
/proc/15439/stat : 15439 (python) S 15423 15423 14075 0 -1 4202560 991 0 0 0 2 2 0 0 18 0 1 0 112817931 294735872 7638 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585231 0 0 16781312 514 7345898757822578946 0 0 17 4 0 0 0
/proc/15439/statm: 71957 7638 383 364 0 7875 0
[pid=15441] ppid=15439 vsize=268424 CPUtime=21.43 cores=4,6
/proc/15441/stat : 15441 (hybridGM3) R 15439 15441 15441 0 -1 4202496 6663 0 0 0 2142 1 0 0 25 0 1 0 112817931 274866176 6397 33554432000 134512640 135204385 4286780864 18446744073709551615 134536656 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15441/statm: 67106 6397 64 169 0 66936 0
[pid=15423/tid=15440] ppid=15421 vsize=296980 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 2 0 112817931 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 25.34
Current children cumulated vsize (KiB) 853232

[startup+51.1009 s]
/proc/loadavg: 4.00 4.00 3.83 6/348 15481
/proc/meminfo: memFree=15171816/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=296980 CPUtime=3.87 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 16698 931 1 0 380 7 0 0 25 0 2 0 112817531 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 74245 9125 1867 364 0 10163 0
[pid=15439] ppid=15423 vsize=287828 CPUtime=0.11 cores=4,6
/proc/15439/stat : 15439 (python) S 15423 15423 14075 0 -1 4202560 1006 0 0 0 5 6 0 0 18 0 1 0 112817931 294735872 7638 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585231 0 0 16781312 514 7345898757822578946 0 0 17 4 0 0 0
/proc/15439/statm: 71957 7638 383 364 0 7875 0
[pid=15441] ppid=15439 vsize=268424 CPUtime=46.97 cores=4,6
/proc/15441/stat : 15441 (hybridGM3) R 15439 15441 15441 0 -1 4202496 6663 0 0 0 4695 2 0 0 25 0 1 0 112817931 274866176 6397 33554432000 134512640 135204385 4286780864 18446744073709551615 134536791 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15441/statm: 67106 6397 64 169 0 66936 0
[pid=15423/tid=15440] ppid=15421 vsize=296980 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 2 0 112817931 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 50.95
Current children cumulated vsize (KiB) 853232

[startup+102.313 s]
/proc/loadavg: 4.00 4.00 3.83 5/352 15521
/proc/meminfo: memFree=15195844/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=296980 CPUtime=3.87 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 16698 931 1 0 380 7 0 0 25 0 2 0 112817531 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 74245 9125 1867 364 0 10163 0
[pid=15439] ppid=15423 vsize=287828 CPUtime=0.22 cores=4,6
/proc/15439/stat : 15439 (python) S 15423 15423 14075 0 -1 4202560 1027 0 0 0 11 11 0 0 18 0 1 0 112817931 294735872 7638 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585231 0 0 16781312 514 7345898757822578946 0 0 17 4 0 0 0
/proc/15439/statm: 71957 7638 383 364 0 7875 0
[pid=15441] ppid=15439 vsize=268424 CPUtime=98.08 cores=4,6
/proc/15441/stat : 15441 (hybridGM3) R 15439 15441 15441 0 -1 4202496 6667 0 0 0 9806 2 0 0 25 0 1 0 112817931 274866176 6401 33554432000 134512640 135204385 4286780864 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15441/statm: 67106 6401 64 169 0 66936 0
[pid=15423/tid=15440] ppid=15421 vsize=296980 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 2 0 112817931 304107520 9125 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 102.17
Current children cumulated vsize (KiB) 853232

[startup+162.301 s]
/proc/loadavg: 4.00 4.00 3.84 5/356 15566
/proc/meminfo: memFree=15121580/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=307844 CPUtime=3.88 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 17589 931 1 0 381 7 0 0 20 0 3 0 112817531 315232256 9267 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 76961 9267 1870 364 0 12876 0
[pid=15439] ppid=15423 vsize=298072 CPUtime=0.23 cores=4,6
/proc/15439/stat : 15439 (python) S 15423 15423 14075 0 -1 4202560 1166 0 0 0 12 11 0 0 19 0 2 0 112817931 305225728 7675 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15439/statm: 74518 7675 415 364 0 10436 0
[pid=15441] ppid=15439 vsize=268424 CPUtime=100.39 cores=4,6
/proc/15441/stat : 15441 (hybridGM3) T 15439 15441 15441 0 -1 4202496 6667 0 0 0 10037 2 0 0 25 0 1 0 112817931 274866176 6401 33554432000 134512640 135204385 4286780864 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/15441/statm: 67106 6401 64 169 0 66936 0
[pid=15439/tid=15522] ppid=15423 vsize=298072 CPUtime=0 cores=4,6
/proc/15439/task/15522/stat : 15522 (python) S 15423 15423 14075 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 112827994 305225728 7675 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604444374951168 0 0 -1 6 0 0 0
[pid=15524] ppid=15423 vsize=297752 CPUtime=0.12 cores=4,6
/proc/15524/stat : 15524 (python) S 15423 15423 14075 0 -1 4202560 921 0 0 0 7 5 0 0 15 0 1 0 112827996 304898048 7766 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585286 0 0 16781312 514 7345898757822578947 0 0 17 4 0 0 0
/proc/15524/statm: 74438 7766 372 364 0 10353 0
[pid=15526] ppid=15524 vsize=268424 CPUtime=57.53 cores=4,6
/proc/15526/stat : 15526 (hybridGM3) R 15524 15526 15526 0 -1 4202496 6658 0 0 0 5752 1 0 0 25 0 1 0 112827996 274866176 6394 33554432000 134512640 135204385 4290433424 18446744073709551615 134546128 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15526/statm: 67106 6394 64 169 0 66936 0
[pid=15423/tid=15440] ppid=15421 vsize=307844 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 25 0 3 0 112817931 315232256 9267 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15525] ppid=15421 vsize=307844 CPUtime=0 cores=4,6
/proc/15423/task/15525/stat : 15525 (python) S 15421 15423 14075 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 3 0 112827996 315232256 9267 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 162.15
Current children cumulated vsize (KiB) 1440516

[startup+222.301 s]
/proc/loadavg: 4.06 4.01 3.85 5/364 15650
/proc/meminfo: memFree=15054696/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=318100 CPUtime=3.89 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 18410 931 1 0 382 7 0 0 20 0 4 0 112817531 325734400 9273 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 79525 9273 1872 364 0 15437 0

################
# More data... #
################

/proc/15524/statm: 74438 7795 400 364 0 10353 0
[pid=15526] ppid=15524 vsize=268424 CPUtime=197.41 cores=4,6
/proc/15526/stat : 15526 (hybridGM3) T 15524 15526 15526 0 -1 4202496 6666 0 0 0 19740 1 0 0 25 0 1 0 112827996 274866176 6402 33554432000 134512640 135204385 4290433424 18446744073709551615 134537224 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15526/statm: 67106 6402 64 169 0 66936 0
[pid=15524/tid=15642] ppid=15423 vsize=297752 CPUtime=0 cores=4,6
/proc/15524/task/15642/stat : 15642 (python) S 15423 15423 14075 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 112838060 304898048 7795 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604453905834432 0 0 -1 4 0 0 0
[pid=15643] ppid=15423 vsize=308616 CPUtime=0.52 cores=4,6
/proc/15643/stat : 15643 (python) S 15423 15423 14075 0 -1 4202560 1125 0 0 0 29 23 0 0 15 0 2 0 112838061 316022784 7799 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15643/statm: 77154 7799 400 364 0 13066 0
[pid=15645] ppid=15643 vsize=276052 CPUtime=196.89 cores=4,6
/proc/15645/stat : 15645 (hybridGM3) T 15643 15645 15645 0 -1 4202496 15809 0 0 0 19688 1 0 0 25 0 1 0 112838061 282677248 8827 33554432000 134512640 135204385 4291603248 18446744073709551615 134536887 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/15645/statm: 69013 8827 85 169 0 68843 0
[pid=15643/tid=15661] ppid=15423 vsize=308616 CPUtime=0 cores=4,6
/proc/15643/task/15661/stat : 15661 (python) S 15423 15423 14075 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 112848126 316022784 7799 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604443223684992 0 0 -1 6 0 0 0
[pid=15687] ppid=15423 vsize=318388 CPUtime=0.24 cores=4,6
/proc/15687/stat : 15687 (python) S 15423 15423 14075 0 -1 4202560 1072 0 0 0 13 11 0 0 19 0 2 0 112867577 326029312 7871 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15687/statm: 79597 7871 399 364 0 15506 0
[pid=15689] ppid=15687 vsize=458972 CPUtime=99.98 cores=4,6
/proc/15689/stat : 15689 (TNM) T 15687 15689 15689 0 -1 4202496 3591 0 0 0 9998 0 0 0 25 0 1 0 112867578 469987328 3328 33554432000 134512640 135008748 4294397184 18446744073709551615 134527542 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15689/statm: 114743 3328 46 122 0 113643 0
[pid=15687/tid=15737] ppid=15423 vsize=318388 CPUtime=0 cores=4,6
/proc/15687/task/15737/stat : 15737 (python) S 15423 15423 14075 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 112877602 326029312 7871 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604443223683200 0 0 -1 4 0 0 0
[pid=15744] ppid=15423 vsize=328852 CPUtime=0.51 cores=4,6
/proc/15744/stat : 15744 (python) S 15423 15423 14075 0 -1 4202560 1125 0 0 0 26 25 0 0 17 0 2 0 112887231 336744448 7876 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15744/statm: 82213 7876 400 364 0 18119 0
[pid=15746] ppid=15744 vsize=268424 CPUtime=196.39 cores=4,6
/proc/15746/stat : 15746 (hybridGM3) T 15744 15746 15746 0 -1 4202496 6662 0 0 0 19638 1 0 0 25 0 1 0 112887231 274866176 6399 33554432000 134512640 135204385 4293040592 18446744073709551615 134547676 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15746/statm: 67106 6399 64 169 0 66936 0
[pid=15744/tid=15782] ppid=15423 vsize=328852 CPUtime=0 cores=4,6
/proc/15744/task/15782/stat : 15782 (python) S 15423 15423 14075 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 112906921 336744448 7876 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604441041740224 0 0 -1 4 0 0 0
[pid=15783] ppid=15423 vsize=339048 CPUtime=0.5 cores=4,6
/proc/15783/stat : 15783 (python) S 15423 15423 14075 0 -1 4202560 1117 0 0 0 28 22 0 0 17 0 2 0 112906922 347185152 7880 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15783/statm: 84762 7880 400 364 0 20665 0
[pid=15785] ppid=15783 vsize=268424 CPUtime=196.39 cores=4,6
/proc/15785/stat : 15785 (hybridGM3) T 15783 15785 15785 0 -1 4202496 6657 0 0 0 19638 1 0 0 25 0 1 0 112906923 274866176 6398 33554432000 134512640 135204385 4288064640 18446744073709551615 134537565 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15785/statm: 67106 6398 64 169 0 66936 0
[pid=15783/tid=15838] ppid=15423 vsize=339048 CPUtime=0 cores=4,6
/proc/15783/task/15838/stat : 15838 (python) S 15423 15423 14075 0 -1 4202560 47 0 0 0 0 0 0 0 17 0 2 0 112926612 347185152 7880 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604441192665344 0 0 -1 4 0 0 0
[pid=15839] ppid=15423 vsize=349416 CPUtime=0.37 cores=4,6
/proc/15839/stat : 15839 (python) S 15423 15423 14075 0 -1 4202560 946 0 0 0 19 18 0 0 15 0 1 0 112926613 357801984 7855 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585286 0 0 16781312 514 7345898757822578947 0 0 17 4 0 0 0
/proc/15839/statm: 87354 7855 372 364 0 23254 0
[pid=15841] ppid=15839 vsize=268424 CPUtime=151.09 cores=4,6
/proc/15841/stat : 15841 (hybridGM3) R 15839 15841 15841 0 -1 4202496 6658 0 0 0 15109 0 0 0 25 0 1 0 112926613 274866176 6397 33554432000 134512640 135204385 4290178048 18446744073709551615 134537292 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15841/statm: 67106 6397 64 169 0 66936 0
[pid=15423/tid=15440] ppid=15421 vsize=359400 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 33 931 0 0 0 0 0 0 18 0 8 0 112817931 368025600 9374 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15525] ppid=15421 vsize=359400 CPUtime=0 cores=4,6
/proc/15423/task/15525/stat : 15525 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 18 0 8 0 112827996 368025600 9374 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15644] ppid=15421 vsize=359400 CPUtime=0 cores=4,6
/proc/15423/task/15644/stat : 15644 (python) S 15421 15423 14075 0 -1 4202560 30 931 0 0 0 0 0 0 18 0 8 0 112838061 368025600 9374 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15688] ppid=15421 vsize=359400 CPUtime=0 cores=4,6
/proc/15423/task/15688/stat : 15688 (python) S 15421 15423 14075 0 -1 4202560 24 931 0 0 0 0 0 0 20 0 8 0 112867578 368025600 9374 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15745] ppid=15421 vsize=359400 CPUtime=0 cores=4,6
/proc/15423/task/15745/stat : 15745 (python) S 15421 15423 14075 0 -1 4202560 30 931 0 0 0 0 0 0 20 0 8 0 112887231 368025600 9374 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15784] ppid=15421 vsize=359400 CPUtime=0 cores=4,6
/proc/15423/task/15784/stat : 15784 (python) S 15421 15423 14075 0 -1 4202560 24 931 0 0 0 0 0 0 20 0 8 0 112906922 368025600 9374 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15840] ppid=15421 vsize=359400 CPUtime=0 cores=4,6
/proc/15423/task/15840/stat : 15840 (python) S 15421 15423 14075 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 8 0 112926613 368025600 9374 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1242.61
Current children cumulated vsize (KiB) 4676688



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+1300 s]
/proc/loadavg: 1.97 2.74 3.35 2/364 15899
/proc/meminfo: memFree=14990264/32950928 swapFree=67111364/67111528
[pid=15423] ppid=15421 vsize=369656 CPUtime=3.99 cores=4,6
/proc/15423/stat : 15423 (python) S 15421 15423 14075 0 -1 4202496 23016 931 1 0 391 8 0 0 20 0 9 0 112817531 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15423/statm: 92414 9380 1890 364 0 28311 0
[pid=15439] ppid=15423 vsize=298072 CPUtime=0.48 cores=4,6
/proc/15439/stat : 15439 (python) S 15423 15423 14075 0 -1 4202560 1204 0 0 0 25 23 0 0 18 0 2 0 112817931 305225728 7676 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15439/statm: 74518 7676 416 364 0 10436 0
[pid=15441] ppid=15439 vsize=268424 CPUtime=197.36 cores=4,6
/proc/15441/stat : 15441 (hybridGM3) T 15439 15441 15441 0 -1 4202496 6667 0 0 0 19734 2 0 0 25 0 1 0 112817931 274866176 6401 33554432000 134512640 135204385 4286780864 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/15441/statm: 67106 6401 64 169 0 66936 0
[pid=15439/tid=15522] ppid=15423 vsize=298072 CPUtime=0 cores=4,6
/proc/15439/task/15522/stat : 15522 (python) S 15423 15423 14075 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 112827994 305225728 7676 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604444374951168 0 0 -1 6 0 0 0
[pid=15524] ppid=15423 vsize=297752 CPUtime=0.5 cores=4,6
/proc/15524/stat : 15524 (python) S 15423 15423 14075 0 -1 4202560 1121 0 0 0 29 21 0 0 15 0 2 0 112827996 304898048 7795 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15524/statm: 74438 7795 400 364 0 10353 0
[pid=15526] ppid=15524 vsize=268424 CPUtime=197.41 cores=4,6
/proc/15526/stat : 15526 (hybridGM3) T 15524 15526 15526 0 -1 4202496 6666 0 0 0 19740 1 0 0 25 0 1 0 112827996 274866176 6402 33554432000 134512640 135204385 4290433424 18446744073709551615 134537224 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15526/statm: 67106 6402 64 169 0 66936 0
[pid=15524/tid=15642] ppid=15423 vsize=297752 CPUtime=0 cores=4,6
/proc/15524/task/15642/stat : 15642 (python) S 15423 15423 14075 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 112838060 304898048 7795 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604453905834432 0 0 -1 4 0 0 0
[pid=15643] ppid=15423 vsize=308616 CPUtime=0.52 cores=4,6
/proc/15643/stat : 15643 (python) S 15423 15423 14075 0 -1 4202560 1125 0 0 0 29 23 0 0 15 0 2 0 112838061 316022784 7799 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/15643/statm: 77154 7799 400 364 0 13066 0
[pid=15645] ppid=15643 vsize=276052 CPUtime=196.89 cores=4,6
/proc/15645/stat : 15645 (hybridGM3) T 15643 15645 15645 0 -1 4202496 15809 0 0 0 19688 1 0 0 25 0 1 0 112838061 282677248 8827 33554432000 134512640 135204385 4291603248 18446744073709551615 134536887 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/15645/statm: 69013 8827 85 169 0 68843 0
[pid=15643/tid=15661] ppid=15423 vsize=308616 CPUtime=0 cores=4,6
/proc/15643/task/15661/stat : 15661 (python) S 15423 15423 14075 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 112848126 316022784 7799 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604443223684992 0 0 -1 6 0 0 0
[pid=15687] ppid=15423 vsize=318388 CPUtime=0.24 cores=4,6
/proc/15687/stat : 15687 (python) S 15423 15423 14075 0 -1 4202560 1072 0 0 0 13 11 0 0 19 0 2 0 112867577 326029312 7871 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15687/statm: 79597 7871 399 364 0 15506 0
[pid=15689] ppid=15687 vsize=458972 CPUtime=99.98 cores=4,6
/proc/15689/stat : 15689 (TNM) T 15687 15689 15689 0 -1 4202496 3591 0 0 0 9998 0 0 0 25 0 1 0 112867578 469987328 3328 33554432000 134512640 135008748 4294397184 18446744073709551615 134527542 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15689/statm: 114743 3328 46 122 0 113643 0
[pid=15687/tid=15737] ppid=15423 vsize=318388 CPUtime=0 cores=4,6
/proc/15687/task/15737/stat : 15737 (python) S 15423 15423 14075 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 112877602 326029312 7871 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604443223683200 0 0 -1 4 0 0 0
[pid=15744] ppid=15423 vsize=328852 CPUtime=0.51 cores=4,6
/proc/15744/stat : 15744 (python) S 15423 15423 14075 0 -1 4202560 1125 0 0 0 26 25 0 0 17 0 2 0 112887231 336744448 7876 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15744/statm: 82213 7876 400 364 0 18119 0
[pid=15746] ppid=15744 vsize=268424 CPUtime=196.39 cores=4,6
/proc/15746/stat : 15746 (hybridGM3) T 15744 15746 15746 0 -1 4202496 6662 0 0 0 19638 1 0 0 25 0 1 0 112887231 274866176 6399 33554432000 134512640 135204385 4293040592 18446744073709551615 134547676 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15746/statm: 67106 6399 64 169 0 66936 0
[pid=15744/tid=15782] ppid=15423 vsize=328852 CPUtime=0 cores=4,6
/proc/15744/task/15782/stat : 15782 (python) S 15423 15423 14075 0 -1 4202560 48 0 0 0 0 0 0 0 17 0 2 0 112906921 336744448 7876 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604441041740224 0 0 -1 4 0 0 0
[pid=15783] ppid=15423 vsize=339048 CPUtime=0.5 cores=4,6
/proc/15783/stat : 15783 (python) S 15423 15423 14075 0 -1 4202560 1117 0 0 0 28 22 0 0 17 0 2 0 112906922 347185152 7880 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15783/statm: 84762 7880 400 364 0 20665 0
[pid=15785] ppid=15783 vsize=268424 CPUtime=196.39 cores=4,6
/proc/15785/stat : 15785 (hybridGM3) T 15783 15785 15785 0 -1 4202496 6657 0 0 0 19638 1 0 0 25 0 1 0 112906923 274866176 6398 33554432000 134512640 135204385 4288064640 18446744073709551615 134537565 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15785/statm: 67106 6398 64 169 0 66936 0
[pid=15783/tid=15838] ppid=15423 vsize=339048 CPUtime=0 cores=4,6
/proc/15783/task/15838/stat : 15838 (python) S 15423 15423 14075 0 -1 4202560 47 0 0 0 0 0 0 0 17 0 2 0 112926612 347185152 7880 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604441192665344 0 0 -1 4 0 0 0
[pid=15839] ppid=15423 vsize=349416 CPUtime=0.49 cores=4,6
/proc/15839/stat : 15839 (python) S 15423 15423 14075 0 -1 4202560 1119 0 0 0 26 23 0 0 17 0 2 0 112926613 357801984 7884 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892391883 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15839/statm: 87354 7884 400 364 0 23254 0
[pid=15841] ppid=15839 vsize=268424 CPUtime=196.38 cores=4,6
/proc/15841/stat : 15841 (hybridGM3) T 15839 15841 15841 0 -1 4202496 6658 0 0 0 19638 0 0 0 25 0 1 0 112926613 274866176 6397 33554432000 134512640 135204385 4290178048 18446744073709551615 134537292 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/15841/statm: 67106 6397 64 169 0 66936 0
[pid=15839/tid=15874] ppid=15423 vsize=349416 CPUtime=0 cores=4,6
/proc/15839/task/15874/stat : 15874 (python) S 15423 15423 14075 0 -1 4202560 47 0 0 0 0 0 0 0 17 0 2 0 112946301 357801984 7884 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 514 18446604453909173184 0 0 -1 4 0 0 0
[pid=15875] ppid=15423 vsize=359664 CPUtime=0.02 cores=4,6
/proc/15875/stat : 15875 (python) S 15423 15423 14075 0 -1 4202560 935 0 0 0 1 1 0 0 15 0 1 0 112946302 368295936 7865 33554432000 4194304 5683644 140733870149904 18446744073709551615 218880585286 0 0 16781312 514 7345898757822578945 0 0 17 4 0 0 0
/proc/15875/statm: 89916 7865 378 364 0 25813 0
[pid=15877] ppid=15875 vsize=268420 CPUtime=12.25 cores=4,6
/proc/15877/stat : 15877 (hybridGM3) R 15875 15877 15877 0 -1 4202496 6665 0 0 0 1224 1 0 0 25 0 1 0 112946303 274862080 6399 33554432000 134512640 135204385 4292399776 18446744073709551615 134537328 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/15877/statm: 67105 6399 64 169 0 66935 0
[pid=15423/tid=15440] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15440/stat : 15440 (python) S 15421 15423 14075 0 -1 4202560 33 931 0 0 0 0 0 0 18 0 9 0 112817931 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15525] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15525/stat : 15525 (python) S 15421 15423 14075 0 -1 4202560 29 931 0 0 0 0 0 0 18 0 9 0 112827996 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15644] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15644/stat : 15644 (python) S 15421 15423 14075 0 -1 4202560 30 931 0 0 0 0 0 0 18 0 9 0 112838061 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15688] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15688/stat : 15688 (python) S 15421 15423 14075 0 -1 4202560 24 931 0 0 0 0 0 0 20 0 9 0 112867578 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15745] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15745/stat : 15745 (python) S 15421 15423 14075 0 -1 4202560 30 931 0 0 0 0 0 0 20 0 9 0 112887231 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15784] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15784/stat : 15784 (python) S 15421 15423 14075 0 -1 4202560 24 931 0 0 0 0 0 0 20 0 9 0 112906922 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15840] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15840/stat : 15840 (python) S 15421 15423 14075 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 9 0 112926613 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
[pid=15423/tid=15876] ppid=15421 vsize=369656 CPUtime=0 cores=4,6
/proc/15423/task/15876/stat : 15876 (python) S 15421 15423 14075 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 9 0 112946302 378527744 9380 33554432000 4194304 5683644 140733870149904 18446744073709551615 218892388785 0 0 16781312 2 18446604444374952064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1300.3
Current children cumulated vsize (KiB) 5315028

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 15423 and gives
#  childrusage.ru_utime.tv_sec=3
#  childrusage.ru_utime.tv_usec=914404
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=91986
# CPU time returned by wait4() is 4.00639
# while last known CPU time is 1300.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1300.06
CPU time (s): 1300.3
CPU user time (s): 1298.66
CPU system time (s): 1.64
CPU usage (%): 100.018
Max. virtual memory (cumulated for all children) (KiB): 5315028

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 3.9144
system time used= 0.091986
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 23953
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2907
involuntary context switches= 41

runsolver used 5.70813 second user time and 11.3453 second system time

The end

Launcher Data 

Begin job on node112 at 2011-05-15 11:34:41
IDJOB=3377287
IDBENCH=82596
IDSOLVER=1857
FILE ID=node112/3377287-1305452081
RUNJOBID= node112-1305451373-15066
PBS_JOBID= 13324849
Free space on /tmp= 73476 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v20000-c84000-S357264423-067-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3377287-1305452081/watcher-3377287-1305452081 -o /tmp/evaluation-result-3377287-1305452081/solver-3377287-1305452081 -C 2400 -W 1300 -M 7680  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3377287-1305452081.cnf -seed 506314655 -budget 2400

TIME LIMIT= 2400 seconds
MEMORY LIMIT= 7680 MiB
NBCORE= 2

MD5SUM BENCH= 425b4c5e5535f7d6b9d17ca8feb0ac3e
RANDOM SEED=506314655

node112.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5180.45
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      15292384 kB
Buffers:        301220 kB
Cached:        3100388 kB
SwapCached:          0 kB
Active:       15593256 kB
Inactive:      1804112 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      15292384 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           99488 kB
Writeback:           0 kB
AnonPages:    13995356 kB
Mapped:          30340 kB
Slab:           145044 kB
PageTables:      51112 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 53903844 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73512 MiB
End job on node112 at 2011-05-15 11:56:23