Trace number 3310270

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 40000.1 5003.72

General information on the benchmark

Nameapplication/jarvisalo/smtqfbv-aigs/
smtlib-qfbv-aigs-countbits128-tseitin.cnf
MD5SUM47da9e46b00ffd3e0b0f453a6c3c9f38
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark2605.99
Satisfiable
(Un)Satisfiability was proved
Number of variables95810
Number of clauses287045
Sum of the clauses size669769
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2191362
Number of clauses of size 395681
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3310270-1304007757/watcher-3310270-1304007757 -o /tmp/evaluation-result-3310270-1304007757/solver-3310270-1304007757 -C 40000 -W 5100 -M 31000 ppfolio HOME/instance-3310270-1304007757.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.43 5.60 3.62 2/170 20589
/proc/meminfo: memFree=11233108/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) R 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0

[startup+0.103592 s]
/proc/loadavg: 6.43 5.60 3.62 2/170 20589
/proc/meminfo: memFree=11233108/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 6 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.200572 s]
/proc/loadavg: 6.43 5.60 3.62 2/170 20589
/proc/meminfo: memFree=11233108/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 24 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300553 s]
/proc/loadavg: 6.43 5.60 3.62 2/170 20589
/proc/meminfo: memFree=11233108/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 24 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.70045 s]
/proc/loadavg: 6.43 5.60 3.62 2/170 20589
/proc/meminfo: memFree=11233108/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 24 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50131 s]
/proc/loadavg: 6.43 5.60 3.62 9/180 20604
/proc/meminfo: memFree=10958244/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 18 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=57932 CPUtime=1.49 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 8960 0 0 0 147 2 0 0 20 0 1 0 493191575 59322368 8653 33554432000 4194304 4587354 140737409364048 18446744073709551615 4237574 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 14483 8653 307 96 0 11045 0
[pid=20591] ppid=20589 vsize=65492 CPUtime=1.49 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 17031 0 0 0 147 2 0 0 19 0 1 0 493191575 67063808 13380 33554432000 4194304 4785740 140735116045776 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 16373 13380 339 145 0 13418 0
[pid=20592] ppid=20589 vsize=471928 CPUtime=1.49 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10064 0 0 0 148 1 0 0 20 0 1 0 493191575 483254272 9960 33554432000 134512640 134534564 4289201728 18446744073709551615 134526037 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 117982 9960 95 6 0 116648 0
[pid=20593] ppid=20589 vsize=186440 CPUtime=1.49 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 81373 0 0 0 139 10 0 0 20 0 1 0 493191575 190914560 21558 33554432000 4194304 4331988 140735969916800 18446744073709551615 4226440 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 46610 21558 169 34 0 45052 0
[pid=20594] ppid=20589 vsize=204472 CPUtime=4.93 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 37238 1250 0 0 488 5 0 0 20 0 5 0 493191575 209379328 23213 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 51118 23213 152 40 0 49662 0
[pid=20594/tid=20601] ppid=20589 vsize=204472 CPUtime=1.14 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 3453 1250 0 0 114 0 0 0 19 0 5 0 493191609 209379328 23213 33554432000 4194304 4356188 140736602621360 18446744073709551615 4264938 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=204472 CPUtime=1.14 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 7245 1250 0 0 114 0 0 0 25 0 5 0 493191609 209379328 23213 33554432000 4194304 4356188 140736602621360 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=204472 CPUtime=1.13 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 3378 1250 0 0 113 0 0 0 19 0 5 0 493191609 209379328 23213 33554432000 4194304 4356188 140736602621360 18446744073709551615 4261924 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=204472 CPUtime=1.14 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 6508 1250 0 0 114 0 0 0 25 0 5 0 493191609 209379328 23213 33554432000 4194304 4356188 140736602621360 18446744073709551615 4244489 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 10.89
Current children cumulated vsize (KiB) 1004588

[startup+3.10102 s]
/proc/loadavg: 6.43 5.60 3.62 9/180 20604
/proc/meminfo: memFree=10887564/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=76168 CPUtime=3.08 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 13519 0 0 0 306 2 0 0 25 0 1 0 493191575 77996032 13212 33554432000 4194304 4587354 140737409364048 18446744073709551615 4367388 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 19042 13212 312 96 0 15604 0
[pid=20591] ppid=20589 vsize=65492 CPUtime=3.09 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 17032 0 0 0 307 2 0 0 25 0 1 0 493191575 67063808 13381 33554432000 4194304 4785740 140735116045776 18446744073709551615 4319122 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 16373 13381 340 145 0 13418 0
[pid=20592] ppid=20589 vsize=475764 CPUtime=3.08 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10065 0 0 0 307 1 0 0 25 0 1 0 493191575 487182336 9961 33554432000 134512640 134534564 4289201728 18446744073709551615 134528955 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118941 9961 95 6 0 117607 0
[pid=20593] ppid=20589 vsize=189488 CPUtime=3.09 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 82208 0 0 0 299 10 0 0 25 0 1 0 493191575 194035712 22393 33554432000 4194304 4331988 140735969916800 18446744073709551615 4211493 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 47372 22393 169 34 0 45814 0
[pid=20594] ppid=20589 vsize=223768 CPUtime=11.32 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 43386 1250 0 0 1126 6 0 0 20 0 5 0 493191575 229138432 28119 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 55942 28119 164 40 0 54486 0
[pid=20594/tid=20601] ppid=20589 vsize=223768 CPUtime=2.74 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 5629 1250 0 0 274 0 0 0 23 0 5 0 493191609 229138432 28119 33554432000 4194304 4356188 140736602621360 18446744073709551615 4322000 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=223768 CPUtime=2.74 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 7849 1250 0 0 273 1 0 0 25 0 5 0 493191609 229138432 28119 33554432000 4194304 4356188 140736602621360 18446744073709551615 4212737 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=223768 CPUtime=2.72 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 6678 1250 0 0 272 0 0 0 23 0 5 0 493191609 229138432 28119 33554432000 4194304 4356188 140736602621360 18446744073709551615 4321963 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=223768 CPUtime=2.74 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 6576 1250 0 0 274 0 0 0 25 0 5 0 493191609 229138432 28119 33554432000 4194304 4356188 140736602621360 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 23.66
Current children cumulated vsize (KiB) 1049004

[startup+6.30044 s]
/proc/loadavg: 6.55 5.64 3.65 9/180 20605
/proc/meminfo: memFree=10815892/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=76168 CPUtime=6.28 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 13519 0 0 0 626 2 0 0 25 0 1 0 493191575 77996032 13212 33554432000 4194304 4587354 140737409364048 18446744073709551615 4421069 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 19042 13212 312 96 0 15604 0
[pid=20591] ppid=20589 vsize=72204 CPUtime=6.28 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 18738 0 0 0 626 2 0 0 25 0 1 0 493191575 73936896 15087 33554432000 4194304 4785740 140735116045776 18446744073709551615 4287244 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 18051 15087 340 145 0 15096 0
[pid=20592] ppid=20589 vsize=475764 CPUtime=6.28 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10065 0 0 0 627 1 0 0 25 0 1 0 493191575 487182336 9961 33554432000 134512640 134534564 4289201728 18446744073709551615 8439254 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118941 9961 95 6 0 117607 0
[pid=20593] ppid=20589 vsize=211576 CPUtime=6.29 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 105408 0 0 0 615 14 0 0 25 0 1 0 493191575 216653824 45593 33554432000 4194304 4331988 140735969916800 18446744073709551615 4240241 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 52894 45593 173 34 0 51336 0
[pid=20594] ppid=20589 vsize=223768 CPUtime=24.1 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 44946 1250 0 0 2404 6 0 0 20 0 5 0 493191575 229138432 29676 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 55942 29676 164 40 0 54486 0
[pid=20594/tid=20601] ppid=20589 vsize=223768 CPUtime=5.94 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 5635 1250 0 0 594 0 0 0 25 0 5 0 493191609 229138432 29676 33554432000 4194304 4356188 140736602621360 18446744073709551615 4305970 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=223768 CPUtime=5.94 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 7849 1250 0 0 593 1 0 0 25 0 5 0 493191609 229138432 29676 33554432000 4194304 4356188 140736602621360 18446744073709551615 4305747 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=223768 CPUtime=5.91 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 8232 1250 0 0 590 1 0 0 25 0 5 0 493191609 229138432 29676 33554432000 4194304 4356188 140736602621360 18446744073709551615 4300701 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=223768 CPUtime=5.94 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 6576 1250 0 0 594 0 0 0 25 0 5 0 493191609 229138432 29676 33554432000 4194304 4356188 140736602621360 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.23
Current children cumulated vsize (KiB) 1077804

[startup+12.7013 s]
/proc/loadavg: 6.67 5.68 3.67 9/180 20605
/proc/meminfo: memFree=10705468/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=156696 CPUtime=12.69 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 32237 0 0 0 1265 4 0 0 25 0 1 0 493191575 160456704 25315 33554432000 4194304 4587354 140737409364048 18446744073709551615 4238965 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 39174 25315 335 96 0 35736 0
[pid=20591] ppid=20589 vsize=109240 CPUtime=12.69 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 28015 0 0 0 1264 5 0 0 25 0 1 0 493191575 111861760 24364 33554432000 4194304 4785740 140735116045776 18446744073709551615 4287050 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 27310 24364 340 145 0 24355 0
[pid=20592] ppid=20589 vsize=475764 CPUtime=12.68 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10065 0 0 0 1267 1 0 0 25 0 1 0 493191575 487182336 9961 33554432000 134512640 134534564 4289201728 18446744073709551615 134528273 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118941 9961 95 6 0 117607 0
[pid=20593] ppid=20589 vsize=233140 CPUtime=12.68 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 110000 0 0 0 1254 14 0 0 25 0 1 0 493191575 238735360 50185 33554432000 4194304 4331988 140735969916800 18446744073709551615 4229294 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 58285 50185 241 34 0 56727 0
[pid=20594] ppid=20589 vsize=198436 CPUtime=49.67 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 44946 1250 0 0 4961 6 0 0 20 0 5 0 493191575 203198464 23343 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 49609 23343 164 40 0 48153 0
[pid=20594/tid=20601] ppid=20589 vsize=198436 CPUtime=12.34 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 5635 1250 0 0 1234 0 0 0 25 0 5 0 493191609 203198464 23343 33554432000 4194304 4356188 140736602621360 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=198436 CPUtime=12.34 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 7849 1250 0 0 1233 1 0 0 25 0 5 0 493191609 203198464 23343 33554432000 4194304 4356188 140736602621360 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=198436 CPUtime=12.28 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 8232 1250 0 0 1227 1 0 0 25 0 5 0 493191609 203198464 23343 33554432000 4194304 4356188 140736602621360 18446744073709551615 4284008 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=198436 CPUtime=12.34 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 6576 1250 0 0 1234 0 0 0 25 0 5 0 493191609 203198464 23343 33554432000 4194304 4356188 140736602621360 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.41
Current children cumulated vsize (KiB) 1191600

[startup+25.501 s]
/proc/loadavg: 6.96 5.79 3.74 9/180 20605
/proc/meminfo: memFree=10635088/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=156696 CPUtime=25.49 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 32279 0 0 0 2544 5 0 0 25 0 1 0 493191575 160456704 25357 33554432000 4194304 4587354 140737409364048 18446744073709551615 4230277 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=20592] ppid=20589 vsize=475768 CPUtime=4782.4 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10075 0 0 0 478238 2 0 0 25 0 1 0 493191575 487186432 9971 33554432000 134512640 134534564 4289201728 18446744073709551615 134530820 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118942 9971 103 6 0 117608 0
[pid=20593] ppid=20589 vsize=1149124 CPUtime=4780.81 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 274954 0 0 0 478012 69 0 0 25 0 1 0 493191575 1176702976 214607 33554432000 4194304 4331988 140735969916800 18446744073709551615 4263258 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 287281 214607 255 34 0 285723 0
[pid=20594] ppid=20589 vsize=650060 CPUtime=19102.1 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 862888 1250 0 0 1910084 126 0 0 20 0 5 0 493191575 665661440 136249 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 162515 136249 164 40 0 161059 0
[pid=20594/tid=20601] ppid=20589 vsize=650060 CPUtime=4782.05 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 314254 1250 0 0 478156 49 0 0 25 0 5 0 493191609 665661440 136249 33554432000 4194304 4356188 140736602621360 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=650060 CPUtime=4782.06 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 58032 1250 0 0 478199 7 0 0 25 0 5 0 493191609 665661440 136249 33554432000 4194304 4356188 140736602621360 18446744073709551615 4331023 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=650060 CPUtime=4756 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 306229 1250 0 0 475557 43 0 0 25 0 5 0 493191609 665661440 136249 33554432000 4194304 4356188 140736602621360 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=650060 CPUtime=4781.6 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 167719 1250 0 0 478138 22 0 0 25 0 5 0 493191609 665661440 136249 33554432000 4194304 4356188 140736602621360 18446744073709551615 4281495 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38230
Current children cumulated vsize (KiB) 4817564

[startup+4842.31 s]
/proc/loadavg: 8.00 8.02 8.00 9/180 20748
/proc/meminfo: memFree=8193216/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=1275056 CPUtime=4842.3 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 1121167 0 0 0 483826 404 0 0 25 0 1 0 493191575 1305657344 79115 33554432000 4194304 4587354 140737409364048 18446744073709551615 4236583 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 318764 79115 338 96 0 315326 0
[pid=20591] ppid=20589 vsize=1249232 CPUtime=4842.42 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 312102 0 0 0 484154 88 0 0 25 0 1 0 493191575 1279213568 308451 33554432000 4194304 4785740 140735116045776 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 312308 308451 340 145 0 309353 0
[pid=20592] ppid=20589 vsize=475768 CPUtime=4842.41 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10075 0 0 0 484239 2 0 0 25 0 1 0 493191575 487186432 9971 33554432000 134512640 134534564 4289201728 18446744073709551615 134526037 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118942 9971 103 6 0 117608 0
[pid=20593] ppid=20589 vsize=1149860 CPUtime=4840.82 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 275580 0 0 0 484012 70 0 0 25 0 1 0 493191575 1177456640 215233 33554432000 4194304 4331988 140735969916800 18446744073709551615 4237136 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 287465 215233 255 34 0 285907 0
[pid=20594] ppid=20589 vsize=695116 CPUtime=19341.8 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 892584 1250 0 0 1934049 130 0 0 20 0 5 0 493191575 711798784 147513 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 173779 147513 164 40 0 172323 0
[pid=20594/tid=20601] ppid=20589 vsize=695116 CPUtime=4842.06 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 318350 1250 0 0 484156 50 0 0 25 0 5 0 493191609 711798784 147513 33554432000 4194304 4356188 140736602621360 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=695116 CPUtime=4842.07 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 62128 1250 0 0 484199 8 0 0 25 0 5 0 493191609 711798784 147513 33554432000 4194304 4356188 140736602621360 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=695116 CPUtime=4815.67 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 306229 1250 0 0 481524 43 0 0 25 0 5 0 493191609 711798784 147513 33554432000 4194304 4356188 140736602621360 18446744073709551615 4210562 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=695116 CPUtime=4841.61 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 189223 1250 0 0 484136 25 0 0 25 0 5 0 493191609 711798784 147513 33554432000 4194304 4356188 140736602621360 18446744073709551615 4234297 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38709.7
Current children cumulated vsize (KiB) 4863356

[startup+4902.3 s]
/proc/loadavg: 8.11 8.05 8.01 9/180 20750
/proc/meminfo: memFree=8236948/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=1318940 CPUtime=4902.29 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 1136684 0 0 0 489820 409 0 0 25 0 1 0 493191575 1350594560 89121 33554432000 4194304 4587354 140737409364048 18446744073709551615 4238778 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 329735 89121 338 96 0 326297 0
[pid=20591] ppid=20589 vsize=1249232 CPUtime=4902.41 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 312155 0 0 0 490153 88 0 0 25 0 1 0 493191575 1279213568 308504 33554432000 4194304 4785740 140735116045776 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 312308 308504 340 145 0 309353 0
[pid=20592] ppid=20589 vsize=475768 CPUtime=4902.4 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10075 0 0 0 490238 2 0 0 25 0 1 0 493191575 487186432 9971 33554432000 134512640 134534564 4289201728 18446744073709551615 134530204 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118942 9971 103 6 0 117608 0
[pid=20593] ppid=20589 vsize=1151092 CPUtime=4900.81 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 276454 0 0 0 490011 70 0 0 25 0 1 0 493191575 1178718208 216107 33554432000 4194304 4331988 140735969916800 18446744073709551615 4237136 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 287773 216107 255 34 0 286215 0
[pid=20594] ppid=20589 vsize=607056 CPUtime=19581.4 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 904873 1250 0 0 1958013 131 0 0 20 0 5 0 493191575 621625344 125498 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 151764 125498 164 40 0 150308 0
[pid=20594/tid=20601] ppid=20589 vsize=607056 CPUtime=4902.06 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 320398 1250 0 0 490156 50 0 0 25 0 5 0 493191609 621625344 125498 33554432000 4194304 4356188 140736602621360 18446744073709551615 4237635 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=607056 CPUtime=4902.07 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 62128 1250 0 0 490199 8 0 0 25 0 5 0 493191609 621625344 125498 33554432000 4194304 4356188 140736602621360 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=607056 CPUtime=4875.34 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 308277 1250 0 0 487491 43 0 0 25 0 5 0 493191609 621625344 125498 33554432000 4194304 4356188 140736602621360 18446744073709551615 4279551 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=607056 CPUtime=4901.6 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 197416 1250 0 0 490134 26 0 0 25 0 5 0 493191609 621625344 125498 33554432000 4194304 4356188 140736602621360 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39189.4
Current children cumulated vsize (KiB) 4820412

[startup+4962.3 s]
/proc/loadavg: 8.11 8.05 8.00 9/180 20752
/proc/meminfo: memFree=8240752/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=1269448 CPUtime=4962.3 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 1140824 0 0 0 495818 412 0 0 25 0 1 0 493191575 1299914752 75811 33554432000 4194304 4587354 140737409364048 18446744073709551615 4236583 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 317362 75811 338 96 0 313924 0
[pid=20591] ppid=20589 vsize=1249232 CPUtime=4962.41 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 312165 0 0 0 496153 88 0 0 25 0 1 0 493191575 1279213568 308514 33554432000 4194304 4785740 140735116045776 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 312308 308514 340 145 0 309353 0
[pid=20592] ppid=20589 vsize=475768 CPUtime=4962.4 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10075 0 0 0 496238 2 0 0 25 0 1 0 493191575 487186432 9971 33554432000 134512640 134534564 4289201728 18446744073709551615 134528608 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118942 9971 103 6 0 117608 0
[pid=20593] ppid=20589 vsize=1151620 CPUtime=4960.81 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 277288 0 0 0 496011 70 0 0 25 0 1 0 493191575 1179258880 216937 33554432000 4194304 4331988 140735969916800 18446744073709551615 4228901 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 287905 216937 251 34 0 286347 0
[pid=20594] ppid=20589 vsize=654160 CPUtime=19821.1 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 929449 1250 0 0 1981977 135 0 0 20 0 5 0 493191575 669859840 137274 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 163540 137274 164 40 0 162084 0
[pid=20594/tid=20601] ppid=20589 vsize=654160 CPUtime=4962.06 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 328078 1250 0 0 496155 51 0 0 25 0 5 0 493191609 669859840 137274 33554432000 4194304 4356188 140736602621360 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=654160 CPUtime=4962.07 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 62128 1250 0 0 496199 8 0 0 25 0 5 0 493191609 669859840 137274 33554432000 4194304 4356188 140736602621360 18446744073709551615 4210557 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=654160 CPUtime=4935.01 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 325173 1250 0 0 493456 45 0 0 25 0 5 0 493191609 669859840 137274 33554432000 4194304 4356188 140736602621360 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=654160 CPUtime=4961.59 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 197416 1250 0 0 496133 26 0 0 25 0 5 0 493191609 669859840 137274 33554432000 4194304 4356188 140736602621360 18446744073709551615 4298607 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39669
Current children cumulated vsize (KiB) 4818552



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.7 s]
/proc/loadavg: 8.05 8.04 8.00 9/180 20753
/proc/meminfo: memFree=8266892/32951124 swapFree=67111528/67111528
[pid=20589] ppid=20587 vsize=18324 CPUtime=0 cores=0-7
/proc/20589/stat : 20589 (ppfolio) S 20587 20589 20191 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 493191574 18763776 390 33554432000 4194304 4287625 140734560536272 18446744073709551615 233510317583 0 0 4096 65536 7519401559541678079 0 0 17 4 0 0 0
/proc/20589/statm: 4581 390 341 23 0 91 0
[pid=20590] ppid=20589 vsize=1325216 CPUtime=5003.71 cores=0
/proc/20590/stat : 20590 (cryptominisat) R 20589 20589 20191 0 -1 4202496 1153039 0 0 0 499955 416 0 0 25 0 1 0 493191575 1357021184 82088 33554432000 4194304 4587354 140737409364048 18446744073709551615 4238755 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20590/statm: 331304 82088 338 96 0 327866 0
[pid=20591] ppid=20589 vsize=1249232 CPUtime=5003.82 cores=1
/proc/20591/stat : 20591 (clasp) R 20589 20589 20191 0 -1 4202496 312191 0 0 0 500294 88 0 0 25 0 1 0 493191575 1279213568 308540 33554432000 4194304 4785740 140735116045776 18446744073709551615 4287042 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20591/statm: 312308 308540 340 145 0 309353 0
[pid=20592] ppid=20589 vsize=475768 CPUtime=5003.81 cores=2
/proc/20592/stat : 20592 (TNM) R 20589 20589 20191 0 -1 4202496 10075 0 0 0 500379 2 0 0 25 0 1 0 493191575 487186432 9971 33554432000 134512640 134534564 4289201728 18446744073709551615 8440184 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20592/statm: 118942 9971 103 6 0 117608 0
[pid=20593] ppid=20589 vsize=1152732 CPUtime=5002.22 cores=3
/proc/20593/stat : 20593 (march_hi) R 20589 20589 20191 0 -1 4202496 278063 0 0 0 500152 70 0 0 25 0 1 0 493191575 1180397568 217712 33554432000 4194304 4331988 140735969916800 18446744073709551615 4229451 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20593/statm: 288183 217712 251 34 0 286625 0
[pid=20594] ppid=20589 vsize=598864 CPUtime=19986.5 cores=4-7
/proc/20594/stat : 20594 (plingeling) S 20589 20589 20191 0 -1 4202496 946857 1250 0 0 1998512 138 0 0 20 0 5 0 493191575 613236736 123450 33554432000 4194304 4356188 140736602621360 18446744073709551615 233522100197 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/20594/statm: 149716 123450 164 40 0 148260 0
[pid=20594/tid=20601] ppid=20589 vsize=598864 CPUtime=5003.46 cores=4-7
/proc/20594/task/20601/stat : 20601 (plingeling) R 20589 20589 20191 0 -1 4202560 333198 1250 0 0 500294 52 0 0 25 0 5 0 493191609 613236736 123450 33554432000 4194304 4356188 140736602621360 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20594/tid=20602] ppid=20589 vsize=598864 CPUtime=5003.47 cores=4-7
/proc/20594/task/20602/stat : 20602 (plingeling) R 20589 20589 20191 0 -1 4202560 66224 1250 0 0 500339 8 0 0 25 0 5 0 493191609 613236736 123450 33554432000 4194304 4356188 140736602621360 18446744073709551615 4284190 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20594/tid=20603] ppid=20589 vsize=598864 CPUtime=4976.2 cores=4-7
/proc/20594/task/20603/stat : 20603 (plingeling) R 20589 20589 20191 0 -1 4202560 325173 1250 0 0 497574 46 0 0 25 0 5 0 493191609 613236736 123450 33554432000 4194304 4356188 140736602621360 18446744073709551615 4233969 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=20594/tid=20604] ppid=20589 vsize=598864 CPUtime=5003 cores=4-7
/proc/20594/task/20604/stat : 20604 (plingeling) R 20589 20589 20191 0 -1 4202560 205608 1250 0 0 500272 28 0 0 25 0 5 0 493191609 613236736 123450 33554432000 4194304 4356188 140736602621360 18446744073709551615 4281782 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 4820136

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 20589 and gives
#  childrusage.ru_utime.tv_sec=5003
#  childrusage.ru_utime.tv_usec=798307
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=33994
# CPU time returned by wait4() is 5003.83
# while last known CPU time is 40000.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5003.72
CPU time (s): 40000.1
CPU user time (s): 39992.9
CPU system time (s): 7.14
CPU usage (%): 799.407
Max. virtual memory (cumulated for all children) (KiB): 5857884

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5003.8
system time used= 0.033994
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 10599
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 569
involuntary context switches= 14991

runsolver used 7.97779 second user time and 19.3191 second system time

The end

Launcher Data 

Begin job on node145 at 2011-04-28 18:22:37
IDJOB=3310270
IDBENCH=82462
IDSOLVER=1638
FILE ID=node145/3310270-1304007757
RUNJOBID= node145-1304007357-20205
PBS_JOBID= 13168830
Free space on /tmp= 72204 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/application/jarvisalo/smtqfbv-aigs/smtlib-qfbv-aigs-countbits128-tseitin.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3310270-1304007757/watcher-3310270-1304007757 -o /tmp/evaluation-result-3310270-1304007757/solver-3310270-1304007757 -C 40000 -W 5100 -M 31000  ppfolio HOME/instance-3310270-1304007757.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 47da9e46b00ffd3e0b0f453a6c3c9f38
RANDOM SEED=530090230

node145.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.57
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.786
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      11233264 kB
Buffers:       2202196 kB
Cached:       18868924 kB
SwapCached:          0 kB
Active:       10930436 kB
Inactive:     10209376 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      11233264 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           14976 kB
Writeback:           0 kB
AnonPages:       67992 kB
Mapped:          14376 kB
Slab:           510648 kB
PageTables:       4292 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   189360 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72192 MiB
End job on node145 at 2011-04-28 19:46:03