Trace number 3219351

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39971.5 5000.02

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S394483079-002-UNKNOWN.cnf
MD5SUM40e955d306a70bd3dffa8283c1221639
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark611.74
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219351-1304150280/watcher-3219351-1304150280 -o /tmp/evaluation-result-3219351-1304150280/solver-3219351-1304150280 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219351-1304150280.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.12 8.09 2/174 13393
/proc/meminfo: memFree=29476748/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) R 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0

[startup+0.086322 s]
/proc/loadavg: 8.10 8.12 8.09 2/174 13393
/proc/meminfo: memFree=29476748/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.186268 s]
/proc/loadavg: 8.10 8.12 8.09 2/174 13393
/proc/meminfo: memFree=29476748/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300249 s]
/proc/loadavg: 8.10 8.12 8.09 2/174 13393
/proc/meminfo: memFree=29476748/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 23 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700173 s]
/proc/loadavg: 8.10 8.12 8.09 2/174 13393
/proc/meminfo: memFree=29476748/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 22 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50103 s]
/proc/loadavg: 8.10 8.12 8.09 9/184 13408
/proc/meminfo: memFree=29415280/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1216636 CPUtime=1.49 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 6456 0 0 0 148 1 0 0 20 0 1 0 526625872 1245835264 4222 33554432000 4194304 4587354 140733263101360 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 304159 4222 327 96 0 300721 0
[pid=13395] ppid=13393 vsize=28064 CPUtime=1.49 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 4660 0 0 0 148 1 0 0 20 0 1 0 526625872 28737536 4364 33554432000 4194304 4785740 140735360754496 18446744073709551615 4550312 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 7016 4365 340 145 0 4061 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=1.49 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 149 0 0 0 20 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134528730 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=19084 CPUtime=1.49 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 7349 0 0 0 149 0 0 0 20 0 1 0 526625872 19542016 2354 33554432000 4194304 4331988 140733427216832 18446744073709551615 4265456 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 4771 2354 174 34 0 3213 0
[pid=13398] ppid=13393 vsize=126116 CPUtime=5.74 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 6143 1252 0 0 573 1 0 0 18 0 5 0 526625872 129142784 3601 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 31529 3601 161 40 0 30073 0
[pid=13398/tid=13405] ppid=13393 vsize=126116 CPUtime=1.41 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 619 1252 0 0 141 0 0 0 23 0 5 0 526625880 129142784 3601 33554432000 4194304 4356188 140733812859824 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=126116 CPUtime=1.41 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 652 1252 0 0 141 0 0 0 20 0 5 0 526625880 129142784 3601 33554432000 4194304 4356188 140733812859824 18446744073709551615 4210605 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=126116 CPUtime=1.41 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 579 1252 0 0 141 0 0 0 23 0 5 0 526625880 129142784 3601 33554432000 4194304 4356188 140733812859824 18446744073709551615 4210820 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=126116 CPUtime=1.4 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 604 1252 0 0 140 0 0 0 23 0 5 0 526625880 129142784 3601 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.7
Current children cumulated vsize (KiB) 1870672

[startup+3.10073 s]
/proc/loadavg: 8.09 8.11 8.09 9/184 13408
/proc/meminfo: memFree=29411064/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1241236 CPUtime=3.08 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 10267 0 0 0 307 1 0 0 25 0 1 0 526625872 1271025664 3794 33554432000 4194304 4587354 140733263101360 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 310309 3794 327 96 0 306871 0
[pid=13395] ppid=13393 vsize=32748 CPUtime=3.09 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 5834 0 0 0 308 1 0 0 25 0 1 0 526625872 33533952 5538 33554432000 4194304 4785740 140735360754496 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 8187 5538 340 145 0 5232 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=3.08 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 308 0 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134528864 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=19084 CPUtime=3.09 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 7349 0 0 0 309 0 0 0 25 0 1 0 526625872 19542016 2354 33554432000 4194304 4331988 140733427216832 18446744073709551615 4265456 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 4771 2354 174 34 0 3213 0
[pid=13398] ppid=13393 vsize=128372 CPUtime=12.13 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 6737 1252 0 0 1212 1 0 0 18 0 5 0 526625872 131452928 4165 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 32093 4165 161 40 0 30637 0
[pid=13398/tid=13405] ppid=13393 vsize=128372 CPUtime=3.01 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 748 1252 0 0 301 0 0 0 25 0 5 0 526625880 131452928 4165 33554432000 4194304 4356188 140733812859824 18446744073709551615 4273850 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=128372 CPUtime=3.01 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 781 1252 0 0 301 0 0 0 25 0 5 0 526625880 131452928 4165 33554432000 4194304 4356188 140733812859824 18446744073709551615 4211015 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=128372 CPUtime=3.01 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 747 1252 0 0 301 0 0 0 25 0 5 0 526625880 131452928 4165 33554432000 4194304 4356188 140733812859824 18446744073709551615 4334062 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=128372 CPUtime=3 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 772 1252 0 0 300 0 0 0 25 0 5 0 526625880 131452928 4165 33554432000 4194304 4356188 140733812859824 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.47
Current children cumulated vsize (KiB) 1902212

[startup+6.30114 s]
/proc/loadavg: 8.09 8.11 8.09 9/184 13408
/proc/meminfo: memFree=29353040/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1286932 CPUtime=6.29 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 22619 0 0 0 626 3 0 0 25 0 1 0 526625872 1317818368 16136 33554432000 4194304 4587354 140733263101360 18446744073709551615 4364768 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 321733 16136 337 96 0 318295 0
[pid=13395] ppid=13393 vsize=37032 CPUtime=6.29 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 6885 0 0 0 628 1 0 0 25 0 1 0 526625872 37920768 6589 33554432000 4194304 4785740 140735360754496 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 9258 6589 340 145 0 6303 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=6.28 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 628 0 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134530504 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=19084 CPUtime=6.29 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 7349 0 0 0 629 0 0 0 25 0 1 0 526625872 19542016 2354 33554432000 4194304 4331988 140733427216832 18446744073709551615 4226369 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 4771 2354 174 34 0 3213 0
[pid=13398] ppid=13393 vsize=131088 CPUtime=24.91 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 7416 1252 0 0 2490 1 0 0 18 0 5 0 526625872 134234112 4844 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 32772 4844 161 40 0 31316 0
[pid=13398/tid=13405] ppid=13393 vsize=131088 CPUtime=6.22 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 908 1252 0 0 622 0 0 0 25 0 5 0 526625880 134234112 4844 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=131088 CPUtime=6.21 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 973 1252 0 0 621 0 0 0 25 0 5 0 526625880 134234112 4844 33554432000 4194304 4356188 140733812859824 18446744073709551615 4332293 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=131088 CPUtime=6.21 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 875 1252 0 0 621 0 0 0 25 0 5 0 526625880 134234112 4844 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331345 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=131088 CPUtime=6.18 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 971 1252 0 0 618 0 0 0 25 0 5 0 526625880 134234112 4844 33554432000 4194304 4356188 140733812859824 18446744073709551615 4281508 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.06
Current children cumulated vsize (KiB) 1954908

[startup+12.701 s]
/proc/loadavg: 8.15 8.13 8.09 9/184 13409
/proc/meminfo: memFree=29358028/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1242968 CPUtime=12.69 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 33584 0 0 0 1264 5 0 0 25 0 1 0 526625872 1272799232 10551 33554432000 4194304 4587354 140733263101360 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 310742 10551 337 96 0 307304 0
[pid=13395] ppid=13393 vsize=46052 CPUtime=12.69 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 9106 0 0 0 1267 2 0 0 25 0 1 0 526625872 47157248 8810 33554432000 4194304 4785740 140735360754496 18446744073709551615 4287241 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 11513 8810 340 145 0 8558 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=12.68 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 1268 0 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134530538 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=20108 CPUtime=12.69 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 7565 0 0 0 1269 0 0 0 25 0 1 0 526625872 20590592 2570 33554432000 4194304 4331988 140733427216832 18446744073709551615 4265477 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 5027 2570 174 34 0 3469 0
[pid=13398] ppid=13393 vsize=138848 CPUtime=50.47 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 10860 1252 0 0 5045 2 0 0 18 0 5 0 526625872 142180352 6752 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 34712 6752 161 40 0 33256 0
[pid=13398/tid=13405] ppid=13393 vsize=138848 CPUtime=12.61 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 1773 1252 0 0 1261 0 0 0 25 0 5 0 526625880 142180352 6752 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=138848 CPUtime=12.61 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 1877 1252 0 0 1261 0 0 0 25 0 5 0 526625880 142180352 6752 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=138848 CPUtime=12.61 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 1708 1252 0 0 1261 0 0 0 25 0 5 0 526625880 142180352 6752 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=138848 CPUtime=12.54 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 1813 1252 0 0 1254 0 0 0 25 0 5 0 526625880 142180352 6752 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.22
Current children cumulated vsize (KiB) 1928748

[startup+25.5006 s]
/proc/loadavg: 8.13 8.12 8.09 9/184 13409
/proc/meminfo: memFree=29319768/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1273764 CPUtime=25.49 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 47182 0 0 0 2541 8 0 0 25 0 1 0 526625872 1304334336 17773 33554432000 4194304 4587354 140733263101360 18446744073709551615 4388336 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=13396] ppid=13393 vsize=462448 CPUtime=4782.42 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 478240 2 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134529024 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=472432 CPUtime=4782.29 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 115765 0 0 0 478153 76 0 0 25 0 1 0 526625872 483770368 110770 33554432000 4194304 4331988 140733427216832 18446744073709551615 4228188 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 118108 110770 176 34 0 116550 0
[pid=13398] ppid=13393 vsize=229224 CPUtime=19101.4 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 618694 1252 0 0 1910045 94 0 0 18 0 5 0 526625872 234725376 29371 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 57306 29371 161 40 0 55850 0
[pid=13398/tid=13405] ppid=13393 vsize=229224 CPUtime=4782.37 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 150471 1252 0 0 478213 24 0 0 25 0 5 0 526625880 234725376 29371 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=229224 CPUtime=4782.37 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 153840 1252 0 0 478213 24 0 0 25 0 5 0 526625880 234725376 29371 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331311 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=229224 CPUtime=4781.91 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 155242 1252 0 0 478168 23 0 0 25 0 5 0 526625880 234725376 29371 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=229224 CPUtime=4754.65 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 155452 1252 0 0 475443 22 0 0 25 0 5 0 526625880 234725376 29371 33554432000 4194304 4356188 140733812859824 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231
Current children cumulated vsize (KiB) 3011088

[startup+4842.3 s]
/proc/loadavg: 8.08 8.13 8.11 10/184 13561
/proc/meminfo: memFree=28262148/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1476040 CPUtime=4842.45 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 1630556 0 0 0 483591 654 0 0 25 0 1 0 526625872 1511464960 87125 33554432000 4194304 4587354 140733263101360 18446744073709551615 4238622 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 369010 87125 337 96 0 365572 0
[pid=13395] ppid=13393 vsize=321684 CPUtime=4842.42 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 75155 0 0 0 484217 25 0 0 25 0 1 0 526625872 329404416 74738 33554432000 4194304 4785740 140735360754496 18446744073709551615 4291944 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 80421 74738 340 145 0 77466 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=4842.42 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 484240 2 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134528913 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=472432 CPUtime=4842.3 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 117213 0 0 0 484153 77 0 0 25 0 1 0 526625872 483770368 112218 33554432000 4194304 4331988 140733427216832 18446744073709551615 4226263 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 118108 112218 176 34 0 116550 0
[pid=13398] ppid=13393 vsize=222184 CPUtime=19341.1 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 621862 1252 0 0 1934011 94 0 0 18 0 5 0 526625872 227516416 27611 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 55546 27611 161 40 0 54090 0
[pid=13398/tid=13405] ppid=13393 vsize=222184 CPUtime=4842.37 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 150727 1252 0 0 484213 24 0 0 25 0 5 0 526625880 227516416 27611 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=222184 CPUtime=4842.37 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 154288 1252 0 0 484213 24 0 0 25 0 5 0 526625880 227516416 27611 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=222184 CPUtime=4841.9 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 155434 1252 0 0 484167 23 0 0 25 0 5 0 526625880 227516416 27611 33554432000 4194304 4356188 140733812859824 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=222184 CPUtime=4814.32 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 157724 1252 0 0 481410 22 0 0 25 0 5 0 526625880 227516416 27611 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.6
Current children cumulated vsize (KiB) 2973112

[startup+4902.3 s]
/proc/loadavg: 8.12 8.13 8.11 9/184 13563
/proc/meminfo: memFree=28281600/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1476040 CPUtime=4902.44 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 1647819 0 0 0 489582 662 0 0 25 0 1 0 526625872 1511464960 82443 33554432000 4194304 4587354 140733263101360 18446744073709551615 4238449 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 369010 82443 337 96 0 365572 0
[pid=13395] ppid=13393 vsize=324064 CPUtime=4902.43 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 75705 0 0 0 490217 26 0 0 25 0 1 0 526625872 331841536 75288 33554432000 4194304 4785740 140735360754496 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 81016 75288 340 145 0 78061 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=4902.43 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 490240 3 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134528708 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=472432 CPUtime=4902.3 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 118699 0 0 0 490152 78 0 0 25 0 1 0 526625872 483770368 113704 33554432000 4194304 4331988 140733427216832 18446744073709551615 4237364 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 118108 113704 176 34 0 116550 0
[pid=13398] ppid=13393 vsize=214760 CPUtime=19580.7 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 622406 1252 0 0 1957979 95 0 0 18 0 5 0 526625872 219914240 25755 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 53690 25755 161 40 0 52234 0
[pid=13398/tid=13405] ppid=13393 vsize=214760 CPUtime=4902.37 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 150727 1252 0 0 490213 24 0 0 25 0 5 0 526625880 219914240 25755 33554432000 4194304 4356188 140733812859824 18446744073709551615 4208624 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=214760 CPUtime=4902.37 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 154416 1252 0 0 490213 24 0 0 25 0 5 0 526625880 219914240 25755 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=214760 CPUtime=4901.89 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 155690 1252 0 0 490166 23 0 0 25 0 5 0 526625880 219914240 25755 33554432000 4194304 4356188 140733812859824 18446744073709551615 4279565 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=214760 CPUtime=4874 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 157884 1252 0 0 487378 22 0 0 25 0 5 0 526625880 219914240 25755 33554432000 4194304 4356188 140733812859824 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.3
Current children cumulated vsize (KiB) 2968068

[startup+4962.3 s]
/proc/loadavg: 8.12 8.13 8.11 9/184 13565
/proc/meminfo: memFree=28294476/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1479768 CPUtime=4962.45 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 1663102 0 0 0 495576 669 0 0 25 0 1 0 526625872 1515282432 75549 33554432000 4194304 4587354 140733263101360 18446744073709551615 4388345 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 369942 75549 337 96 0 366504 0
[pid=13395] ppid=13393 vsize=324064 CPUtime=4962.43 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 75719 0 0 0 496217 26 0 0 25 0 1 0 526625872 331841536 75302 33554432000 4194304 4785740 140735360754496 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 81016 75302 340 145 0 78061 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=4962.43 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 496240 3 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134530573 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=472432 CPUtime=4962.3 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 120092 0 0 0 496151 79 0 0 25 0 1 0 526625872 483770368 115097 33554432000 4194304 4331988 140733427216832 18446744073709551615 4210516 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 118108 115097 176 34 0 116550 0
[pid=13398] ppid=13393 vsize=222568 CPUtime=19820.4 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 626790 1252 0 0 1981946 95 0 0 18 0 5 0 526625872 227909632 27707 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 55642 27707 161 40 0 54186 0
[pid=13398/tid=13405] ppid=13393 vsize=222568 CPUtime=4962.38 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 150759 1252 0 0 496214 24 0 0 25 0 5 0 526625880 227909632 27707 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331082 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=222568 CPUtime=4962.37 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 156528 1252 0 0 496213 24 0 0 25 0 5 0 526625880 227909632 27707 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=222568 CPUtime=4961.89 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 157930 1252 0 0 496166 23 0 0 25 0 5 0 526625880 227909632 27707 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331355 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=222568 CPUtime=4933.67 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 157884 1252 0 0 493345 22 0 0 25 0 5 0 526625880 227909632 27707 33554432000 4194304 4356188 140733812859824 18446744073709551615 4332149 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670
Current children cumulated vsize (KiB) 2979604



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.20 8.15 8.11 9/184 13566
/proc/meminfo: memFree=28279840/32951124 swapFree=67111352/67111528
[pid=13393] ppid=13391 vsize=18324 CPUtime=0 cores=0-7
/proc/13393/stat : 13393 (ppfolio) S 13391 13393 13180 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 526625872 18763776 390 33554432000 4194304 4287625 140734211298528 18446744073709551615 255377321487 0 0 4096 65536 4610393409364951039 0 0 17 6 0 0 0
/proc/13393/statm: 4581 390 341 23 0 91 0
[pid=13394] ppid=13393 vsize=1479768 CPUtime=5000.15 cores=0
/proc/13394/stat : 13394 (cryptominisat) R 13393 13393 13180 0 -1 4202496 1663102 0 0 0 499346 669 0 0 25 0 1 0 526625872 1515282432 75549 33554432000 4194304 4587354 140733263101360 18446744073709551615 4388352 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13394/statm: 369942 75549 337 96 0 366504 0
[pid=13395] ppid=13393 vsize=324856 CPUtime=5000.14 cores=1
/proc/13395/stat : 13395 (clasp) R 13393 13393 13180 0 -1 4202496 75923 0 0 0 499988 26 0 0 25 0 1 0 526625872 332652544 75506 33554432000 4194304 4785740 140735360754496 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13395/statm: 81214 75506 340 145 0 78259 0
[pid=13396] ppid=13393 vsize=462448 CPUtime=5000.14 cores=2
/proc/13396/stat : 13396 (TNM) R 13393 13393 13180 0 -1 4202496 2355 0 0 0 500011 3 0 0 25 0 1 0 526625872 473546752 2251 33554432000 134512640 134534564 4293594592 18446744073709551615 134528719 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13396/statm: 115612 2251 95 6 0 114278 0
[pid=13397] ppid=13393 vsize=696196 CPUtime=5000.01 cores=3
/proc/13397/stat : 13397 (march_hi) R 13393 13393 13180 0 -1 4202496 121032 0 0 0 499921 80 0 0 25 0 1 0 526625872 712904704 116037 33554432000 4194304 4331988 140733427216832 18446744073709551615 4265444 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13397/statm: 174049 116037 176 34 0 172491 0
[pid=13398] ppid=13393 vsize=232808 CPUtime=19971 cores=4-7
/proc/13398/stat : 13398 (plingeling) S 13393 13393 13180 0 -1 4202496 629350 1252 0 0 1997007 96 0 0 18 0 5 0 526625872 238395392 30267 33554432000 4194304 4356188 140733812859824 18446744073709551615 255389104101 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13398/statm: 58202 30267 161 40 0 56746 0
[pid=13398/tid=13405] ppid=13393 vsize=232808 CPUtime=5000.08 cores=4-7
/proc/13398/task/13405/stat : 13405 (plingeling) R 13393 13393 13180 0 -1 4202560 152999 1252 0 0 499984 24 0 0 25 0 5 0 526625880 238395392 30267 33554432000 4194304 4356188 140733812859824 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13398/tid=13406] ppid=13393 vsize=232808 CPUtime=5000.08 cores=4-7
/proc/13398/task/13406/stat : 13406 (plingeling) R 13393 13393 13180 0 -1 4202560 156688 1252 0 0 499984 24 0 0 25 0 5 0 526625880 238395392 30267 33554432000 4194304 4356188 140733812859824 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13398/tid=13407] ppid=13393 vsize=232808 CPUtime=4999.59 cores=4-7
/proc/13398/task/13407/stat : 13407 (plingeling) R 13393 13393 13180 0 -1 4202560 158090 1252 0 0 499936 23 0 0 25 0 5 0 526625880 238395392 30267 33554432000 4194304 4356188 140733812859824 18446744073709551615 4279985 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13398/tid=13408] ppid=13393 vsize=232808 CPUtime=4971.17 cores=4-7
/proc/13398/task/13408/stat : 13408 (plingeling) R 13393 13393 13180 0 -1 4202560 157884 1252 0 0 497095 22 0 0 25 0 5 0 526625880 238395392 30267 33554432000 4194304 4356188 140733812859824 18446744073709551615 4210587 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39971.5
Current children cumulated vsize (KiB) 3214400

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 13393 and gives
#  childrusage.ru_utime.tv_sec=5000
#  childrusage.ru_utime.tv_usec=119866
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=37994
# CPU time returned by wait4() is 5000.16
# while last known CPU time is 39971.5
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39971.5
CPU user time (s): 39962.7
CPU system time (s): 8.74
CPU usage (%): 799.427
Max. virtual memory (cumulated for all children) (KiB): 4213384

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5000.12
system time used= 0.037994
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2879
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 397
involuntary context switches= 220834

runsolver used 7.75782 second user time and 19.525 second system time

The end

Launcher Data 

Begin job on node135 at 2011-04-30 09:58:00
IDJOB=3219351
IDBENCH=82575
IDSOLVER=1638
FILE ID=node135/3219351-1304150280
RUNJOBID= node135-1304145274-13194
PBS_JOBID= 13170876
Free space on /tmp= 72100 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S394483079-002-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219351-1304150280/watcher-3219351-1304150280 -o /tmp/evaluation-result-3219351-1304150280/solver-3219351-1304150280 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219351-1304150280.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 40e955d306a70bd3dffa8283c1221639
RANDOM SEED=953636724

node135.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6640.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29477028 kB
Buffers:        946672 kB
Cached:        1942364 kB
SwapCached:        172 kB
Active:         185956 kB
Inactive:      2768920 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29477028 kB
SwapTotal:    67111528 kB
SwapFree:     67111352 kB
Dirty:           10896 kB
Writeback:           0 kB
AnonPages:       65868 kB
Mapped:          14688 kB
Slab:           454304 kB
PageTables:       4792 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   190116 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72088 MiB
End job on node135 at 2011-04-30 11:21:22