Trace number 3276001

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 40000.5 5003.62

General information on the benchmark

NameSAT04/handmade/kexu/frb-0.8/
frb59-26-2.used-as.sat04-892.cnf
MD5SUMfd8d5783ed404e5c6590d627ffdcca50
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1534
Number of clauses132295
Sum of the clauses size266006
Maximum clause length26
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2132236
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 559

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3276001-1304172774/watcher-3276001-1304172774 -o /tmp/evaluation-result-3276001-1304172774/solver-3276001-1304172774 -C 40000 -W 5100 -M 31000 ppfolio HOME/instance-3276001-1304172774.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.08 8.02 2/170 5530
/proc/meminfo: memFree=24803308/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 4 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0

[startup+0.0348669 s]
/proc/loadavg: 8.10 8.08 8.02 2/170 5530
/proc/meminfo: memFree=24803308/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 4 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100855 s]
/proc/loadavg: 8.10 8.08 8.02 2/170 5530
/proc/meminfo: memFree=24803308/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300821 s]
/proc/loadavg: 8.10 8.08 8.02 2/170 5530
/proc/meminfo: memFree=24803308/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 23 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700749 s]
/proc/loadavg: 8.10 8.08 8.02 2/170 5530
/proc/meminfo: memFree=24803308/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 20 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50061 s]
/proc/loadavg: 8.10 8.08 8.02 9/180 5545
/proc/meminfo: memFree=24719652/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=44744 CPUtime=1.49 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 9015 0 0 0 148 1 0 0 20 0 1 0 607508184 45817856 6469 33554432000 4194304 4587354 140737469757040 18446744073709551615 4236479 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 11186 6469 326 96 0 7748 0
[pid=5532] ppid=5530 vsize=20908 CPUtime=1.49 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 3427 0 0 0 149 0 0 0 20 0 1 0 607508184 21409792 2573 33554432000 4194304 4785740 140735102053824 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 5227 2573 340 145 0 2272 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=1.48 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 148 0 0 0 20 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=26060 CPUtime=1.49 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 10210 0 0 0 148 1 0 0 20 0 1 0 607508184 26685440 2756 33554432000 4194304 4331988 140736894350928 18446744073709551615 4226457 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 6515 2756 175 34 0 4957 0
[pid=5535] ppid=5530 vsize=143768 CPUtime=5.7 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 10245 1248 0 0 569 1 0 0 19 0 5 0 607508184 147218432 7984 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 35942 7984 163 40 0 34486 0
[pid=5535/tid=5542] ppid=5530 vsize=143768 CPUtime=1.4 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 1420 1248 0 0 140 0 0 0 23 0 5 0 607508193 147218432 7984 33554432000 4194304 4356188 140733421168016 18446744073709551615 4270898 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=143768 CPUtime=1.4 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 1417 1248 0 0 140 0 0 0 23 0 5 0 607508193 147218432 7984 33554432000 4194304 4356188 140733421168016 18446744073709551615 4270960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=143768 CPUtime=1.4 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 1033 1248 0 0 140 0 0 0 25 0 5 0 607508193 147218432 7984 33554432000 4194304 4356188 140733421168016 18446744073709551615 4210575 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=143768 CPUtime=1.39 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 1753 1248 0 0 139 0 0 0 25 0 5 0 607508193 147218432 7984 33554432000 4194304 4356188 140733421168016 18446744073709551615 4283929 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.65
Current children cumulated vsize (KiB) 716756

[startup+3.10133 s]
/proc/loadavg: 8.10 8.08 8.02 9/180 5545
/proc/meminfo: memFree=24711220/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1628264 CPUtime=3.09 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 10307 0 0 0 308 1 0 0 25 0 1 0 607508184 1667342336 6677 33554432000 4194304 4587354 140737469757040 18446744073709551615 4429920 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 407066 6677 337 96 0 403628 0
[pid=5532] ppid=5530 vsize=25956 CPUtime=3.09 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 4686 0 0 0 309 0 0 0 25 0 1 0 607508184 26578944 3832 33554432000 4194304 4785740 140735102053824 18446744073709551615 4286998 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 6489 3832 340 145 0 3534 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=3.08 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 308 0 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134530741 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=26412 CPUtime=3.09 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 10308 0 0 0 308 1 0 0 25 0 1 0 607508184 27045888 2854 33554432000 4194304 4331988 140736894350928 18446744073709551615 4239523 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 6603 2854 175 34 0 5045 0
[pid=5535] ppid=5530 vsize=153112 CPUtime=12.09 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 12581 1248 0 0 1208 1 0 0 19 0 5 0 607508184 156786688 10320 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 38278 10320 163 40 0 36822 0
[pid=5535/tid=5542] ppid=5530 vsize=153112 CPUtime=3 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 1964 1248 0 0 300 0 0 0 25 0 5 0 607508193 156786688 10320 33554432000 4194304 4356188 140733421168016 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=153112 CPUtime=3 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 2057 1248 0 0 300 0 0 0 25 0 5 0 607508193 156786688 10320 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331076 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=153112 CPUtime=3 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 1737 1248 0 0 300 0 0 0 25 0 5 0 607508193 156786688 10320 33554432000 4194304 4356188 140733421168016 18446744073709551615 4281548 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=153112 CPUtime=2.98 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 2201 1248 0 0 298 0 0 0 25 0 5 0 607508193 156786688 10320 33554432000 4194304 4356188 140733421168016 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.44
Current children cumulated vsize (KiB) 2315020

[startup+6.30077 s]
/proc/loadavg: 8.09 8.08 8.02 9/180 5545
/proc/meminfo: memFree=24674144/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1646992 CPUtime=6.29 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 14971 0 0 0 627 2 0 0 25 0 1 0 607508184 1686519808 11331 33554432000 4194304 4587354 140737469757040 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 411748 11331 337 96 0 408310 0
[pid=5532] ppid=5530 vsize=34824 CPUtime=6.28 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 6883 0 0 0 628 0 0 0 25 0 1 0 607508184 35659776 6029 33554432000 4194304 4785740 140735102053824 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 8706 6029 340 145 0 5751 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=6.28 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 628 0 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134530573 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=27284 CPUtime=6.29 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 10515 0 0 0 628 1 0 0 25 0 1 0 607508184 27938816 3061 33554432000 4194304 4331988 140736894350928 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 6821 3061 175 34 0 5263 0
[pid=5535] ppid=5530 vsize=157368 CPUtime=24.88 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 13645 1248 0 0 2486 2 0 0 19 0 5 0 607508184 161144832 11384 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 39342 11384 163 40 0 37886 0
[pid=5535/tid=5542] ppid=5530 vsize=157368 CPUtime=6.2 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 2476 1248 0 0 620 0 0 0 25 0 5 0 607508193 161144832 11384 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331023 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=157368 CPUtime=6.2 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 2441 1248 0 0 620 0 0 0 25 0 5 0 607508193 161144832 11384 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=157368 CPUtime=6.2 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 1905 1248 0 0 620 0 0 0 25 0 5 0 607508193 161144832 11384 33554432000 4194304 4356188 140733421168016 18446744073709551615 4225511 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=157368 CPUtime=6.17 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 2201 1248 0 0 617 0 0 0 25 0 5 0 607508193 161144832 11384 33554432000 4194304 4356188 140733421168016 18446744073709551615 4279432 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.02
Current children cumulated vsize (KiB) 2347744

[startup+12.7007 s]
/proc/loadavg: 8.08 8.08 8.02 9/180 5545
/proc/meminfo: memFree=24652616/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1659440 CPUtime=12.68 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 19032 0 0 0 1266 2 0 0 25 0 1 0 607508184 1699266560 11985 33554432000 4194304 4587354 140737469757040 18446744073709551615 4388345 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 414860 11985 337 96 0 411422 0
[pid=5532] ppid=5530 vsize=38656 CPUtime=12.69 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 7824 0 0 0 1268 1 0 0 25 0 1 0 607508184 39583744 6970 33554432000 4194304 4785740 140735102053824 18446744073709551615 4278129 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 9664 6970 340 145 0 6709 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=12.68 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 1268 0 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=29236 CPUtime=12.69 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 10911 0 0 0 1268 1 0 0 25 0 1 0 607508184 29937664 3457 33554432000 4194304 4331988 140736894350928 18446744073709551615 4228235 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 7309 3457 175 34 0 5751 0
[pid=5535] ppid=5530 vsize=168632 CPUtime=50.44 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 16461 1248 0 0 5042 2 0 0 19 0 5 0 607508184 172679168 14200 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 42158 14200 163 40 0 40702 0
[pid=5535/tid=5542] ppid=5530 vsize=168632 CPUtime=12.6 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 3438 1248 0 0 1260 0 0 0 25 0 5 0 607508193 172679168 14200 33554432000 4194304 4356188 140733421168016 18446744073709551615 4210575 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=168632 CPUtime=12.6 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 2696 1248 0 0 1260 0 0 0 25 0 5 0 607508193 172679168 14200 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331002 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=168632 CPUtime=12.6 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 2032 1248 0 0 1260 0 0 0 25 0 5 0 607508193 172679168 14200 33554432000 4194304 4356188 140733421168016 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=168632 CPUtime=12.53 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 3673 1248 0 0 1253 0 0 0 25 0 5 0 607508193 172679168 14200 33554432000 4194304 4356188 140733421168016 18446744073709551615 4330909 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.18
Current children cumulated vsize (KiB) 2377240

[startup+25.5014 s]
/proc/loadavg: 8.06 8.07 8.01 9/180 5545
/proc/meminfo: memFree=24604072/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1673944 CPUtime=25.49 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 28492 0 0 0 2545 4 0 0 25 0 1 0 607508184 1714118656 19525 33554432000 4194304 4587354 140737469757040 18446744073709551615 4238582 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=5533] ppid=5530 vsize=462952 CPUtime=4781.3 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 478128 2 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134528687 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=1109196 CPUtime=4782.39 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 228417 0 0 0 478078 161 0 0 25 0 1 0 607508184 1135816704 220412 33554432000 4194304 4331988 140736894350928 18446744073709551615 4228224 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 277299 220412 175 34 0 275741 0
[pid=5535] ppid=5530 vsize=465572 CPUtime=19102.8 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 2888937 1248 0 0 1909870 408 0 0 19 0 5 0 607508184 476745728 88467 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 116393 88467 163 40 0 114937 0
[pid=5535/tid=5542] ppid=5530 vsize=465572 CPUtime=4782.31 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 754623 1248 0 0 478128 103 0 0 25 0 5 0 607508193 476745728 88467 33554432000 4194304 4356188 140733421168016 18446744073709551615 4332260 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=465572 CPUtime=4781.84 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 748817 1248 0 0 478077 107 0 0 25 0 5 0 607508193 476745728 88467 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=465572 CPUtime=4782.3 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 710442 1248 0 0 478130 100 0 0 25 0 5 0 607508193 476745728 88467 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=465572 CPUtime=4756.23 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 670433 1248 0 0 475526 97 0 0 25 0 5 0 607508193 476745728 88467 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331281 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231.2
Current children cumulated vsize (KiB) 4086752

[startup+4842.3 s]
/proc/loadavg: 8.09 8.07 8.00 9/180 5686
/proc/meminfo: memFree=23145948/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1770224 CPUtime=4842.39 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 779058 0 0 0 483857 382 0 0 25 0 1 0 607508184 1812709376 47396 33554432000 4194304 4587354 140737469757040 18446744073709551615 4237665 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 442556 47396 337 96 0 439118 0
[pid=5532] ppid=5530 vsize=234548 CPUtime=4842.38 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 53219 0 0 0 484223 15 0 0 25 0 1 0 607508184 240177152 52365 33554432000 4194304 4785740 140735102053824 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 58637 52365 340 145 0 55682 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=4841.3 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 484128 2 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134530980 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=1109196 CPUtime=4842.39 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 231215 0 0 0 484076 163 0 0 25 0 1 0 607508184 1135816704 223210 33554432000 4194304 4331988 140736894350928 18446744073709551615 215987987661 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 277299 223210 175 34 0 275741 0
[pid=5535] ppid=5530 vsize=469668 CPUtime=19342.4 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 2930921 1248 0 0 1933830 414 0 0 19 0 5 0 607508184 480940032 89491 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 117417 89491 163 40 0 115961 0
[pid=5535/tid=5542] ppid=5530 vsize=469668 CPUtime=4842.31 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 765887 1248 0 0 484127 104 0 0 25 0 5 0 607508193 480940032 89491 33554432000 4194304 4356188 140733421168016 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=469668 CPUtime=4841.83 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 760081 1248 0 0 484075 108 0 0 25 0 5 0 607508193 480940032 89491 33554432000 4194304 4356188 140733421168016 18446744073709551615 4280568 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=469668 CPUtime=4842.29 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 720682 1248 0 0 484128 101 0 0 25 0 5 0 607508193 480940032 89491 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331309 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=469668 CPUtime=4815.9 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 679649 1248 0 0 481491 99 0 0 25 0 5 0 607508193 480940032 89491 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.9
Current children cumulated vsize (KiB) 4064912

[startup+4902.3 s]
/proc/loadavg: 8.08 8.07 8.00 9/180 5687
/proc/meminfo: memFree=23134624/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1763260 CPUtime=4902.39 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 786789 0 0 0 489852 387 0 0 25 0 1 0 607508184 1805578240 49234 33554432000 4194304 4587354 140737469757040 18446744073709551615 4231420 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 440815 49234 337 96 0 437377 0
[pid=5532] ppid=5530 vsize=234976 CPUtime=4902.38 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 53318 0 0 0 490223 15 0 0 25 0 1 0 607508184 240615424 52464 33554432000 4194304 4785740 140735102053824 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 58744 52464 340 145 0 55789 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=4901.29 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 490127 2 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=1258372 CPUtime=4902.4 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 234015 0 0 0 490074 166 0 0 25 0 1 0 607508184 1288572928 226010 33554432000 4194304 4331988 140736894350928 18446744073709551615 4228224 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 314593 226010 175 34 0 313035 0
[pid=5535] ppid=5530 vsize=461476 CPUtime=19582.1 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 2966249 1248 0 0 1957793 419 0 0 19 0 5 0 607508184 472551424 87443 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 115369 87443 163 40 0 113913 0
[pid=5535/tid=5542] ppid=5530 vsize=461476 CPUtime=4902.31 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 776639 1248 0 0 490126 105 0 0 25 0 5 0 607508193 472551424 87443 33554432000 4194304 4356188 140733421168016 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=461476 CPUtime=4901.83 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 769809 1248 0 0 490073 110 0 0 25 0 5 0 607508193 472551424 87443 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331028 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=461476 CPUtime=4902.3 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 732458 1248 0 0 490127 103 0 0 25 0 5 0 607508193 472551424 87443 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=461476 CPUtime=4875.57 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 682721 1248 0 0 487458 99 0 0 25 0 5 0 607508193 472551424 87443 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.6
Current children cumulated vsize (KiB) 4199360

[startup+4962.3 s]
/proc/loadavg: 8.03 8.05 8.00 9/180 5689
/proc/meminfo: memFree=23152320/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1766488 CPUtime=4962.39 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 799449 0 0 0 495846 393 0 0 25 0 1 0 607508184 1808883712 43563 33554432000 4194304 4587354 140737469757040 18446744073709551615 4243888 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 441622 43563 337 96 0 438184 0
[pid=5532] ppid=5530 vsize=235544 CPUtime=4962.39 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 53449 0 0 0 496223 16 0 0 25 0 1 0 607508184 241197056 52595 33554432000 4194304 4785740 140735102053824 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 58886 52595 340 145 0 55931 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=4961.29 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 496127 2 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134530593 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=1258372 CPUtime=4962.4 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 236850 0 0 0 496072 168 0 0 25 0 1 0 607508184 1288572928 228845 33554432000 4194304 4331988 140736894350928 18446744073709551615 4265490 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 314593 228845 175 34 0 313035 0
[pid=5535] ppid=5530 vsize=453284 CPUtime=19821.8 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 3006185 1248 0 0 1981755 424 0 0 19 0 5 0 607508184 464162816 85395 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 113321 85395 163 40 0 111865 0
[pid=5535/tid=5542] ppid=5530 vsize=453284 CPUtime=4962.31 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 790975 1248 0 0 496124 107 0 0 25 0 5 0 607508193 464162816 85395 33554432000 4194304 4356188 140733421168016 18446744073709551615 4332264 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=453284 CPUtime=4961.81 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 782097 1248 0 0 496070 111 0 0 25 0 5 0 607508193 464162816 85395 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=453284 CPUtime=4962.29 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 742698 1248 0 0 496125 104 0 0 25 0 5 0 607508193 464162816 85395 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=453284 CPUtime=4935.24 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 685793 1248 0 0 493425 99 0 0 25 0 5 0 607508193 464162816 85395 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331323 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670.3
Current children cumulated vsize (KiB) 4194964



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.6 s]
/proc/loadavg: 8.01 8.04 8.00 9/180 5690
/proc/meminfo: memFree=23126508/32950940 swapFree=67111364/67111528
[pid=5530] ppid=5528 vsize=18324 CPUtime=0 cores=0-7
/proc/5530/stat : 5530 (ppfolio) S 5528 5530 5127 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 607508184 18763776 390 33554432000 4194304 4287625 140737432242240 18446744073709551615 215988612623 0 0 4096 65536 17250747693551058943 0 0 17 6 0 0 0
/proc/5530/statm: 4581 390 341 23 0 91 0
[pid=5531] ppid=5530 vsize=1773288 CPUtime=5003.7 cores=0
/proc/5531/stat : 5531 (cryptominisat) R 5530 5530 5127 0 -1 4202496 805437 0 0 0 499973 397 0 0 25 0 1 0 607508184 1815846912 43563 33554432000 4194304 4587354 140737469757040 18446744073709551615 4243808 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/5531/statm: 443322 43563 337 96 0 439884 0
[pid=5532] ppid=5530 vsize=237608 CPUtime=5003.69 cores=1
/proc/5532/stat : 5532 (clasp) R 5530 5530 5127 0 -1 4202496 53941 0 0 0 500353 16 0 0 25 0 1 0 607508184 243310592 53087 33554432000 4194304 4785740 140735102053824 18446744073709551615 4287024 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/5532/statm: 59402 53087 340 145 0 56447 0
[pid=5533] ppid=5530 vsize=462952 CPUtime=5002.59 cores=2
/proc/5533/stat : 5533 (TNM) R 5530 5530 5127 0 -1 4202496 3271 0 0 0 500257 2 0 0 25 0 1 0 607508184 474062848 3168 33554432000 134512640 134534564 4288057200 18446744073709551615 134530268 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/5533/statm: 115738 3168 95 6 0 114404 0
[pid=5534] ppid=5530 vsize=1258372 CPUtime=5003.7 cores=3
/proc/5534/stat : 5534 (march_hi) R 5530 5530 5127 0 -1 4202496 239319 0 0 0 500200 170 0 0 25 0 1 0 607508184 1288572928 231314 33554432000 4194304 4331988 140736894350928 18446744073709551615 4228235 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/5534/statm: 314593 231314 175 34 0 313035 0
[pid=5535] ppid=5530 vsize=467620 CPUtime=19986.8 cores=4-7
/proc/5535/stat : 5535 (plingeling) S 5530 5530 5127 0 -1 4202496 3033321 1248 0 0 1998250 428 0 0 19 0 5 0 607508184 478842880 88979 33554432000 4194304 4356188 140733421168016 18446744073709551615 216000395237 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/5535/statm: 116905 88979 163 40 0 115449 0
[pid=5535/tid=5542] ppid=5530 vsize=467620 CPUtime=5003.62 cores=4-7
/proc/5535/task/5542/stat : 5542 (plingeling) R 5530 5530 5127 0 -1 4202560 793023 1248 0 0 500255 107 0 0 25 0 5 0 607508193 478842880 88979 33554432000 4194304 4356188 140733421168016 18446744073709551615 4279769 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5535/tid=5543] ppid=5530 vsize=467620 CPUtime=5003.12 cores=4-7
/proc/5535/task/5543/stat : 5543 (plingeling) R 5530 5530 5127 0 -1 4202560 788753 1248 0 0 500200 112 0 0 25 0 5 0 607508193 478842880 88979 33554432000 4194304 4356188 140733421168016 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5535/tid=5544] ppid=5530 vsize=467620 CPUtime=5003.61 cores=4-7
/proc/5535/task/5544/stat : 5544 (plingeling) R 5530 5530 5127 0 -1 4202560 753962 1248 0 0 500255 106 0 0 25 0 5 0 607508193 478842880 88979 33554432000 4194304 4356188 140733421168016 18446744073709551615 4332007 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5535/tid=5545] ppid=5530 vsize=467620 CPUtime=4976.33 cores=4-7
/proc/5535/task/5545/stat : 5545 (plingeling) R 5530 5530 5127 0 -1 4202560 692961 1248 0 0 497532 101 0 0 25 0 5 0 607508193 478842880 88979 33554432000 4194304 4356188 140733421168016 18446744073709551615 4331992 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 4218164

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5530 and gives
#  childrusage.ru_utime.tv_sec=10002
#  childrusage.ru_utime.tv_usec=312417
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=18389
# CPU time returned by wait4() is 10006.3
# while last known CPU time is 40000.5
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5003.62
CPU time (s): 40000.5
CPU user time (s): 39990.3
CPU system time (s): 10.13
CPU usage (%): 799.431
Max. virtual memory (cumulated for all children) (KiB): 5859252

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 10002.3
system time used= 4.01839
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 809232
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 609
involuntary context switches= 75257

runsolver used 8.23875 second user time and 19.0581 second system time

The end

Launcher Data 

Begin job on node117 at 2011-04-30 16:12:54
IDJOB=3276001
IDBENCH=84560
IDSOLVER=1638
FILE ID=node117/3276001-1304172774
RUNJOBID= node117-1304169570-5141
PBS_JOBID= 13170959
Free space on /tmp= 72628 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT04/handmade/kexu/frb-0.8/frb59-26-2.used-as.sat04-892.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3276001-1304172774/watcher-3276001-1304172774 -o /tmp/evaluation-result-3276001-1304172774/solver-3276001-1304172774 -C 40000 -W 5100 -M 31000  ppfolio HOME/instance-3276001-1304172774.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= fd8d5783ed404e5c6590d627ffdcca50
RANDOM SEED=2107040477

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      24803340 kB
Buffers:       1895400 kB
Cached:        5620432 kB
SwapCached:        160 kB
Active:         906772 kB
Inactive:      6672844 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      24803340 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           11908 kB
Writeback:           0 kB
AnonPages:       64256 kB
Mapped:          14384 kB
Slab:           500868 kB
PageTables:       4916 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS:   187636 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72620 MiB
End job on node117 at 2011-04-30 17:36:20