Trace number 3310337

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 40000.3 5003.52

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
32/aes_32_4_keyfind_1.cnf
MD5SUMd71ab4cbe872e5620884d4e4d370890a
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables912
Number of clauses3488
Sum of the clauses size11040
Maximum clause length5
Minimum clause length1
Number of clauses of size 180
Number of clauses of size 2160
Number of clauses of size 32432
Number of clauses of size 4736
Number of clauses of size 580
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3310337-1304013988/watcher-3310337-1304013988 -o /tmp/evaluation-result-3310337-1304013988/solver-3310337-1304013988 -C 40000 -W 5100 -M 31000 ppfolio HOME/instance-3310337-1304013988.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.44 6.91 6.96 2/170 11232
/proc/meminfo: memFree=17986616/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0

[startup+0.0224879 s]
/proc/loadavg: 6.44 6.91 6.96 2/170 11232
/proc/meminfo: memFree=17986616/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100439 s]
/proc/loadavg: 6.44 6.91 6.96 2/170 11232
/proc/meminfo: memFree=17986616/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) R 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300404 s]
/proc/loadavg: 6.44 6.91 6.96 2/170 11232
/proc/meminfo: memFree=17986616/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 24 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700334 s]
/proc/loadavg: 6.44 6.91 6.96 2/170 11232
/proc/meminfo: memFree=17986616/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 20 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50119 s]
/proc/loadavg: 6.44 6.91 6.96 9/180 11247
/proc/meminfo: memFree=17969548/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 17 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1372896 CPUtime=1.49 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 2043 0 0 0 149 0 0 0 20 0 1 0 513001437 1405845504 1094 33554432000 4194304 4587354 140733281996320 18446744073709551615 4238736 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 343224 1094 332 96 0 339786 0
[pid=11234] ppid=11232 vsize=14092 CPUtime=1.49 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 989 0 0 0 149 0 0 0 25 0 1 0 513001437 14430208 872 33554432000 4194304 4785740 140735375414144 18446744073709551615 4287229 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 3523 872 340 145 0 568 0
[pid=11235] ppid=11232 vsize=455988 CPUtime=1.49 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 387 0 0 0 149 0 0 0 25 0 1 0 513001437 466931712 283 33554432000 134512640 134534564 4286744864 18446744073709551615 134530309 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113997 283 95 6 0 112663 0
[pid=11236] ppid=11232 vsize=9696 CPUtime=1.49 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 1043 0 0 0 149 0 0 0 25 0 1 0 513001437 9928704 745 33554432000 4194304 4331988 140734194153344 18446744073709551615 4243196 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 2424 745 174 34 0 866 0
[pid=11237] ppid=11232 vsize=122424 CPUtime=5.93 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 3471 1259 0 0 593 0 0 0 18 0 5 0 513001437 125362176 2717 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 30606 2717 164 40 0 29150 0
[pid=11237/tid=11244] ppid=11232 vsize=122424 CPUtime=1.48 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 670 1259 0 0 148 0 0 0 20 0 5 0 513001438 125362176 2717 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331383 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=122424 CPUtime=1.47 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 687 1259 0 0 147 0 0 0 25 0 5 0 513001438 125362176 2717 33554432000 4194304 4356188 140734699494416 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=122424 CPUtime=1.48 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 870 1259 0 0 148 0 0 0 25 0 5 0 513001438 125362176 2717 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=122424 CPUtime=1.48 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 693 1259 0 0 148 0 0 0 20 0 5 0 513001438 125362176 2717 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331389 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.89
Current children cumulated vsize (KiB) 1993420

[startup+3.10091 s]
/proc/loadavg: 6.44 6.91 6.96 9/180 11247
/proc/meminfo: memFree=17960868/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1377964 CPUtime=3.09 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 4461 0 0 0 309 0 0 0 25 0 1 0 513001437 1411035136 2630 33554432000 4194304 4587354 140733281996320 18446744073709551615 4238668 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 344491 2630 338 96 0 341053 0
[pid=11234] ppid=11232 vsize=14772 CPUtime=3.09 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 1158 0 0 0 309 0 0 0 25 0 1 0 513001437 15126528 1041 33554432000 4194304 4785740 140735375414144 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 3693 1041 340 145 0 738 0
[pid=11235] ppid=11232 vsize=455988 CPUtime=3.08 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 387 0 0 0 308 0 0 0 25 0 1 0 513001437 466931712 283 33554432000 134512640 134534564 4286744864 18446744073709551615 134528913 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113997 283 95 6 0 112663 0
[pid=11236] ppid=11232 vsize=9868 CPUtime=3.09 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 1068 0 0 0 309 0 0 0 25 0 1 0 513001437 10104832 770 33554432000 4194304 4331988 140734194153344 18446744073709551615 4210960 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 2467 770 174 34 0 909 0
[pid=11237] ppid=11232 vsize=123788 CPUtime=12.32 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 4260 1259 0 0 1232 0 0 0 18 0 5 0 513001437 126758912 3058 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 30947 3058 164 40 0 29491 0
[pid=11237/tid=11244] ppid=11232 vsize=123788 CPUtime=3.08 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 800 1259 0 0 308 0 0 0 25 0 5 0 513001438 126758912 3058 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=123788 CPUtime=3.06 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 879 1259 0 0 306 0 0 0 25 0 5 0 513001438 126758912 3058 33554432000 4194304 4356188 140734699494416 18446744073709551615 4306228 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=123788 CPUtime=3.08 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 1079 1259 0 0 308 0 0 0 25 0 5 0 513001438 126758912 3058 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=123788 CPUtime=3.08 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 951 1259 0 0 308 0 0 0 25 0 5 0 513001438 126758912 3058 33554432000 4194304 4356188 140734699494416 18446744073709551615 4210842 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.67
Current children cumulated vsize (KiB) 2000704

[startup+6.30035 s]
/proc/loadavg: 6.56 6.93 6.97 9/180 11247
/proc/meminfo: memFree=17953172/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1382772 CPUtime=6.29 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 6998 0 0 0 629 0 0 0 25 0 1 0 513001437 1415958528 3487 33554432000 4194304 4587354 140733281996320 18446744073709551615 4238616 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 345693 3487 338 96 0 342255 0
[pid=11234] ppid=11232 vsize=15180 CPUtime=6.29 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 1269 0 0 0 629 0 0 0 25 0 1 0 513001437 15544320 1152 33554432000 4194304 4785740 140735375414144 18446744073709551615 4553396 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 3795 1152 340 145 0 840 0
[pid=11235] ppid=11232 vsize=455988 CPUtime=6.28 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 387 0 0 0 628 0 0 0 25 0 1 0 513001437 466931712 283 33554432000 134512640 134534564 4286744864 18446744073709551615 134530573 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113997 283 95 6 0 112663 0
[pid=11236] ppid=11232 vsize=9868 CPUtime=6.29 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 1093 0 0 0 629 0 0 0 25 0 1 0 513001437 10104832 795 33554432000 4194304 4331988 140734194153344 18446744073709551615 4210960 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 2467 795 174 34 0 909 0
[pid=11237] ppid=11232 vsize=124960 CPUtime=25.11 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 6569 1259 0 0 2510 1 0 0 18 0 5 0 513001437 127959040 3351 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 31240 3351 164 40 0 29784 0
[pid=11237/tid=11244] ppid=11232 vsize=124960 CPUtime=6.28 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 1344 1259 0 0 628 0 0 0 25 0 5 0 513001438 127959040 3351 33554432000 4194304 4356188 140734699494416 18446744073709551615 4280608 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=124960 CPUtime=6.24 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 1648 1259 0 0 624 0 0 0 25 0 5 0 513001438 127959040 3351 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=124960 CPUtime=6.28 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 1497 1259 0 0 628 0 0 0 25 0 5 0 513001438 127959040 3351 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=124960 CPUtime=6.28 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 1529 1259 0 0 628 0 0 0 25 0 5 0 513001438 127959040 3351 33554432000 4194304 4356188 140734699494416 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.26
Current children cumulated vsize (KiB) 2007092

[startup+12.7012 s]
/proc/loadavg: 6.68 6.94 6.97 9/180 11247
/proc/meminfo: memFree=17947516/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1383116 CPUtime=12.69 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 9835 0 0 0 1268 1 0 0 25 0 1 0 513001437 1416310784 3482 33554432000 4194304 4587354 140733281996320 18446744073709551615 4238597 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 345779 3482 338 96 0 342341 0
[pid=11234] ppid=11232 vsize=15900 CPUtime=12.69 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 1416 0 0 0 1269 0 0 0 25 0 1 0 513001437 16281600 1299 33554432000 4194304 4785740 140735375414144 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 3975 1299 340 145 0 1020 0
[pid=11235] ppid=11232 vsize=455988 CPUtime=12.68 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 387 0 0 0 1268 0 0 0 25 0 1 0 513001437 466931712 283 33554432000 134512640 134534564 4286744864 18446744073709551615 134530494 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113997 283 95 6 0 112663 0
[pid=11236] ppid=11232 vsize=10324 CPUtime=12.69 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 1215 0 0 0 1269 0 0 0 25 0 1 0 513001437 10571776 917 33554432000 4194304 4331988 140734194153344 18446744073709551615 4284385 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 2581 917 175 34 0 1023 0
[pid=11237] ppid=11232 vsize=128948 CPUtime=50.68 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 9966 1259 0 0 5067 1 0 0 18 0 5 0 513001437 132042752 4348 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 32237 4348 164 40 0 30781 0
[pid=11237/tid=11244] ppid=11232 vsize=128948 CPUtime=12.68 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 2370 1259 0 0 1268 0 0 0 25 0 5 0 513001438 132042752 4348 33554432000 4194304 4356188 140734699494416 18446744073709551615 4279461 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=128948 CPUtime=12.61 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 2289 1259 0 0 1261 0 0 0 25 0 5 0 513001438 132042752 4348 33554432000 4194304 4356188 140734699494416 18446744073709551615 4279427 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=128948 CPUtime=12.67 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 2362 1259 0 0 1267 0 0 0 25 0 5 0 513001438 132042752 4348 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331281 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=128948 CPUtime=12.68 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 2394 1259 0 0 1268 0 0 0 25 0 5 0 513001438 132042752 4348 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331305 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.43
Current children cumulated vsize (KiB) 2012600

[startup+25.501 s]
/proc/loadavg: 7.04 7.01 6.99 9/180 11248
/proc/meminfo: memFree=17939536/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1386360 CPUtime=25.49 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 15019 0 0 0 2547 2 0 0 25 0 1 0 513001437 1419632640 4106 33554432000 4194304 4587354 140733281996320 18446744073709551615 4243446 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=11235] ppid=11232 vsize=455992 CPUtime=4841.78 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 396 0 0 0 484176 2 0 0 25 0 1 0 513001437 466935808 292 33554432000 134512640 134534564 4286744864 18446744073709551615 134514044 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113998 292 103 6 0 112664 0
[pid=11236] ppid=11232 vsize=257852 CPUtime=4842.4 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 53581 0 0 0 484200 40 0 0 25 0 1 0 513001437 264040448 53283 33554432000 4194304 4331988 140734194153344 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 64463 53283 175 34 0 62905 0
[pid=11237] ppid=11232 vsize=263644 CPUtime=19342.7 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 651200 1259 0 0 1934162 105 0 0 18 0 5 0 513001437 269971456 38022 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 65911 38022 164 40 0 64455 0
[pid=11237/tid=11244] ppid=11232 vsize=263644 CPUtime=4842.39 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 142632 1259 0 0 484219 20 0 0 25 0 5 0 513001438 269971456 38022 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=263644 CPUtime=4816.11 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 176438 1259 0 0 481582 29 0 0 25 0 5 0 513001438 269971456 38022 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331203 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=263644 CPUtime=4842.39 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 159262 1259 0 0 484211 28 0 0 25 0 5 0 513001438 269971456 38022 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331076 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=263644 CPUtime=4841.74 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 172317 1259 0 0 484147 27 0 0 25 0 5 0 513001438 269971456 38022 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331028 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38711.5
Current children cumulated vsize (KiB) 2595164

[startup+4902.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/180 11390
/proc/meminfo: memFree=17404272/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1559448 CPUtime=4902.26 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 427689 0 0 0 490059 167 0 0 25 0 1 0 513001437 1596874752 48469 33554432000 4194304 4587354 140733281996320 18446744073709551615 4239225 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 389862 48469 338 96 0 386424 0
[pid=11234] ppid=11232 vsize=37892 CPUtime=4902.4 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 6418 0 0 0 490239 1 0 0 25 0 1 0 513001437 38801408 6301 33554432000 4194304 4785740 140735375414144 18446744073709551615 4549066 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 9473 6301 341 145 0 6518 0
[pid=11235] ppid=11232 vsize=455992 CPUtime=4901.78 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 396 0 0 0 490176 2 0 0 25 0 1 0 513001437 466935808 292 33554432000 134512640 134534564 4286744864 18446744073709551615 134529887 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113998 292 103 6 0 112664 0
[pid=11236] ppid=11232 vsize=257852 CPUtime=4902.4 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 54205 0 0 0 490199 41 0 0 25 0 1 0 513001437 264040448 53907 33554432000 4194304 4331988 140734194153344 18446744073709551615 4287057 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 64463 53907 175 34 0 62905 0
[pid=11237] ppid=11232 vsize=254428 CPUtime=19582.3 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 655136 1259 0 0 1958129 106 0 0 18 0 5 0 513001437 260534272 35718 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 63607 35718 164 40 0 62151 0
[pid=11237/tid=11244] ppid=11232 vsize=254428 CPUtime=4902.39 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 143080 1259 0 0 490219 20 0 0 25 0 5 0 513001438 260534272 35718 33554432000 4194304 4356188 140734699494416 18446744073709551615 4332260 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=254428 CPUtime=4875.79 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 176438 1259 0 0 487550 29 0 0 25 0 5 0 513001438 260534272 35718 33554432000 4194304 4356188 140734699494416 18446744073709551615 4330995 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=254428 CPUtime=4902.39 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 160062 1259 0 0 490211 28 0 0 25 0 5 0 513001438 260534272 35718 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=254428 CPUtime=4901.75 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 175005 1259 0 0 490147 28 0 0 25 0 5 0 513001438 260534272 35718 33554432000 4194304 4356188 140734699494416 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39191.2
Current children cumulated vsize (KiB) 2583936

[startup+4962.3 s]
/proc/loadavg: 8.00 8.01 8.00 9/180 11391
/proc/meminfo: memFree=17389976/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1559448 CPUtime=4962.26 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 428681 0 0 0 496059 167 0 0 25 0 1 0 513001437 1596874752 49429 33554432000 4194304 4587354 140733281996320 18446744073709551615 4238770 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 389862 49429 338 96 0 386424 0
[pid=11234] ppid=11232 vsize=37892 CPUtime=4962.4 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 6435 0 0 0 496239 1 0 0 25 0 1 0 513001437 38801408 6318 33554432000 4194304 4785740 140735375414144 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 9473 6318 341 145 0 6518 0
[pid=11235] ppid=11232 vsize=455992 CPUtime=4961.77 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 396 0 0 0 496175 2 0 0 25 0 1 0 513001437 466935808 292 33554432000 134512640 134534564 4286744864 18446744073709551615 134530619 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113998 292 103 6 0 112664 0
[pid=11236] ppid=11232 vsize=257852 CPUtime=4962.41 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 54869 0 0 0 496199 42 0 0 25 0 1 0 513001437 264040448 54571 33554432000 4194304 4331988 140734194153344 18446744073709551615 4242612 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 64463 54571 175 34 0 62905 0
[pid=11237] ppid=11232 vsize=270684 CPUtime=19822 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 664160 1259 0 0 1982096 107 0 0 18 0 5 0 513001437 277180416 39782 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 67671 39782 164 40 0 66215 0
[pid=11237/tid=11244] ppid=11232 vsize=270684 CPUtime=4962.4 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 144008 1259 0 0 496220 20 0 0 25 0 5 0 513001438 277180416 39782 33554432000 4194304 4356188 140734699494416 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=270684 CPUtime=4935.46 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 180630 1259 0 0 493517 29 0 0 25 0 5 0 513001438 277180416 39782 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=270684 CPUtime=4962.39 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 161982 1259 0 0 496211 28 0 0 25 0 5 0 513001438 277180416 39782 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=270684 CPUtime=4961.74 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 176989 1259 0 0 496146 28 0 0 25 0 5 0 513001438 277180416 39782 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39670.9
Current children cumulated vsize (KiB) 2600192



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.5 s]
/proc/loadavg: 8.00 8.01 8.00 9/180 11392
/proc/meminfo: memFree=17404844/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1559456 CPUtime=5003.47 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 429221 0 0 0 500179 168 0 0 25 0 1 0 513001437 1596882944 48511 33554432000 4194304 4587354 140733281996320 18446744073709551615 4238418 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 389864 48511 338 96 0 386426 0
[pid=11234] ppid=11232 vsize=37892 CPUtime=5003.61 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 6435 0 0 0 500360 1 0 0 25 0 1 0 513001437 38801408 6318 33554432000 4194304 4785740 140735375414144 18446744073709551615 4287008 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 9473 6318 341 145 0 6518 0
[pid=11235] ppid=11232 vsize=455992 CPUtime=5002.98 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 396 0 0 0 500296 2 0 0 25 0 1 0 513001437 466935808 292 33554432000 134512640 134534564 4286744864 18446744073709551615 134529632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113998 292 103 6 0 112664 0
[pid=11236] ppid=11232 vsize=257852 CPUtime=5003.61 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 55296 0 0 0 500319 42 0 0 25 0 1 0 513001437 264040448 54998 33554432000 4194304 4331988 140734194153344 18446744073709551615 4299840 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 64463 54998 175 34 0 62905 0
[pid=11237] ppid=11232 vsize=256476 CPUtime=19986.6 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 664896 1259 0 0 1998556 107 0 0 18 0 5 0 513001437 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 64119 36230 164 40 0 62663 0
[pid=11237/tid=11244] ppid=11232 vsize=256476 CPUtime=5003.6 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 144040 1259 0 0 500340 20 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4332260 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=256476 CPUtime=4976.44 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 181078 1259 0 0 497615 29 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331023 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=256476 CPUtime=5003.6 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 161982 1259 0 0 500332 28 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=256476 CPUtime=5002.95 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 177245 1259 0 0 500267 28 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 2585992

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5003.5 s]
/proc/loadavg: 8.00 8.01 8.00 9/180 11392
/proc/meminfo: memFree=17404844/32951124 swapFree=67111528/67111528
[pid=11232] ppid=11230 vsize=18324 CPUtime=0 cores=0-7
/proc/11232/stat : 11232 (ppfolio) S 11230 11232 11004 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 513001437 18763776 390 33554432000 4194304 4287625 140733977593152 18446744073709551615 216634535439 0 0 4096 65536 4426185354415964159 0 0 17 4 0 0 0
/proc/11232/statm: 4581 390 341 23 0 91 0
[pid=11233] ppid=11232 vsize=1559456 CPUtime=5003.47 cores=0
/proc/11233/stat : 11233 (cryptominisat) R 11232 11232 11004 0 -1 4202496 429221 0 0 0 500179 168 0 0 25 0 1 0 513001437 1596882944 48511 33554432000 4194304 4587354 140733281996320 18446744073709551615 4238418 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11233/statm: 389864 48511 338 96 0 386426 0
[pid=11234] ppid=11232 vsize=37892 CPUtime=5003.61 cores=1
/proc/11234/stat : 11234 (clasp) R 11232 11232 11004 0 -1 4202496 6435 0 0 0 500360 1 0 0 25 0 1 0 513001437 38801408 6318 33554432000 4194304 4785740 140735375414144 18446744073709551615 4287008 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11234/statm: 9473 6318 341 145 0 6518 0
[pid=11235] ppid=11232 vsize=455992 CPUtime=5002.98 cores=2
/proc/11235/stat : 11235 (TNM) R 11232 11232 11004 0 -1 4202496 396 0 0 0 500296 2 0 0 25 0 1 0 513001437 466935808 292 33554432000 134512640 134534564 4286744864 18446744073709551615 134529632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11235/statm: 113998 292 103 6 0 112664 0
[pid=11236] ppid=11232 vsize=257852 CPUtime=5003.61 cores=3
/proc/11236/stat : 11236 (march_hi) R 11232 11232 11004 0 -1 4202496 55296 0 0 0 500319 42 0 0 25 0 1 0 513001437 264040448 54998 33554432000 4194304 4331988 140734194153344 18446744073709551615 4299840 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11236/statm: 64463 54998 175 34 0 62905 0
[pid=11237] ppid=11232 vsize=256476 CPUtime=19986.6 cores=4-7
/proc/11237/stat : 11237 (plingeling) S 11232 11232 11004 0 -1 4202496 664896 1259 0 0 1998556 107 0 0 18 0 5 0 513001437 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/11237/statm: 64119 36230 164 40 0 62663 0
[pid=11237/tid=11244] ppid=11232 vsize=256476 CPUtime=5003.6 cores=4-7
/proc/11237/task/11244/stat : 11244 (plingeling) R 11232 11232 11004 0 -1 4202560 144040 1259 0 0 500340 20 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4332260 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11237/tid=11245] ppid=11232 vsize=256476 CPUtime=4976.44 cores=4-7
/proc/11237/task/11245/stat : 11245 (plingeling) R 11232 11232 11004 0 -1 4202560 181078 1259 0 0 497615 29 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331023 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11237/tid=11246] ppid=11232 vsize=256476 CPUtime=5003.6 cores=4-7
/proc/11237/task/11246/stat : 11246 (plingeling) R 11232 11232 11004 0 -1 4202560 161982 1259 0 0 500332 28 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11237/tid=11247] ppid=11232 vsize=256476 CPUtime=5002.95 cores=4-7
/proc/11237/task/11247/stat : 11247 (plingeling) R 11232 11232 11004 0 -1 4202560 177245 1259 0 0 500267 28 0 0 25 0 5 0 513001438 262631424 36230 33554432000 4194304 4356188 140734699494416 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 2585992

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 11232 and gives
#  childrusage.ru_utime.tv_sec=10006
#  childrusage.ru_utime.tv_usec=568770
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=41993
# CPU time returned by wait4() is 10006.6
# while last known CPU time is 40000.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5003.52
CPU time (s): 40000.3
CPU user time (s): 39997.1
CPU system time (s): 3.2
CPU usage (%): 799.444
Max. virtual memory (cumulated for all children) (KiB): 3947236

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 10006.6
system time used= 0.041993
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 7370
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 551
involuntary context switches= 81432

runsolver used 8.10077 second user time and 19.0621 second system time

The end

Launcher Data 

Begin job on node103 at 2011-04-28 20:06:28
IDJOB=3310337
IDBENCH=83431
IDSOLVER=1638
FILE ID=node103/3310337-1304013988
RUNJOBID= node103-1304008886-11018
PBS_JOBID= 13168844
Free space on /tmp= 71520 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/32/aes_32_4_keyfind_1.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3310337-1304013988/watcher-3310337-1304013988 -o /tmp/evaluation-result-3310337-1304013988/solver-3310337-1304013988 -C 40000 -W 5100 -M 31000  ppfolio HOME/instance-3310337-1304013988.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= d71ab4cbe872e5620884d4e4d370890a
RANDOM SEED=805971443

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17986772 kB
Buffers:       2143436 kB
Cached:       12172808 kB
SwapCached:          0 kB
Active:        5497324 kB
Inactive:      8882400 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17986772 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           10124 kB
Writeback:           0 kB
AnonPages:       63108 kB
Mapped:          14672 kB
Slab:           517596 kB
PageTables:       4284 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183520 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71508 MiB
End job on node103 at 2011-04-28 21:29:54