Trace number 3219939

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39965.3 5000.02

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v15000-c63000-S594553190-015-UNKNOWN.cnf
MD5SUM0a98d6d9a7bdb797305aed69a94f9a69
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark451.768
Satisfiable
(Un)Satisfiability was proved
Number of variables15000
Number of clauses63000
Sum of the clauses size189000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 363000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219939-1304155132/watcher-3219939-1304155132 -o /tmp/evaluation-result-3219939-1304155132/solver-3219939-1304155132 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219939-1304155132.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.06 8.06 8.00 2/168 19064
/proc/meminfo: memFree=30930328/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) R 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0

[startup+0.045406 s]
/proc/loadavg: 8.06 8.06 8.00 2/168 19064
/proc/meminfo: memFree=30930328/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100395 s]
/proc/loadavg: 8.06 8.06 8.00 2/168 19064
/proc/meminfo: memFree=30930328/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300328 s]
/proc/loadavg: 8.06 8.06 8.00 2/168 19064
/proc/meminfo: memFree=30930328/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 23 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.70026 s]
/proc/loadavg: 8.06 8.06 8.00 2/168 19064
/proc/meminfo: memFree=30930328/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 20 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50112 s]
/proc/loadavg: 8.06 8.06 8.00 9/178 19079
/proc/meminfo: memFree=30840720/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=38604 CPUtime=1.49 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 6831 0 0 0 148 1 0 0 20 0 1 0 527111900 39530496 5348 33554432000 4194304 4587354 140735254763632 18446744073709551615 4237686 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 9651 5348 326 96 0 6213 0
[pid=19066] ppid=19064 vsize=25280 CPUtime=1.49 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 4472 0 0 0 149 0 0 0 20 0 1 0 527111900 25886720 3647 33554432000 4194304 4785740 140735416026000 18446744073709551615 242970211013 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 6320 3647 340 145 0 3365 0
[pid=19067] ppid=19064 vsize=457964 CPUtime=1.49 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2680 0 0 0 149 0 0 0 20 0 1 0 527111900 468955136 2577 33554432000 134512640 134534564 4286689312 18446744073709551615 134528864 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/19067/statm: 114491 2577 95 6 0 113157 0
[pid=19068] ppid=19064 vsize=39296 CPUtime=1.49 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 21186 0 0 0 145 4 0 0 20 0 1 0 527111900 40239104 7583 33554432000 4194304 4331988 140735898847280 18446744073709551615 4237226 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 9824 7583 173 34 0 8266 0
[pid=19069] ppid=19064 vsize=134840 CPUtime=5.42 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 9872 1259 0 0 541 1 0 0 18 0 5 0 527111900 138076160 5800 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 33710 5800 164 40 0 32254 0
[pid=19069/tid=19076] ppid=19064 vsize=134840 CPUtime=1.32 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 1126 1259 0 0 132 0 0 0 25 0 5 0 527111917 138076160 5800 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=134840 CPUtime=1.32 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 1080 1259 0 0 132 0 0 0 19 0 5 0 527111917 138076160 5800 33554432000 4194304 4356188 140736156637552 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=134840 CPUtime=1.32 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 1068 1259 0 0 132 0 0 0 19 0 5 0 527111917 138076160 5800 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=134840 CPUtime=1.29 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 1062 1259 0 0 129 0 0 0 25 0 5 0 527111917 138076160 5800 33554432000 4194304 4356188 140736156637552 18446744073709551615 4332333 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.38
Current children cumulated vsize (KiB) 714308

[startup+3.10085 s]
/proc/loadavg: 8.06 8.06 8.00 9/178 19079
/proc/meminfo: memFree=30828072/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=76108 CPUtime=3.08 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 8109 0 0 0 307 1 0 0 25 0 1 0 527111900 77934592 6614 33554432000 4194304 4587354 140735254763632 18446744073709551615 4238392 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 19027 6614 326 96 0 15589 0
[pid=19066] ppid=19064 vsize=34128 CPUtime=3.09 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 6668 0 0 0 308 1 0 0 25 0 1 0 527111900 34947072 5843 33554432000 4194304 4785740 140735416026000 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 8532 5843 340 145 0 5577 0
[pid=19067] ppid=19064 vsize=457964 CPUtime=3.09 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2680 0 0 0 309 0 0 0 25 0 1 0 527111900 468955136 2577 33554432000 134512640 134534564 4286689312 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/19067/statm: 114491 2577 95 6 0 113157 0
[pid=19068] ppid=19064 vsize=40372 CPUtime=3.09 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 21474 0 0 0 305 4 0 0 25 0 1 0 527111900 41340928 7871 33554432000 4194304 4331988 140735898847280 18446744073709551615 4212616 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 10093 7871 173 34 0 8535 0
[pid=19069] ppid=19064 vsize=135496 CPUtime=11.82 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 10044 1259 0 0 1181 1 0 0 18 0 5 0 527111900 138747904 5972 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 33874 5972 164 40 0 32418 0
[pid=19069/tid=19076] ppid=19064 vsize=135496 CPUtime=2.92 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 1214 1259 0 0 292 0 0 0 25 0 5 0 527111917 138747904 5972 33554432000 4194304 4356188 140736156637552 18446744073709551615 4332288 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=135496 CPUtime=2.92 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 1144 1259 0 0 292 0 0 0 25 0 5 0 527111917 138747904 5972 33554432000 4194304 4356188 140736156637552 18446744073709551615 4234244 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=135496 CPUtime=2.92 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 1074 1259 0 0 292 0 0 0 24 0 5 0 527111917 138747904 5972 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=135496 CPUtime=2.88 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 1076 1259 0 0 288 0 0 0 25 0 5 0 527111917 138747904 5972 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331019 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.17
Current children cumulated vsize (KiB) 762392

[startup+6.3013 s]
/proc/loadavg: 8.05 8.05 8.00 9/178 19079
/proc/meminfo: memFree=30808728/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=43284 CPUtime=6.29 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 11667 0 0 0 627 2 0 0 25 0 1 0 527111900 44322816 6040 33554432000 4194304 4587354 140735254763632 18446744073709551615 4238736 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 10821 6040 331 96 0 7383 0
[pid=19066] ppid=19064 vsize=46128 CPUtime=6.29 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 9678 0 0 0 628 1 0 0 25 0 1 0 527111900 47235072 8853 33554432000 4194304 4785740 140735416026000 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 11532 8853 340 145 0 8577 0
[pid=19067] ppid=19064 vsize=457964 CPUtime=6.28 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2680 0 0 0 628 0 0 0 25 0 1 0 527111900 468955136 2577 33554432000 134512640 134534564 4286689312 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/19067/statm: 114491 2577 95 6 0 113157 0
[pid=19068] ppid=19064 vsize=41032 CPUtime=6.29 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 21621 0 0 0 625 4 0 0 25 0 1 0 527111900 42016768 8018 33554432000 4194304 4331988 140735898847280 18446744073709551615 4237130 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 10258 8018 173 34 0 8700 0
[pid=19069] ppid=19064 vsize=140264 CPUtime=23.41 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 13276 1259 0 0 2333 8 0 0 18 0 5 0 527111900 143630336 7029 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 35066 7029 164 40 0 33610 0
[pid=19069/tid=19076] ppid=19064 vsize=140264 CPUtime=6.11 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 1950 1259 0 0 611 0 0 0 25 0 5 0 527111917 143630336 7029 33554432000 4194304 4356188 140736156637552 18446744073709551615 4279685 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=140264 CPUtime=6.06 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 1940 1259 0 0 606 0 0 0 25 0 5 0 527111917 143630336 7029 33554432000 4194304 4356188 140736156637552 18446744073709551615 4283885 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=140264 CPUtime=5 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 1842 1259 0 0 494 6 0 0 25 0 5 0 527111917 143630336 7029 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=140264 CPUtime=6.06 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 2008 1259 0 0 606 0 0 0 25 0 5 0 527111917 143630336 7029 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330823 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 48.56
Current children cumulated vsize (KiB) 746996

[startup+12.7012 s]
/proc/loadavg: 8.21 8.09 8.01 9/178 19079
/proc/meminfo: memFree=30796664/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=44640 CPUtime=12.68 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 32269 0 0 0 1263 5 0 0 25 0 1 0 527111900 45711360 5707 33554432000 4194304 4587354 140735254763632 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 11160 5707 337 96 0 7722 0
[pid=19066] ppid=19064 vsize=52900 CPUtime=12.69 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 11368 0 0 0 1267 2 0 0 25 0 1 0 527111900 54169600 10543 33554432000 4194304 4785740 140735416026000 18446744073709551615 4416823 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 13225 10543 340 145 0 10270 0
[pid=19067] ppid=19064 vsize=457964 CPUtime=12.68 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2680 0 0 0 1268 0 0 0 25 0 1 0 527111900 468955136 2577 33554432000 134512640 134534564 4286689312 18446744073709551615 8717814 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/19067/statm: 114491 2577 95 6 0 113157 0
[pid=19068] ppid=19064 vsize=41692 CPUtime=12.69 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 21778 0 0 0 1265 4 0 0 25 0 1 0 527111900 42692608 8175 33554432000 4194304 4331988 140735898847280 18446744073709551615 4286573 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 10423 8175 173 34 0 8865 0
[pid=19069] ppid=19064 vsize=146048 CPUtime=47.45 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 15666 1259 0 0 4730 15 0 0 18 0 5 0 527111900 149553152 8625 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 36512 8625 164 40 0 35056 0
[pid=19069/tid=19076] ppid=19064 vsize=146048 CPUtime=12.49 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 2908 1259 0 0 1247 2 0 0 25 0 5 0 527111917 149553152 8625 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=146048 CPUtime=11.38 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 2452 1259 0 0 1133 5 0 0 25 0 5 0 527111917 149553152 8625 33554432000 4194304 4356188 140736156637552 18446744073709551615 4332309 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=146048 CPUtime=10.98 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 1945 1259 0 0 1092 6 0 0 25 0 5 0 527111917 149553152 8625 33554432000 4194304 4356188 140736156637552 18446744073709551615 4273850 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=146048 CPUtime=12.42 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 2825 1259 0 0 1242 0 0 0 25 0 5 0 527111917 149553152 8625 33554432000 4194304 4356188 140736156637552 18446744073709551615 4234244 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 98.19
Current children cumulated vsize (KiB) 761568

[startup+25.501 s]
/proc/loadavg: 8.23 8.10 8.01 9/178 19079
/proc/meminfo: memFree=30754852/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=164064 CPUtime=25.49 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 61256 0 0 0 2538 11 0 0 25 0 1 0 527111900 168001536 11786 33554432000 4194304 4587354 140735254763632 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=19067] ppid=19064 vsize=457968 CPUtime=4779.89 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2689 0 1 0 477879 110 0 0 25 0 1 0 527111900 468959232 2587 33554432000 134512640 134534564 4286689312 18446744073709551615 134530379 0 0 4096 0 0 0 0 17 2 0 0 2
/proc/19067/statm: 114492 2587 103 6 0 113158 0
[pid=19068] ppid=19064 vsize=47368 CPUtime=4782.25 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 23208 0 0 0 478220 5 0 0 25 0 1 0 527111900 48504832 9605 33554432000 4194304 4331988 140735898847280 18446744073709551615 4211525 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 11842 9605 173 34 0 10284 0
[pid=19069] ppid=19064 vsize=218000 CPUtime=19098.1 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 196920 1259 0 0 1909766 46 0 0 18 0 5 0 527111900 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 54500 26937 164 40 0 53044 0
[pid=19069/tid=19076] ppid=19064 vsize=218000 CPUtime=4782.16 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 40947 1259 0 0 478208 8 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=218000 CPUtime=4781.06 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 46031 1259 0 0 478094 12 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4211045 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=218000 CPUtime=4778.65 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 50551 1259 0 0 477850 15 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=218000 CPUtime=4756.04 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 53855 1259 0 0 475596 8 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330982 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38225
Current children cumulated vsize (KiB) 1414088

[startup+4842.3 s]
/proc/loadavg: 8.10 8.05 8.00 9/178 19220
/proc/meminfo: memFree=30231644/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=173676 CPUtime=4842.38 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 2258593 0 0 0 483319 919 0 0 25 0 1 0 527111900 177844224 18077 33554432000 4194304 4587354 140735254763632 18446744073709551615 4238392 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 43419 18077 337 96 0 39981 0
[pid=19066] ppid=19064 vsize=478768 CPUtime=4842.34 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 115700 0 0 0 484186 48 0 0 25 0 1 0 527111900 490258432 114875 33554432000 4194304 4785740 140735416026000 18446744073709551615 4327247 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 119692 114875 340 145 0 116737 0
[pid=19067] ppid=19064 vsize=457968 CPUtime=4839.88 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2689 0 1 0 483877 111 0 0 25 0 1 0 527111900 468959232 2587 33554432000 134512640 134534564 4286689312 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 2
/proc/19067/statm: 114492 2587 103 6 0 113158 0
[pid=19068] ppid=19064 vsize=47368 CPUtime=4842.25 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 23211 0 0 0 484220 5 0 0 25 0 1 0 527111900 48504832 9608 33554432000 4194304 4331988 140735898847280 18446744073709551615 4265456 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 11842 9608 173 34 0 10284 0
[pid=19069] ppid=19064 vsize=230288 CPUtime=19337.8 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 201016 1259 0 0 1933730 46 0 0 18 0 5 0 527111900 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 57572 30009 164 40 0 56116 0
[pid=19069/tid=19076] ppid=19064 vsize=230288 CPUtime=4842.17 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 42995 1259 0 0 484208 9 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=230288 CPUtime=4841.06 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 46031 1259 0 0 484094 12 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4274287 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=230288 CPUtime=4838.63 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 50551 1259 0 0 483848 15 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330817 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=230288 CPUtime=4815.71 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 55903 1259 0 0 481563 8 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4334609 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38704.6
Current children cumulated vsize (KiB) 1406392

[startup+4902.3 s]
/proc/loadavg: 8.10 8.06 8.00 9/178 19221
/proc/meminfo: memFree=30171488/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=173676 CPUtime=4902.38 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 2275994 0 0 0 489310 928 0 0 25 0 1 0 527111900 177844224 35399 33554432000 4194304 4587354 140735254763632 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 43419 35399 337 96 0 39981 0
[pid=19066] ppid=19064 vsize=478768 CPUtime=4902.34 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 115700 0 0 0 490186 48 0 0 25 0 1 0 527111900 490258432 114875 33554432000 4194304 4785740 140735416026000 18446744073709551615 4287204 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 119692 114875 340 145 0 116737 0
[pid=19067] ppid=19064 vsize=457968 CPUtime=4899.87 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2689 0 1 0 489874 113 0 0 25 0 1 0 527111900 468959232 2587 33554432000 134512640 134534564 4286689312 18446744073709551615 134530268 0 0 4096 0 0 0 0 17 2 0 0 2
/proc/19067/statm: 114492 2587 103 6 0 113158 0
[pid=19068] ppid=19064 vsize=47368 CPUtime=4902.25 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 23215 0 0 0 490220 5 0 0 25 0 1 0 527111900 48504832 9612 33554432000 4194304 4331988 140735898847280 18446744073709551615 4240760 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 11842 9612 173 34 0 10284 0
[pid=19069] ppid=19064 vsize=230288 CPUtime=19577.4 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 206136 1259 0 0 1957695 47 0 0 18 0 5 0 527111900 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 57572 30009 164 40 0 56116 0
[pid=19069/tid=19076] ppid=19064 vsize=230288 CPUtime=4902.17 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 45043 1259 0 0 490208 9 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=230288 CPUtime=4901.06 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 47055 1259 0 0 490094 12 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=230288 CPUtime=4898.6 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 52599 1259 0 0 489845 15 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4334477 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=230288 CPUtime=4875.38 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 55903 1259 0 0 487530 8 0 0 25 0 5 0 527111917 235814912 30009 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39184.3
Current children cumulated vsize (KiB) 1406392

[startup+4962.3 s]
/proc/loadavg: 8.09 8.06 8.00 9/178 19223
/proc/meminfo: memFree=30196764/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=208868 CPUtime=4962.39 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 2297495 0 0 0 495298 941 0 0 25 0 1 0 527111900 213880832 28738 33554432000 4194304 4587354 140735254763632 18446744073709551615 4243943 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 52217 28738 337 96 0 48779 0
[pid=19066] ppid=19064 vsize=478768 CPUtime=4962.34 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 115701 0 0 0 496186 48 0 0 25 0 1 0 527111900 490258432 114876 33554432000 4194304 4785740 140735416026000 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 119692 114876 340 145 0 116737 0
[pid=19067] ppid=19064 vsize=457968 CPUtime=4959.86 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2689 0 1 0 495873 113 0 0 25 0 1 0 527111900 468959232 2587 33554432000 134512640 134534564 4286689312 18446744073709551615 134528934 0 0 4096 0 0 0 0 17 2 0 0 2
/proc/19067/statm: 114492 2587 103 6 0 113158 0
[pid=19068] ppid=19064 vsize=47368 CPUtime=4962.25 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 23217 0 0 0 496220 5 0 0 25 0 1 0 527111900 48504832 9614 33554432000 4194304 4331988 140735898847280 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 11842 9614 173 34 0 10284 0
[pid=19069] ppid=19064 vsize=222096 CPUtime=19817.1 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 210232 1259 0 0 1981659 48 0 0 18 0 5 0 527111900 227426304 27961 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 55524 27961 164 40 0 54068 0
[pid=19069/tid=19076] ppid=19064 vsize=222096 CPUtime=4962.17 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 45043 1259 0 0 496208 9 0 0 25 0 5 0 527111917 227426304 27961 33554432000 4194304 4356188 140736156637552 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=222096 CPUtime=4961.07 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 49103 1259 0 0 496094 13 0 0 25 0 5 0 527111917 227426304 27961 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=222096 CPUtime=4958.58 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 53623 1259 0 0 495843 15 0 0 25 0 5 0 527111917 227426304 27961 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=222096 CPUtime=4935.05 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 56927 1259 0 0 493497 8 0 0 25 0 5 0 527111917 227426304 27961 33554432000 4194304 4356188 140736156637552 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39663.9
Current children cumulated vsize (KiB) 1433392



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.05 8.05 8.00 9/178 19224
/proc/meminfo: memFree=30266556/32951132 swapFree=67111364/67111528
[pid=19064] ppid=19062 vsize=18324 CPUtime=0 cores=0-7
/proc/19064/stat : 19064 (ppfolio) S 19062 19064 18651 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527111900 18763776 390 33554432000 4194304 4287625 140735941591376 18446744073709551615 242970570255 0 0 4096 65536 1740536806669025279 0 0 17 6 0 0 0
/proc/19064/statm: 4581 390 341 23 0 91 0
[pid=19065] ppid=19064 vsize=187132 CPUtime=5000.09 cores=0
/proc/19065/stat : 19065 (cryptominisat) R 19064 19064 18651 0 -1 4202496 2309706 0 0 0 499063 946 0 0 25 0 1 0 527111900 191623168 12244 33554432000 4194304 4587354 140735254763632 18446744073709551615 4238658 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/19065/statm: 46783 12244 337 96 0 43345 0
[pid=19066] ppid=19064 vsize=479408 CPUtime=5000.05 cores=1
/proc/19066/stat : 19066 (clasp) R 19064 19064 18651 0 -1 4202496 115815 0 0 0 499957 48 0 0 25 0 1 0 527111900 490913792 114990 33554432000 4194304 4785740 140735416026000 18446744073709551615 4552955 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/19066/statm: 119852 114990 340 145 0 116897 0
[pid=19067] ppid=19064 vsize=457968 CPUtime=4997.55 cores=2
/proc/19067/stat : 19067 (TNM) R 19064 19064 18651 0 -1 4202496 2689 0 1 0 499640 115 0 0 25 0 1 0 527111900 468959232 2587 33554432000 134512640 134534564 4286689312 18446744073709551615 134530151 0 0 4096 0 0 0 0 17 2 0 0 2
/proc/19067/statm: 114492 2587 103 6 0 113158 0
[pid=19068] ppid=19064 vsize=47368 CPUtime=4999.96 cores=3
/proc/19068/stat : 19068 (march_hi) R 19064 19064 18651 0 -1 4202496 23220 0 0 0 499991 5 0 0 25 0 1 0 527111900 48504832 9617 33554432000 4194304 4331988 140735898847280 18446744073709551615 4240514 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/19068/statm: 11842 9617 173 34 0 10284 0
[pid=19069] ppid=19064 vsize=218000 CPUtime=19967.7 cores=4-7
/proc/19069/stat : 19069 (plingeling) S 19064 19064 18651 0 -1 4202496 212280 1259 0 0 1996720 48 0 0 18 0 5 0 527111900 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 242982352869 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/19069/statm: 54500 26937 164 40 0 53044 0
[pid=19069/tid=19076] ppid=19064 vsize=218000 CPUtime=4999.88 cores=4-7
/proc/19069/task/19076/stat : 19076 (plingeling) R 19064 19064 18651 0 -1 4202560 45043 1259 0 0 499979 9 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=19069/tid=19077] ppid=19064 vsize=218000 CPUtime=4998.77 cores=4-7
/proc/19069/task/19077/stat : 19077 (plingeling) R 19064 19064 18651 0 -1 4202560 49103 1259 0 0 499864 13 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4279738 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=19069/tid=19078] ppid=19064 vsize=218000 CPUtime=4996.28 cores=4-7
/proc/19069/task/19078/stat : 19078 (plingeling) R 19064 19064 18651 0 -1 4202560 53623 1259 0 0 499613 15 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4210830 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=19069/tid=19079] ppid=19064 vsize=218000 CPUtime=4972.55 cores=4-7
/proc/19069/task/19079/stat : 19079 (plingeling) R 19064 19064 18651 0 -1 4202560 58975 1259 0 0 497246 9 0 0 25 0 5 0 527111917 223232000 26937 33554432000 4194304 4356188 140736156637552 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39965.3
Current children cumulated vsize (KiB) 1408200

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 19064 and gives
#  childrusage.ru_utime.tv_sec=14986
#  childrusage.ru_utime.tv_usec=958635
#  childrusage.ru_stime.tv_sec=10
#  childrusage.ru_stime.tv_usec=681376
# CPU time returned by wait4() is 14997.6
# while last known CPU time is 39965.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39965.3
CPU user time (s): 39953.7
CPU system time (s): 11.62
CPU usage (%): 799.304
Max. virtual memory (cumulated for all children) (KiB): 1568556

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14987
system time used= 10.6814
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2336138
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 627
involuntary context switches= 37478

runsolver used 8.09977 second user time and 19.1691 second system time

The end

Launcher Data 

Begin job on node128 at 2011-04-30 11:18:52
IDJOB=3219939
IDBENCH=82673
IDSOLVER=1638
FILE ID=node128/3219939-1304155132
RUNJOBID= node128-1304145123-18665
PBS_JOBID= 13170867
Free space on /tmp= 67532 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v15000-c63000-S594553190-015-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219939-1304155132/watcher-3219939-1304155132 -o /tmp/evaluation-result-3219939-1304155132/solver-3219939-1304155132 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219939-1304155132.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 0a98d6d9a7bdb797305aed69a94f9a69
RANDOM SEED=745227086

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30930732 kB
Buffers:        451200 kB
Cached:        1087676 kB
SwapCached:          0 kB
Active:         181164 kB
Inactive:      1421836 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30930732 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           11284 kB
Writeback:           0 kB
AnonPages:       65048 kB
Mapped:          14996 kB
Slab:           352464 kB
PageTables:       4732 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   188544 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 67520 MiB
End job on node128 at 2011-04-30 12:42:15