Trace number 3220119

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39972.8 5000.02

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v90-c8010-S818753762-096.UNKNOWN.cnf
MD5SUMffd67922a8bb7c4660bea122f8a5ce0e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables90
Number of clauses8010
Sum of the clauses size56070
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58010

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3220119-1304156023/watcher-3220119-1304156023 -o /tmp/evaluation-result-3220119-1304156023/solver-3220119-1304156023 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3220119-1304156023.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 8.01 8.00 2/170 818
/proc/meminfo: memFree=30094292/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18240 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) R 816 818 328 0 -1 4194304 283 0 0 0 0 0 0 0 18 0 1 0 527199425 18677760 231 33554432000 4194304 4287625 140733592934992 18446744073709551615 239471732200 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/818/statm: 4560 232 196 23 0 57 0

[startup+0.0366231 s]
/proc/loadavg: 8.04 8.01 8.00 2/170 818
/proc/meminfo: memFree=30094292/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 21 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100609 s]
/proc/loadavg: 8.04 8.01 8.00 2/170 818
/proc/meminfo: memFree=30094292/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 21 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300571 s]
/proc/loadavg: 8.04 8.01 8.00 2/170 818
/proc/meminfo: memFree=30094292/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 20 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700496 s]
/proc/loadavg: 8.04 8.01 8.00 2/170 818
/proc/meminfo: memFree=30094292/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50134 s]
/proc/loadavg: 8.04 8.01 8.00 9/180 842
/proc/meminfo: memFree=30075124/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=27416 CPUtime=1.49 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 3300 0 0 0 149 0 0 0 20 0 1 0 527199425 28073984 1261 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 6854 1261 330 96 0 3416 0
[pid=820] ppid=818 vsize=15328 CPUtime=1.49 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 1287 0 0 0 149 0 0 0 20 0 1 0 527199425 15695872 1179 33554432000 4194304 4785740 140734966656960 18446744073709551615 4287053 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 3832 1179 341 145 0 877 0
[pid=821] ppid=818 vsize=456768 CPUtime=1.49 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 149 0 0 0 20 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134528497 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=8800 CPUtime=1.49 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 1833 0 0 0 149 0 0 0 20 0 1 0 527199425 9011200 610 33554432000 4194304 4331988 140736810022432 18446744073709551615 4299264 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 2200 610 174 34 0 642 0
[pid=823] ppid=818 vsize=127732 CPUtime=5.91 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 4622 1251 0 0 591 0 0 0 18 0 5 0 527199425 130797568 4006 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 31933 4006 162 40 0 30477 0
[pid=823/tid=830] ppid=818 vsize=127732 CPUtime=1.46 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 862 1251 0 0 146 0 0 0 25 0 5 0 527199427 130797568 4006 33554432000 4194304 4356188 140735952138640 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=127732 CPUtime=1.47 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 873 1251 0 0 147 0 0 0 20 0 5 0 527199427 130797568 4006 33554432000 4194304 4356188 140735952138640 18446744073709551615 4332041 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=127732 CPUtime=1.47 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 880 1251 0 0 147 0 0 0 20 0 5 0 527199427 130797568 4006 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331822 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=127732 CPUtime=1.47 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 819 1251 0 0 147 0 0 0 23 0 5 0 527199427 130797568 4006 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331170 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.87
Current children cumulated vsize (KiB) 654368

[startup+3.10104 s]
/proc/loadavg: 8.04 8.01 8.00 9/180 842
/proc/meminfo: memFree=30060148/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=1405840 CPUtime=3.08 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 5961 0 0 0 308 0 0 0 25 0 1 0 527199425 1439580160 3380 33554432000 4194304 4587354 140733285526464 18446744073709551615 239476391580 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 351460 3380 336 96 0 348022 0
[pid=820] ppid=818 vsize=16260 CPUtime=3.09 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 1502 0 0 0 309 0 0 0 25 0 1 0 527199425 16650240 1394 33554432000 4194304 4785740 140734966656960 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 4065 1394 341 145 0 1110 0
[pid=821] ppid=818 vsize=456768 CPUtime=3.09 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 309 0 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=9256 CPUtime=3.09 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 1966 0 0 0 309 0 0 0 25 0 1 0 527199425 9478144 743 33554432000 4194304 4331988 140736810022432 18446744073709551615 4226242 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 2314 743 175 34 0 756 0
[pid=823] ppid=818 vsize=127436 CPUtime=12.31 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 5204 1251 0 0 1230 1 0 0 18 0 5 0 527199425 130494464 3932 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 31859 3932 162 40 0 30403 0
[pid=823/tid=830] ppid=818 vsize=127436 CPUtime=3.06 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 991 1251 0 0 306 0 0 0 25 0 5 0 527199427 130494464 3932 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331264 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=127436 CPUtime=3.07 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 1035 1251 0 0 307 0 0 0 25 0 5 0 527199427 130494464 3932 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=127436 CPUtime=3.07 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 1042 1251 0 0 307 0 0 0 25 0 5 0 527199427 130494464 3932 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331141 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=127436 CPUtime=3.07 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 948 1251 0 0 307 0 0 0 25 0 5 0 527199427 130494464 3932 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.66
Current children cumulated vsize (KiB) 2033884

[startup+6.30043 s]
/proc/loadavg: 8.04 8.01 8.00 9/180 842
/proc/meminfo: memFree=30056884/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=1406960 CPUtime=6.29 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 7995 0 0 0 628 1 0 0 25 0 1 0 527199425 1440727040 3589 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 351740 3589 336 96 0 348302 0
[pid=820] ppid=818 vsize=17292 CPUtime=6.29 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 1750 0 0 0 629 0 0 0 25 0 1 0 527199425 17707008 1642 33554432000 4194304 4785740 140734966656960 18446744073709551615 239476392566 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 4323 1642 341 145 0 1368 0
[pid=821] ppid=818 vsize=456768 CPUtime=6.29 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 629 0 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=9824 CPUtime=6.29 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 2066 0 0 0 629 0 0 0 25 0 1 0 527199425 10059776 843 33554432000 4194304 4331988 140736810022432 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 2456 843 175 34 0 898 0
[pid=823] ppid=818 vsize=128860 CPUtime=25.09 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 6616 1251 0 0 2508 1 0 0 18 0 5 0 527199425 131952640 4288 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 32215 4288 162 40 0 30759 0
[pid=823/tid=830] ppid=818 vsize=128860 CPUtime=6.24 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 1377 1251 0 0 624 0 0 0 25 0 5 0 527199427 131952640 4288 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=128860 CPUtime=6.27 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 1356 1251 0 0 627 0 0 0 25 0 5 0 527199427 131952640 4288 33554432000 4194304 4356188 140735952138640 18446744073709551615 4238482 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=128860 CPUtime=6.27 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 1362 1251 0 0 627 0 0 0 25 0 5 0 527199427 131952640 4288 33554432000 4194304 4356188 140735952138640 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=128860 CPUtime=6.27 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 1333 1251 0 0 627 0 0 0 25 0 5 0 527199427 131952640 4288 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331072 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.25
Current children cumulated vsize (KiB) 2038028

[startup+12.7012 s]
/proc/loadavg: 8.03 8.01 8.00 9/180 842
/proc/meminfo: memFree=30043940/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=1415404 CPUtime=12.69 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 13271 0 0 0 1267 2 0 0 25 0 1 0 527199425 1449373696 5807 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 353851 5807 336 96 0 350413 0
[pid=820] ppid=818 vsize=19228 CPUtime=12.69 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 2188 0 0 0 1269 0 0 0 25 0 1 0 527199425 19689472 2080 33554432000 4194304 4785740 140734966656960 18446744073709551615 4287055 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 4807 2080 341 145 0 1852 0
[pid=821] ppid=818 vsize=456768 CPUtime=12.69 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 1269 0 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134530549 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=10336 CPUtime=12.69 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 2264 0 0 0 1269 0 0 0 25 0 1 0 527199425 10584064 1041 33554432000 4194304 4331988 140736810022432 18446744073709551615 4287500 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 2584 1041 175 34 0 1026 0
[pid=823] ppid=818 vsize=130524 CPUtime=50.65 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 9080 1251 0 0 5064 1 0 0 18 0 5 0 527199425 133656576 4704 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 32631 4704 162 40 0 31175 0
[pid=823/tid=830] ppid=818 vsize=130524 CPUtime=12.61 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 1953 1251 0 0 1261 0 0 0 25 0 5 0 527199427 133656576 4704 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=130524 CPUtime=12.67 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 1996 1251 0 0 1267 0 0 0 25 0 5 0 527199427 133656576 4704 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=130524 CPUtime=12.67 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 1938 1251 0 0 1267 0 0 0 25 0 5 0 527199427 133656576 4704 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=130524 CPUtime=12.67 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 2005 1251 0 0 1267 0 0 0 25 0 5 0 527199427 133656576 4704 33554432000 4194304 4356188 140735952138640 18446744073709551615 4225618 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.41
Current children cumulated vsize (KiB) 2050584

[startup+25.5007 s]
/proc/loadavg: 8.02 8.01 8.00 9/180 843
/proc/meminfo: memFree=30029232/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=45552 CPUtime=25.49 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 19290 0 0 0 2547 2 0 0 25 0 1 0 527199425 46645248 5745 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=821] ppid=818 vsize=456768 CPUtime=4782.47 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 478246 1 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=697560 CPUtime=4782.47 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 125890 0 0 0 478150 97 0 0 25 0 1 0 527199425 714301440 124667 33554432000 4194304 4331988 140736810022432 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 174390 124667 175 34 0 172832 0
[pid=823] ppid=818 vsize=312620 CPUtime=19102.4 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 472064 1251 0 0 1910170 73 0 0 18 0 5 0 527199425 320122880 50228 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 78155 50228 162 40 0 76699 0
[pid=823/tid=830] ppid=818 vsize=312620 CPUtime=4755.6 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 119389 1251 0 0 475544 16 0 0 25 0 5 0 527199427 320122880 50228 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=312620 CPUtime=4782.45 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 115720 1251 0 0 478226 19 0 0 25 0 5 0 527199427 320122880 50228 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=312620 CPUtime=4782.32 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 120734 1251 0 0 478216 16 0 0 25 0 5 0 527199427 320122880 50228 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=312620 CPUtime=4782.01 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 115033 1251 0 0 478181 20 0 0 25 0 5 0 527199427 320122880 50228 33554432000 4194304 4356188 140735952138640 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38232.3
Current children cumulated vsize (KiB) 1680076

[startup+4842.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/180 993
/proc/meminfo: memFree=29222696/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=129552 CPUtime=4842.47 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 315930 0 0 0 484118 129 0 0 25 0 1 0 527199425 132661248 28723 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 32388 28723 336 96 0 28950 0
[pid=820] ppid=818 vsize=65828 CPUtime=4842.45 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 14735 0 0 0 484243 2 0 0 25 0 1 0 527199425 67407872 12129 33554432000 4194304 4785740 140734966656960 18446744073709551615 4286964 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 16457 12129 341 145 0 13502 0
[pid=821] ppid=818 vsize=456768 CPUtime=4842.47 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 484246 1 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134528842 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=697560 CPUtime=4842.47 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 127269 0 0 0 484149 98 0 0 25 0 1 0 527199425 714301440 126046 33554432000 4194304 4331988 140736810022432 18446744073709551615 4237398 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 174390 126046 175 34 0 172832 0
[pid=823] ppid=818 vsize=314668 CPUtime=19342.1 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 474432 1251 0 0 1934136 74 0 0 18 0 5 0 527199425 322220032 50740 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 78667 50740 162 40 0 77211 0
[pid=823/tid=830] ppid=818 vsize=314668 CPUtime=4815.27 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 119709 1251 0 0 481511 16 0 0 25 0 5 0 527199427 322220032 50740 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331309 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=314668 CPUtime=4842.45 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 115976 1251 0 0 484226 19 0 0 25 0 5 0 527199427 322220032 50740 33554432000 4194304 4356188 140735952138640 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=314668 CPUtime=4842.32 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 121566 1251 0 0 484216 16 0 0 25 0 5 0 527199427 322220032 50740 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331309 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=314668 CPUtime=4842.01 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 115993 1251 0 0 484181 20 0 0 25 0 5 0 527199427 322220032 50740 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38712
Current children cumulated vsize (KiB) 1682700

[startup+4902.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/180 995
/proc/meminfo: memFree=29211956/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=205380 CPUtime=4902.48 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 316791 0 0 0 490118 130 0 0 25 0 1 0 527199425 210309120 29563 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 51345 29563 336 96 0 47907 0
[pid=820] ppid=818 vsize=65828 CPUtime=4902.45 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 14735 0 0 0 490243 2 0 0 25 0 1 0 527199425 67407872 12129 33554432000 4194304 4785740 140734966656960 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 16457 12129 341 145 0 13502 0
[pid=821] ppid=818 vsize=456768 CPUtime=4902.47 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 490246 1 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134528483 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=697560 CPUtime=4902.47 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 128494 0 0 0 490148 99 0 0 25 0 1 0 527199425 714301440 127271 33554432000 4194304 4331988 140736810022432 18446744073709551615 4210430 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 174390 127271 175 34 0 172832 0
[pid=823] ppid=818 vsize=317228 CPUtime=19581.8 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 479040 1251 0 0 1958103 74 0 0 18 0 5 0 527199425 324841472 51380 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 79307 51380 162 40 0 77851 0
[pid=823/tid=830] ppid=818 vsize=317228 CPUtime=4874.95 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 120669 1251 0 0 487479 16 0 0 25 0 5 0 527199427 324841472 51380 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=317228 CPUtime=4902.45 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 116936 1251 0 0 490226 19 0 0 25 0 5 0 527199427 324841472 51380 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=317228 CPUtime=4902.33 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 123486 1251 0 0 490216 17 0 0 25 0 5 0 527199427 324841472 51380 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=317228 CPUtime=4902 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 116761 1251 0 0 490180 20 0 0 25 0 5 0 527199427 324841472 51380 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331199 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39191.6
Current children cumulated vsize (KiB) 1761088

[startup+4962.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/180 996
/proc/meminfo: memFree=29213892/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=129552 CPUtime=4962.48 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 317685 0 0 0 496118 130 0 0 25 0 1 0 527199425 132661248 28723 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238578 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 32388 28723 336 96 0 28950 0
[pid=820] ppid=818 vsize=65828 CPUtime=4962.45 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 14748 0 0 0 496243 2 0 0 25 0 1 0 527199425 67407872 12142 33554432000 4194304 4785740 140734966656960 18446744073709551615 4287050 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 16457 12142 341 145 0 13502 0
[pid=821] ppid=818 vsize=456768 CPUtime=4962.47 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 496246 1 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=697560 CPUtime=4962.47 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 130135 0 0 0 496147 100 0 0 25 0 1 0 527199425 714301440 128912 33554432000 4194304 4331988 140736810022432 18446744073709551615 4299275 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 174390 128912 175 34 0 172832 0
[pid=823] ppid=818 vsize=311340 CPUtime=19821.4 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 481088 1251 0 0 1982070 75 0 0 18 0 5 0 527199425 318812160 49908 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 77835 49908 162 40 0 76379 0
[pid=823/tid=830] ppid=818 vsize=311340 CPUtime=4934.62 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 120669 1251 0 0 493446 16 0 0 25 0 5 0 527199427 318812160 49908 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=311340 CPUtime=4962.46 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 117704 1251 0 0 496226 20 0 0 25 0 5 0 527199427 318812160 49908 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=311340 CPUtime=4962.33 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 123806 1251 0 0 496216 17 0 0 25 0 5 0 527199427 318812160 49908 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=311340 CPUtime=4962 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 117721 1251 0 0 496180 20 0 0 25 0 5 0 527199427 318812160 49908 33554432000 4194304 4356188 140735952138640 18446744073709551615 4284134 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39671.3
Current children cumulated vsize (KiB) 1679372



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.09 8.06 8.00 9/180 997
/proc/meminfo: memFree=29194124/32950928 swapFree=67086612/67111528
[pid=818] ppid=816 vsize=18324 CPUtime=0 cores=0-7
/proc/818/stat : 818 (ppfolio) S 816 818 328 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 527199425 18763776 389 33554432000 4194304 4287625 140733592934992 18446744073709551615 239476715023 0 0 4096 65536 4581893518217183231 0 0 17 6 0 0 0
/proc/818/statm: 4581 389 341 23 0 91 0
[pid=819] ppid=818 vsize=205588 CPUtime=5000.18 cores=0
/proc/819/stat : 819 (cryptominisat) R 818 818 328 0 -1 4202496 318677 0 0 0 499888 130 0 0 25 0 1 0 527199425 210522112 29682 33554432000 4194304 4587354 140733285526464 18446744073709551615 4238752 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/819/statm: 51397 29682 336 96 0 47959 0
[pid=820] ppid=818 vsize=65828 CPUtime=5000.16 cores=1
/proc/820/stat : 820 (clasp) R 818 818 328 0 -1 4202496 14748 0 0 0 500014 2 0 0 25 0 1 0 527199425 67407872 12142 33554432000 4194304 4785740 140734966656960 18446744073709551615 4287140 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/820/statm: 16457 12142 341 145 0 13502 0
[pid=821] ppid=818 vsize=456768 CPUtime=5000.18 cores=2
/proc/821/stat : 821 (TNM) R 818 818 328 0 -1 4202496 596 0 0 0 500017 1 0 0 25 0 1 0 527199425 467730432 493 33554432000 134512640 134534564 4288119824 18446744073709551615 134530522 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/821/statm: 114192 493 95 6 0 112858 0
[pid=822] ppid=818 vsize=697560 CPUtime=5000.18 cores=3
/proc/822/stat : 822 (march_hi) R 818 818 328 0 -1 4202496 131141 0 0 0 499917 101 0 0 25 0 1 0 527199425 714301440 129918 33554432000 4194304 4331988 140736810022432 18446744073709551615 4228099 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/822/statm: 174390 129918 175 34 0 172832 0
[pid=823] ppid=818 vsize=324908 CPUtime=19972.1 cores=4-7
/proc/823/stat : 823 (plingeling) S 818 818 328 0 -1 4202496 484480 1251 0 0 1997130 75 0 0 18 0 5 0 527199425 332705792 53300 33554432000 4194304 4356188 140735952138640 18446744073709551615 239488497637 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/823/statm: 81227 53300 162 40 0 79771 0
[pid=823/tid=830] ppid=818 vsize=324908 CPUtime=4972.12 cores=4-7
/proc/823/task/830/stat : 830 (plingeling) R 818 818 328 0 -1 4202560 121629 1251 0 0 497196 16 0 0 25 0 5 0 527199427 332705792 53300 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=823/tid=840] ppid=818 vsize=324908 CPUtime=5000.17 cores=4-7
/proc/823/task/840/stat : 840 (plingeling) R 818 818 328 0 -1 4202560 118472 1251 0 0 499997 20 0 0 25 0 5 0 527199427 332705792 53300 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=823/tid=841] ppid=818 vsize=324908 CPUtime=5000.03 cores=4-7
/proc/823/task/841/stat : 841 (plingeling) R 818 818 328 0 -1 4202560 124702 1251 0 0 499986 17 0 0 25 0 5 0 527199427 332705792 53300 33554432000 4194304 4356188 140735952138640 18446744073709551615 4332275 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=823/tid=842] ppid=818 vsize=324908 CPUtime=4999.7 cores=4-7
/proc/823/task/842/stat : 842 (plingeling) R 818 818 328 0 -1 4202560 118489 1251 0 0 499950 20 0 0 25 0 5 0 527199427 332705792 53300 33554432000 4194304 4356188 140735952138640 18446744073709551615 4331323 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39972.8
Current children cumulated vsize (KiB) 1768976

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 818 and gives
#  childrusage.ru_utime.tv_sec=14999
#  childrusage.ru_utime.tv_usec=206773
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=364792
# CPU time returned by wait4() is 15000.6
# while last known CPU time is 39972.8
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39972.8
CPU user time (s): 39969.7
CPU system time (s): 3.09
CPU usage (%): 799.452
Max. virtual memory (cumulated for all children) (KiB): 14164072

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14999.2
system time used= 1.36479
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 334556
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 844
involuntary context switches= 37199

runsolver used 8.04778 second user time and 19.1861 second system time

The end

Launcher Data 

Begin job on node148 at 2011-04-30 11:33:44
IDJOB=3220119
IDBENCH=82703
IDSOLVER=1638
FILE ID=node148/3220119-1304156023
RUNJOBID= node148-1304145444-342
PBS_JOBID= 13170884
Free space on /tmp= 72964 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k7-r89-v90-c8010-S818753762-096.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3220119-1304156023/watcher-3220119-1304156023 -o /tmp/evaluation-result-3220119-1304156023/solver-3220119-1304156023 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3220119-1304156023.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ffd67922a8bb7c4660bea122f8a5ce0e
RANDOM SEED=1563097787

node148.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30094448 kB
Buffers:        762080 kB
Cached:        1681056 kB
SwapCached:       3696 kB
Active:         157108 kB
Inactive:      2328032 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30094448 kB
SwapTotal:    67111528 kB
SwapFree:     67086612 kB
Dirty:           10096 kB
Writeback:           0 kB
AnonPages:       40352 kB
Mapped:          13748 kB
Slab:           304952 kB
PageTables:       5000 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   186388 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72952 MiB
End job on node148 at 2011-04-30 12:57:06