Trace number 3372401

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.9 5009.53

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.070-NOTKNOWN.cnf
MD5SUMf52ec4dacdcf65c3aedb61fb46ade01b
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1007.46
Satisfiable
(Un)Satisfiability was proved
Number of variables153284
Number of clauses2473656
Sum of the clauses size5647736
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 22387280
Number of clauses of size 36720
Number of clauses of size 431080
Number of clauses of size 513440
Number of clauses of size over 534440

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372401-1305437454/watcher-3372401-1305437454 -o /tmp/evaluation-result-3372401-1305437454/solver-3372401-1305437454 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372401-1305437454.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.11 6.47 5.23 1/277 14938
/proc/meminfo: memFree=21423764/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=11968 CPUtime=0 cores=0-7
/proc/14938/stat : 14938 (plingeling) R 14936 14938 14543 0 -1 4202496 1077 1251 0 0 0 0 0 0 22 0 1 0 655343749 12255232 1006 33554432000 4194304 4417452 140737183236400 18446744073709551615 232335586280 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/14938/statm: 3040 1017 144 55 0 926 0

[startup+0.014937 s]
/proc/loadavg: 7.11 6.47 5.23 1/277 14938
/proc/meminfo: memFree=21423764/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=13436 CPUtime=0 cores=0-7
/proc/14938/stat : 14938 (plingeling) R 14936 14938 14543 0 -1 4202496 1450 1251 0 0 0 0 0 0 22 0 1 0 655343749 13758464 1379 33554432000 4194304 4417452 140737183236400 18446744073709551615 232335892224 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/14938/statm: 3359 1379 144 55 0 1245 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 13436

[startup+0.100942 s]
/proc/loadavg: 7.11 6.47 5.23 1/277 14938
/proc/meminfo: memFree=21423764/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=23280 CPUtime=0.09 cores=0-7
/proc/14938/stat : 14938 (plingeling) R 14936 14938 14543 0 -1 4202496 3888 1251 0 0 9 0 0 0 22 0 1 0 655343749 23838720 3813 33554432000 4194304 4417452 140737183236400 18446744073709551615 4227801 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/14938/statm: 5820 3813 149 55 0 3706 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 23280

[startup+0.300908 s]
/proc/loadavg: 7.11 6.47 5.23 1/277 14938
/proc/meminfo: memFree=21423764/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=39652 CPUtime=0.28 cores=0-7
/proc/14938/stat : 14938 (plingeling) R 14936 14938 14543 0 -1 4202496 8021 1251 0 0 28 0 0 0 23 0 1 0 655343749 40603648 7938 33554432000 4194304 4417452 140737183236400 18446744073709551615 4209644 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/14938/statm: 9913 7938 149 55 0 7799 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 39652

[startup+0.700838 s]
/proc/loadavg: 7.11 6.47 5.23 1/277 14938
/proc/meminfo: memFree=21423764/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=90900 CPUtime=0.69 cores=0-7
/proc/14938/stat : 14938 (plingeling) R 14936 14938 14543 0 -1 4202496 20833 1251 0 0 67 2 0 0 25 0 1 0 655343749 93081600 20750 33554432000 4194304 4417452 140737183236400 18446744073709551615 4209681 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/14938/statm: 22725 20750 149 55 0 20611 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 90900

[startup+1.50069 s]
/proc/loadavg: 7.11 6.47 5.23 2/278 14945
/proc/meminfo: memFree=21303732/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=238880 CPUtime=1.48 cores=0-7
/proc/14938/stat : 14938 (plingeling) R 14936 14938 14543 0 -1 4202496 57828 1251 0 0 142 6 0 0 25 0 1 0 655343749 244613120 57745 33554432000 4194304 4417452 140737183236400 18446744073709551615 4209644 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/14938/statm: 59720 57745 149 55 0 57606 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 238880

[startup+3.10041 s]
/proc/loadavg: 6.62 6.38 5.21 2/278 14945
/proc/meminfo: memFree=21191508/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=386336 CPUtime=3.09 cores=0-7
/proc/14938/stat : 14938 (plingeling) R 14936 14938 14543 0 -1 4202496 94692 1251 0 0 297 12 0 0 25 0 1 0 655343749 395608064 94609 33554432000 4194304 4417452 140737183236400 18446744073709551615 4235968 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/14938/statm: 96584 94609 149 55 0 94470 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 386336

[startup+6.30081 s]
/proc/loadavg: 6.62 6.38 5.21 9/286 14953
/proc/meminfo: memFree=20990952/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=636732 CPUtime=23.38 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 156364 1251 0 0 2315 23 0 0 18 0 9 0 655343749 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 159183 110425 177 55 0 157069 0
[pid=14938/tid=14946] ppid=14936 vsize=636732 CPUtime=2.44 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 4936 1251 0 0 243 1 0 0 22 0 9 0 655344133 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4277503 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=636732 CPUtime=2.42 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 5354 1251 0 0 242 0 0 0 25 0 9 0 655344133 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=636732 CPUtime=2.43 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 9985 1251 0 0 243 0 0 0 25 0 9 0 655344133 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4275475 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=636732 CPUtime=2.41 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2401 1251 0 0 241 0 0 0 25 0 9 0 655344133 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4250936 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=636732 CPUtime=2.44 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 6177 1251 0 0 243 1 0 0 22 0 9 0 655344134 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236878 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=636732 CPUtime=2.44 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 3389 1251 0 0 244 0 0 0 25 0 9 0 655344134 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4275606 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=636732 CPUtime=2.43 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 5826 1251 0 0 243 0 0 0 22 0 9 0 655344134 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4277864 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=636732 CPUtime=2.43 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 4833 1251 0 0 243 0 0 0 22 0 9 0 655344134 652013568 110425 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236891 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 23.38
Current children cumulated vsize (KiB) 636732

[startup+12.7007 s]
/proc/loadavg: 6.83 6.43 5.23 9/286 14953
/proc/meminfo: memFree=20999580/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=614408 CPUtime=74.49 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 217542 1251 0 0 7418 31 0 0 18 0 9 0 655343749 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 153602 107187 180 55 0 151488 0
[pid=14938/tid=14946] ppid=14936 vsize=614408 CPUtime=8.82 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 14473 1251 0 0 880 2 0 0 25 0 9 0 655344133 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4232334 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=614408 CPUtime=8.83 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 20943 1251 0 0 880 3 0 0 25 0 9 0 655344133 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4235648 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=614408 CPUtime=8.84 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 13078 1251 0 0 883 1 0 0 25 0 9 0 655344133 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=614408 CPUtime=8.75 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2756 1251 0 0 875 0 0 0 25 0 9 0 655344133 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=614408 CPUtime=8.84 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 10379 1251 0 0 883 1 0 0 25 0 9 0 655344134 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=614408 CPUtime=8.84 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 11587 1251 0 0 882 2 0 0 25 0 9 0 655344134 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=614408 CPUtime=8.83 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 11055 1251 0 0 882 1 0 0 25 0 9 0 655344134 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4259140 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=614408 CPUtime=8.83 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 19808 1251 0 0 881 2 0 0 25 0 9 0 655344134 629153792 107187 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 74.49
Current children cumulated vsize (KiB) 614408

[startup+25.5004 s]
/proc/loadavg: 7.01 6.48 5.26 9/286 14953
/proc/meminfo: memFree=20974128/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=645052 CPUtime=176.77 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 225208 1251 0 0 17644 33 0 0 18 0 9 0 655343749 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 161263 113585 180 55 0 159149 0
[pid=14938/tid=14946] ppid=14936 vsize=645052 CPUtime=21.62 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 15721 1251 0 0 2160 2 0 0 25 0 9 0 655344133 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=645052 CPUtime=21.63 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 21088 1251 0 0 2160 3 0 0 25 0 9 0 655344133 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=645052 CPUtime=21.63 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 13644 1251 0 0 2162 1 0 0 25 0 9 0 655344133 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=645052 CPUtime=21.43 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 7072 1251 0 0 2142 1 0 0 25 0 9 0 655344133 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4237214 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=645052 CPUtime=21.64 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 10797 1251 0 0 2162 2 0 0 25 0 9 0 655344134 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4281668 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=645052 CPUtime=21.64 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 11683 1251 0 0 2162 2 0 0 25 0 9 0 655344134 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=645052 CPUtime=21.63 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 11480 1251 0 0 2162 1 0 0 25 0 9 0 655344134 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4265429 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=645052 CPUtime=21.62 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 20260 1251 0 0 2160 2 0 0 25 0 9 0 655344134 660533248 113585 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 176.77
Current children cumulated vsize (KiB) 645052

[startup+51.1009 s]
/proc/loadavg: 7.35 6.60 5.34 9/286 14954
/proc/meminfo: memFree=20923704/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=719652 CPUtime=381.33 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 242031 1251 0 0 38098 35 0 0 18 0 9 0 655343749 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 179913 130188 180 55 0 177799 0
[pid=14938/tid=14946] ppid=14936 vsize=719652 CPUtime=47.22 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 17179 1251 0 0 4720 2 0 0 25 0 9 0 655344133 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=719652 CPUtime=47.22 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 22715 1251 0 0 4719 3 0 0 25 0 9 0 655344133 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357260 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=719652 CPUtime=47.23 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 14568 1251 0 0 4722 1 0 0 25 0 9 0 655344133 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236949 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=719652 CPUtime=46.8 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 11660 1251 0 0 4679 1 0 0 25 0 9 0 655344133 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=719652 CPUtime=47.24 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 15650 1251 0 0 4721 3 0 0 25 0 9 0 655344134 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4234231 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=719652 CPUtime=47.23 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 13216 1251 0 0 4721 2 0 0 25 0 9 0 655344134 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4264232 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=719652 CPUtime=47.23 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 12078 1251 0 0 4722 1 0 0 25 0 9 0 655344134 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=719652 CPUtime=47.22 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 21502 1251 0 0 4720 2 0 0 25 0 9 0 655344134 736923648 130188 33554432000 4194304 4417452 140737183236400 18446744073709551615 4275781 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 381.33
Current children cumulated vsize (KiB) 719652

[startup+102.311 s]
/proc/loadavg: 7.81 6.86 5.50 9/286 14955
/proc/meminfo: memFree=20866756/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=754752 CPUtime=790.54 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 276628 1251 0 0 79014 40 0 0 18 0 9 0 655343749 772866048 140173 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 188688 140173 180 55 0 186574 0
[pid=14938/tid=14946] ppid=14936 vsize=754752 CPUtime=98.44 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 19953 1251 0 0 9841 3 0 0 25 0 9 0 655344133 772866048 140173 33554432000 4194304 4417452 140737183236400 18446744073709551615 4366014 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=754752 CPUtime=98.44 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 31752 1251 0 0 9839 5 0 0 25 0 9 0 655344133 772866048 140173 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=754752 CPUtime=98.44 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 15519 1251 0 0 9843 1 0 0 25 0 9 0 655344133 772866048 140173 33554432000 4194304 4417452 140737183236400 18446744073709551615 4354515 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=754752 CPUtime=97.55 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 16023 1251 0 0 9753 2 0 0 25 0 9 0 655344133 772866048 140173 33554432000 4194304 4417452 140737183236400 18446744073709551615 4356852 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=754752 CPUtime=98.45 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 15650 1251 0 0 9842 3 0 0 25 0 9 0 655344134 772866048 140173 33554432000 4194304 4417452 140737183236400 18446744073709551615 4237148 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=754752 CPUtime=98.44 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 20529 1251 0 0 9841 3 0 0 25 0 9 0 655344134 772866048 140173 33554432000 4194304 4417452 140737183236400 18446744073709551615 4232654 0 0 4096 17442 0 0 0 -1 6 0 0 0

################
# More data... #
################

[pid=14938/tid=14948] ppid=14936 vsize=3513272 CPUtime=4718.35 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 1095180 1251 0 0 471651 184 0 0 25 0 9 0 655344133 3597590528 829748 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=3513272 CPUtime=4716.48 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2169609 1251 0 0 471230 418 0 0 25 0 9 0 655344133 3597590528 829748 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=3513272 CPUtime=4718.31 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 1207046 1251 0 0 471582 249 0 0 25 0 9 0 655344134 3597590528 829748 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=3513272 CPUtime=4677.59 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 838203 1251 0 0 467622 137 0 0 25 0 9 0 655344134 3597590528 829748 33554432000 4194304 4417452 140737183236400 18446744073709551615 4356852 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=3513272 CPUtime=4718.42 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 1207616 1251 0 0 471650 192 0 0 25 0 9 0 655344134 3597590528 829748 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357024 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=3513272 CPUtime=4717.12 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 733543 1251 0 0 471598 114 0 0 25 0 9 0 655344134 3597590528 829748 33554432000 4194304 4417452 140737183236400 18446744073709551615 4365959 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37707.2
Current children cumulated vsize (KiB) 3513272

[startup+4782.3 s]
/proc/loadavg: 8.12 8.12 8.07 9/286 15092
/proc/meminfo: memFree=17744332/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=3898048 CPUtime=38186.6 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 9568370 1251 0 0 3816997 1658 0 0 18 0 9 0 655343749 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 974512 925942 180 55 0 972398 0
[pid=14938/tid=14946] ppid=14936 vsize=3898048 CPUtime=4778.48 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 1409483 1251 0 0 477627 221 0 0 25 0 9 0 655344133 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=3898048 CPUtime=4778.47 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 686629 1251 0 0 477737 110 0 0 25 0 9 0 655344133 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=3898048 CPUtime=4778.34 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 1102232 1251 0 0 477649 185 0 0 25 0 9 0 655344133 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4232651 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=3898048 CPUtime=4776.48 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2220701 1251 0 0 477225 423 0 0 25 0 9 0 655344133 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4356867 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=3898048 CPUtime=4778.3 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 1209770 1251 0 0 477580 250 0 0 25 0 9 0 655344134 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=3898048 CPUtime=4737.04 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 844503 1251 0 0 473566 138 0 0 25 0 9 0 655344134 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=3898048 CPUtime=4778.41 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 1227691 1251 0 0 477647 194 0 0 25 0 9 0 655344134 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=3898048 CPUtime=4777.1 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 753898 1251 0 0 477594 116 0 0 25 0 9 0 655344134 3991601152 925942 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38186.5
Current children cumulated vsize (KiB) 3898048

[startup+4842.3 s]
/proc/loadavg: 8.17 8.14 8.08 9/286 15094
/proc/meminfo: memFree=17443372/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=4171856 CPUtime=38666 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 9721966 1251 0 0 3864914 1685 0 0 18 0 9 0 655343749 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 1042964 994394 180 55 0 1040850 0
[pid=14938/tid=14946] ppid=14936 vsize=4171856 CPUtime=4838.48 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 1417752 1251 0 0 483626 222 0 0 25 0 9 0 655344133 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4264940 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=4171856 CPUtime=4838.47 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 686629 1251 0 0 483737 110 0 0 25 0 9 0 655344133 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4232666 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=4171856 CPUtime=4838.35 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 1108371 1251 0 0 483648 187 0 0 25 0 9 0 655344133 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4232601 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=4171856 CPUtime=4836.48 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2260261 1251 0 0 483220 428 0 0 25 0 9 0 655344133 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357283 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=4171856 CPUtime=4838.29 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 1216334 1251 0 0 483577 252 0 0 25 0 9 0 655344134 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4232680 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=4171856 CPUtime=4796.5 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 864536 1251 0 0 479508 142 0 0 25 0 9 0 655344134 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=4171856 CPUtime=4838.42 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 1259528 1251 0 0 483643 199 0 0 25 0 9 0 655344134 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4354244 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=4171856 CPUtime=4837.1 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 795092 1251 0 0 483584 126 0 0 25 0 9 0 655344134 4271980544 994394 33554432000 4194304 4417452 140737183236400 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38666
Current children cumulated vsize (KiB) 4171856

[startup+4902.3 s]
/proc/loadavg: 8.17 8.14 8.08 9/286 15095
/proc/meminfo: memFree=17377744/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=4236780 CPUtime=39145.4 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 9914984 1251 0 0 3912835 1710 0 0 18 0 9 0 655343749 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 1059195 1010625 180 55 0 1057081 0
[pid=14938/tid=14946] ppid=14936 vsize=4236780 CPUtime=4898.48 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 1425860 1251 0 0 489625 223 0 0 25 0 9 0 655344133 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=4236780 CPUtime=4898.48 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 703288 1251 0 0 489736 112 0 0 25 0 9 0 655344133 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4232680 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=4236780 CPUtime=4898.35 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 1108908 1251 0 0 489648 187 0 0 25 0 9 0 655344133 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4275892 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=4236780 CPUtime=4896.48 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2387805 1251 0 0 489205 443 0 0 25 0 9 0 655344133 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4365150 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=4236780 CPUtime=4898.29 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 1221174 1251 0 0 489576 253 0 0 25 0 9 0 655344134 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4250972 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=4236780 CPUtime=4855.96 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 875116 1251 0 0 485453 143 0 0 25 0 9 0 655344134 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4236675 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=4236780 CPUtime=4898.42 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 1282770 1251 0 0 489639 203 0 0 25 0 9 0 655344134 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4355852 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=4236780 CPUtime=4897.08 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 796600 1251 0 0 489581 127 0 0 25 0 9 0 655344134 4338462720 1010625 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364474 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39145.5
Current children cumulated vsize (KiB) 4236780

[startup+4962.3 s]
/proc/loadavg: 8.11 8.13 8.08 9/286 15097
/proc/meminfo: memFree=17017992/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=4596728 CPUtime=39624.9 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 10008925 1251 0 0 3960763 1727 0 0 18 0 9 0 655343749 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 1149182 1100586 180 55 0 1147068 0
[pid=14938/tid=14946] ppid=14936 vsize=4596728 CPUtime=4958.49 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 1446160 1251 0 0 495623 226 0 0 25 0 9 0 655344133 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357970 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=4596728 CPUtime=4958.48 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 707398 1251 0 0 495735 113 0 0 25 0 9 0 655344133 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=4596728 CPUtime=4958.35 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 1115807 1251 0 0 495646 189 0 0 25 0 9 0 655344133 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4233432 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=4596728 CPUtime=4956.48 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2387805 1251 0 0 495205 443 0 0 25 0 9 0 655344133 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4357976 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=4596728 CPUtime=4958.3 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 1225966 1251 0 0 495575 255 0 0 25 0 9 0 655344134 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364440 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=4596728 CPUtime=4915.4 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 888822 1251 0 0 491394 146 0 0 25 0 9 0 655344134 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=4596728 CPUtime=4958.41 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 1320843 1251 0 0 495633 208 0 0 25 0 9 0 655344134 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=4596728 CPUtime=4957.08 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 802661 1251 0 0 495580 128 0 0 25 0 9 0 655344134 4707049472 1100586 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39624.9
Current children cumulated vsize (KiB) 4596728



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009.3 s]
/proc/loadavg: 8.05 8.11 8.08 10/286 15098
/proc/meminfo: memFree=16991052/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=4622964 CPUtime=40000.6 cores=0-7
/proc/14938/stat : 14938 (plingeling) S 14936 14938 14543 0 -1 4202496 10113879 1251 0 0 3998317 1744 0 0 18 0 9 0 655343749 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 1155741 1107146 180 55 0 1153627 0
[pid=14938/tid=14946] ppid=14936 vsize=4622964 CPUtime=5005.51 cores=0-7
/proc/14938/task/14946/stat : 14946 (plingeling) R 14936 14938 14543 0 -1 4202560 1451052 1251 0 0 500324 227 0 0 25 0 9 0 655344133 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4250967 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=14938/tid=14947] ppid=14936 vsize=4622964 CPUtime=5005.5 cores=0-7
/proc/14938/task/14947/stat : 14947 (plingeling) R 14936 14938 14543 0 -1 4202560 722392 1251 0 0 500435 115 0 0 25 0 9 0 655344133 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4356962 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=14938/tid=14948] ppid=14936 vsize=4622964 CPUtime=5005.36 cores=0-7
/proc/14938/task/14948/stat : 14948 (plingeling) R 14936 14938 14543 0 -1 4202560 1157904 1251 0 0 500341 195 0 0 25 0 9 0 655344133 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=14938/tid=14949] ppid=14936 vsize=4622964 CPUtime=5003.5 cores=0-7
/proc/14938/task/14949/stat : 14949 (plingeling) R 14936 14938 14543 0 -1 4202560 2387805 1251 0 0 499907 443 0 0 25 0 9 0 655344133 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=14938/tid=14950] ppid=14936 vsize=4622964 CPUtime=5005.32 cores=0-7
/proc/14938/task/14950/stat : 14950 (plingeling) R 14936 14938 14543 0 -1 4202560 1227798 1251 0 0 500276 256 0 0 25 0 9 0 655344134 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=4622964 CPUtime=4961.99 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4202560 908970 1251 0 0 496048 151 0 0 25 0 9 0 655344134 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4355860 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=14938/tid=14952] ppid=14936 vsize=4622964 CPUtime=5005.44 cores=0-7
/proc/14938/task/14952/stat : 14952 (plingeling) R 14936 14938 14543 0 -1 4202560 1324121 1251 0 0 500335 209 0 0 25 0 9 0 655344134 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4356852 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=14938/tid=14953] ppid=14936 vsize=4622964 CPUtime=5004.1 cores=0-7
/proc/14938/task/14953/stat : 14953 (plingeling) R 14936 14938 14543 0 -1 4202560 820374 1251 0 0 500280 130 0 0 25 0 9 0 655344134 4733915136 1107146 33554432000 4194304 4417452 140737183236400 18446744073709551615 4311296 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 4622964

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.43 s]
/proc/loadavg: 8.05 8.11 8.08 10/286 15098
/proc/meminfo: memFree=16991052/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=0 CPUtime=40000.8 cores=0-7
/proc/14938/stat : 14938 (plingeling) Z 14936 14938 14543 0 -1 4203548 10113879 1251 0 0 3998324 1754 0 0 18 0 2 0 655343749 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 0 0 0 0 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=0 CPUtime=4962.1 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4203588 908970 1251 0 0 496049 161 0 0 25 0 2 0 655344134 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 0

[startup+5009.5 s]
/proc/loadavg: 8.05 8.11 8.08 10/286 15098
/proc/meminfo: memFree=16991052/32951124 swapFree=66457188/67111528
[pid=14938] ppid=14936 vsize=0 CPUtime=40000.9 cores=0-7
/proc/14938/stat : 14938 (plingeling) Z 14936 14938 14543 0 -1 4203548 10113879 1251 0 0 3998324 1761 0 0 18 0 2 0 655343749 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/14938/statm: 0 0 0 0 0 0 0
[pid=14938/tid=14951] ppid=14936 vsize=0 CPUtime=4962.17 cores=0-7
/proc/14938/task/14951/stat : 14951 (plingeling) R 14936 14938 14543 0 -1 4203588 908970 1251 0 0 496049 168 0 0 25 0 2 0 655344134 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5009.53
CPU time (s): 40000.9
CPU user time (s): 39983.2
CPU system time (s): 17.6403
CPU usage (%): 798.496
Max. virtual memory (cumulated for all children) (KiB): 7025000

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39983.2
system time used= 17.6403
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 10115130
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 827
involuntary context switches= 350850

runsolver used 12.4171 second user time and 33.135 second system time

The end

Launcher Data 

Begin job on node132 at 2011-05-15 07:30:56
IDJOB=3372401
IDBENCH=82363
IDSOLVER=1851
FILE ID=node132/3372401-1305437454
RUNJOBID= node132-1305437109-14557
PBS_JOBID= 13324765
Free space on /tmp= 73328 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.070-NOTKNOWN.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372401-1305437454/watcher-3372401-1305437454 -o /tmp/evaluation-result-3372401-1305437454/solver-3372401-1305437454 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372401-1305437454.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f52ec4dacdcf65c3aedb61fb46ade01b
RANDOM SEED=312532182

node132.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21424168 kB
Buffers:        290620 kB
Cached:        5683684 kB
SwapCached:       6904 kB
Active:        4851484 kB
Inactive:      6365996 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21424168 kB
SwapTotal:    67111528 kB
SwapFree:     66457188 kB
Dirty:           43972 kB
Writeback:           0 kB
AnonPages:     5241208 kB
Mapped:          18528 kB
Slab:           219936 kB
PageTables:      23968 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 34735028 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73328 MiB
End job on node132 at 2011-05-15 08:54:27