Trace number 3394702

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.6 5003.83

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_26_40_6-dir.shuffled-as.sat03-163.cnf
MD5SUMc06088cac8521bfe72bbb4c9ad8d39fb
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark3288.15
Satisfiable
(Un)Satisfiability was proved
Number of variables780
Number of clauses66810
Sum of the clauses size134348
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 266784
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 526

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394702-1307666288/watcher-3394702-1307666288 -o /tmp/evaluation-result-3394702-1307666288/solver-3394702-1307666288 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394702-1307666288.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.40 1.83 1.94 2/171 11568
/proc/meminfo: memFree=32147176/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=8728 CPUtime=0 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 216 0 0 0 0 0 0 0 25 0 1 0 878226996 8937472 146 33554432000 4194304 4417452 140735672222848 18446744073709551615 246516834048 0 0 4096 0 18446744071562362840 0 0 17 6 0 0 0
/proc/11568/statm: 2182 146 122 55 0 68 0

[startup+0.0556849 s]
/proc/loadavg: 1.40 1.83 1.94 2/171 11568
/proc/meminfo: memFree=32147176/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=13632 CPUtime=0.03 cores=0-7
/proc/11568/stat : 11568 (plingeling) R 11566 11568 11538 0 -1 4202496 1610 1253 0 2 3 0 0 0 25 0 1 0 878226996 13959168 1431 33554432000 4194304 4417452 140735672222848 18446744073709551615 246516452869 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/11568/statm: 3408 1431 148 55 0 1294 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 13632

[startup+0.10068 s]
/proc/loadavg: 1.40 1.83 1.94 2/171 11568
/proc/meminfo: memFree=32147176/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=25920 CPUtime=0.07 cores=0-7
/proc/11568/stat : 11568 (plingeling) R 11566 11568 11538 0 -1 4202496 4682 1253 0 2 7 0 0 0 25 0 1 0 878226996 26542080 4503 33554432000 4194304 4417452 140735672222848 18446744073709551615 4262065 0 0 4096 17442 0 0 0 17 2 0 0 0
/proc/11568/statm: 6480 4503 148 55 0 4366 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 25920

[startup+0.332595 s]
/proc/loadavg: 1.40 1.83 1.94 2/171 11568
/proc/meminfo: memFree=32147176/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=181180 CPUtime=1.75 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 9903 1253 0 2 174 1 0 0 19 0 9 0 878226996 185528320 6566 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 45295 6566 180 55 0 43181 0
Current children cumulated CPU time (s) 1.75
Current children cumulated vsize (KiB) 181180

[startup+0.700528 s]
/proc/loadavg: 1.40 1.83 1.94 2/171 11568
/proc/meminfo: memFree=32147176/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=187516 CPUtime=4.69 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 12120 1253 0 2 468 1 0 0 19 0 9 0 878226996 192016384 8165 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 46879 8165 180 55 0 44765 0
Current children cumulated CPU time (s) 4.69
Current children cumulated vsize (KiB) 187516

[startup+1.50038 s]
/proc/loadavg: 1.40 1.83 1.94 9/180 11582
/proc/meminfo: memFree=32114944/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=194212 CPUtime=11.08 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 15009 1253 0 2 1107 1 0 0 19 0 9 0 878226996 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 48553 9828 180 55 0 46439 0
[pid=11568/tid=11575] ppid=11566 vsize=194212 CPUtime=1.38 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 948 1253 0 2 138 0 0 0 19 0 9 0 878227007 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249991 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=194212 CPUtime=1.36 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 215 1253 0 2 136 0 0 0 25 0 9 0 878227007 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249193 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=194212 CPUtime=1.38 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 847 1253 0 2 138 0 0 0 25 0 9 0 878227007 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4321714 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=194212 CPUtime=1.32 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 5482 1253 0 2 132 0 0 0 25 0 9 0 878227007 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364861 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=194212 CPUtime=1.38 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 767 1253 0 2 138 0 0 0 22 0 9 0 878227007 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249130 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=194212 CPUtime=1.38 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 642 1253 0 2 138 0 0 0 25 0 9 0 878227007 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4337287 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=194212 CPUtime=1.38 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 519 1253 0 2 138 0 0 0 25 0 9 0 878227008 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4356837 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=194212 CPUtime=1.37 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 892 1253 0 2 137 0 0 0 25 0 9 0 878227008 198873088 9828 33554432000 4194304 4417452 140735672222848 18446744073709551615 4250072 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.08
Current children cumulated vsize (KiB) 194212

[startup+3.10109 s]
/proc/loadavg: 1.93 1.93 1.97 9/180 11582
/proc/meminfo: memFree=32098824/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=210024 CPUtime=23.89 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 22544 1253 0 2 2386 3 0 0 19 0 9 0 878226996 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 52506 13809 180 55 0 50392 0
[pid=11568/tid=11575] ppid=11566 vsize=210024 CPUtime=2.98 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 1352 1253 0 2 298 0 0 0 24 0 9 0 878227007 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249982 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=210024 CPUtime=2.97 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 619 1253 0 2 297 0 0 0 25 0 9 0 878227007 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249935 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=210024 CPUtime=2.98 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 1395 1253 0 2 298 0 0 0 25 0 9 0 878227007 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249570 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=210024 CPUtime=2.92 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 10230 1253 0 2 291 1 0 0 25 0 9 0 878227007 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=210024 CPUtime=2.98 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1171 1253 0 2 298 0 0 0 25 0 9 0 878227007 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4250047 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=210024 CPUtime=2.98 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 642 1253 0 2 298 0 0 0 25 0 9 0 878227007 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4236267 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=210024 CPUtime=2.98 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 1142 1253 0 2 298 0 0 0 25 0 9 0 878227008 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4236326 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=210024 CPUtime=2.96 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 1296 1253 0 2 296 0 0 0 25 0 9 0 878227008 215064576 13809 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249958 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.89
Current children cumulated vsize (KiB) 210024

[startup+6.30051 s]
/proc/loadavg: 1.93 1.93 1.97 9/180 11582
/proc/meminfo: memFree=32081232/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=229048 CPUtime=49.46 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 35983 1253 0 2 4941 5 0 0 19 0 9 0 878226996 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 57262 18815 180 55 0 55148 0
[pid=11568/tid=11575] ppid=11566 vsize=229048 CPUtime=6.18 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 1455 1253 0 2 618 0 0 0 25 0 9 0 878227007 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=229048 CPUtime=6.16 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 848 1253 0 2 616 0 0 0 25 0 9 0 878227007 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=229048 CPUtime=6.18 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 3251 1253 0 2 618 0 0 0 25 0 9 0 878227007 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4320836 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=229048 CPUtime=6.13 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 18821 1253 0 2 610 3 0 0 25 0 9 0 878227007 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=229048 CPUtime=6.18 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1171 1253 0 2 618 0 0 0 25 0 9 0 878227007 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=229048 CPUtime=6.18 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 1134 1253 0 2 618 0 0 0 25 0 9 0 878227007 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4356867 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=229048 CPUtime=6.17 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 2554 1253 0 2 617 0 0 0 25 0 9 0 878227008 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366512 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=229048 CPUtime=6.14 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 2052 1253 0 2 614 0 0 0 25 0 9 0 878227008 234545152 18815 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 49.46
Current children cumulated vsize (KiB) 229048

[startup+12.7003 s]
/proc/loadavg: 2.94 2.15 2.04 9/180 11583
/proc/meminfo: memFree=32040980/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=280368 CPUtime=100.62 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 74851 1253 0 2 10049 13 0 0 19 0 9 0 878226996 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 70092 31545 180 55 0 67978 0
[pid=11568/tid=11575] ppid=11566 vsize=280368 CPUtime=12.57 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 5325 1253 0 2 1257 0 0 0 25 0 9 0 878227007 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=280368 CPUtime=12.56 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 3998 1253 0 2 1256 0 0 0 25 0 9 0 878227007 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=280368 CPUtime=12.57 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 9478 1253 0 2 1256 1 0 0 25 0 9 0 878227007 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=280368 CPUtime=12.53 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 26801 1253 0 2 1247 6 0 0 25 0 9 0 878227007 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=280368 CPUtime=12.58 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 5510 1253 0 2 1257 1 0 0 25 0 9 0 878227007 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=280368 CPUtime=12.57 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 6611 1253 0 2 1257 0 0 0 25 0 9 0 878227007 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364749 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=280368 CPUtime=12.57 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 6314 1253 0 2 1257 0 0 0 25 0 9 0 878227008 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=280368 CPUtime=12.5 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 6117 1253 0 2 1250 0 0 0 25 0 9 0 878227008 287096832 31545 33554432000 4194304 4417452 140735672222848 18446744073709551615 4236569 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.62
Current children cumulated vsize (KiB) 280368

[startup+25.501 s]
/proc/loadavg: 3.72 2.34 2.10 9/180 11583
/proc/meminfo: memFree=31921148/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=393504 CPUtime=202.96 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 136594 1253 0 2 20271 25 0 0 19 0 9 0 878226996 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 98376 61363 180 55 0 96262 0
[pid=11568/tid=11575] ppid=11566 vsize=393504 CPUtime=25.38 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 10564 1253 0 2 2536 2 0 0 25 0 9 0 878227007 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 18446744073699065939 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=393504 CPUtime=25.36 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 9381 1253 0 2 2535 1 0 0 25 0 9 0 878227007 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 4278271 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=393504 CPUtime=25.37 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 18111 1253 0 2 2534 3 0 0 25 0 9 0 878227007 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 4365092 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=393504 CPUtime=25.33 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 41494 1253 0 2 2525 8 0 0 25 0 9 0 878227007 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 4250464 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=393504 CPUtime=25.37 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 10553 1253 0 2 2536 1 0 0 25 0 9 0 878227007 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 4353568 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=393504 CPUtime=25.37 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 16785 1253 0 2 2535 2 0 0 25 0 9 0 878227007 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=393504 CPUtime=25.37 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 12178 1253 0 2 2536 1 0 0 25 0 9 0 878227008 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 4249997 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=393504 CPUtime=25.23 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 12831 1253 0 2 2522 1 0 0 25 0 9 0 878227008 402948096 61363 33554432000 4194304 4417452 140735672222848 18446744073709551615 4259600 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 202.96
Current children cumulated vsize (KiB) 393504

[startup+51.1004 s]
/proc/loadavg: 5.18 2.79 2.26 9/180 11584
/proc/meminfo: memFree=31748040/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=554356 CPUtime=407.61 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 259951 1253 0 2 40717 44 0 0 19 0 9 0 878226996 567660544 103364 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 138589 103364 180 55 0 136475 0
[pid=11568/tid=11575] ppid=11566 vsize=554356 CPUtime=50.97 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 28870 1253 0 2 5093 4 0 0 25 0 9 0 878227007 567660544 103364 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=554356 CPUtime=50.96 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 20824 1253 0 2 5093 3 0 0 25 0 9 0 878227007 567660544 103364 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=554356 CPUtime=50.98 cores=0-7

################
# More data... #
################


[startup+4722.31 s]
/proc/loadavg: 8.30 8.19 8.11 9/180 11720
/proc/meminfo: memFree=27804768/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=4489768 CPUtime=37751.4 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 12332472 1253 0 2 3773123 2021 0 0 19 0 9 0 878226996 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 1122442 1083717 180 55 0 1120328 0
[pid=11568/tid=11575] ppid=11566 vsize=4489768 CPUtime=4722.31 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 2826236 1253 0 2 471850 381 0 0 25 0 9 0 878227007 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=4489768 CPUtime=4721.2 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 1451288 1253 0 2 471863 257 0 0 25 0 9 0 878227007 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=4489768 CPUtime=4722.3 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 1735627 1253 0 2 471956 274 0 0 25 0 9 0 878227007 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=4489768 CPUtime=4722.28 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 834921 1253 0 2 472055 173 0 0 25 0 9 0 878227007 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=4489768 CPUtime=4722.16 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1258536 1253 0 2 471979 237 0 0 25 0 9 0 878227007 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=4489768 CPUtime=4722.16 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 1814586 1253 0 2 471931 285 0 0 25 0 9 0 878227007 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=4489768 CPUtime=4722.31 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 1075562 1253 0 2 472029 202 0 0 25 0 9 0 878227008 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=4489768 CPUtime=4696.55 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 1331019 1253 0 2 469447 208 0 0 25 0 9 0 878227008 4597522432 1083717 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 37751.4
Current children cumulated vsize (KiB) 4489768

[startup+4782.3 s]
/proc/loadavg: 8.29 8.20 8.11 9/180 11722
/proc/meminfo: memFree=27657684/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=4636968 CPUtime=38231.1 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 12442336 1253 0 2 3821069 2039 0 0 19 0 9 0 878226996 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 1159242 1120517 180 55 0 1157128 0
[pid=11568/tid=11575] ppid=11566 vsize=4636968 CPUtime=4782.3 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 2826236 1253 0 2 477849 381 0 0 25 0 9 0 878227007 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=4636968 CPUtime=4781.19 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 1484056 1253 0 2 477859 260 0 0 25 0 9 0 878227007 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=4636968 CPUtime=4782.29 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 1789982 1253 0 2 477944 285 0 0 25 0 9 0 878227007 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=4636968 CPUtime=4782.27 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 834921 1253 0 2 478054 173 0 0 25 0 9 0 878227007 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=4636968 CPUtime=4782.16 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1258536 1253 0 2 477979 237 0 0 25 0 9 0 878227007 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=4636968 CPUtime=4782.16 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 1814586 1253 0 2 477931 285 0 0 25 0 9 0 878227007 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=4636968 CPUtime=4782.31 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 1075562 1253 0 2 478029 202 0 0 25 0 9 0 878227008 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=4636968 CPUtime=4756.22 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 1353760 1253 0 2 475411 211 0 0 25 0 9 0 878227008 4748255232 1120517 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364861 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231.1
Current children cumulated vsize (KiB) 4636968

[startup+4842.3 s]
/proc/loadavg: 8.25 8.21 8.12 9/180 11723
/proc/meminfo: memFree=27561296/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=4722980 CPUtime=38710.8 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 12533463 1253 0 2 3869022 2053 0 0 19 0 9 0 878226996 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 1180745 1142020 180 55 0 1178631 0
[pid=11568/tid=11575] ppid=11566 vsize=4722980 CPUtime=4842.31 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 2826236 1253 0 2 483850 381 0 0 25 0 9 0 878227007 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=4722980 CPUtime=4841.19 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 1500440 1253 0 2 483857 262 0 0 25 0 9 0 878227007 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366804 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=4722980 CPUtime=4842.29 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 1789982 1253 0 2 483944 285 0 0 25 0 9 0 878227007 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=4722980 CPUtime=4842.27 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 845784 1253 0 2 484051 176 0 0 25 0 9 0 878227007 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=4722980 CPUtime=4842.16 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1258536 1253 0 2 483979 237 0 0 25 0 9 0 878227007 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=4722980 CPUtime=4842.16 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 1814586 1253 0 2 483931 285 0 0 25 0 9 0 878227007 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=4722980 CPUtime=4842.31 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 1133680 1253 0 2 484021 210 0 0 25 0 9 0 878227008 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=4722980 CPUtime=4815.9 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 1359522 1253 0 2 481378 212 0 0 25 0 9 0 878227008 4836331520 1142020 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.8
Current children cumulated vsize (KiB) 4722980

[startup+4902.3 s]
/proc/loadavg: 8.27 8.21 8.12 9/180 11725
/proc/meminfo: memFree=27330644/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=4963400 CPUtime=39190.4 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 12822736 1253 0 2 3916947 2096 0 0 19 0 9 0 878226996 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 1240850 1202124 180 55 0 1238736 0
[pid=11568/tid=11575] ppid=11566 vsize=4963400 CPUtime=4902.31 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 2924329 1253 0 2 489838 393 0 0 25 0 9 0 878227007 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=4963400 CPUtime=4901.18 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 1533208 1253 0 2 489853 265 0 0 25 0 9 0 878227007 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=4963400 CPUtime=4902.29 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 1847666 1253 0 2 489934 295 0 0 25 0 9 0 878227007 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364791 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=4963400 CPUtime=4902.27 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 845784 1253 0 2 490051 176 0 0 25 0 9 0 878227007 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=4963400 CPUtime=4902.16 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1335775 1253 0 2 489964 252 0 0 25 0 9 0 878227007 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=4963400 CPUtime=4902.16 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 1814586 1253 0 2 489931 285 0 0 25 0 9 0 878227007 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=4963400 CPUtime=4902.31 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 1133680 1253 0 2 490021 210 0 0 25 0 9 0 878227008 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366400 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=4963400 CPUtime=4875.58 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 1383011 1253 0 2 487343 215 0 0 25 0 9 0 878227008 5082521600 1202124 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.4
Current children cumulated vsize (KiB) 4963400

[startup+4962.3 s]
/proc/loadavg: 8.22 8.22 8.12 9/180 11727
/proc/meminfo: memFree=27217656/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=5074696 CPUtime=39670.1 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 12954550 1253 0 2 3964894 2117 0 0 19 0 9 0 878226996 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 1268674 1229948 180 55 0 1266560 0
[pid=11568/tid=11575] ppid=11566 vsize=5074696 CPUtime=4962.32 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 2957097 1253 0 2 495834 398 0 0 25 0 9 0 878227007 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364597 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=5074696 CPUtime=4961.18 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 1549592 1253 0 2 495851 267 0 0 25 0 9 0 878227007 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=5074696 CPUtime=4962.3 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 1847666 1253 0 2 495934 296 0 0 25 0 9 0 878227007 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364752 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=5074696 CPUtime=4962.28 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 845784 1253 0 2 496051 177 0 0 25 0 9 0 878227007 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=5074696 CPUtime=4962.16 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1335775 1253 0 2 495964 252 0 0 25 0 9 0 878227007 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=5074696 CPUtime=4962.16 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 1845043 1253 0 2 495928 288 0 0 25 0 9 0 878227007 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=5074696 CPUtime=4962.31 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 1150064 1253 0 2 496016 215 0 0 25 0 9 0 878227008 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=5074696 CPUtime=4935.24 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 1418832 1253 0 2 493304 220 0 0 25 0 9 0 878227008 5196488704 1229948 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670.1
Current children cumulated vsize (KiB) 5074696



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.6 s]
/proc/loadavg: 8.23 8.22 8.13 9/180 11737
/proc/meminfo: memFree=27151584/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=5139580 CPUtime=40000.3 cores=0-7
/proc/11568/stat : 11568 (plingeling) S 11566 11568 11538 0 -1 4202496 13047929 1253 0 2 3997898 2132 0 0 19 0 9 0 878226996 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 246528636901 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 1284895 1246169 180 55 0 1282781 0
[pid=11568/tid=11575] ppid=11566 vsize=5139580 CPUtime=5003.61 cores=0-7
/proc/11568/task/11575/stat : 11575 (plingeling) R 11566 11568 11538 0 -1 4202560 2973481 1253 0 2 499961 400 0 0 25 0 9 0 878227007 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=11568/tid=11576] ppid=11566 vsize=5139580 CPUtime=5002.48 cores=0-7
/proc/11568/task/11576/stat : 11576 (plingeling) R 11566 11568 11538 0 -1 4202560 1582360 1253 0 2 499977 271 0 0 25 0 9 0 878227007 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11568/tid=11577] ppid=11566 vsize=5139580 CPUtime=5003.6 cores=0-7
/proc/11568/task/11577/stat : 11577 (plingeling) R 11566 11568 11538 0 -1 4202560 1847666 1253 0 2 500064 296 0 0 25 0 9 0 878227007 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=11568/tid=11578] ppid=11566 vsize=5139580 CPUtime=5003.59 cores=0-7
/proc/11568/task/11578/stat : 11578 (plingeling) R 11566 11568 11538 0 -1 4202560 845784 1253 0 2 500182 177 0 0 25 0 9 0 878227007 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=11568/tid=11579] ppid=11566 vsize=5139580 CPUtime=5003.47 cores=0-7
/proc/11568/task/11579/stat : 11579 (plingeling) R 11566 11568 11538 0 -1 4202560 1335775 1253 0 2 500095 252 0 0 25 0 9 0 878227007 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11568/tid=11580] ppid=11566 vsize=5139580 CPUtime=5003.46 cores=0-7
/proc/11568/task/11580/stat : 11580 (plingeling) R 11566 11568 11538 0 -1 4202560 1889270 1253 0 2 500050 296 0 0 25 0 9 0 878227007 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=11568/tid=11581] ppid=11566 vsize=5139580 CPUtime=5003.62 cores=0-7
/proc/11568/task/11581/stat : 11581 (plingeling) R 11566 11568 11538 0 -1 4202560 1150064 1253 0 2 500146 216 0 0 25 0 9 0 878227008 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=5139580 CPUtime=4976.31 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4202560 1418832 1253 0 2 497411 220 0 0 25 0 9 0 878227008 5262929920 1246169 33554432000 4194304 4417452 140735672222848 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 5139580

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5003.8 s]
/proc/loadavg: 8.23 8.22 8.13 2/174 11738
/proc/meminfo: memFree=31543956/32951124 swapFree=67063556/67111528
[pid=11568] ppid=11566 vsize=0 CPUtime=40000.5 cores=0-7
/proc/11568/stat : 11568 (plingeling) Z 11566 11568 11538 0 -1 4203548 13047929 1253 0 2 3997902 2151 0 0 18 0 2 0 878226996 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/11568/statm: 0 0 0 0 0 0 0
[pid=11568/tid=11582] ppid=11566 vsize=0 CPUtime=4976.52 cores=0-7
/proc/11568/task/11582/stat : 11582 (plingeling) R 11566 11568 11538 0 -1 4203588 1418832 1253 0 2 497412 240 0 0 25 0 2 0 878227008 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5003.83
CPU time (s): 40000.6
CPU user time (s): 39979
CPU system time (s): 21.5437
CPU usage (%): 799.399
Max. virtual memory (cumulated for all children) (KiB): 5140232

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39979
system time used= 21.5437
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 13049182
page faults= 2
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 238
involuntary context switches= 364691

runsolver used 8.66068 second user time and 18.6442 second system time

The end

Launcher Data 

Begin job on node134 at 2011-06-10 02:38:08
IDJOB=3394702
IDBENCH=83903
IDSOLVER=1851
FILE ID=node134/3394702-1307666288
RUNJOBID= node134-1307666288-11552
PBS_JOBID= 13549474
Free space on /tmp= 74204 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_26_40_6-dir.shuffled-as.sat03-163.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394702-1307666288/watcher-3394702-1307666288 -o /tmp/evaluation-result-3394702-1307666288/solver-3394702-1307666288 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394702-1307666288.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c06088cac8521bfe72bbb4c9ad8d39fb
RANDOM SEED=1270796259

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      32147348 kB
Buffers:        246596 kB
Cached:         337640 kB
SwapCached:      10280 kB
Active:         352008 kB
Inactive:       250180 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      32147348 kB
SwapTotal:    67111528 kB
SwapFree:     67063556 kB
Dirty:             832 kB
Writeback:           0 kB
AnonPages:       16116 kB
Mapped:           9844 kB
Slab:           134968 kB
PageTables:       4460 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   157920 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74204 MiB
End job on node134 at 2011-06-10 04:01:34