Trace number 3372743

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.9 5006.65

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
64/aes_64_4_keyfind_1.cnf
MD5SUM07ea67092ce3fd4529e72f69434e3f99
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1808
Number of clauses9968
Sum of the clauses size39152
Maximum clause length5
Minimum clause length1
Number of clauses of size 1144
Number of clauses of size 2320
Number of clauses of size 33904
Number of clauses of size 41344
Number of clauses of size 54256
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372743-1305441859/watcher-3372743-1305441859 -o /tmp/evaluation-result-3372743-1305441859/solver-3372743-1305441859 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372743-1305441859.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.36 7.88 7.96 2/309 17793
/proc/meminfo: memFree=27035436/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=9928 CPUtime=0 cores=0-7
/proc/17793/stat : 17793 (plingeling) R 17791 17793 17445 0 -1 4202496 559 1250 0 0 0 0 0 0 18 0 1 0 388377156 10166272 488 33554432000 4194304 4417452 140733638304640 18446744073709551615 268660356047 0 0 4096 17442 0 0 0 17 4 0 0 0
/proc/17793/statm: 2482 488 148 55 0 368 0

[startup+0.071561 s]
/proc/loadavg: 7.36 7.88 7.96 2/309 17793
/proc/meminfo: memFree=27035436/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=163160 CPUtime=0.82 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 2428 1250 0 0 82 0 0 0 18 0 9 0 388377156 167075840 2066 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 40790 2066 180 55 0 38676 0
Current children cumulated CPU time (s) 0.82
Current children cumulated vsize (KiB) 163160

[startup+0.232585 s]
/proc/loadavg: 7.36 7.88 7.96 2/309 17793
/proc/meminfo: memFree=27035436/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=165192 CPUtime=1.62 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 3034 1250 0 0 162 0 0 0 18 0 9 0 388377156 169156608 2544 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 41298 2544 180 55 0 39184 0
Current children cumulated CPU time (s) 1.62
Current children cumulated vsize (KiB) 165192

[startup+0.332568 s]
/proc/loadavg: 7.36 7.88 7.96 2/309 17793
/proc/meminfo: memFree=27035436/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=167588 CPUtime=2.42 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 3944 1250 0 0 242 0 0 0 18 0 9 0 388377156 171610112 3198 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 41897 3198 180 55 0 39783 0
Current children cumulated CPU time (s) 2.42
Current children cumulated vsize (KiB) 167588

[startup+0.700506 s]
/proc/loadavg: 7.36 7.88 7.96 2/309 17793
/proc/meminfo: memFree=27035436/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=173576 CPUtime=5.37 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 7259 1250 0 0 536 1 0 0 18 0 9 0 388377156 177741824 4874 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 43394 4874 180 55 0 41280 0
Current children cumulated CPU time (s) 5.37
Current children cumulated vsize (KiB) 173576

[startup+1.50036 s]
/proc/loadavg: 7.36 7.88 7.96 9/318 17807
/proc/meminfo: memFree=27013912/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=178340 CPUtime=11.76 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 12597 1250 0 0 1173 3 0 0 18 0 9 0 388377156 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 44585 6059 180 55 0 42471 0
[pid=17793/tid=17800] ppid=17791 vsize=178340 CPUtime=1.46 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 1063 1250 0 0 146 0 0 0 20 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=178340 CPUtime=1.46 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 1034 1250 0 0 146 0 0 0 23 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4252408 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=178340 CPUtime=1.45 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 1491 1250 0 0 145 0 0 0 23 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=178340 CPUtime=1.46 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1823 1250 0 0 146 0 0 0 20 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4356852 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=178340 CPUtime=1.45 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 1681 1250 0 0 145 0 0 0 25 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4357562 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=178340 CPUtime=1.46 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 1220 1250 0 0 146 0 0 0 25 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=178340 CPUtime=1.46 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 1474 1250 0 0 146 0 0 0 25 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4353568 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=178340 CPUtime=1.46 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1360 1250 0 0 146 0 0 0 23 0 9 0 388377159 182620160 6059 33554432000 4194304 4417452 140733638304640 18446744073709551615 4320926 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.76
Current children cumulated vsize (KiB) 178340

[startup+3.10108 s]
/proc/loadavg: 7.36 7.88 7.96 9/318 17807
/proc/meminfo: memFree=27008456/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=187660 CPUtime=24.54 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 21209 1250 0 0 2450 4 0 0 18 0 9 0 388377156 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 46915 8372 180 55 0 44801 0
[pid=17793/tid=17800] ppid=17791 vsize=187660 CPUtime=3.06 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 1849 1250 0 0 306 0 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=187660 CPUtime=3.06 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 2042 1250 0 0 306 0 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=187660 CPUtime=3.05 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 2275 1250 0 0 305 0 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=187660 CPUtime=3.07 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 3177 1250 0 0 307 0 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366146 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=187660 CPUtime=3.04 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 3319 1250 0 0 303 1 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4232791 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=187660 CPUtime=3.06 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 2133 1250 0 0 306 0 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=187660 CPUtime=3.06 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 2355 1250 0 0 306 0 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364583 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=187660 CPUtime=3.06 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 2608 1250 0 0 306 0 0 0 25 0 9 0 388377159 192163840 8372 33554432000 4194304 4417452 140733638304640 18446744073709551615 4236549 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.54
Current children cumulated vsize (KiB) 187660

[startup+6.30052 s]
/proc/loadavg: 7.41 7.88 7.96 9/318 17807
/proc/meminfo: memFree=26993116/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=188592 CPUtime=50.11 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 37207 1250 0 0 5004 7 0 0 18 0 9 0 388377156 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 47148 8649 180 55 0 45034 0
[pid=17793/tid=17800] ppid=17791 vsize=188592 CPUtime=6.26 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 3454 1250 0 0 626 0 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=188592 CPUtime=6.26 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 3864 1250 0 0 626 0 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4365135 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=188592 CPUtime=6.24 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 3795 1250 0 0 624 0 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=188592 CPUtime=6.26 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 5460 1250 0 0 626 0 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=188592 CPUtime=6.21 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 6241 1250 0 0 620 1 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4250476 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=188592 CPUtime=6.26 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 4345 1250 0 0 625 1 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=188592 CPUtime=6.26 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 3690 1250 0 0 626 0 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4357536 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=188592 CPUtime=6.26 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 4907 1250 0 0 625 1 0 0 25 0 9 0 388377159 193118208 8649 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364583 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 50.11
Current children cumulated vsize (KiB) 188592

[startup+12.7004 s]
/proc/loadavg: 7.46 7.88 7.96 9/318 17808
/proc/meminfo: memFree=26979972/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=213008 CPUtime=101.24 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 67353 1250 0 0 10111 13 0 0 18 0 9 0 388377156 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 53252 14837 180 55 0 51138 0
[pid=17793/tid=17800] ppid=17791 vsize=213008 CPUtime=12.66 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 7524 1250 0 0 1265 1 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=213008 CPUtime=12.66 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 6452 1250 0 0 1265 1 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=213008 CPUtime=12.65 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 7092 1250 0 0 1264 1 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=213008 CPUtime=12.67 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 8755 1250 0 0 1266 1 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=213008 CPUtime=12.54 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 9655 1250 0 0 1252 2 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4236816 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=213008 CPUtime=12.66 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 8973 1250 0 0 1264 2 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4365106 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=213008 CPUtime=12.66 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 8222 1250 0 0 1265 1 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4353568 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=213008 CPUtime=12.66 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 9229 1250 0 0 1265 1 0 0 25 0 9 0 388377159 218120192 14837 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 101.24
Current children cumulated vsize (KiB) 213008

[startup+25.5012 s]
/proc/loadavg: 7.58 7.89 7.96 9/318 17808
/proc/meminfo: memFree=26958648/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=229516 CPUtime=203.51 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 122635 1250 0 0 20328 23 0 0 18 0 9 0 388377156 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 57379 19847 180 55 0 55265 0
[pid=17793/tid=17800] ppid=17791 vsize=229516 CPUtime=25.47 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 11083 1250 0 0 2545 2 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4353624 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=229516 CPUtime=25.46 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 11657 1250 0 0 2544 2 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=229516 CPUtime=25.44 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 15112 1250 0 0 2542 2 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=229516 CPUtime=25.47 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 16514 1250 0 0 2545 2 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4353561 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=229516 CPUtime=25.21 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 14650 1250 0 0 2518 3 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=229516 CPUtime=25.46 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 17390 1250 0 0 2541 5 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=229516 CPUtime=25.46 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 15386 1250 0 0 2544 2 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=229516 CPUtime=25.46 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 19392 1250 0 0 2544 2 0 0 25 0 9 0 388377159 235024384 19847 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364850 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.51
Current children cumulated vsize (KiB) 229516

[startup+51.1007 s]
/proc/loadavg: 7.79 7.91 7.96 9/318 17809
/proc/meminfo: memFree=26938200/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=245204 CPUtime=408.07 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 211343 1250 0 0 40765 42 0 0 18 0 9 0 388377156 251088896 23987 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 61301 23987 180 55 0 59187 0
[pid=17793/tid=17800] ppid=17791 vsize=245204 CPUtime=51.07 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 20886 1250 0 0 5103 4 0 0 25 0 9 0 388377159 251088896 23987 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=245204 CPUtime=51.06 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 20143 1250 0 0 5102 4 0 0 25 0 9 0 388377159 251088896 23987 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=245204 CPUtime=51.04 cores=0-7

################
# More data... #
################

/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1052355 1250 0 0 472052 186 0 0 25 0 9 0 388377159 798810112 162434 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=780088 CPUtime=4675.16 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 636377 1250 0 0 467401 115 0 0 25 0 9 0 388377159 798810112 162434 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=780088 CPUtime=4722.37 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 526416 1250 0 0 472132 105 0 0 25 0 9 0 388377159 798810112 162434 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=780088 CPUtime=4722.31 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 680727 1250 0 0 472108 123 0 0 25 0 9 0 388377159 798810112 162434 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=780088 CPUtime=4722.36 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1099752 1250 0 0 472065 171 0 0 25 0 9 0 388377159 798810112 162434 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37729.2
Current children cumulated vsize (KiB) 780088

[startup+4782.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/318 17959
/proc/meminfo: memFree=26333180/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=864284 CPUtime=38208.6 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 6134853 1250 0 0 3819775 1085 0 0 18 0 9 0 388377156 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 216071 175452 180 55 0 213957 0
[pid=17793/tid=17800] ppid=17791 vsize=864284 CPUtime=4782.36 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 703049 1250 0 0 478118 118 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=864284 CPUtime=4779.86 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 684772 1250 0 0 477856 130 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=864284 CPUtime=4782.35 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 711642 1250 0 0 478110 125 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4365959 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=864284 CPUtime=4782.37 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1065772 1250 0 0 478048 189 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=864284 CPUtime=4734.55 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 637495 1250 0 0 473340 115 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=864284 CPUtime=4782.36 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 530524 1250 0 0 478130 106 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=864284 CPUtime=4782.3 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 688263 1250 0 0 478105 125 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=864284 CPUtime=4782.34 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1111885 1250 0 0 478062 172 0 0 25 0 9 0 388377159 885026816 175452 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38208.6
Current children cumulated vsize (KiB) 864284

[startup+4842.31 s]
/proc/loadavg: 8.02 8.05 8.00 9/318 17960
/proc/meminfo: memFree=26425616/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=729392 CPUtime=38688.1 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 6173064 1250 0 0 3867716 1092 0 0 18 0 9 0 388377156 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 182348 152418 180 55 0 180234 0
[pid=17793/tid=17800] ppid=17791 vsize=729392 CPUtime=4842.37 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 707311 1250 0 0 484118 119 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=729392 CPUtime=4839.86 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 688096 1250 0 0 483856 130 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364693 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=729392 CPUtime=4842.37 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 717726 1250 0 0 484110 127 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=729392 CPUtime=4842.38 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1073337 1250 0 0 484048 190 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=729392 CPUtime=4793.96 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 638618 1250 0 0 479281 115 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366480 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=729392 CPUtime=4842.38 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 533536 1250 0 0 484131 107 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=729392 CPUtime=4842.31 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 693159 1250 0 0 484105 126 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=729392 CPUtime=4842.36 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1119830 1250 0 0 484062 174 0 0 25 0 9 0 388377159 746897408 152418 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38688.1
Current children cumulated vsize (KiB) 729392

[startup+4902.3 s]
/proc/loadavg: 8.01 8.03 8.00 9/318 17962
/proc/meminfo: memFree=26330696/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=856696 CPUtime=39167.4 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 6230184 1250 0 0 3915638 1102 0 0 18 0 9 0 388377156 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 214174 176053 180 55 0 212060 0
[pid=17793/tid=17800] ppid=17791 vsize=856696 CPUtime=4902.36 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 712689 1250 0 0 490116 120 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=856696 CPUtime=4899.84 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 699971 1250 0 0 489853 131 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=856696 CPUtime=4902.36 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 720130 1250 0 0 490108 128 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=856696 CPUtime=4902.38 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1086779 1250 0 0 490044 194 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=856696 CPUtime=4853.36 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 639745 1250 0 0 485220 116 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=856696 CPUtime=4902.36 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 540449 1250 0 0 490129 107 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=856696 CPUtime=4902.3 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 698055 1250 0 0 490103 127 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=856696 CPUtime=4902.34 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1130915 1250 0 0 490059 175 0 0 25 0 9 0 388377159 877256704 176053 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39167.4
Current children cumulated vsize (KiB) 856696

[startup+4962.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/318 17964
/proc/meminfo: memFree=26425608/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=712628 CPUtime=39646.8 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 6282152 1250 0 0 3963568 1111 0 0 18 0 9 0 388377156 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 178157 152344 180 55 0 176043 0
[pid=17793/tid=17800] ppid=17791 vsize=712628 CPUtime=4962.36 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 716853 1250 0 0 496115 121 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=712628 CPUtime=4959.82 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 711017 1250 0 0 495850 132 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=712628 CPUtime=4962.35 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 732058 1250 0 0 496106 129 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366633 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=712628 CPUtime=4962.37 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1098128 1250 0 0 496041 196 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=712628 CPUtime=4912.75 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 640874 1250 0 0 491159 116 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=712628 CPUtime=4962.37 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 543189 1250 0 0 496129 108 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=712628 CPUtime=4962.31 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 699647 1250 0 0 496103 128 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4250569 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=712628 CPUtime=4962.34 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1138935 1250 0 0 496058 176 0 0 25 0 9 0 388377159 729731072 152344 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39646.8
Current children cumulated vsize (KiB) 712628



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.6 s]
/proc/loadavg: 8.05 8.04 8.00 9/318 17965
/proc/meminfo: memFree=26410920/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=718556 CPUtime=40000.8 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 6304507 1250 0 0 3998965 1114 0 0 18 0 9 0 388377156 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 179639 155874 180 55 0 177525 0
[pid=17793/tid=17800] ppid=17791 vsize=718556 CPUtime=5006.67 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 719185 1250 0 0 500546 121 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364619 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=718556 CPUtime=5004.13 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 712311 1250 0 0 500280 133 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=718556 CPUtime=5006.66 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 732058 1250 0 0 500537 129 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=718556 CPUtime=5006.69 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1107141 1250 0 0 500472 197 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=718556 CPUtime=4956.61 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 640874 1250 0 0 495545 116 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=718556 CPUtime=5006.68 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 543189 1250 0 0 500560 108 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364693 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=718556 CPUtime=5006.62 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 704423 1250 0 0 500533 129 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=718556 CPUtime=5006.66 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1143875 1250 0 0 500489 177 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366126 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 718556

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5006.6 s]
/proc/loadavg: 8.05 8.04 8.00 9/318 17965
/proc/meminfo: memFree=26410920/32951124 swapFree=46575252/67111528
[pid=17793] ppid=17791 vsize=718556 CPUtime=40000.8 cores=0-7
/proc/17793/stat : 17793 (plingeling) S 17791 17793 17445 0 -1 4202496 6304507 1250 0 0 3998965 1114 0 0 18 0 9 0 388377156 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 0 0 0 0
/proc/17793/statm: 179639 155874 180 55 0 177525 0
[pid=17793/tid=17800] ppid=17791 vsize=718556 CPUtime=5006.67 cores=0-7
/proc/17793/task/17800/stat : 17800 (plingeling) R 17791 17793 17445 0 -1 4202560 719185 1250 0 0 500546 121 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364619 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=17793/tid=17801] ppid=17791 vsize=718556 CPUtime=5004.13 cores=0-7
/proc/17793/task/17801/stat : 17801 (plingeling) R 17791 17793 17445 0 -1 4202560 712311 1250 0 0 500280 133 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17793/tid=17802] ppid=17791 vsize=718556 CPUtime=5006.66 cores=0-7
/proc/17793/task/17802/stat : 17802 (plingeling) R 17791 17793 17445 0 -1 4202560 732058 1250 0 0 500537 129 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17793/tid=17803] ppid=17791 vsize=718556 CPUtime=5006.69 cores=0-7
/proc/17793/task/17803/stat : 17803 (plingeling) R 17791 17793 17445 0 -1 4202560 1107141 1250 0 0 500472 197 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17793/tid=17804] ppid=17791 vsize=718556 CPUtime=4956.61 cores=0-7
/proc/17793/task/17804/stat : 17804 (plingeling) R 17791 17793 17445 0 -1 4202560 640874 1250 0 0 495545 116 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17793/tid=17805] ppid=17791 vsize=718556 CPUtime=5006.68 cores=0-7
/proc/17793/task/17805/stat : 17805 (plingeling) R 17791 17793 17445 0 -1 4202560 543189 1250 0 0 500560 108 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364693 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=17793/tid=17806] ppid=17791 vsize=718556 CPUtime=5006.62 cores=0-7
/proc/17793/task/17806/stat : 17806 (plingeling) R 17791 17793 17445 0 -1 4202560 704423 1250 0 0 500533 129 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=17793/tid=17807] ppid=17791 vsize=718556 CPUtime=5006.66 cores=0-7
/proc/17793/task/17807/stat : 17807 (plingeling) R 17791 17793 17445 0 -1 4202560 1143875 1250 0 0 500489 177 0 0 25 0 9 0 388377159 735801344 155874 33554432000 4194304 4417452 140733638304640 18446744073709551615 4366126 0 0 4096 17442 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 718556

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5006.65
CPU time (s): 40000.9
CPU user time (s): 39989.7
CPU system time (s): 11.1743
CPU usage (%): 798.956
Max. virtual memory (cumulated for all children) (KiB): 941960

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39989.7
system time used= 11.1743
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6305757
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 294
involuntary context switches= 191208

runsolver used 13.5919 second user time and 36.4915 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-15 08:44:19
IDJOB=3372743
IDBENCH=83416
IDSOLVER=1851
FILE ID=node107/3372743-1305441859
RUNJOBID= node107-1305436255-17459
PBS_JOBID= 13324746
Free space on /tmp= 73492 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/64/aes_64_4_keyfind_1.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372743-1305441859/watcher-3372743-1305441859 -o /tmp/evaluation-result-3372743-1305441859/solver-3372743-1305441859 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372743-1305441859.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 07ea67092ce3fd4529e72f69434e3f99
RANDOM SEED=1234529

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      27035716 kB
Buffers:        155028 kB
Cached:        1191524 kB
SwapCached:       9832 kB
Active:        5243836 kB
Inactive:       389496 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      27035716 kB
SwapTotal:    67111528 kB
SwapFree:     46575252 kB
Dirty:             432 kB
Writeback:           0 kB
AnonPages:     4283712 kB
Mapped:          18964 kB
Slab:           146192 kB
PageTables:      68020 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 64915308 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73492 MiB
End job on node107 at 2011-05-15 10:07:48