Trace number 3394128

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.5 5011.14

General information on the benchmark

NameCRAFTED/rbsat/crafted/
forced/rbsat-v2640c305320gyes10.cnf
MD5SUM98bac4f2d533fe7abb12552567f9e1aa
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2640
Number of clauses305320
Sum of the clauses size613120
Maximum clause length33
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2305240
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 580

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394128-1306091208/watcher-3394128-1306091208 -o /tmp/evaluation-result-3394128-1306091208/solver-3394128-1306091208 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394128-1306091208.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.76 2.08 2.05 2/482 16598
/proc/meminfo: memFree=19924420/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=9620 CPUtime=0 cores=0-7
/proc/16598/stat : 16598 (plingeling) R 16596 16598 16572 0 -1 4202496 482 1259 0 0 0 0 0 0 18 0 1 0 720719314 9850880 411 33554432000 4194304 4417452 140735282320400 18446744073709551615 210204647168 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16598/statm: 2405 411 147 55 0 291 0

[startup+0.0720639 s]
/proc/loadavg: 1.76 2.08 2.05 2/482 16598
/proc/meminfo: memFree=19924420/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=19612 CPUtime=0.06 cores=0-7
/proc/16598/stat : 16598 (plingeling) R 16596 16598 16572 0 -1 4202496 2997 1259 0 0 6 0 0 0 18 0 1 0 720719314 20082688 2926 33554432000 4194304 4417452 140735282320400 18446744073709551615 4262041 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16598/statm: 4903 2926 148 55 0 2789 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 19612

[startup+0.101066 s]
/proc/loadavg: 1.76 2.08 2.05 2/482 16598
/proc/meminfo: memFree=19924420/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=27804 CPUtime=0.09 cores=0-7
/proc/16598/stat : 16598 (plingeling) R 16596 16598 16572 0 -1 4202496 5045 1259 0 0 9 0 0 0 18 0 1 0 720719314 28471296 4974 33554432000 4194304 4417452 140735282320400 18446744073709551615 210204647168 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16598/statm: 6951 4974 148 55 0 4837 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27804

[startup+0.301015 s]
/proc/loadavg: 1.76 2.08 2.05 2/482 16598
/proc/meminfo: memFree=19924420/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=76956 CPUtime=0.28 cores=0-7
/proc/16598/stat : 16598 (plingeling) R 16596 16598 16572 0 -1 4202496 17333 1259 0 0 27 1 0 0 19 0 1 0 720719314 78802944 17262 33554432000 4194304 4417452 140735282320400 18446744073709551615 4235968 0 0 4096 17442 0 0 0 17 6 0 0 0
/proc/16598/statm: 19239 17262 148 55 0 17125 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 76956

[startup+0.700877 s]
/proc/loadavg: 1.76 2.08 2.05 2/482 16598
/proc/meminfo: memFree=19924420/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=237648 CPUtime=2.51 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 26266 1259 0 0 248 3 0 0 20 0 9 0 720719314 243351552 20592 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 59412 20592 177 55 0 57298 0
Current children cumulated CPU time (s) 2.51
Current children cumulated vsize (KiB) 237648

[startup+1.50074 s]
/proc/loadavg: 1.76 2.08 2.05 9/491 16612
/proc/meminfo: memFree=19830736/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=258896 CPUtime=8.88 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 45449 1259 0 0 883 5 0 0 20 0 9 0 720719314 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 64724 25985 180 55 0 62610 0
[pid=16598/tid=16605] ppid=16596 vsize=258896 CPUtime=1.1 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 2725 1259 0 0 110 0 0 0 19 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4320455 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=258896 CPUtime=0.96 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 3448 1259 0 0 96 0 0 0 23 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4258258 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=258896 CPUtime=1.1 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 3356 1259 0 0 110 0 0 0 19 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4258241 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=258896 CPUtime=1.1 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 8187 1259 0 0 109 1 0 0 25 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4366126 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=258896 CPUtime=1.08 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 3302 1259 0 0 108 0 0 0 25 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4258392 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=258896 CPUtime=1.05 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 518 1259 0 0 105 0 0 0 25 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4236255 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=258896 CPUtime=0.95 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 3684 1259 0 0 95 0 0 0 25 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=258896 CPUtime=1.09 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 2880 1259 0 0 109 0 0 0 25 0 9 0 720719353 265109504 25985 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 8.88
Current children cumulated vsize (KiB) 258896

[startup+3.10047 s]
/proc/loadavg: 1.76 2.08 2.05 9/491 16612
/proc/meminfo: memFree=19803080/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=280192 CPUtime=21.67 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 61452 1259 0 0 2160 7 0 0 20 0 9 0 720719314 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 70048 31309 180 55 0 67934 0
[pid=16598/tid=16605] ppid=16596 vsize=280192 CPUtime=2.7 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 4086 1259 0 0 270 0 0 0 23 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=280192 CPUtime=2.56 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 4809 1259 0 0 256 0 0 0 25 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=280192 CPUtime=2.7 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 3356 1259 0 0 270 0 0 0 23 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4376709 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=280192 CPUtime=2.7 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 20107 1259 0 0 268 2 0 0 25 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=280192 CPUtime=2.66 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 3302 1259 0 0 266 0 0 0 25 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249209 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=280192 CPUtime=2.65 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 518 1259 0 0 265 0 0 0 25 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249601 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=280192 CPUtime=2.55 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 3684 1259 0 0 255 0 0 0 25 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4377348 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=280192 CPUtime=2.69 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 4241 1259 0 0 269 0 0 0 25 0 9 0 720719353 286916608 31309 33554432000 4194304 4417452 140735282320400 18446744073709551615 4258258 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 21.67
Current children cumulated vsize (KiB) 280192

[startup+6.30093 s]
/proc/loadavg: 2.26 2.18 2.08 9/491 16612
/proc/meminfo: memFree=19781392/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=295148 CPUtime=47.22 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 79498 1259 0 0 4712 10 0 0 20 0 9 0 720719314 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 73787 35080 180 55 0 71673 0
[pid=16598/tid=16605] ppid=16596 vsize=295148 CPUtime=5.9 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 4086 1259 0 0 590 0 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4239414 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=295148 CPUtime=5.76 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 4809 1259 0 0 576 0 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249181 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=295148 CPUtime=5.9 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 3818 1259 0 0 590 0 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4278215 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=295148 CPUtime=5.9 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 37476 1259 0 0 585 5 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=295148 CPUtime=5.81 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 3302 1259 0 0 581 0 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=295148 CPUtime=5.85 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 518 1259 0 0 585 0 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=295148 CPUtime=5.75 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 3899 1259 0 0 575 0 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4343760 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=295148 CPUtime=5.89 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 4241 1259 0 0 589 0 0 0 25 0 9 0 720719353 302231552 35080 33554432000 4194304 4417452 140735282320400 18446744073709551615 4235225 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 47.22
Current children cumulated vsize (KiB) 295148

[startup+12.7009 s]
/proc/loadavg: 2.72 2.28 2.11 9/491 16612
/proc/meminfo: memFree=19750084/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=329052 CPUtime=98.32 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 109415 1259 0 0 9817 15 0 0 20 0 9 0 720719314 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 82263 43466 180 55 0 80149 0
[pid=16598/tid=16605] ppid=16596 vsize=329052 CPUtime=12.3 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 4086 1259 0 0 1230 0 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4250010 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=329052 CPUtime=12.16 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 4809 1259 0 0 1216 0 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4258401 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=329052 CPUtime=12.3 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 4603 1259 0 0 1230 0 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4365078 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=329052 CPUtime=12.3 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 66266 1259 0 0 1220 10 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=329052 CPUtime=12.11 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 3302 1259 0 0 1211 0 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249597 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=329052 CPUtime=12.25 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 518 1259 0 0 1225 0 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=329052 CPUtime=12.15 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 4241 1259 0 0 1215 0 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=329052 CPUtime=12.29 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 4241 1259 0 0 1229 0 0 0 25 0 9 0 720719353 336949248 43466 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249151 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 98.32
Current children cumulated vsize (KiB) 329052

[startup+25.5007 s]
/proc/loadavg: 3.89 2.56 2.21 9/491 16612
/proc/meminfo: memFree=19628568/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=452952 CPUtime=200.5 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 183485 1259 0 0 20023 27 0 0 20 0 9 0 720719314 463822848 74150 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 113238 74176 180 55 0 111124 0
[pid=16598/tid=16605] ppid=16596 vsize=452952 CPUtime=25.11 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 6952 1259 0 0 2510 1 0 0 25 0 9 0 720719353 463822848 74188 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=452952 CPUtime=24.96 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 7018 1259 0 0 2496 0 0 0 25 0 9 0 720719353 463822848 74200 33554432000 4194304 4417452 140735282320400 18446744073709551615 4232523 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=452952 CPUtime=25.1 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 5658 1259 0 0 2510 0 0 0 25 0 9 0 720719353 463822848 74214 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=452952 CPUtime=25.1 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 119723 1259 0 0 2491 19 0 0 25 0 9 0 720719353 463822848 74229 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=452952 CPUtime=24.7 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 7611 1259 0 0 2470 0 0 0 25 0 9 0 720719353 463822848 74244 33554432000 4194304 4417452 140735282320400 18446744073709551615 4366065 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=452952 CPUtime=25.04 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 6015 1259 0 0 2504 0 0 0 25 0 9 0 720719353 463822848 74260 33554432000 4194304 4417452 140735282320400 18446744073709551615 210204341311 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=452952 CPUtime=24.95 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 5607 1259 0 0 2495 0 0 0 25 0 9 0 720719353 463822848 74275 33554432000 4194304 4417452 140735282320400 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=452952 CPUtime=25.09 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 7663 1259 0 0 2508 1 0 0 25 0 9 0 720719353 463822848 74291 33554432000 4194304 4417452 140735282320400 18446744073709551615 4365512 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 200.5
Current children cumulated vsize (KiB) 452952

[startup+51.1004 s]
/proc/loadavg: 5.37 3.01 2.37 9/491 16613
/proc/meminfo: memFree=19517776/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=561976 CPUtime=404.88 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 262588 1259 0 0 40449 39 0 0 20 0 9 0 720719314 575463424 101731 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 140494 101731 180 55 0 138380 0
[pid=16598/tid=16605] ppid=16596 vsize=561976 CPUtime=50.7 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 8095 1259 0 0 5069 1 0 0 25 0 9 0 720719353 575463424 101731 33554432000 4194304 4417452 140735282320400 18446744073709551615 4237814 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=561976 CPUtime=50.56 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 8489 1259 0 0 5055 1 0 0 25 0 9 0 720719353 575463424 101731 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=561976 CPUtime=50.69 cores=0-7

################
# More data... #
################

[pid=16598/tid=16607] ppid=16596 vsize=5096156 CPUtime=4720.56 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 391940 1259 0 0 471997 59 0 0 25 0 9 0 720719353 5218463744 1250755 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364420 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=5096156 CPUtime=4722 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 2535101 1259 0 0 471654 546 0 0 25 0 9 0 720719353 5218463744 1250755 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=5096156 CPUtime=4667.46 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 578092 1259 0 0 466652 94 0 0 25 0 9 0 720719353 5218463744 1250755 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=5096156 CPUtime=4721.84 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 514735 1259 0 0 472107 77 0 0 25 0 9 0 720719353 5218463744 1250755 33554432000 4194304 4417452 140735282320400 18446744073709551615 4354244 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=5096156 CPUtime=4721.14 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 898401 1259 0 0 471969 145 0 0 25 0 9 0 720719353 5218463744 1250755 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=5096156 CPUtime=4721.84 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 923781 1259 0 0 472046 138 0 0 25 0 9 0 720719353 5218463744 1250755 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37696.2
Current children cumulated vsize (KiB) 5096156

[startup+4782.3 s]
/proc/loadavg: 8.08 8.08 8.00 9/491 16751
/proc/meminfo: memFree=14705960/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=5301580 CPUtime=38175.1 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 7513710 1259 0 0 3816202 1309 0 0 20 0 9 0 720719314 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 1325395 1302111 180 55 0 1323281 0
[pid=16598/tid=16605] ppid=16596 vsize=5301580 CPUtime=4781.87 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 827614 1259 0 0 478065 122 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=5301580 CPUtime=4758.08 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 718627 1259 0 0 475700 108 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=5301580 CPUtime=4780.53 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 399417 1259 0 0 477992 61 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=5301580 CPUtime=4781.98 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 2548296 1259 0 0 477651 547 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=5301580 CPUtime=4727.44 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 580248 1259 0 0 472650 94 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=5301580 CPUtime=4781.82 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 524298 1259 0 0 478104 78 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=5301580 CPUtime=4781.12 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 970328 1259 0 0 477955 157 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=5301580 CPUtime=4781.82 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 927533 1259 0 0 478044 138 0 0 25 0 9 0 720719353 5428817920 1302111 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364594 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38175.1
Current children cumulated vsize (KiB) 5301580

[startup+4842.3 s]
/proc/loadavg: 8.07 8.08 8.00 9/491 16753
/proc/meminfo: memFree=14492392/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=5515360 CPUtime=38654.1 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 7575453 1259 0 0 3864091 1320 0 0 20 0 9 0 720719314 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 1378840 1355556 180 55 0 1376726 0
[pid=16598/tid=16605] ppid=16596 vsize=5515360 CPUtime=4841.87 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 827614 1259 0 0 484065 122 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=5515360 CPUtime=4817.09 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 733884 1259 0 0 481597 112 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=5515360 CPUtime=4840.51 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 403417 1259 0 0 483990 61 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364589 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=5515360 CPUtime=4841.98 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 2573439 1259 0 0 483648 550 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=5515360 CPUtime=4787.44 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 582712 1259 0 0 478649 95 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=5515360 CPUtime=4841.82 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 534889 1259 0 0 484103 79 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4343748 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=5515360 CPUtime=4841.12 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 970328 1259 0 0 483955 157 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4357007 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=5515360 CPUtime=4841.82 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 931821 1259 0 0 484044 138 0 0 25 0 9 0 720719353 5647728640 1355556 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364765 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38654.1
Current children cumulated vsize (KiB) 5515360

[startup+4902.3 s]
/proc/loadavg: 8.07 8.08 8.00 9/491 16754
/proc/meminfo: memFree=14107564/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=5902468 CPUtime=39133.1 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 7834656 1259 0 0 3911947 1364 0 0 20 0 9 0 720719314 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 1475617 1452334 180 55 0 1473503 0
[pid=16598/tid=16605] ppid=16596 vsize=5902468 CPUtime=4901.88 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 840526 1259 0 0 490063 125 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4320940 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=5902468 CPUtime=4876.11 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 847233 1259 0 0 487473 138 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=5902468 CPUtime=4900.5 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 428927 1259 0 0 489984 66 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=5902468 CPUtime=4901.98 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 2573439 1259 0 0 489648 550 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=5902468 CPUtime=4847.44 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 620410 1259 0 0 484646 98 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=5902468 CPUtime=4901.82 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 539649 1259 0 0 490102 80 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4236778 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=5902468 CPUtime=4901.12 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 970328 1259 0 0 489955 157 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=5902468 CPUtime=4901.81 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 996795 1259 0 0 490036 145 0 0 25 0 9 0 720719353 6044127232 1452334 33554432000 4194304 4417452 140735282320400 18446744073709551615 4355852 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39133.1
Current children cumulated vsize (KiB) 5902468

[startup+4962.3 s]
/proc/loadavg: 8.06 8.08 8.00 9/491 16756
/proc/meminfo: memFree=13632884/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=6373080 CPUtime=39612.1 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 8014895 1259 0 0 3959806 1406 0 0 20 0 9 0 720719314 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 1593270 1569987 180 55 0 1591156 0
[pid=16598/tid=16605] ppid=16596 vsize=6373080 CPUtime=4961.88 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 844942 1259 0 0 496063 125 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=6373080 CPUtime=4935.13 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 847233 1259 0 0 493375 138 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=6373080 CPUtime=4960.49 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 451931 1259 0 0 495980 69 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=6373080 CPUtime=4961.98 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 2599829 1259 0 0 495641 557 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=6373080 CPUtime=4907.44 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 628602 1259 0 0 490644 100 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=6373080 CPUtime=4961.83 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 544398 1259 0 0 496102 81 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=6373080 CPUtime=4961.12 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 1083816 1259 0 0 495927 185 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=6373080 CPUtime=4961.81 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 996795 1259 0 0 496036 145 0 0 25 0 9 0 720719353 6526033920 1569987 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364587 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39612.1
Current children cumulated vsize (KiB) 6373080



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5010.9 s]
/proc/loadavg: 8.11 8.09 8.01 9/491 16757
/proc/meminfo: memFree=13901904/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=6104804 CPUtime=40000.2 cores=0-7
/proc/16598/stat : 16598 (plingeling) S 16596 16598 16572 0 -1 4202496 8039777 1259 0 0 3998611 1410 0 0 20 0 9 0 720719314 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 210216450021 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 1526201 1502918 180 55 0 1524087 0
[pid=16598/tid=16605] ppid=16596 vsize=6104804 CPUtime=5010.5 cores=0-7
/proc/16598/task/16605/stat : 16605 (plingeling) R 16596 16598 16572 0 -1 4202560 850830 1259 0 0 500924 126 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=6104804 CPUtime=4982.94 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4202560 847233 1259 0 0 498156 138 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4307594 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=16598/tid=16607] ppid=16596 vsize=6104804 CPUtime=5009.09 cores=0-7
/proc/16598/task/16607/stat : 16607 (plingeling) R 16596 16598 16572 0 -1 4202560 461551 1259 0 0 500839 70 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=16598/tid=16608] ppid=16596 vsize=6104804 CPUtime=5010.6 cores=0-7
/proc/16598/task/16608/stat : 16608 (plingeling) R 16596 16598 16572 0 -1 4202560 2599829 1259 0 0 500503 557 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=16598/tid=16609] ppid=16596 vsize=6104804 CPUtime=4956.05 cores=0-7
/proc/16598/task/16609/stat : 16609 (plingeling) R 16596 16598 16572 0 -1 4202560 637976 1259 0 0 495503 102 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=16598/tid=16610] ppid=16596 vsize=6104804 CPUtime=5010.45 cores=0-7
/proc/16598/task/16610/stat : 16610 (plingeling) R 16596 16598 16572 0 -1 4202560 544398 1259 0 0 500964 81 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=16598/tid=16611] ppid=16596 vsize=6104804 CPUtime=5009.74 cores=0-7
/proc/16598/task/16611/stat : 16611 (plingeling) R 16596 16598 16572 0 -1 4202560 1083816 1259 0 0 500789 185 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=16598/tid=16612] ppid=16596 vsize=6104804 CPUtime=5010.43 cores=0-7
/proc/16598/task/16612/stat : 16612 (plingeling) R 16596 16598 16572 0 -1 4202560 996795 1259 0 0 500898 145 0 0 25 0 9 0 720719353 6251319296 1502918 33554432000 4194304 4417452 140735282320400 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.2
Current children cumulated vsize (KiB) 6104804

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5011.02 s]
/proc/loadavg: 8.11 8.09 8.01 9/491 16757
/proc/meminfo: memFree=13901904/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=0 CPUtime=40000.4 cores=0-7
/proc/16598/stat : 16598 (plingeling) Z 16596 16598 16572 0 -1 4203548 8039777 1259 0 0 3998619 1420 0 0 18 0 2 0 720719314 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 0 0 0 0 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=0 CPUtime=4983.04 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4203588 847233 1259 0 0 498156 148 0 0 25 0 2 0 720719353 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 0

[startup+5011.1 s]
/proc/loadavg: 8.11 8.09 8.01 9/491 16757
/proc/meminfo: memFree=13901904/32951124 swapFree=63649292/67111528
[pid=16598] ppid=16596 vsize=0 CPUtime=40000.5 cores=0-7
/proc/16598/stat : 16598 (plingeling) Z 16596 16598 16572 0 -1 4203548 8039777 1259 0 0 3998619 1428 0 0 18 0 2 0 720719314 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/16598/statm: 0 0 0 0 0 0 0
[pid=16598/tid=16606] ppid=16596 vsize=0 CPUtime=4983.12 cores=0-7
/proc/16598/task/16606/stat : 16606 (plingeling) R 16596 16598 16572 0 -1 4203588 847233 1259 0 0 498156 156 0 0 25 0 2 0 720719353 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5011.14
CPU time (s): 40000.5
CPU user time (s): 39986.2
CPU system time (s): 14.3248
CPU usage (%): 798.231
Max. virtual memory (cumulated for all children) (KiB): 7335296

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39986.2
system time used= 14.3248
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 8041036
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 828
involuntary context switches= 194006

runsolver used 23.4044 second user time and 58.3981 second system time

The end

Launcher Data 

Begin job on node130 at 2011-05-22 21:06:48
IDJOB=3394128
IDBENCH=71605
IDSOLVER=1851
FILE ID=node130/3394128-1306091208
RUNJOBID= node130-1306091208-16586
PBS_JOBID= 13387938
Free space on /tmp= 73104 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v2640c305320gyes10.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394128-1306091208/watcher-3394128-1306091208 -o /tmp/evaluation-result-3394128-1306091208/solver-3394128-1306091208 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394128-1306091208.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 98bac4f2d533fe7abb12552567f9e1aa
RANDOM SEED=1879501812

node130.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19924700 kB
Buffers:        343896 kB
Cached:        7582756 kB
SwapCached:      10260 kB
Active:        6947240 kB
Inactive:      5759416 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19924700 kB
SwapTotal:    67111528 kB
SwapFree:     63649292 kB
Dirty:            4280 kB
Writeback:           0 kB
AnonPages:     4778412 kB
Mapped:          29908 kB
Slab:           195132 kB
PageTables:      53596 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 145065064 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73104 MiB
End job on node130 at 2011-05-22 22:30:21