Trace number 3370935

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 4800.44 1205.29

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-7x39.cnf
MD5SUMf717554c4e7f802b3078c48464dbcd9d
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1092
Number of clauses64155
Sum of the clauses size253344
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 21638
Number of clauses of size 30
Number of clauses of size 462517
Number of clauses of size 50
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3370935-1305425256/watcher-3370935-1305425256 -o /tmp/evaluation-result-3370935-1305425256/solver-3370935-1305425256 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3370935-1305425256.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.57 7.94 7.46 1/344 29229
/proc/meminfo: memFree=21556384/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=11056 CPUtime=0 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) R 29227 29229 27864 0 -1 4202496 857 1252 0 0 0 0 0 0 25 0 1 0 634941773 11321344 788 33554432000 4194304 4417452 140736221138208 18446744073709551615 255106768640 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/29229/statm: 2764 788 148 55 0 650 0

[startup+0.183386 s]
/proc/loadavg: 7.57 7.94 7.46 1/344 29229
/proc/meminfo: memFree=21556384/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=70908 CPUtime=0.48 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 6627 1252 0 0 48 0 0 0 25 0 5 0 634941773 72609792 5537 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 17727 5537 178 55 0 15613 0
Current children cumulated CPU time (s) 0.48
Current children cumulated vsize (KiB) 70908

[startup+0.200386 s]
/proc/loadavg: 7.57 7.94 7.46 1/344 29229
/proc/meminfo: memFree=21556384/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=71516 CPUtime=0.55 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 6756 1252 0 0 55 0 0 0 25 0 5 0 634941773 73232384 5666 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 17879 5666 179 55 0 15765 0
Current children cumulated CPU time (s) 0.55
Current children cumulated vsize (KiB) 71516

[startup+0.300373 s]
/proc/loadavg: 7.57 7.94 7.46 1/344 29229
/proc/meminfo: memFree=21556384/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=73092 CPUtime=0.94 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 7395 1252 0 0 94 0 0 0 25 0 5 0 634941773 74846208 6073 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 18273 6073 180 55 0 16159 0
Current children cumulated CPU time (s) 0.94
Current children cumulated vsize (KiB) 73092

[startup+0.700294 s]
/proc/loadavg: 7.57 7.94 7.46 1/344 29229
/proc/meminfo: memFree=21556384/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=84392 CPUtime=2.55 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 11433 1252 0 0 254 1 0 0 25 0 5 0 634941773 86417408 8886 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 21098 8886 180 55 0 18984 0
Current children cumulated CPU time (s) 2.55
Current children cumulated vsize (KiB) 84392

[startup+1.50116 s]
/proc/loadavg: 7.57 7.94 7.46 9/358 29294
/proc/meminfo: memFree=21454992/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=92836 CPUtime=5.74 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 16009 1252 0 0 572 2 0 0 25 0 5 0 634941773 95064064 11019 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 23209 11019 180 55 0 21095 0
[pid=29229/tid=29236] ppid=29227 vsize=92836 CPUtime=1.42 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 1506 1252 0 0 142 0 0 0 25 0 5 0 634941781 95064064 11019 33554432000 4194304 4417452 140736221138208 18446744073709551615 4263523 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=92836 CPUtime=1.41 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 1430 1252 0 0 141 0 0 0 25 0 5 0 634941781 95064064 11019 33554432000 4194304 4417452 140736221138208 18446744073709551615 4336693 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=92836 CPUtime=1.4 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 2831 1252 0 0 140 0 0 0 25 0 5 0 634941781 95064064 11019 33554432000 4194304 4417452 140736221138208 18446744073709551615 4356916 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=92836 CPUtime=1.41 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 5497 1252 0 0 141 0 0 0 25 0 5 0 634941781 95064064 11019 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.74
Current children cumulated vsize (KiB) 92836

[startup+3.10087 s]
/proc/loadavg: 7.57 7.94 7.46 9/358 29294
/proc/meminfo: memFree=21399440/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=162908 CPUtime=12.13 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 30377 1252 0 0 1209 4 0 0 25 0 5 0 634941773 166817792 12164 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 40727 12164 180 55 0 38613 0
[pid=29229/tid=29236] ppid=29227 vsize=162908 CPUtime=3.01 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 4605 1252 0 0 301 0 0 0 25 0 5 0 634941781 166817792 12164 33554432000 4194304 4417452 140736221138208 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=162908 CPUtime=3.01 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 1901 1252 0 0 301 0 0 0 25 0 5 0 634941781 166817792 12164 33554432000 4194304 4417452 140736221138208 18446744073709551615 4336699 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=162908 CPUtime=2.99 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 8100 1252 0 0 298 1 0 0 25 0 5 0 634941781 166817792 12164 33554432000 4194304 4417452 140736221138208 18446744073709551615 4366087 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=162908 CPUtime=3.01 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 11026 1252 0 0 300 1 0 0 25 0 5 0 634941781 166817792 12164 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364488 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 12.13
Current children cumulated vsize (KiB) 162908

[startup+6.30031 s]
/proc/loadavg: 7.61 7.94 7.46 9/358 29294
/proc/meminfo: memFree=21337800/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=194124 CPUtime=24.89 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 58118 1252 0 0 2480 9 0 0 25 0 5 0 634941773 198782976 19985 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 48531 19985 180 55 0 46417 0
[pid=29229/tid=29236] ppid=29227 vsize=194124 CPUtime=6.22 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 10018 1252 0 0 621 1 0 0 25 0 5 0 634941781 198782976 19985 33554432000 4194304 4417452 140736221138208 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=194180 CPUtime=6.2 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 4807 1252 0 0 620 0 0 0 25 0 5 0 634941781 198840320 19998 33554432000 4194304 4417452 140736221138208 18446744073709551615 4248923 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=194180 CPUtime=6.15 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 16587 1252 0 0 612 3 0 0 25 0 5 0 634941781 198840320 19999 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=194180 CPUtime=6.21 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 21975 1252 0 0 618 3 0 0 25 0 5 0 634941781 198840320 19999 33554432000 4194304 4417452 140736221138208 18446744073709551615 255106462671 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.89
Current children cumulated vsize (KiB) 194124

[startup+12.7012 s]
/proc/loadavg: 7.64 7.94 7.46 9/358 29295
/proc/meminfo: memFree=21295940/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=241884 CPUtime=50.42 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 111432 1252 0 0 5024 18 0 0 25 0 5 0 634941773 247689216 31908 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 60471 31908 180 55 0 58357 0
[pid=29229/tid=29236] ppid=29227 vsize=241884 CPUtime=12.61 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 25521 1252 0 0 1258 3 0 0 25 0 5 0 634941781 247689216 31908 33554432000 4194304 4417452 140736221138208 18446744073709551615 4258641 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=241884 CPUtime=12.61 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 14466 1252 0 0 1259 2 0 0 25 0 5 0 634941781 247689216 31908 33554432000 4194304 4417452 140736221138208 18446744073709551615 4249171 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=241884 CPUtime=12.48 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 32017 1252 0 0 1242 6 0 0 25 0 5 0 634941781 247689216 31908 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=241884 CPUtime=12.62 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 34683 1252 0 0 1256 6 0 0 25 0 5 0 634941781 247689216 31908 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50.42
Current children cumulated vsize (KiB) 241884

[startup+25.5009 s]
/proc/loadavg: 7.72 7.95 7.47 9/358 29295
/proc/meminfo: memFree=21214232/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=265988 CPUtime=101.47 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 213482 1252 0 0 10112 35 0 0 25 0 5 0 634941773 272371712 37934 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 66497 37934 180 55 0 64383 0
[pid=29229/tid=29236] ppid=29227 vsize=265988 CPUtime=25.41 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 48832 1252 0 0 2534 7 0 0 25 0 5 0 634941781 272371712 37934 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=265988 CPUtime=25.41 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 37978 1252 0 0 2536 5 0 0 25 0 5 0 634941781 272371712 37934 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=265988 CPUtime=25.13 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 62614 1252 0 0 2502 11 0 0 25 0 5 0 634941781 272371712 37934 33554432000 4194304 4417452 140736221138208 18446744073709551615 4249165 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=265988 CPUtime=25.41 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 59313 1252 0 0 2531 10 0 0 25 0 5 0 634941781 272371712 37934 33554432000 4194304 4417452 140736221138208 18446744073709551615 4250507 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 101.47
Current children cumulated vsize (KiB) 265988

[startup+51.1004 s]
/proc/loadavg: 7.89 7.97 7.49 9/358 29296
/proc/meminfo: memFree=21015160/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=340428 CPUtime=203.59 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 348151 1252 0 0 20304 55 0 0 25 0 5 0 634941773 348598272 56574 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 85107 56574 180 55 0 82993 0
[pid=29229/tid=29236] ppid=29227 vsize=340428 CPUtime=51.02 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 76344 1252 0 0 5091 11 0 0 25 0 5 0 634941781 348598272 56574 33554432000 4194304 4417452 140736221138208 18446744073709551615 4265373 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=340428 CPUtime=51.01 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 70938 1252 0 0 5091 10 0 0 25 0 5 0 634941781 348598272 56574 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=340428 CPUtime=50.45 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 106190 1252 0 0 5027 18 0 0 25 0 5 0 634941781 348598272 56574 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=340428 CPUtime=51.01 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 89934 1252 0 0 5086 15 0 0 25 0 5 0 634941781 348598272 56574 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.59
Current children cumulated vsize (KiB) 340428

[startup+102.3 s]
/proc/loadavg: 7.99 7.98 7.52 9/358 29297
/proc/meminfo: memFree=20731688/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=576852 CPUtime=407.86 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 577759 1252 0 0 40693 93 0 0 25 0 5 0 634941773 590696448 115650 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 144213 115650 180 55 0 142099 0
[pid=29229/tid=29236] ppid=29227 vsize=576852 CPUtime=102.22 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 131035 1252 0 0 10203 19 0 0 25 0 5 0 634941781 590696448 115650 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=576852 CPUtime=102.23 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 132627 1252 0 0 10204 19 0 0 25 0 5 0 634941781 590696448 115650 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=576852 CPUtime=101.09 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 165660 1252 0 0 10080 29 0 0 25 0 5 0 634941781 590696448 115650 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=576852 CPUtime=102.22 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 143692 1252 0 0 10198 24 0 0 25 0 5 0 634941781 590696448 115650 33554432000 4194304 4417452 140736221138208 18446744073709551615 4358184 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.86
Current children cumulated vsize (KiB) 576852

[startup+162.301 s]
/proc/loadavg: 8.22 8.04 7.56 9/358 29299
/proc/meminfo: memFree=20232932/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=825924 CPUtime=647.16 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 803555 1252 0 0 64585 131 0 0 25 0 5 0 634941773 845746176 177918 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 206481 177918 180 55 0 204367 0
[pid=29229/tid=29236] ppid=29227 vsize=825924 CPUtime=161.88 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 190020 1252 0 0 16160 28 0 0 25 0 5 0 634941781 845746176 177918 33554432000 4194304 4417452 140736221138208 18446744073709551615 4262493 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=825924 CPUtime=162.21 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 195838 1252 0 0 16191 30 0 0 25 0 5 0 634941781 845746176 177918 33554432000 4194304 4417452 140736221138208 18446744073709551615 4357204 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=825924 CPUtime=160.75 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 220038 1252 0 0 16037 38 0 0 25 0 5 0 634941781 845746176 177918 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364885 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=825924 CPUtime=162.22 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 192914 1252 0 0 16189 33 0 0 25 0 5 0 634941781 845746176 177918 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 647.16
Current children cumulated vsize (KiB) 825924


################
# More data... #
################

[pid=29229/tid=29237] ppid=29227 vsize=1305384 CPUtime=762.24 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 645557 1252 0 0 76121 103 0 0 25 0 5 0 634941781 1336713216 297783 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1305384 CPUtime=760.75 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 665031 1252 0 0 75967 108 0 0 25 0 5 0 634941781 1336713216 297783 33554432000 4194304 4417452 140736221138208 18446744073709551615 4259603 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1305384 CPUtime=762.24 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 437845 1252 0 0 76149 75 0 0 25 0 5 0 634941781 1336713216 297783 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3040.36
Current children cumulated vsize (KiB) 1305384

[startup+822.301 s]
/proc/loadavg: 8.11 8.12 7.83 9/358 29316
/proc/meminfo: memFree=18421900/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1449904 CPUtime=3279.63 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 2569862 1252 0 0 327549 414 0 0 25 0 5 0 634941773 1484701696 333913 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 362476 333913 180 55 0 360362 0
[pid=29229/tid=29236] ppid=29227 vsize=1449904 CPUtime=814.33 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 716892 1252 0 0 81323 110 0 0 25 0 5 0 634941781 1484701696 333913 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1449904 CPUtime=822.22 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 704730 1252 0 0 82109 113 0 0 25 0 5 0 634941781 1484701696 333913 33554432000 4194304 4417452 140736221138208 18446744073709551615 4250464 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1449904 CPUtime=820.74 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 678498 1252 0 0 81964 110 0 0 25 0 5 0 634941781 1484701696 333913 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1449904 CPUtime=822.23 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 464997 1252 0 0 82145 78 0 0 25 0 5 0 634941781 1484701696 333913 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3279.63
Current children cumulated vsize (KiB) 1449904

[startup+882.313 s]
/proc/loadavg: 8.30 8.16 7.86 10/365 29904
/proc/meminfo: memFree=18453120/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1816280 CPUtime=3518.46 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 2716366 1252 0 0 351409 437 0 0 25 0 5 0 634941773 1859870720 425507 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 454070 425507 180 55 0 451956 0
[pid=29229/tid=29236] ppid=29227 vsize=1816280 CPUtime=873.44 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 778202 1252 0 0 87224 120 0 0 25 0 5 0 634941781 1859870720 425507 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1816280 CPUtime=882.2 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 751042 1252 0 0 88101 119 0 0 25 0 5 0 634941781 1859870720 425507 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1816280 CPUtime=880.73 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 717380 1252 0 0 87956 117 0 0 25 0 5 0 634941781 1859870720 425507 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1816280 CPUtime=881.97 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 464997 1252 0 0 88118 79 0 0 25 0 5 0 634941781 1859870720 425507 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3518.46
Current children cumulated vsize (KiB) 1816280

[startup+942.301 s]
/proc/loadavg: 9.01 8.38 7.96 10/366 31613
/proc/meminfo: memFree=18302576/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1806248 CPUtime=3756.76 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 2822484 1252 0 0 375222 454 0 0 25 0 5 0 634941773 1849597952 422999 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 451562 422999 180 55 0 449448 0
[pid=29229/tid=29236] ppid=29227 vsize=1806248 CPUtime=932.54 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 778202 1252 0 0 93134 120 0 0 25 0 5 0 634941781 1849597952 422999 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1806248 CPUtime=942.11 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 780425 1252 0 0 94087 124 0 0 25 0 5 0 634941781 1849597952 422999 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1806248 CPUtime=940.63 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 770316 1252 0 0 93937 126 0 0 25 0 5 0 634941781 1849597952 422999 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1806248 CPUtime=941.38 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 488796 1252 0 0 94056 82 0 0 25 0 5 0 634941781 1849597952 422999 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3756.76
Current children cumulated vsize (KiB) 1806248

[startup+1002.3 s]
/proc/loadavg: 9.19 8.56 8.04 10/365 1020
/proc/meminfo: memFree=18251784/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1823372 CPUtime=3995.1 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 2935166 1252 0 0 399040 470 0 0 25 0 5 0 634941773 1867132928 427280 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 455843 427280 180 55 0 453729 0
[pid=29229/tid=29236] ppid=29227 vsize=1823372 CPUtime=991.73 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 822062 1252 0 0 99047 126 0 0 25 0 5 0 634941781 1867132928 427280 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1823372 CPUtime=1002.05 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 813844 1252 0 0 100077 128 0 0 25 0 5 0 634941781 1867132928 427280 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1823372 CPUtime=1000.54 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 805719 1252 0 0 99922 132 0 0 25 0 5 0 634941781 1867132928 427280 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1823372 CPUtime=1000.68 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 488796 1252 0 0 99986 82 0 0 25 0 5 0 634941781 1867132928 427280 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3995.1
Current children cumulated vsize (KiB) 1823372

[startup+1062.3 s]
/proc/loadavg: 9.57 8.78 8.15 11/366 2712
/proc/meminfo: memFree=18200196/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1711096 CPUtime=4232.81 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 3102102 1252 0 0 422780 501 0 0 25 0 5 0 634941773 1752162304 399211 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 427774 399211 180 55 0 425660 0
[pid=29229/tid=29236] ppid=29227 vsize=1711096 CPUtime=1050.92 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 854830 1252 0 0 104960 132 0 0 25 0 5 0 634941781 1752162304 399211 33554432000 4194304 4417452 140736221138208 18446744073709551615 4366616 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1711096 CPUtime=1061.92 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 865725 1252 0 0 106055 137 0 0 25 0 5 0 634941781 1752162304 399211 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1711096 CPUtime=1060.27 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 823598 1252 0 0 105891 136 0 0 25 0 5 0 634941781 1752162304 399211 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364741 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1711096 CPUtime=1059.61 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 553204 1252 0 0 105867 94 0 0 25 0 5 0 634941781 1752162304 399211 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4232.81
Current children cumulated vsize (KiB) 1711096

[startup+1122.3 s]
/proc/loadavg: 9.51 8.89 8.23 10/366 4463
/proc/meminfo: memFree=18196928/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1543040 CPUtime=4471.03 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 3221911 1252 0 0 446582 521 0 0 25 0 5 0 634941773 1580072960 357197 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 385760 357197 180 55 0 383646 0
[pid=29229/tid=29236] ppid=29227 vsize=1543040 CPUtime=1110.11 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 877618 1252 0 0 110873 138 0 0 25 0 5 0 634941781 1580072960 357197 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1543040 CPUtime=1121.83 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 876410 1252 0 0 112045 138 0 0 25 0 5 0 634941781 1580072960 357197 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1543040 CPUtime=1120.05 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 896749 1252 0 0 111859 146 0 0 25 0 5 0 634941781 1580072960 357197 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1543040 CPUtime=1118.93 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 566389 1252 0 0 111796 97 0 0 25 0 5 0 634941781 1580072960 357197 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4471.03
Current children cumulated vsize (KiB) 1543040

[startup+1182.3 s]
/proc/loadavg: 9.30 8.94 8.28 10/364 6131
/proc/meminfo: memFree=18027892/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1575900 CPUtime=4709.36 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 3291037 1252 0 0 470402 534 0 0 25 0 5 0 634941773 1613721600 365412 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 393975 365412 180 55 0 391861 0
[pid=29229/tid=29236] ppid=29227 vsize=1575900 CPUtime=1169.38 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 913548 1252 0 0 116794 144 0 0 25 0 5 0 634941781 1613721600 365412 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1575900 CPUtime=1181.77 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 885760 1252 0 0 118037 140 0 0 25 0 5 0 634941781 1613721600 365412 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1575900 CPUtime=1180.03 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 896749 1252 0 0 117857 146 0 0 25 0 5 0 634941781 1613721600 365412 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1575900 CPUtime=1178.07 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 590235 1252 0 0 117705 102 0 0 25 0 5 0 634941781 1613721600 365412 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4709.36
Current children cumulated vsize (KiB) 1575900



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.2 s]
/proc/loadavg: 9.33 8.98 8.31 10/365 6836
/proc/meminfo: memFree=17697592/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1691812 CPUtime=4800.34 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 3324792 1252 0 0 479494 540 0 0 25 0 5 0 634941773 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 422953 394390 180 55 0 420839 0
[pid=29229/tid=29236] ppid=29227 vsize=1691812 CPUtime=1192 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 933594 1252 0 0 119054 146 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1691812 CPUtime=1204.69 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 899469 1252 0 0 120326 143 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1691812 CPUtime=1202.94 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 896749 1252 0 0 120148 146 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1691812 CPUtime=1200.6 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 590235 1252 0 0 119958 102 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 1691812

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.2 s]
/proc/loadavg: 9.33 8.98 8.31 10/365 6836
/proc/meminfo: memFree=17697592/32951124 swapFree=67111348/67111528
[pid=29229] ppid=29227 vsize=1691812 CPUtime=4800.34 cores=1,3,5,7
/proc/29229/stat : 29229 (plingeling) S 29227 29229 27864 0 -1 4202496 3324792 1252 0 0 479494 540 0 0 25 0 5 0 634941773 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 3 0 0 0
/proc/29229/statm: 422953 394390 180 55 0 420839 0
[pid=29229/tid=29236] ppid=29227 vsize=1691812 CPUtime=1192 cores=1,3,5,7
/proc/29229/task/29236/stat : 29236 (plingeling) R 29227 29229 27864 0 -1 4202560 933594 1252 0 0 119054 146 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=29229/tid=29237] ppid=29227 vsize=1691812 CPUtime=1204.69 cores=1,3,5,7
/proc/29229/task/29237/stat : 29237 (plingeling) R 29227 29229 27864 0 -1 4202560 899469 1252 0 0 120326 143 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29229/tid=29238] ppid=29227 vsize=1691812 CPUtime=1202.94 cores=1,3,5,7
/proc/29229/task/29238/stat : 29238 (plingeling) R 29227 29229 27864 0 -1 4202560 896749 1252 0 0 120148 146 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=29229/tid=29239] ppid=29227 vsize=1691812 CPUtime=1200.6 cores=1,3,5,7
/proc/29229/task/29239/stat : 29239 (plingeling) R 29227 29229 27864 0 -1 4202560 590235 1252 0 0 119958 102 0 0 25 0 5 0 634941781 1732415488 394390 33554432000 4194304 4417452 140736221138208 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 1691812

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1205.29
CPU time (s): 4800.44
CPU user time (s): 4794.97
CPU system time (s): 5.47317
CPU usage (%): 398.281
Max. virtual memory (cumulated for all children) (KiB): 1947012

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4794.97
system time used= 5.47317
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3326044
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 111
involuntary context switches= 40470

runsolver used 3.9144 second user time and 9.94949 second system time

The end

Launcher Data 

Begin job on node144 at 2011-05-15 04:07:36
IDJOB=3370935
IDBENCH=83170
IDSOLVER=1848
FILE ID=node144/3370935-1305425256
RUNJOBID= node144-1305424050-28560
PBS_JOBID= 13324583
Free space on /tmp= 73256 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-7x39.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3370935-1305425256/watcher-3370935-1305425256 -o /tmp/evaluation-result-3370935-1305425256/solver-3370935-1305425256 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3370935-1305425256.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f717554c4e7f802b3078c48464dbcd9d
RANDOM SEED=984111158

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21556664 kB
Buffers:        293036 kB
Cached:        3627240 kB
SwapCached:          0 kB
Active:        8214944 kB
Inactive:      2833468 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21556664 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:             824 kB
Writeback:           0 kB
AnonPages:     7127200 kB
Mapped:          27008 kB
Slab:           244352 kB
PageTables:      36868 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 81972416 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73248 MiB
End job on node144 at 2011-05-15 04:27:44