Trace number 3251428

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587? (TO) 4800.37 1201.92

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m42-q663-n40-p30-s67876261.cnf
MD5SUMed0227339111fb337104188c725538d0
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1.24481
Satisfiable
(Un)Satisfiability was proved
Number of variables1634
Number of clauses212885
Sum of the clauses size427324
Maximum clause length42
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2212845
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251428-1304118165/watcher-3251428-1304118165 -o /tmp/evaluation-result-3251428-1304118165/solver-3251428-1304118165 -C 4800 -W 1300 -M 15500 plingeling -t 4 HOME/instance-3251428-1304118165.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.37 7.76 7.89 2/174 5327
/proc/meminfo: memFree=22828560/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=8728 CPUtime=0 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 216 0 0 0 0 0 0 0 25 0 1 0 523416167 8937472 146 33554432000 4194304 4417332 140733427726576 18446744073709551615 274473967360 0 0 4096 0 18446744071562362840 0 0 17 1 0 0 0
/proc/5327/statm: 2182 146 122 55 0 68 0

[startup+0.0885171 s]
/proc/loadavg: 6.37 7.76 7.89 2/174 5327
/proc/meminfo: memFree=22828560/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=17684 CPUtime=0.05 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) R 5325 5327 5295 0 -1 4202496 2562 1251 0 0 5 0 0 0 25 0 1 0 523416167 18108416 2443 33554432000 4194304 4417332 140733427726576 18446744073709551615 4231537 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/5327/statm: 4421 2443 147 55 0 2307 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 17684

[startup+0.100512 s]
/proc/loadavg: 6.37 7.76 7.89 2/174 5327
/proc/meminfo: memFree=22828560/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=17684 CPUtime=0.07 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) R 5325 5327 5295 0 -1 4202496 2562 1251 0 0 7 0 0 0 25 0 1 0 523416167 18108416 2443 33554432000 4194304 4417332 140733427726576 18446744073709551615 4259618 0 0 4096 17442 0 0 0 17 5 0 0 0
/proc/5327/statm: 4421 2443 147 55 0 2307 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 17684

[startup+0.30048 s]
/proc/loadavg: 6.37 7.76 7.89 2/174 5327
/proc/meminfo: memFree=22828560/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=69708 CPUtime=0.67 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 5803 1251 0 0 67 0 0 0 25 0 5 0 523416167 71380992 5241 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 17427 5241 175 55 0 15313 0
Current children cumulated CPU time (s) 0.67
Current children cumulated vsize (KiB) 69708

[startup+0.70041 s]
/proc/loadavg: 6.37 7.76 7.89 2/174 5327
/proc/meminfo: memFree=22828560/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=84752 CPUtime=2.27 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 17785 1251 0 0 226 1 0 0 25 0 5 0 523416167 86786048 8974 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 21188 8974 179 55 0 19074 0
Current children cumulated CPU time (s) 2.27
Current children cumulated vsize (KiB) 84752

[startup+1.50027 s]
/proc/loadavg: 6.37 7.76 7.89 9/186 5352
/proc/meminfo: memFree=22727140/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=102960 CPUtime=5.46 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 26385 1251 0 0 544 2 0 0 25 0 5 0 523416167 105431040 13527 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 25740 13527 180 55 0 23626 0
[pid=5327/tid=5345] ppid=5325 vsize=102960 CPUtime=1.33 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 4237 1251 0 0 133 0 0 0 25 0 5 0 523416183 105431040 13527 33554432000 4194304 4417332 140733427726576 18446744073709551615 4250072 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=102960 CPUtime=1.32 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 3508 1251 0 0 132 0 0 0 25 0 5 0 523416183 105431040 13527 33554432000 4194304 4417332 140733427726576 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=102960 CPUtime=1.32 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 4308 1251 0 0 132 0 0 0 25 0 5 0 523416183 105431040 13527 33554432000 4194304 4417332 140733427726576 18446744073709551615 4250022 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=102960 CPUtime=1.33 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 9710 1251 0 0 132 1 0 0 25 0 5 0 523416183 105431040 13527 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364541 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.46
Current children cumulated vsize (KiB) 102960

[startup+3.10099 s]
/proc/loadavg: 6.58 7.78 7.89 9/186 5352
/proc/meminfo: memFree=22702848/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=116956 CPUtime=11.86 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 34102 1251 0 0 1183 3 0 0 25 0 5 0 523416167 119762944 17026 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 29239 17026 180 55 0 27125 0
[pid=5327/tid=5345] ppid=5325 vsize=116956 CPUtime=2.93 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 4237 1251 0 0 293 0 0 0 25 0 5 0 523416183 119762944 17026 33554432000 4194304 4417332 140733427726576 18446744073709551615 4248897 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=116956 CPUtime=2.91 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 3508 1251 0 0 291 0 0 0 25 0 5 0 523416183 119762944 17026 33554432000 4194304 4417332 140733427726576 18446744073709551615 4258401 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=116956 CPUtime=2.93 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 4308 1251 0 0 293 0 0 0 25 0 5 0 523416183 119762944 17026 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364732 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=116956 CPUtime=2.93 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 17427 1251 0 0 291 2 0 0 25 0 5 0 523416183 119762944 17026 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364800 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.86
Current children cumulated vsize (KiB) 116956

[startup+6.30043 s]
/proc/loadavg: 6.58 7.78 7.89 9/186 5353
/proc/meminfo: memFree=22638624/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=148100 CPUtime=24.64 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 49675 1251 0 0 2458 6 0 0 25 0 5 0 523416167 151654400 24301 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 37025 24301 180 55 0 34911 0
[pid=5327/tid=5345] ppid=5325 vsize=148100 CPUtime=6.13 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 4630 1251 0 0 613 0 0 0 25 0 5 0 523416183 151654400 24301 33554432000 4194304 4417332 140733427726576 18446744073709551615 4239395 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=148100 CPUtime=6.1 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 4345 1251 0 0 610 0 0 0 25 0 5 0 523416183 151654400 24301 33554432000 4194304 4417332 140733427726576 18446744073709551615 4236319 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=148100 CPUtime=6.13 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 4679 1251 0 0 613 0 0 0 25 0 5 0 523416183 151654400 24301 33554432000 4194304 4417332 140733427726576 18446744073709551615 4336643 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=148100 CPUtime=6.13 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 31399 1251 0 0 609 4 0 0 25 0 5 0 523416183 151654400 24301 33554432000 4194304 4417332 140733427726576 18446744073709551615 4321685 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.64
Current children cumulated vsize (KiB) 148100

[startup+12.7003 s]
/proc/loadavg: 6.80 7.79 7.89 9/186 5353
/proc/meminfo: memFree=22554444/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=215448 CPUtime=50.2 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 73108 1251 0 0 5011 9 0 0 25 0 5 0 523416167 220618752 41649 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 53862 41649 180 55 0 51748 0
[pid=5327/tid=5345] ppid=5325 vsize=215448 CPUtime=12.53 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 4630 1251 0 0 1253 0 0 0 25 0 5 0 523416183 220618752 41649 33554432000 4194304 4417332 140733427726576 18446744073709551615 4248198 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=215448 CPUtime=12.46 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 4345 1251 0 0 1246 0 0 0 25 0 5 0 523416183 220618752 41649 33554432000 4194304 4417332 140733427726576 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=215448 CPUtime=12.52 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 5886 1251 0 0 1252 0 0 0 25 0 5 0 523416183 220618752 41649 33554432000 4194304 4417332 140733427726576 18446744073709551615 4303811 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=215448 CPUtime=12.53 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 53625 1251 0 0 1246 7 0 0 25 0 5 0 523416183 220618752 41649 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364568 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50.2
Current children cumulated vsize (KiB) 215448

[startup+25.501 s]
/proc/loadavg: 6.98 7.79 7.89 9/186 5353
/proc/meminfo: memFree=22454008/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=275552 CPUtime=101.34 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 104745 1251 0 0 10119 15 0 0 25 0 5 0 523416167 282165248 40292 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 68888 40292 180 55 0 66774 0
[pid=5327/tid=5345] ppid=5325 vsize=275552 CPUtime=25.33 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 6112 1251 0 0 2533 0 0 0 25 0 5 0 523416183 282165248 40292 33554432000 4194304 4417332 140733427726576 18446744073709551615 4237764 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=275552 CPUtime=25.19 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 4345 1251 0 0 2519 0 0 0 25 0 5 0 523416183 282165248 40292 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364565 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=275552 CPUtime=25.32 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 5886 1251 0 0 2532 0 0 0 25 0 5 0 523416183 282165248 40292 33554432000 4194304 4417332 140733427726576 18446744073709551615 4306349 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=275552 CPUtime=25.33 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 83780 1251 0 0 2520 13 0 0 25 0 5 0 523416183 282165248 40292 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364680 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 101.34
Current children cumulated vsize (KiB) 275552

[startup+51.1005 s]
/proc/loadavg: 7.47 7.84 7.91 9/186 5354
/proc/meminfo: memFree=22206080/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=381080 CPUtime=203.6 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 151041 1251 0 0 20338 22 0 0 25 0 5 0 523416167 390225920 66638 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 95270 66638 181 55 0 93156 0
[pid=5327/tid=5345] ppid=5325 vsize=381080 CPUtime=50.92 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 6113 1251 0 0 5092 0 0 0 25 0 5 0 523416183 390225920 66638 33554432000 4194304 4417332 140733427726576 18446744073709551615 4249143 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=381080 CPUtime=50.65 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 4345 1251 0 0 5065 0 0 0 25 0 5 0 523416183 390225920 66638 33554432000 4194304 4417332 140733427726576 18446744073709551615 4249222 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=381080 CPUtime=50.92 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 5886 1251 0 0 5092 0 0 0 25 0 5 0 523416183 390225920 66638 33554432000 4194304 4417332 140733427726576 18446744073709551615 4357133 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=381080 CPUtime=50.92 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 130075 1251 0 0 5073 19 0 0 25 0 5 0 523416183 390225920 66638 33554432000 4194304 4417332 140733427726576 18446744073709551615 4366297 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.6
Current children cumulated vsize (KiB) 381080

[startup+102.307 s]
/proc/loadavg: 8.05 7.93 7.93 9/186 5358
/proc/meminfo: memFree=21813328/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=586212 CPUtime=408.12 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 259795 1251 0 0 40776 36 0 0 25 0 5 0 523416167 600281088 117997 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 146553 117997 181 55 0 144439 0
[pid=5327/tid=5345] ppid=5325 vsize=586212 CPUtime=102.13 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 16193 1251 0 0 10212 1 0 0 25 0 5 0 523416183 600281088 117997 33554432000 4194304 4417332 140733427726576 18446744073709551615 4357164 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=586212 CPUtime=101.57 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 14974 1251 0 0 10156 1 0 0 25 0 5 0 523416183 600281088 117997 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=586212 CPUtime=102.13 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 16053 1251 0 0 10211 2 0 0 25 0 5 0 523416183 600281088 117997 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=586212 CPUtime=102.14 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 207953 1251 0 0 10183 31 0 0 25 0 5 0 523416183 600281088 117997 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364656 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 408.12
Current children cumulated vsize (KiB) 586212

[startup+162.301 s]
/proc/loadavg: 8.09 7.96 7.94 9/186 5360
/proc/meminfo: memFree=21676092/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=611480 CPUtime=647.78 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 379357 1251 0 0 64723 55 0 0 25 0 5 0 523416167 626155520 124339 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 152870 124339 181 55 0 150756 0
[pid=5327/tid=5345] ppid=5325 vsize=611480 CPUtime=162.12 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 39130 1251 0 0 16208 4 0 0 25 0 5 0 523416183 626155520 124339 33554432000 4194304 4417332 140733427726576 18446744073709551615 4366384 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=611480 CPUtime=161.24 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 28261 1251 0 0 16121 3 0 0 25 0 5 0 523416183 626155520 124339 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364463 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=611480 CPUtime=162.12 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 48711 1251 0 0 16207 5 0 0 25 0 5 0 523416183 626155520 124339 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=611480 CPUtime=162.13 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 258633 1251 0 0 16172 41 0 0 25 0 5 0 523416183 626155520 124339 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364473 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 647.78
Current children cumulated vsize (KiB) 611480


################
# More data... #
################

/proc/meminfo: memFree=19257096/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=1510096 CPUtime=3044.58 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 1444014 1251 0 0 304236 222 0 0 25 0 5 0 523416167 1546338304 348993 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 377524 348993 181 55 0 375410 0
[pid=5327/tid=5345] ppid=5325 vsize=1510096 CPUtime=762.14 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 246965 1251 0 0 76179 35 0 0 25 0 5 0 523416183 1546338304 348993 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=1510096 CPUtime=757.99 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 298533 1251 0 0 75758 41 0 0 25 0 5 0 523416183 1546338304 348993 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=1510096 CPUtime=762.14 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 255215 1251 0 0 76180 34 0 0 25 0 5 0 523416183 1546338304 348993 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=1510096 CPUtime=762.15 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 638679 1251 0 0 76105 110 0 0 25 0 5 0 523416183 1546338304 348993 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364509 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3044.58
Current children cumulated vsize (KiB) 1510096

[startup+822.301 s]
/proc/loadavg: 8.06 8.09 8.01 9/186 5377
/proc/meminfo: memFree=19224936/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=1712392 CPUtime=3284.24 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 1526311 1251 0 0 328190 234 0 0 25 0 5 0 523416167 1753489408 399567 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 428098 399567 181 55 0 425984 0
[pid=5327/tid=5345] ppid=5325 vsize=1712392 CPUtime=822.13 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 269721 1251 0 0 82174 39 0 0 25 0 5 0 523416183 1753489408 399567 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364742 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=1712392 CPUtime=817.66 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 300381 1251 0 0 81725 41 0 0 25 0 5 0 523416183 1753489408 399567 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=1712392 CPUtime=822.13 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 271599 1251 0 0 82178 35 0 0 25 0 5 0 523416183 1753489408 399567 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364750 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=1712392 CPUtime=822.14 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 679988 1251 0 0 82098 116 0 0 25 0 5 0 523416183 1753489408 399567 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3284.24
Current children cumulated vsize (KiB) 1712392

[startup+882.3 s]
/proc/loadavg: 8.08 8.10 8.02 9/186 5379
/proc/meminfo: memFree=19044596/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=1812400 CPUtime=3523.92 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 1607650 1251 0 0 352145 247 0 0 25 0 5 0 523416167 1855897600 424569 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 453100 424569 181 55 0 450986 0
[pid=5327/tid=5345] ppid=5325 vsize=1812400 CPUtime=882.14 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 302489 1251 0 0 88170 44 0 0 25 0 5 0 523416183 1855897600 424569 33554432000 4194304 4417332 140733427726576 18446744073709551615 18446744073699065939 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=1812400 CPUtime=877.33 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 340760 1251 0 0 87685 48 0 0 25 0 5 0 523416183 1855897600 424569 33554432000 4194304 4417332 140733427726576 18446744073709551615 4366042 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=1812400 CPUtime=882.13 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 279791 1251 0 0 88177 36 0 0 25 0 5 0 523416183 1855897600 424569 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=1812400 CPUtime=882.14 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 679988 1251 0 0 88098 116 0 0 25 0 5 0 523416183 1855897600 424569 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3523.92
Current children cumulated vsize (KiB) 1812400

[startup+942.301 s]
/proc/loadavg: 8.07 8.10 8.02 9/186 5380
/proc/meminfo: memFree=18538760/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=2087776 CPUtime=3763.6 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 1762330 1251 0 0 376095 265 0 0 25 0 5 0 523416167 2137882624 493381 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 521944 493381 181 55 0 519830 0
[pid=5327/tid=5345] ppid=5325 vsize=2087776 CPUtime=942.14 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 378601 1251 0 0 94160 54 0 0 25 0 5 0 523416183 2137882624 493381 33554432000 4194304 4417332 140733427726576 18446744073709551615 4365960 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=2087776 CPUtime=937.01 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 357144 1251 0 0 93651 50 0 0 25 0 5 0 523416183 2137882624 493381 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=2087776 CPUtime=942.14 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 287983 1251 0 0 94177 37 0 0 25 0 5 0 523416183 2137882624 493381 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364544 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=2087776 CPUtime=942.14 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 733980 1251 0 0 94092 122 0 0 25 0 5 0 523416183 2137882624 493381 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3763.6
Current children cumulated vsize (KiB) 2087776

[startup+1002.3 s]
/proc/loadavg: 8.12 8.10 8.02 9/186 5382
/proc/meminfo: memFree=17816056/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=2385020 CPUtime=4003.27 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 1869409 1251 0 0 400046 281 0 0 25 0 5 0 523416167 2442260480 567692 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 596255 567692 181 55 0 594141 0
[pid=5327/tid=5345] ppid=5325 vsize=2385020 CPUtime=1002.14 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 394985 1251 0 0 100159 55 0 0 25 0 5 0 523416183 2442260480 567692 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364494 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=2385020 CPUtime=996.68 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 389690 1251 0 0 99614 54 0 0 25 0 5 0 523416183 2442260480 567692 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=2385020 CPUtime=1002.14 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 312543 1251 0 0 100172 42 0 0 25 0 5 0 523416183 2442260480 567692 33554432000 4194304 4417332 140733427726576 18446744073709551615 4365007 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=2385020 CPUtime=1002.14 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 767569 1251 0 0 100086 128 0 0 25 0 5 0 523416183 2442260480 567692 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364526 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4003.27
Current children cumulated vsize (KiB) 2385020

[startup+1062.3 s]
/proc/loadavg: 8.12 8.10 8.02 9/186 5384
/proc/meminfo: memFree=17518532/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=2373144 CPUtime=4242.95 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 1939712 1251 0 0 424002 293 0 0 25 0 5 0 523416167 2430099456 564723 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 593286 564723 181 55 0 591172 0
[pid=5327/tid=5345] ppid=5325 vsize=2373144 CPUtime=1062.14 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 394985 1251 0 0 106159 55 0 0 25 0 5 0 523416183 2430099456 564723 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364512 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=2373144 CPUtime=1056.36 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 454963 1251 0 0 105571 65 0 0 25 0 5 0 523416183 2430099456 564723 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=2373144 CPUtime=1062.14 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 317573 1251 0 0 106171 43 0 0 25 0 5 0 523416183 2430099456 564723 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364822 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=2373144 CPUtime=1062.14 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 767569 1251 0 0 106086 128 0 0 25 0 5 0 523416183 2430099456 564723 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364470 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4242.95
Current children cumulated vsize (KiB) 2373144

[startup+1122.3 s]
/proc/loadavg: 8.10 8.09 8.02 9/186 5385
/proc/meminfo: memFree=16956036/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=2495488 CPUtime=4482.64 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 1991733 1251 0 0 447960 304 0 0 25 0 5 0 523416167 2555379712 595309 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 623872 595309 181 55 0 621758 0
[pid=5327/tid=5345] ppid=5325 vsize=2495488 CPUtime=1122.15 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 403121 1251 0 0 112158 57 0 0 25 0 5 0 523416183 2555379712 595309 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364560 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=2495488 CPUtime=1116.03 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 454963 1251 0 0 111538 65 0 0 25 0 5 0 523416183 2555379712 595309 33554432000 4194304 4417332 140733427726576 18446744073709551615 4365963 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=2495488 CPUtime=1122.15 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 337778 1251 0 0 112168 47 0 0 25 0 5 0 523416183 2555379712 595309 33554432000 4194304 4417332 140733427726576 18446744073709551615 4310248 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=2495488 CPUtime=1122.15 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 791249 1251 0 0 112082 133 0 0 25 0 5 0 523416183 2555379712 595309 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364544 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4482.64
Current children cumulated vsize (KiB) 2495488

[startup+1182.3 s]
/proc/loadavg: 8.10 8.09 8.02 9/186 5387
/proc/meminfo: memFree=16445736/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=2651976 CPUtime=4722.31 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 2080757 1251 0 0 471913 318 0 0 25 0 5 0 523416167 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 662994 634431 181 55 0 660880 0
[pid=5327/tid=5345] ppid=5325 vsize=2651976 CPUtime=1182.15 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 403121 1251 0 0 118158 57 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364519 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=2651976 CPUtime=1175.71 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 521783 1251 0 0 117495 76 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4250505 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=2651976 CPUtime=1182.14 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 359982 1251 0 0 118164 50 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=2651976 CPUtime=1182.15 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 791249 1251 0 0 118082 133 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364736 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4722.31
Current children cumulated vsize (KiB) 2651976



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.8 s]
/proc/loadavg: 8.07 8.08 8.02 9/186 5387
/proc/meminfo: memFree=16342548/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=2651976 CPUtime=4800.22 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) S 5325 5327 5295 0 -1 4202496 2080757 1251 0 0 479704 318 0 0 25 0 5 0 523416167 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 274485770213 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 662994 634431 181 55 0 660880 0
[pid=5327/tid=5345] ppid=5325 vsize=2651976 CPUtime=1201.65 cores=1,3,5,7
/proc/5327/task/5345/stat : 5345 (plingeling) R 5325 5327 5295 0 -1 4202560 403121 1251 0 0 120108 57 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4263227 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=2651976 CPUtime=1195.1 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4202560 521783 1251 0 0 119434 76 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364536 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5327/tid=5347] ppid=5325 vsize=2651976 CPUtime=1201.65 cores=1,3,5,7
/proc/5327/task/5347/stat : 5347 (plingeling) R 5325 5327 5295 0 -1 4202560 359982 1251 0 0 120115 50 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5327/tid=5348] ppid=5325 vsize=2651976 CPUtime=1201.66 cores=1,3,5,7
/proc/5327/task/5348/stat : 5348 (plingeling) R 5325 5327 5295 0 -1 4202560 791249 1251 0 0 120033 133 0 0 25 0 5 0 523416183 2715623424 634431 33554432000 4194304 4417332 140733427726576 18446744073709551615 4364642 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 2651976

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.91 s]
/proc/loadavg: 8.07 8.08 8.02 9/186 5387
/proc/meminfo: memFree=16342548/32950928 swapFree=67111524/67111528
[pid=5327] ppid=5325 vsize=0 CPUtime=4800.35 cores=1,3,5,7
/proc/5327/stat : 5327 (plingeling) Z 5325 5327 5295 0 -1 4203548 2080757 1251 0 0 479707 328 0 0 18 0 2 0 523416167 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/5327/statm: 0 0 0 0 0 0 0
[pid=5327/tid=5346] ppid=5325 vsize=0 CPUtime=1195.21 cores=1,3,5,7
/proc/5327/task/5346/stat : 5346 (plingeling) R 5325 5327 5295 0 -1 4203588 521783 1251 0 0 119435 86 0 0 25 0 2 0 523416183 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.35
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.92
CPU time (s): 4800.37
CPU user time (s): 4797.07
CPU system time (s): 3.2955
CPU usage (%): 399.391
Max. virtual memory (cumulated for all children) (KiB): 2651976

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4797.07
system time used= 3.2955
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2082008
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 72
involuntary context switches= 35229

runsolver used 2.03469 second user time and 4.52631 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-30 01:02:45
IDJOB=3251428
IDBENCH=88222
IDSOLVER=1605
FILE ID=node125/3251428-1304118165
RUNJOBID= node125-1304118165-5314
PBS_JOBID= 13170720
Free space on /tmp= 72760 MiB

SOLVER NAME= Plingeling 587
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m42-q663-n40-p30-s67876261.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251428-1304118165/watcher-3251428-1304118165 -o /tmp/evaluation-result-3251428-1304118165/solver-3251428-1304118165 -C 4800 -W 1300 -M 15500  plingeling -t 4 HOME/instance-3251428-1304118165.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ed0227339111fb337104188c725538d0
RANDOM SEED=1714915801

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      22829088 kB
Buffers:       2124540 kB
Cached:        7404072 kB
SwapCached:          4 kB
Active:        2123324 kB
Inactive:      7472340 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      22829088 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            5604 kB
Writeback:           0 kB
AnonPages:       67244 kB
Mapped:          14388 kB
Slab:           460168 kB
PageTables:       5032 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   187772 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72756 MiB
End job on node125 at 2011-04-30 01:22:49