Trace number 3380671

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.4 5017.17

General information on the benchmark

NameAPPLICATIONS/satComp09_BioInstances/
ndhf_xits_19_UNKNOWN.cnf
MD5SUM4cf2a693df1b6e5a084b88e84ec5f2af
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark13.4859
Satisfiable
(Un)Satisfiability was proved
Number of variables4020
Number of clauses466486
Sum of the clauses size1393798
Maximum clause length20
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 26187
Number of clauses of size 3460268
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 531

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...486622/instance-3380671-1305486622.cnf
10.48/10.53	c Solving...
40000.08/5015.91	
40000.08/5015.91	*** INTERRUPTED! ***
40000.08/5015.91	*** Shutting down threads...
40001.33/5017.13	s UNKNOWN
40001.33/5017.13	
40001.33/5017.13	c INTERRUPTED   : 1
40001.33/5017.13	c Models        : 0+    
40001.33/5017.13	c Time          : 5015.916s (Solving: 5005.39s 1st Model: 0.00s Unsat: 0.00s)
40001.33/5017.13	c CPU Time      : 40000.130s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380671-1305486622/watcher-3380671-1305486622 -o /tmp/evaluation-result-3380671-1305486622/solver-3380671-1305486622 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380671-1305486622.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.40 5.65 6.65 2/295 14211
/proc/meminfo: memFree=24438352/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=22932 CPUtime=0 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1091 0 0 0 0 0 0 0 21 0 1 0 738892797 23482368 954 33554432000 4194304 4964800 140735694090576 18446744073709551615 260020056773 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14211/statm: 5733 956 386 189 0 607 0

[startup+0.109118 s]
/proc/loadavg: 7.40 5.65 6.65 2/295 14211
/proc/meminfo: memFree=24438352/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=44052 CPUtime=0.1 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 7981 0 0 0 9 1 0 0 22 0 1 0 738892797 45109248 6122 33554432000 4194304 4964800 140735694090576 18446744073709551615 260020056773 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14211/statm: 11013 6124 386 189 0 5887 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 44052

[startup+0.201118 s]
/proc/loadavg: 7.40 5.65 6.65 2/295 14211
/proc/meminfo: memFree=24438352/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=58408 CPUtime=0.19 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 11748 0 0 0 18 1 0 0 22 0 1 0 738892797 59809792 9851 33554432000 4194304 4964800 140735694090576 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14211/statm: 14602 9851 399 189 0 9476 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 58408

[startup+0.3011 s]
/proc/loadavg: 7.40 5.65 6.65 2/295 14211
/proc/meminfo: memFree=24438352/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=58408 CPUtime=0.29 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 11748 0 0 0 28 1 0 0 22 0 1 0 738892797 59809792 9851 33554432000 4194304 4964800 140735694090576 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14211/statm: 14602 9851 399 189 0 9476 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 58408

[startup+0.701007 s]
/proc/loadavg: 7.40 5.65 6.65 2/295 14211
/proc/meminfo: memFree=24438352/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=58408 CPUtime=0.69 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 11748 0 0 0 68 1 0 0 23 0 1 0 738892797 59809792 9851 33554432000 4194304 4964800 140735694090576 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14211/statm: 14602 9851 399 189 0 9476 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 58408

[startup+1.50086 s]
/proc/loadavg: 7.40 5.65 6.65 2/296 14212
/proc/meminfo: memFree=24400904/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=58408 CPUtime=1.49 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 11748 0 0 0 148 1 0 0 25 0 1 0 738892797 59809792 9851 33554432000 4194304 4964800 140735694090576 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14211/statm: 14602 9851 399 189 0 9476 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 58408

[startup+3.10057 s]
/proc/loadavg: 7.40 5.65 6.65 2/296 14212
/proc/meminfo: memFree=24400284/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=58408 CPUtime=3.09 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 11748 0 0 0 308 1 0 0 25 0 1 0 738892797 59809792 9851 33554432000 4194304 4964800 140735694090576 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/14211/statm: 14602 9851 399 189 0 9476 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 58408

[startup+6.30098 s]
/proc/loadavg: 6.89 5.57 6.62 2/296 14212
/proc/meminfo: memFree=24396368/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=63204 CPUtime=6.29 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 12939 0 0 0 628 1 0 0 25 0 1 0 738892797 64720896 11042 33554432000 4194304 4964800 140735694090576 18446744073709551615 4600362 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14211/statm: 15801 11042 400 189 0 10675 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 63204

[startup+12.7008 s]
/proc/loadavg: 6.42 5.50 6.59 9/303 14220
/proc/meminfo: memFree=24333024/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=495404 CPUtime=26.95 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 31673 0 0 0 2685 10 0 0 21 0 8 0 738892797 507293696 28830 33554432000 4194304 4964800 140735694090576 18446744073709551615 4668753 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 123851 28831 464 189 0 118200 0
[pid=14211/tid=14213] ppid=14209 vsize=495404 CPUtime=2.06 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 2261 0 0 0 205 1 0 0 19 0 8 0 738893850 507293696 28831 33554432000 4194304 4964800 140735694090576 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=495404 CPUtime=2.02 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 2730 0 0 0 201 1 0 0 19 0 8 0 738893850 507293696 28831 33554432000 4194304 4964800 140735694090576 18446744073709551615 260020057121 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=495404 CPUtime=2.03 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 1941 0 0 0 202 1 0 0 19 0 8 0 738893850 507293696 28831 33554432000 4194304 4964800 140735694090576 18446744073709551615 4660835 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=495404 CPUtime=2.02 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 2288 0 0 0 201 1 0 0 19 0 8 0 738893850 507293696 28831 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=495404 CPUtime=2.14 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 1135 0 0 0 214 0 0 0 21 0 8 0 738893850 507293696 28831 33554432000 4194304 4964800 140735694090576 18446744073709551615 4668672 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=495404 CPUtime=2.03 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 1866 0 0 0 203 0 0 0 19 0 8 0 738893850 507293696 28831 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627269 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=495404 CPUtime=2.03 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 2189 0 0 0 202 1 0 0 19 0 8 0 738893850 507293696 28831 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 26.95
Current children cumulated vsize (KiB) 495404

[startup+25.5004 s]
/proc/loadavg: 6.77 5.62 6.61 9/303 14220
/proc/meminfo: memFree=24110248/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=513344 CPUtime=129.22 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 86417 0 0 0 12884 38 0 0 25 0 8 0 738892797 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 128336 83574 466 189 0 122685 0
[pid=14211/tid=14213] ppid=14209 vsize=513344 CPUtime=14.84 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 7574 0 0 0 1481 3 0 0 25 0 8 0 738893850 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627309 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=513344 CPUtime=14.82 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 13783 0 0 0 1476 6 0 0 25 0 8 0 738893850 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=513344 CPUtime=14.7 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 7219 0 0 0 1468 2 0 0 25 0 8 0 738893850 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=513344 CPUtime=14.83 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 7819 0 0 0 1477 6 0 0 25 0 8 0 738893850 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4660110 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=513344 CPUtime=14.94 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 9400 0 0 0 1490 4 0 0 25 0 8 0 738893850 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=513344 CPUtime=14.83 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 7552 0 0 0 1480 3 0 0 25 0 8 0 738893850 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=513344 CPUtime=14.82 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 9134 0 0 0 1478 4 0 0 25 0 8 0 738893850 525664256 83574 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 129.22
Current children cumulated vsize (KiB) 513344

[startup+51.1007 s]
/proc/loadavg: 7.19 5.81 6.64 9/303 14221
/proc/meminfo: memFree=23783024/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=953848 CPUtime=333.76 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 167543 0 0 0 33285 91 0 0 25 0 8 0 738892797 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670340 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 238462 164700 466 189 0 232811 0
[pid=14211/tid=14213] ppid=14209 vsize=953848 CPUtime=40.45 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 19042 0 0 0 4033 12 0 0 25 0 8 0 738893850 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=953848 CPUtime=40.41 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 27832 0 0 0 4026 15 0 0 25 0 8 0 738893850 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627987 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=953848 CPUtime=40.06 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 15135 0 0 0 4000 6 0 0 25 0 8 0 738893850 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=953848 CPUtime=40.42 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 18021 0 0 0 4030 12 0 0 25 0 8 0 738893850 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=953848 CPUtime=40.53 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 18524 0 0 0 4042 11 0 0 25 0 8 0 738893850 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4320848 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=953848 CPUtime=40.43 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 13277 0 0 0 4036 7 0 0 25 0 8 0 738893850 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=953848 CPUtime=40.42 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 20632 0 0 0 4030 12 0 0 25 0 8 0 738893850 976740352 164700 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 333.76
Current children cumulated vsize (KiB) 953848

[startup+102.301 s]
/proc/loadavg: 7.65 6.14 6.71 9/303 14222
/proc/meminfo: memFree=23585704/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=987412 CPUtime=742.89 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 216552 0 0 0 74160 129 0 0 25 0 8 0 738892797 1011109888 213557 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 246853 213557 466 189 0 241202 0
[pid=14211/tid=14213] ppid=14209 vsize=987412 CPUtime=91.64 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 25189 0 0 0 9147 17 0 0 25 0 8 0 738893850 1011109888 213558 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=987412 CPUtime=91.62 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 37749 0 0 0 9141 21 0 0 25 0 8 0 738893850 1011109888 213558 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=987412 CPUtime=90.8 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 18193 0 0 0 9072 8 0 0 25 0 8 0 738893850 1011109888 213558 33554432000 4194304 4964800 140735694090576 18446744073709551615 4320984 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=987412 CPUtime=91.62 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 21358 0 0 0 9148 14 0 0 25 0 8 0 738893850 1011109888 213558 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=987412 CPUtime=91.74 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 25280 0 0 0 9158 16 0 0 25 0 8 0 738893850 1011109888 213558 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=987412 CPUtime=91.62 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 17242 0 0 0 9152 10 0 0 25 0 8 0 738893850 1011109888 213559 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=987412 CPUtime=91.63 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 30336 0 0 0 9142 21 0 0 25 0 8 0 738893850 1011109888 213559 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 742.89
Current children cumulated vsize (KiB) 987412

[startup+162.301 s]
/proc/loadavg: 7.87 6.48 6.79 9/303 14224
/proc/meminfo: memFree=23362204/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=1349908 CPUtime=1222.23 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 273178 0 0 0 122050 173 0 0 25 0 8 0 738892797 1382305792 269924 33554432000 4194304 4964800 140735694090576 18446744073709551615 260020057121 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 337477 269924 466 189 0 331826 0
[pid=14211/tid=14213] ppid=14209 vsize=1349908 CPUtime=151.62 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 34213 0 0 0 15136 26 0 0 25 0 8 0 738893850 1382305792 269924 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=1349908 CPUtime=151.6 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 46190 0 0 0 15134 26 0 0 25 0 8 0 738893850 1382305792 269924 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=1349908 CPUtime=150.23 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 22563 0 0 0 15012 11 0 0 25 0 8 0 738893850 1382305792 269924 33554432000 4194304 4964800 140735694090576 18446744073709551615 4321069 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=1349908 CPUtime=151.61 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 29100 0 0 0 15141 20 0 0 25 0 8 0 738893850 1382305792 269924 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=1349908 CPUtime=151.73 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 29442 0 0 0 15155 18 0 0 25 0 8 0 738893850 1382305792 269924 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=1349908 CPUtime=151.61 cores=0-7

################
# More data... #
################

/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 108514 0 0 0 464976 112 0 0 25 0 8 0 738893850 5060677632 1138425 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=4942068 CPUtime=4651.43 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 221321 0 0 0 464894 249 0 0 25 0 8 0 738893850 5060677632 1138425 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37174.9
Current children cumulated vsize (KiB) 4942068

[startup+4722.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/303 14359
/proc/meminfo: memFree=19840636/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=4942068 CPUtime=37654.4 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1162620 0 0 0 3764230 1211 0 0 25 0 8 0 738892797 5060677632 1147649 33554432000 4194304 4964800 140735694090576 18446744073709551615 4668712 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 1235517 1147649 466 189 0 1229866 0
[pid=14211/tid=14213] ppid=14209 vsize=4942068 CPUtime=4706.73 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 164119 0 0 0 470472 201 0 0 25 0 8 0 738893850 5060677632 1147650 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=4942068 CPUtime=4710.71 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 136684 0 0 0 470981 90 0 0 25 0 8 0 738893850 5060677632 1147650 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=4942068 CPUtime=4690.3 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 103012 0 0 0 468928 102 0 0 25 0 8 0 738893850 5060677632 1147650 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=4942068 CPUtime=4690.96 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 114645 0 0 0 468971 125 0 0 25 0 8 0 738893850 5060677632 1147650 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=4942068 CPUtime=4711.74 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 115146 0 0 0 471084 90 0 0 25 0 8 0 738893850 5060677632 1147650 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=4942068 CPUtime=4710.89 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 110002 0 0 0 470974 115 0 0 25 0 8 0 738893850 5060677632 1147650 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=4942068 CPUtime=4711.44 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 223223 0 0 0 470891 253 0 0 25 0 8 0 738893850 5060677632 1147650 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37654.4
Current children cumulated vsize (KiB) 4942068

[startup+4782.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/303 14361
/proc/meminfo: memFree=19809472/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=4946304 CPUtime=38133.7 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1170521 0 0 0 3812145 1225 0 0 25 0 8 0 738892797 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4668712 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 1236576 1155426 466 189 0 1230925 0
[pid=14211/tid=14213] ppid=14209 vsize=4946304 CPUtime=4766.72 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 165253 0 0 0 476471 201 0 0 25 0 8 0 738893850 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4653512 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=4946304 CPUtime=4770.7 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 136923 0 0 0 476980 90 0 0 25 0 8 0 738893850 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=4946304 CPUtime=4749.7 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 103135 0 0 0 474867 103 0 0 25 0 8 0 738893850 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=4946304 CPUtime=4750.94 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 114655 0 0 0 474969 125 0 0 25 0 8 0 738893850 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=4946304 CPUtime=4771.69 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 115199 0 0 0 477078 91 0 0 25 0 8 0 738893850 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4644886 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=4946304 CPUtime=4770.88 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 110040 0 0 0 476973 115 0 0 25 0 8 0 738893850 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=4946304 CPUtime=4771.43 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 223233 0 0 0 476890 253 0 0 25 0 8 0 738893850 5065015296 1155426 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38133.7
Current children cumulated vsize (KiB) 4946304

[startup+4842.31 s]
/proc/loadavg: 8.07 8.05 8.00 9/303 14363
/proc/meminfo: memFree=19807680/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=4946304 CPUtime=38613.1 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1171291 0 0 0 3860083 1228 0 0 25 0 8 0 738892797 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627601 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 1236576 1155647 466 189 0 1230925 0
[pid=14211/tid=14213] ppid=14209 vsize=4946304 CPUtime=4826.74 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 165778 0 0 0 482472 202 0 0 25 0 8 0 738893850 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4321097 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=4946304 CPUtime=4830.71 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 136935 0 0 0 482981 90 0 0 25 0 8 0 738893850 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=4946304 CPUtime=4809.11 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 103207 0 0 0 480808 103 0 0 25 0 8 0 738893850 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=4946304 CPUtime=4810.94 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 114655 0 0 0 480969 125 0 0 25 0 8 0 738893850 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=4946304 CPUtime=4831.65 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 115348 0 0 0 483073 92 0 0 25 0 8 0 738893850 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=4946304 CPUtime=4830.89 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 110051 0 0 0 482974 115 0 0 25 0 8 0 738893850 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=4946304 CPUtime=4831.43 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 223233 0 0 0 482889 254 0 0 25 0 8 0 738893850 5065015296 1155647 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38613.1
Current children cumulated vsize (KiB) 4946304

[startup+4902.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/303 14364
/proc/meminfo: memFree=19803836/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=4946304 CPUtime=39092.4 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1172507 0 0 0 3908010 1230 0 0 25 0 8 0 738892797 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4320981 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 1236576 1156800 466 189 0 1230925 0
[pid=14211/tid=14213] ppid=14209 vsize=4946304 CPUtime=4886.72 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 166662 0 0 0 488470 202 0 0 25 0 8 0 738893850 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4644912 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=4946304 CPUtime=4890.7 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 136935 0 0 0 488980 90 0 0 25 0 8 0 738893850 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=4946304 CPUtime=4868.51 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 103298 0 0 0 486748 103 0 0 25 0 8 0 738893850 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=4946304 CPUtime=4870.92 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 114839 0 0 0 486967 125 0 0 25 0 8 0 738893850 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=4946304 CPUtime=4891.64 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 115348 0 0 0 489072 92 0 0 25 0 8 0 738893850 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=4946304 CPUtime=4890.85 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 110108 0 0 0 488969 116 0 0 25 0 8 0 738893850 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=4946304 CPUtime=4891.42 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 223233 0 0 0 488888 254 0 0 25 0 8 0 738893850 5065015296 1156800 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39092.4
Current children cumulated vsize (KiB) 4946304

[startup+4962.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/303 14366
/proc/meminfo: memFree=19798924/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=4946304 CPUtime=39571.7 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1173892 0 0 0 3955940 1233 0 0 25 0 8 0 738892797 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4627394 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 1236576 1158061 466 189 0 1230925 0
[pid=14211/tid=14213] ppid=14209 vsize=4946304 CPUtime=4946.66 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 166662 0 0 0 494463 203 0 0 25 0 8 0 738893850 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=4946304 CPUtime=4950.71 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 137321 0 0 0 494980 91 0 0 25 0 8 0 738893850 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=4946304 CPUtime=4927.91 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 104005 0 0 0 492687 104 0 0 25 0 8 0 738893850 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=4946304 CPUtime=4930.92 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 115116 0 0 0 492966 126 0 0 25 0 8 0 738893850 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=4946304 CPUtime=4951.65 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 115350 0 0 0 495072 93 0 0 25 0 8 0 738893850 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=4946304 CPUtime=4950.85 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 110108 0 0 0 494969 116 0 0 25 0 8 0 738893850 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=4946304 CPUtime=4951.41 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 223234 0 0 0 494887 254 0 0 25 0 8 0 738893850 5065015296 1158061 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39571.7
Current children cumulated vsize (KiB) 4946304



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5015.9 s]
/proc/loadavg: 8.03 8.05 8.00 9/303 14367
/proc/meminfo: memFree=19792160/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=4947396 CPUtime=40000.1 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1175573 0 0 0 3998774 1234 0 0 25 0 8 0 738892797 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/14211/statm: 1236849 1159742 466 189 0 1231198 0
[pid=14211/tid=14213] ppid=14209 vsize=4947396 CPUtime=5000.27 cores=0-7
/proc/14211/task/14213/stat : 14213 (claspmt) R 14209 14211 13835 0 -1 4202560 166848 0 0 0 499824 203 0 0 25 0 8 0 738893850 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=14211/tid=14214] ppid=14209 vsize=4947396 CPUtime=5004.32 cores=0-7
/proc/14211/task/14214/stat : 14214 (claspmt) R 14209 14211 13835 0 -1 4202560 137922 0 0 0 500341 91 0 0 25 0 8 0 738893850 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14211/tid=14215] ppid=14209 vsize=4947396 CPUtime=4980.99 cores=0-7
/proc/14211/task/14215/stat : 14215 (claspmt) R 14209 14211 13835 0 -1 4202560 104660 0 0 0 497995 104 0 0 25 0 8 0 738893850 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14211/tid=14216] ppid=14209 vsize=4947396 CPUtime=4984.52 cores=0-7
/proc/14211/task/14216/stat : 14216 (claspmt) R 14209 14211 13835 0 -1 4202560 115343 0 0 0 498326 126 0 0 25 0 8 0 738893850 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14211/tid=14217] ppid=14209 vsize=4947396 CPUtime=5005.26 cores=0-7
/proc/14211/task/14217/stat : 14217 (claspmt) R 14209 14211 13835 0 -1 4202560 115356 0 0 0 500433 93 0 0 25 0 8 0 738893850 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=14211/tid=14218] ppid=14209 vsize=4947396 CPUtime=5004.47 cores=0-7
/proc/14211/task/14218/stat : 14218 (claspmt) R 14209 14211 13835 0 -1 4202560 110108 0 0 0 500331 116 0 0 25 0 8 0 738893850 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14211/tid=14219] ppid=14209 vsize=4947396 CPUtime=5005.03 cores=0-7
/proc/14211/task/14219/stat : 14219 (claspmt) R 14209 14211 13835 0 -1 4202560 223240 0 0 0 500248 255 0 0 25 0 8 0 738893850 5066133504 1159742 33554432000 4194304 4964800 140735694090576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 4947396

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5016.7 s]
/proc/loadavg: 8.03 8.05 8.00 2/297 14368
/proc/meminfo: memFree=19792084/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=4934588 CPUtime=40000.9 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1175583 0 0 0 3998857 1234 0 0 25 0 1 0 738892797 5053018112 1154820 33554432000 4194304 4964800 140735694090576 18446744073709551615 260020051225 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/14211/statm: 1233647 1154820 476 189 0 1227996 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 4934588

[startup+5017.1 s]
/proc/loadavg: 7.46 7.93 7.96 2/297 14368
/proc/meminfo: memFree=23011744/32951124 swapFree=67083432/67111528
[pid=14211] ppid=14209 vsize=1579756 CPUtime=40001.3 cores=0-7
/proc/14211/stat : 14211 (claspmt) R 14209 14211 13835 0 -1 4202496 1175583 0 0 0 3998884 1249 0 0 25 0 1 0 738892797 1617670144 303568 33554432000 4194304 4964800 140735694090576 18446744073709551615 260020439175 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/14211/statm: 394939 303056 476 189 0 389288 0
Current children cumulated CPU time (s) 40001.3
Current children cumulated vsize (KiB) 1579756

Child status: 0
Real time (s): 5017.17
CPU time (s): 40001.4
CPU user time (s): 39988.8
CPU system time (s): 12.5471
CPU usage (%): 797.29
Max. virtual memory (cumulated for all children) (KiB): 4947396

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39988.8
system time used= 12.5471
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1175584
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 115643
involuntary context switches= 184628

runsolver used 13.5239 second user time and 35.7156 second system time

The end

Launcher Data 

Begin job on node110 at 2011-05-15 21:10:22
IDJOB=3380671
IDBENCH=71095
IDSOLVER=1865
FILE ID=node110/3380671-1305486622
RUNJOBID= node110-1305481206-13849
PBS_JOBID= 13325370
Free space on /tmp= 73608 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/APPLICATIONS/satComp09_BioInstances/ndhf_xits_19_UNKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380671-1305486622/watcher-3380671-1305486622 -o /tmp/evaluation-result-3380671-1305486622/solver-3380671-1305486622 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380671-1305486622.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 4cf2a693df1b6e5a084b88e84ec5f2af
RANDOM SEED=79555689

node110.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24438632 kB
Buffers:        301012 kB
Cached:        4473176 kB
SwapCached:       1440 kB
Active:        4718164 kB
Inactive:      3460464 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24438632 kB
SwapTotal:    67111528 kB
SwapFree:     67083432 kB
Dirty:            8564 kB
Writeback:           0 kB
AnonPages:     3402816 kB
Mapped:          26728 kB
Slab:           246676 kB
PageTables:      23516 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 63397616 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73604 MiB
End job on node110 at 2011-05-15 22:34:00