Trace number 3380621

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.1 5008.66

General information on the benchmark

NameCRAFTED/edgematching/
compact/em_11_3_4_cmp.cnf
MD5SUM3b5e80540ef2fcb09cfe50ac7e3686c1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark333.04
Satisfiable
(Un)Satisfiability was proved
Number of variables8713
Number of clauses99699
Sum of the clauses size335345
Maximum clause length81
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 210866
Number of clauses of size 367270
Number of clauses of size 410556
Number of clauses of size 510773
Number of clauses of size over 5234

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...485814/instance-3380621-1305485814.cnf
0.09/0.13	c Solving...
40000.44/5008.11	
40000.44/5008.11	*** INTERRUPTED! ***
40000.44/5008.11	*** Shutting down threads...
40000.98/5008.66	s UNKNOWN
40000.98/5008.66	
40000.98/5008.66	c INTERRUPTED   : 1
40000.98/5008.66	c Models        : 0+    
40000.98/5008.66	c Time          : 5008.116s (Solving: 5007.99s 1st Model: 0.00s Unsat: 0.00s)
40000.98/5008.66	c CPU Time      : 40000.500s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380621-1305485814/watcher-3380621-1305485814 -o /tmp/evaluation-result-3380621-1305485814/solver-3380621-1305485814 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380621-1305485814.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.06 1.04 1.00 2/323 29240
/proc/meminfo: memFree=19935264/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=23492 CPUtime=0 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 1232 0 0 0 0 0 0 0 21 0 1 0 660179294 24055808 1094 33554432000 4194304 4964800 140735942918016 18446744073709551615 271550198469 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/29240/statm: 5873 1095 386 189 0 747 0

[startup+0.0777391 s]
/proc/loadavg: 1.06 1.04 1.00 2/323 29240
/proc/meminfo: memFree=19935264/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=32008 CPUtime=0.07 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 3786 0 0 0 7 0 0 0 21 0 1 0 660179294 32776192 3091 33554432000 4194304 4964800 140735942918016 18446744073709551615 4607761 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/29240/statm: 8002 3091 399 189 0 2876 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 32008

[startup+0.100733 s]
/proc/loadavg: 1.06 1.04 1.00 2/323 29240
/proc/meminfo: memFree=19935264/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=32008 CPUtime=0.09 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 3786 0 0 0 9 0 0 0 21 0 1 0 660179294 32776192 3091 33554432000 4194304 4964800 140735942918016 18446744073709551615 4607744 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/29240/statm: 8002 3091 399 189 0 2876 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 32008

[startup+0.301254 s]
/proc/loadavg: 1.06 1.04 1.00 2/323 29240
/proc/meminfo: memFree=19935264/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=394944 CPUtime=0.88 cores=0-7
/proc/29240/stat : 29240 (claspmt) D 29238 29240 28989 0 -1 4202496 10077 0 0 0 83 5 0 0 18 0 8 0 660179294 404422656 8937 33554432000 4194304 4964800 140735942918016 18446744073709551615 271550580919 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/29240/statm: 98736 8937 464 189 0 93085 0
Current children cumulated CPU time (s) 0.88
Current children cumulated vsize (KiB) 394944

[startup+0.700588 s]
/proc/loadavg: 1.06 1.04 1.00 2/323 29240
/proc/meminfo: memFree=19935264/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=401332 CPUtime=3.98 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 18200 0 0 0 389 9 0 0 18 0 8 0 660179294 410963968 17060 33554432000 4194304 4964800 140735942918016 18446744073709551615 4653388 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29240/statm: 100333 17061 464 189 0 94682 0
Current children cumulated CPU time (s) 3.98
Current children cumulated vsize (KiB) 401332

[startup+1.50044 s]
/proc/loadavg: 1.62 1.15 1.04 9/331 29248
/proc/meminfo: memFree=19856644/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=472984 CPUtime=10.33 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 27283 0 0 0 1019 14 0 0 19 0 8 0 660179294 484335616 26143 33554432000 4194304 4964800 140735942918016 18446744073709551615 4325338 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29240/statm: 118246 26143 464 189 0 112595 0
[pid=29240/tid=29242] ppid=29238 vsize=472984 CPUtime=1.27 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 3051 0 0 0 126 1 0 0 19 0 8 0 660179308 484335616 26143 33554432000 4194304 4964800 140735942918016 18446744073709551615 4359861 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=472984 CPUtime=1.23 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 1622 0 0 0 123 0 0 0 19 0 8 0 660179308 484335616 26143 33554432000 4194304 4964800 140735942918016 18446744073709551615 4362960 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=472984 CPUtime=1.27 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 3152 0 0 0 125 2 0 0 19 0 8 0 660179308 484335616 26143 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=472984 CPUtime=1.27 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 4318 0 0 0 125 2 0 0 19 0 8 0 660179308 484335616 26144 33554432000 4194304 4964800 140735942918016 18446744073709551615 4332144 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=472984 CPUtime=1.34 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 810 0 0 0 134 0 0 0 19 0 8 0 660179308 484335616 26144 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=472984 CPUtime=1.26 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 2864 0 0 0 125 1 0 0 18 0 8 0 660179308 484335616 26145 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670144 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=472984 CPUtime=1.26 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 4571 0 0 0 123 3 0 0 18 0 8 0 660179308 484335616 26145 33554432000 4194304 4964800 140735942918016 18446744073709551615 271550198817 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 10.33
Current children cumulated vsize (KiB) 472984

[startup+3.10115 s]
/proc/loadavg: 1.62 1.15 1.04 9/331 29248
/proc/meminfo: memFree=19800968/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=482092 CPUtime=23.09 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 43740 0 0 0 2288 21 0 0 23 0 8 0 660179294 493662208 42600 33554432000 4194304 4964800 140735942918016 18446744073709551615 4380234 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29240/statm: 120523 42601 464 189 0 114872 0
[pid=29240/tid=29242] ppid=29238 vsize=482092 CPUtime=2.86 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 4785 0 0 0 283 3 0 0 22 0 8 0 660179308 493662208 42602 33554432000 4194304 4964800 140735942918016 18446744073709551615 4325409 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=482092 CPUtime=2.83 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 1766 0 0 0 283 0 0 0 22 0 8 0 660179308 493662208 42602 33554432000 4194304 4964800 140735942918016 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=482092 CPUtime=2.87 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 4404 0 0 0 284 3 0 0 22 0 8 0 660179308 493662208 42602 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=482092 CPUtime=2.87 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 6519 0 0 0 284 3 0 0 23 0 8 0 660179308 493662208 42602 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=482092 CPUtime=2.93 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 5258 0 0 0 292 1 0 0 25 0 8 0 660179308 493662208 42603 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=482092 CPUtime=2.87 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 4271 0 0 0 285 2 0 0 21 0 8 0 660179308 493662208 42603 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=482092 CPUtime=2.86 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 7299 0 0 0 282 4 0 0 20 0 8 0 660179308 493662208 42603 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670378 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.09
Current children cumulated vsize (KiB) 482092

[startup+6.30057 s]
/proc/loadavg: 1.62 1.15 1.04 9/331 29249
/proc/meminfo: memFree=19723412/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=493132 CPUtime=48.63 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 61494 0 0 0 4834 29 0 0 19 0 8 0 660179294 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320981 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29240/statm: 123283 60354 466 189 0 117632 0
[pid=29240/tid=29242] ppid=29238 vsize=493132 CPUtime=6.02 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 6626 0 0 0 598 4 0 0 18 0 8 0 660179308 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=493132 CPUtime=6.04 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 1949 0 0 0 603 1 0 0 21 0 8 0 660179308 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=493132 CPUtime=6.06 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 6418 0 0 0 603 3 0 0 25 0 8 0 660179308 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=493132 CPUtime=6.07 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 9144 0 0 0 603 4 0 0 25 0 8 0 660179308 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=493132 CPUtime=6.13 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 7543 0 0 0 611 2 0 0 25 0 8 0 660179308 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=493132 CPUtime=6.06 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 6408 0 0 0 603 3 0 0 21 0 8 0 660179308 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=493132 CPUtime=6.05 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 11869 0 0 0 600 5 0 0 25 0 8 0 660179308 504967168 60354 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 48.63
Current children cumulated vsize (KiB) 493132

[startup+12.7004 s]
/proc/loadavg: 2.60 1.38 1.11 10/331 29249
/proc/meminfo: memFree=19655672/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=495440 CPUtime=99.74 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 72556 0 0 0 9938 36 0 0 25 0 8 0 660179294 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670328 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29240/statm: 123860 71416 466 189 0 118209 0
[pid=29240/tid=29242] ppid=29238 vsize=495440 CPUtime=12.35 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 7447 0 0 0 1230 5 0 0 25 0 8 0 660179308 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=495440 CPUtime=12.43 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 2301 0 0 0 1242 1 0 0 25 0 8 0 660179308 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=495440 CPUtime=12.46 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 8120 0 0 0 1241 5 0 0 25 0 8 0 660179308 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=495440 CPUtime=12.47 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 10705 0 0 0 1242 5 0 0 25 0 8 0 660179308 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=495440 CPUtime=12.53 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 8461 0 0 0 1250 3 0 0 25 0 8 0 660179308 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=495440 CPUtime=12.46 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 7965 0 0 0 1242 4 0 0 25 0 8 0 660179308 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=495440 CPUtime=12.45 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 14053 0 0 0 1238 7 0 0 25 0 8 0 660179308 507330560 71416 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 99.74
Current children cumulated vsize (KiB) 495440

[startup+25.5011 s]
/proc/loadavg: 3.43 1.59 1.19 9/331 29249
/proc/meminfo: memFree=19572624/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=508228 CPUtime=201.99 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 94304 0 0 0 20150 49 0 0 25 0 8 0 660179294 520425472 93164 33554432000 4194304 4964800 140735942918016 18446744073709551615 4653296 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29240/statm: 127057 93164 466 189 0 121406 0
[pid=29240/tid=29242] ppid=29238 vsize=508228 CPUtime=25.01 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 10480 0 0 0 2494 7 0 0 25 0 8 0 660179308 520425472 93164 33554432000 4194304 4964800 140735942918016 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=508228 CPUtime=25.23 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 2723 0 0 0 2522 1 0 0 25 0 8 0 660179308 520425472 93165 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=508228 CPUtime=25.26 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 10841 0 0 0 2519 7 0 0 25 0 8 0 660179308 520425472 93165 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=508228 CPUtime=25.27 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 14312 0 0 0 2520 7 0 0 25 0 8 0 660179308 520425472 93165 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=508228 CPUtime=25.32 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 12097 0 0 0 2528 4 0 0 25 0 8 0 660179308 520425472 93165 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=508228 CPUtime=25.25 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 8872 0 0 0 2520 5 0 0 25 0 8 0 660179308 520425472 93166 33554432000 4194304 4964800 140735942918016 18446744073709551615 4653584 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=508228 CPUtime=25.25 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 18392 0 0 0 2515 10 0 0 25 0 8 0 660179308 520425472 93168 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670412 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 201.99
Current children cumulated vsize (KiB) 508228

[startup+51.1004 s]
/proc/loadavg: 5.14 2.14 1.38 9/331 29250
/proc/meminfo: memFree=19486884/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=654180 CPUtime=406.46 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 113965 0 0 0 40586 60 0 0 19 0 8 0 660179294 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/29240/statm: 163545 112816 466 189 0 157894 0
[pid=29240/tid=29242] ppid=29238 vsize=654180 CPUtime=50.33 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 11940 0 0 0 5026 7 0 0 25 0 8 0 660179308 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=654180 CPUtime=50.83 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 3204 0 0 0 5082 1 0 0 25 0 8 0 660179308 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=654180 CPUtime=50.85 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 13558 0 0 0 5076 9 0 0 25 0 8 0 660179308 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4343768 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=654180 CPUtime=50.85 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 17410 0 0 0 5077 8 0 0 25 0 8 0 660179308 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4658531 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=654180 CPUtime=50.92 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 16412 0 0 0 5086 6 0 0 25 0 8 0 660179308 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=654180 CPUtime=50.83 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 10687 0 0 0 5077 6 0 0 25 0 8 0 660179308 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=654180 CPUtime=50.85 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 19091 0 0 0 5075 10 0 0 25 0 8 0 660179308 669880320 112816 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 406.46

################
# More data... #
################

[pid=29240] ppid=29238 vsize=4141944 CPUtime=37717.6 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 981242 0 0 0 3770847 914 0 0 25 0 8 0 660179294 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4653482 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/29240/statm: 1035486 975269 466 189 0 1029835 0
[pid=29240/tid=29242] ppid=29238 vsize=4141944 CPUtime=4712.51 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 52549 0 0 0 471180 71 0 0 25 0 8 0 660179308 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=4141944 CPUtime=4721.66 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 49092 0 0 0 472118 48 0 0 25 0 8 0 660179308 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=4141944 CPUtime=4703.22 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 108088 0 0 0 470201 121 0 0 25 0 8 0 660179308 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=4141944 CPUtime=4720.48 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 176298 0 0 0 471912 136 0 0 25 0 8 0 660179308 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=4141944 CPUtime=4717.08 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 71271 0 0 0 471639 69 0 0 25 0 8 0 660179308 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=4141944 CPUtime=4708.01 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 96282 0 0 0 470686 115 0 0 25 0 8 0 660179308 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4658589 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=4141944 CPUtime=4721.03 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 200721 0 0 0 471953 150 0 0 25 0 8 0 660179308 4241350656 975269 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37717.6
Current children cumulated vsize (KiB) 4141944

[startup+4782.3 s]
/proc/loadavg: 8.10 8.16 8.10 9/331 29414
/proc/meminfo: memFree=16028460/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=4141944 CPUtime=38196.8 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 981297 0 0 0 3818762 916 0 0 25 0 8 0 660179294 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669479 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/29240/statm: 1035486 975324 466 189 0 1029835 0
[pid=29240/tid=29242] ppid=29238 vsize=4141944 CPUtime=4771.78 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 52549 0 0 0 477106 72 0 0 25 0 8 0 660179308 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=4141944 CPUtime=4781.65 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 49092 0 0 0 478117 48 0 0 25 0 8 0 660179308 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=4141944 CPUtime=4763.21 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 108088 0 0 0 476200 121 0 0 25 0 8 0 660179308 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=4141944 CPUtime=4780.45 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 176299 0 0 0 477909 136 0 0 25 0 8 0 660179308 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=4141944 CPUtime=4777.07 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 71324 0 0 0 477638 69 0 0 25 0 8 0 660179308 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=4141944 CPUtime=4767.98 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 96282 0 0 0 476683 115 0 0 25 0 8 0 660179308 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=4141944 CPUtime=4781.01 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 200722 0 0 0 477951 150 0 0 25 0 8 0 660179308 4241350656 975324 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38196.8
Current children cumulated vsize (KiB) 4141944

[startup+4842.31 s]
/proc/loadavg: 8.13 8.15 8.09 9/331 29416
/proc/meminfo: memFree=16025828/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=4141944 CPUtime=38676.2 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 981997 0 0 0 3866699 917 0 0 25 0 8 0 660179294 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4342904 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/29240/statm: 1035486 976024 466 189 0 1029835 0
[pid=29240/tid=29242] ppid=29238 vsize=4141944 CPUtime=4831.14 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 52560 0 0 0 483042 72 0 0 25 0 8 0 660179308 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=4141944 CPUtime=4841.67 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 49536 0 0 0 484119 48 0 0 25 0 8 0 660179308 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=4141944 CPUtime=4823.21 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 108088 0 0 0 482200 121 0 0 25 0 8 0 660179308 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=4141944 CPUtime=4840.44 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 176299 0 0 0 483908 136 0 0 25 0 8 0 660179308 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=4141944 CPUtime=4837.08 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 71533 0 0 0 483639 69 0 0 25 0 8 0 660179308 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=4141944 CPUtime=4828 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 96282 0 0 0 482685 115 0 0 25 0 8 0 660179308 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=4141944 CPUtime=4841.01 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 200743 0 0 0 483951 150 0 0 25 0 8 0 660179308 4241350656 976024 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38676.2
Current children cumulated vsize (KiB) 4141944

[startup+4902.3 s]
/proc/loadavg: 8.05 8.12 8.09 9/331 29417
/proc/meminfo: memFree=16011916/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=4154312 CPUtime=39155.4 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 985706 0 0 0 3914615 920 0 0 25 0 8 0 660179294 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4342157 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/29240/statm: 1038578 979575 466 189 0 1032927 0
[pid=29240/tid=29242] ppid=29238 vsize=4154312 CPUtime=4890.62 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 52560 0 0 0 488990 72 0 0 25 0 8 0 660179308 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=4154312 CPUtime=4901.49 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 50247 0 0 0 490100 49 0 0 25 0 8 0 660179308 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=4154312 CPUtime=4883.19 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 108406 0 0 0 488197 122 0 0 25 0 8 0 660179308 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=4154312 CPUtime=4900.42 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 176301 0 0 0 489906 136 0 0 25 0 8 0 660179308 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=4154312 CPUtime=4897.07 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 71779 0 0 0 489638 69 0 0 25 0 8 0 660179308 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=4154312 CPUtime=4887.97 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 96354 0 0 0 488682 115 0 0 25 0 8 0 660179308 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=4154312 CPUtime=4901 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 200743 0 0 0 489949 151 0 0 25 0 8 0 660179308 4254015488 979575 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39155.3
Current children cumulated vsize (KiB) 4154312

[startup+4962.3 s]
/proc/loadavg: 8.17 8.14 8.09 9/331 29419
/proc/meminfo: memFree=15952992/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=4287260 CPUtime=39634.6 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 1000556 0 0 0 3962524 935 0 0 25 0 8 0 660179294 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/29240/statm: 1071815 994386 466 189 0 1066164 0
[pid=29240/tid=29242] ppid=29238 vsize=4287260 CPUtime=4950.58 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 52869 0 0 0 494985 73 0 0 25 0 8 0 660179308 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4370288 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=4287260 CPUtime=4960.87 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 50262 0 0 0 496038 49 0 0 25 0 8 0 660179308 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=4287260 CPUtime=4943.18 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 108415 0 0 0 494196 122 0 0 25 0 8 0 660179308 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=4287260 CPUtime=4960.42 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 180575 0 0 0 495901 141 0 0 25 0 8 0 660179308 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=4287260 CPUtime=4957 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 73152 0 0 0 495630 70 0 0 25 0 8 0 660179308 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4641840 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=4287260 CPUtime=4947.96 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 97929 0 0 0 494677 119 0 0 25 0 8 0 660179308 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=4287260 CPUtime=4960.99 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 206482 0 0 0 495946 153 0 0 25 0 8 0 660179308 4390154240 994386 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39634.6
Current children cumulated vsize (KiB) 4287260



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5008.1 s]
/proc/loadavg: 8.08 8.12 8.09 9/331 29420
/proc/meminfo: memFree=15942188/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=4287260 CPUtime=40000.4 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 1003203 0 0 0 3999105 939 0 0 25 0 8 0 660179294 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4320966 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/29240/statm: 1071815 997033 466 189 0 1066164 0
[pid=29240/tid=29242] ppid=29238 vsize=4287260 CPUtime=4996.39 cores=0-7
/proc/29240/task/29242/stat : 29242 (claspmt) R 29238 29240 28989 0 -1 4202560 53000 0 0 0 499566 73 0 0 25 0 8 0 660179308 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=29240/tid=29243] ppid=29238 vsize=4287260 CPUtime=5006.68 cores=0-7
/proc/29240/task/29243/stat : 29243 (claspmt) R 29238 29240 28989 0 -1 4202560 50268 0 0 0 500619 49 0 0 25 0 8 0 660179308 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=29240/tid=29244] ppid=29238 vsize=4287260 CPUtime=4988.96 cores=0-7
/proc/29240/task/29244/stat : 29244 (claspmt) R 29238 29240 28989 0 -1 4202560 108416 0 0 0 498774 122 0 0 23 0 8 0 660179308 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=29240/tid=29245] ppid=29238 vsize=4287260 CPUtime=5006.19 cores=0-7
/proc/29240/task/29245/stat : 29245 (claspmt) R 29238 29240 28989 0 -1 4202560 182943 0 0 0 500476 143 0 0 25 0 8 0 660179308 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=29240/tid=29246] ppid=29238 vsize=4287260 CPUtime=5002.28 cores=0-7
/proc/29240/task/29246/stat : 29246 (claspmt) R 29238 29240 28989 0 -1 4202560 73155 0 0 0 500158 70 0 0 25 0 8 0 660179308 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4670378 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=29240/tid=29247] ppid=29238 vsize=4287260 CPUtime=4993.74 cores=0-7
/proc/29240/task/29247/stat : 29247 (claspmt) R 29238 29240 28989 0 -1 4202560 98065 0 0 0 499255 119 0 0 25 0 8 0 660179308 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=29240/tid=29248] ppid=29238 vsize=4287260 CPUtime=5006.81 cores=0-7
/proc/29240/task/29248/stat : 29248 (claspmt) R 29238 29240 28989 0 -1 4202560 206483 0 0 0 500527 154 0 0 25 0 8 0 660179308 4390154240 997033 33554432000 4194304 4964800 140735942918016 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 4287260

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5008.3 s]
/proc/loadavg: 8.08 8.12 8.09 2/325 29421
/proc/meminfo: memFree=15952280/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=4279428 CPUtime=40000.7 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 1003213 0 0 0 3999130 940 0 0 25 0 1 0 660179294 4382134272 994127 33554432000 4194304 4964800 140735942918016 18446744073709551615 271550192843 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29240/statm: 1069857 994127 476 189 0 1064206 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 4279428

[startup+5008.5 s]
/proc/loadavg: 8.08 8.12 8.09 2/325 29421
/proc/meminfo: memFree=15952280/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=4130864 CPUtime=40000.9 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 1003213 0 0 0 3999148 940 0 0 25 0 1 0 660179294 4230004736 960413 33554432000 4194304 4964800 140735942918016 18446744073709551615 271550581095 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29240/statm: 1032716 959901 476 189 0 1027065 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 4130864

[startup+5008.6 s]
/proc/loadavg: 8.08 8.12 8.09 2/325 29421
/proc/meminfo: memFree=15952280/32951124 swapFree=67056788/67111528
[pid=29240] ppid=29238 vsize=1652576 CPUtime=40001 cores=0-7
/proc/29240/stat : 29240 (claspmt) R 29238 29240 28989 0 -1 4202496 1003213 0 0 0 3999148 950 0 0 25 0 1 0 660179294 1692237824 354105 33554432000 4194304 4964800 140735942918016 18446744073709551615 271550580871 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/29240/statm: 413144 353593 476 189 0 407493 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 1652576

Child status: 0
Real time (s): 5008.66
CPU time (s): 40001.1
CPU user time (s): 39991.5
CPU system time (s): 9.56454
CPU usage (%): 798.637
Max. virtual memory (cumulated for all children) (KiB): 4287260

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39991.5
system time used= 9.56455
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1003214
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 217748
involuntary context switches= 406248

runsolver used 15.0827 second user time and 39.262 second system time

The end

Launcher Data 

Begin job on node133 at 2011-05-15 20:56:54
IDJOB=3380621
IDBENCH=70910
IDSOLVER=1865
FILE ID=node133/3380621-1305485814
RUNJOBID= node133-1305481203-29003
PBS_JOBID= 13325347
Free space on /tmp= 73624 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/CRAFTED/edgematching/compact/em_11_3_4_cmp.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380621-1305485814/watcher-3380621-1305485814 -o /tmp/evaluation-result-3380621-1305485814/solver-3380621-1305485814 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380621-1305485814.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 3b5e80540ef2fcb09cfe50ac7e3686c1
RANDOM SEED=421321427

node133.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6741.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19935544 kB
Buffers:        311396 kB
Cached:        4209456 kB
SwapCached:       8756 kB
Active:        9495040 kB
Inactive:      3219404 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19935544 kB
SwapTotal:    67111528 kB
SwapFree:     67056788 kB
Dirty:            2720 kB
Writeback:           0 kB
AnonPages:     8191236 kB
Mapped:          27996 kB
Slab:           201436 kB
PageTables:      33668 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 63747536 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73620 MiB
End job on node133 at 2011-05-15 22:20:24