Trace number 3380799

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 39085.3 5102.21

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.080-SAT.cnf
MD5SUM1ca09a2b911e29894c6603aee8d3e672
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1503.62
Satisfiable
(Un)Satisfiability was proved
Number of variables175084
Number of clauses2826936
Sum of the clauses size6454456
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 22728320
Number of clauses of size 37680
Number of clauses of size 435520
Number of clauses of size 515360
Number of clauses of size over 539360

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...492622/instance-3380799-1305492622.cnf
2.89/2.90	c Solving...
39082.98/5100.02	
39082.98/5100.02	*** INTERRUPTED! ***
39082.98/5100.02	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380799-1305492622/watcher-3380799-1305492622 -o /tmp/evaluation-result-3380799-1305492622/solver-3380799-1305492622 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380799-1305492622.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.74 7.95 6.12 3/481 28739
/proc/meminfo: memFree=15048836/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=43108 CPUtime=0 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 4214 0 0 0 0 0 0 0 21 0 1 0 660860078 44142592 4144 33554432000 4194304 4964800 140736063470288 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 10777 4151 376 189 0 5651 0

[startup+0.0871181 s]
/proc/loadavg: 7.74 7.95 6.12 3/481 28739
/proc/meminfo: memFree=15048836/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=56252 CPUtime=0.08 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 10598 0 0 0 7 1 0 0 21 0 1 0 660860078 57602048 9125 33554432000 4194304 4964800 140736063470288 18446744073709551615 4581782 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 14063 9127 389 189 0 8937 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 56252

[startup+0.101096 s]
/proc/loadavg: 7.74 7.95 6.12 3/481 28739
/proc/meminfo: memFree=15048836/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=57312 CPUtime=0.09 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 10983 0 0 0 8 1 0 0 21 0 1 0 660860078 58687488 9375 33554432000 4194304 4964800 140736063470288 18446744073709551615 218905730816 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 14328 9375 389 189 0 9202 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 57312

[startup+0.301062 s]
/proc/loadavg: 7.74 7.95 6.12 3/481 28739
/proc/meminfo: memFree=15048836/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=90244 CPUtime=0.3 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 22671 0 0 0 27 3 0 0 22 0 1 0 660860078 92409856 17196 33554432000 4194304 4964800 140736063470288 18446744073709551615 218905391813 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 22561 17198 389 189 0 17435 0
Current children cumulated CPU time (s) 0.3
Current children cumulated vsize (KiB) 90244

[startup+0.70099 s]
/proc/loadavg: 7.74 7.95 6.12 3/481 28739
/proc/meminfo: memFree=15048836/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=148676 CPUtime=0.69 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 42714 0 0 0 64 5 0 0 23 0 1 0 660860078 152244224 32112 33554432000 4194304 4964800 140736063470288 18446744073709551615 218905391813 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 37169 32113 389 189 0 32043 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 148676

[startup+1.50084 s]
/proc/loadavg: 7.74 7.95 6.12 2/482 28740
/proc/meminfo: memFree=14878452/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=236764 CPUtime=1.49 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 67937 0 0 0 141 8 0 0 25 0 1 0 660860078 242446336 50969 33554432000 4194304 4964800 140736063470288 18446744073709551615 4607513 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 59191 50969 403 189 0 54065 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 236764

[startup+3.10052 s]
/proc/loadavg: 7.20 7.83 6.10 2/482 28740
/proc/meminfo: memFree=14847080/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=568764 CPUtime=3.46 cores=0-7
/proc/28739/stat : 28739 (claspmt) S 28737 28739 28586 0 -1 4202496 108526 0 0 0 322 24 0 0 18 0 8 0 660860078 582414336 78127 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 142191 78127 443 189 0 136540 0
Current children cumulated CPU time (s) 3.46
Current children cumulated vsize (KiB) 568764

[startup+6.30098 s]
/proc/loadavg: 7.20 7.83 6.10 2/489 28747
/proc/meminfo: memFree=14739144/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=568764 CPUtime=6.7 cores=0-7
/proc/28739/stat : 28739 (claspmt) S 28737 28739 28586 0 -1 4202496 109138 0 0 0 646 24 0 0 18 0 8 0 660860078 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 142191 78739 443 189 0 136540 0
[pid=28739/tid=28741] ppid=28737 vsize=568764 CPUtime=0.04 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) S 28737 28739 28586 0 -1 4202560 5 0 0 0 4 0 0 0 18 0 8 0 660860368 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) S 28737 28739 28586 0 -1 4202560 8021 0 0 0 5 4 0 0 18 0 8 0 660860368 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 4 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=568764 CPUtime=0.06 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) S 28737 28739 28586 0 -1 4202560 5600 0 0 0 5 1 0 0 18 0 8 0 660860368 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 6 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=568764 CPUtime=0.03 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) S 28737 28739 28586 0 -1 4202560 6032 0 0 0 1 2 0 0 18 0 8 0 660860368 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 2 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=568764 CPUtime=3.29 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 7516 0 0 0 327 2 0 0 25 0 8 0 660860368 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670144 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) S 28737 28739 28586 0 -1 4202560 7239 0 0 0 8 1 0 0 18 0 8 0 660860368 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 7 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=568764 CPUtime=0.12 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) S 28737 28739 28586 0 -1 4202560 6701 0 0 0 10 2 0 0 18 0 8 0 660860368 582414336 78739 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 6.7
Current children cumulated vsize (KiB) 568764

[startup+12.7008 s]
/proc/loadavg: 6.25 7.60 6.04 2/489 28747
/proc/meminfo: memFree=14739920/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=568764 CPUtime=13.09 cores=0-7
/proc/28739/stat : 28739 (claspmt) S 28737 28739 28586 0 -1 4202496 109207 0 0 0 1285 24 0 0 18 0 8 0 660860078 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 142191 78808 443 189 0 136540 0
[pid=28739/tid=28741] ppid=28737 vsize=568764 CPUtime=0.04 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) S 28737 28739 28586 0 -1 4202560 5 0 0 0 4 0 0 0 18 0 8 0 660860368 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) S 28737 28739 28586 0 -1 4202560 8021 0 0 0 5 4 0 0 18 0 8 0 660860368 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 4 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=568764 CPUtime=0.06 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) S 28737 28739 28586 0 -1 4202560 5600 0 0 0 5 1 0 0 18 0 8 0 660860368 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 6 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=568764 CPUtime=0.03 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) S 28737 28739 28586 0 -1 4202560 6032 0 0 0 1 2 0 0 18 0 8 0 660860368 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 2 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=568764 CPUtime=9.69 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 7585 0 0 0 967 2 0 0 25 0 8 0 660860368 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218905392161 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) S 28737 28739 28586 0 -1 4202560 7239 0 0 0 8 1 0 0 18 0 8 0 660860368 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 7 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=568764 CPUtime=0.12 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) S 28737 28739 28586 0 -1 4202560 6701 0 0 0 10 2 0 0 18 0 8 0 660860368 582414336 78808 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 13.09
Current children cumulated vsize (KiB) 568764

[startup+25.5005 s]
/proc/loadavg: 5.44 7.39 5.99 2/489 28747
/proc/meminfo: memFree=14741424/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=568764 CPUtime=25.89 cores=0-7
/proc/28739/stat : 28739 (claspmt) S 28737 28739 28586 0 -1 4202496 109356 0 0 0 2565 24 0 0 18 0 8 0 660860078 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 142191 78957 443 189 0 136540 0
[pid=28739/tid=28741] ppid=28737 vsize=568764 CPUtime=0.04 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) S 28737 28739 28586 0 -1 4202560 5 0 0 0 4 0 0 0 18 0 8 0 660860368 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) S 28737 28739 28586 0 -1 4202560 8021 0 0 0 5 4 0 0 18 0 8 0 660860368 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 4 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=568764 CPUtime=0.06 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) S 28737 28739 28586 0 -1 4202560 5600 0 0 0 5 1 0 0 18 0 8 0 660860368 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 6 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=568764 CPUtime=0.03 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) S 28737 28739 28586 0 -1 4202560 6032 0 0 0 1 2 0 0 18 0 8 0 660860368 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 2 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=568764 CPUtime=22.48 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 7734 0 0 0 2246 2 0 0 25 0 8 0 660860368 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) S 28737 28739 28586 0 -1 4202560 7239 0 0 0 8 1 0 0 18 0 8 0 660860368 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 7 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=568764 CPUtime=0.12 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) S 28737 28739 28586 0 -1 4202560 6701 0 0 0 10 2 0 0 18 0 8 0 660860368 582414336 78957 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 25.89
Current children cumulated vsize (KiB) 568764

[startup+51.1009 s]
/proc/loadavg: 3.93 6.87 5.85 2/489 28748
/proc/meminfo: memFree=14739728/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=568764 CPUtime=51.49 cores=0-7
/proc/28739/stat : 28739 (claspmt) S 28737 28739 28586 0 -1 4202496 109567 0 0 0 5125 24 0 0 18 0 8 0 660860078 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 142191 79168 443 189 0 136540 0
[pid=28739/tid=28741] ppid=28737 vsize=568764 CPUtime=0.04 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) S 28737 28739 28586 0 -1 4202560 5 0 0 0 4 0 0 0 18 0 8 0 660860368 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) S 28737 28739 28586 0 -1 4202560 8021 0 0 0 5 4 0 0 18 0 8 0 660860368 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 4 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=568764 CPUtime=0.06 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) S 28737 28739 28586 0 -1 4202560 5600 0 0 0 5 1 0 0 18 0 8 0 660860368 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 6 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=568764 CPUtime=0.03 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) S 28737 28739 28586 0 -1 4202560 6032 0 0 0 1 2 0 0 18 0 8 0 660860368 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 2 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=568764 CPUtime=48.08 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 7945 0 0 0 4806 2 0 0 25 0 8 0 660860368 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670077 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) S 28737 28739 28586 0 -1 4202560 7239 0 0 0 8 1 0 0 18 0 8 0 660860368 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 7 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=568764 CPUtime=0.12 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) S 28737 28739 28586 0 -1 4202560 6701 0 0 0 10 2 0 0 18 0 8 0 660860368 582414336 79168 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 51.49
Current children cumulated vsize (KiB) 568764

[startup+102.319 s]
/proc/loadavg: 2.17 5.88 5.57 2/489 28749
/proc/meminfo: memFree=14738000/32951124 swapFree=63082284/67111528
[pid=28739] ppid=28737 vsize=568764 CPUtime=102.72 cores=0-7
/proc/28739/stat : 28739 (claspmt) S 28737 28739 28586 0 -1 4202496 109837 0 0 0 10247 25 0 0 18 0 8 0 660860078 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 142191 79438 443 189 0 136540 0
[pid=28739/tid=28741] ppid=28737 vsize=568764 CPUtime=0.04 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) S 28737 28739 28586 0 -1 4202560 5 0 0 0 4 0 0 0 18 0 8 0 660860368 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) S 28737 28739 28586 0 -1 4202560 8021 0 0 0 5 4 0 0 18 0 8 0 660860368 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 4 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=568764 CPUtime=0.06 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) S 28737 28739 28586 0 -1 4202560 5600 0 0 0 5 1 0 0 18 0 8 0 660860368 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 6 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=568764 CPUtime=0.03 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) S 28737 28739 28586 0 -1 4202560 6032 0 0 0 1 2 0 0 18 0 8 0 660860368 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 2 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=568764 CPUtime=99.31 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 8215 0 0 0 9928 3 0 0 25 0 8 0 660860368 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=568764 CPUtime=0.09 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) S 28737 28739 28586 0 -1 4202560 7239 0 0 0 8 1 0 0 18 0 8 0 660860368 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 7 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=568764 CPUtime=0.12 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) S 28737 28739 28586 0 -1 4202560 6701 0 0 0 10 2 0 0 18 0 8 0 660860368 582414336 79438 33554432000 4194304 4964800 140736063470288 18446744073709551615 218917546905 0 0 4096 18946 18446604447565668480 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 102.72
Current children cumulated vsize (KiB) 568764

[startup+162.301 s]
/proc/loadavg: 1.46 5.06 5.31 2/489 28751
/proc/meminfo: memFree=14737952/32951124 swapFree=63082284/67111528

################
# More data... #
################

[pid=28739/tid=28746] ppid=28737 vsize=15537792 CPUtime=4533.82 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) R 28737 28739 28586 0 -1 4202560 673326 0 0 0 452754 628 0 0 25 0 8 0 660860368 15910699008 3782442 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=15537792 CPUtime=4609.31 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) R 28737 28739 28586 0 -1 4202560 131758 0 0 0 460859 72 0 0 25 0 8 0 660860368 15910699008 3782442 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37027.8
Current children cumulated vsize (KiB) 15537792

[startup+4902.3 s]
/proc/loadavg: 8.00 8.04 8.00 9/489 28901
/proc/meminfo: memFree=94244/32951124 swapFree=62707224/67111528
[pid=28739] ppid=28737 vsize=15668864 CPUtime=37506.3 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 3880197 0 0 0 3747934 2700 0 0 25 0 8 0 660860078 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4668568 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 3917216 3830825 466 189 0 3911565 0
[pid=28739/tid=28741] ppid=28737 vsize=15668864 CPUtime=4669.19 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) R 28737 28739 28586 0 -1 4202560 90123 0 0 0 466815 104 0 0 25 0 8 0 660860368 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=15668864 CPUtime=4667.84 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) R 28737 28739 28586 0 -1 4202560 1773140 0 0 0 465893 891 0 0 25 0 8 0 660860368 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4320953 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=15668864 CPUtime=4667.28 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) R 28737 28739 28586 0 -1 4202560 637982 0 0 0 466232 496 0 0 25 0 8 0 660860368 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=15668864 CPUtime=4668.63 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) R 28737 28739 28586 0 -1 4202560 483044 0 0 0 466383 480 0 0 25 0 8 0 660860368 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=15668864 CPUtime=4899.43 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 8232 0 0 0 489940 3 0 0 25 0 8 0 660860368 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670225 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=15668864 CPUtime=4592.85 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) R 28737 28739 28586 0 -1 4202560 673326 0 0 0 458657 628 0 0 25 0 8 0 660860368 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=15668864 CPUtime=4669.3 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) R 28737 28739 28586 0 -1 4202560 135105 0 0 0 466858 72 0 0 25 0 8 0 660860368 16044916736 3830825 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 37506.3
Current children cumulated vsize (KiB) 15668864

[startup+4962.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/489 28903
/proc/meminfo: memFree=95616/32951124 swapFree=62355720/67111528
[pid=28739] ppid=28737 vsize=16127616 CPUtime=37984 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 3967850 0 0 0 3795647 2751 0 0 25 0 8 0 660860078 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4632355 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 4031904 3918351 466 189 0 4026253 0
[pid=28739/tid=28741] ppid=28737 vsize=16127616 CPUtime=4729.19 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) R 28737 28739 28586 0 -1 4202560 96043 0 0 0 472808 111 0 0 25 0 8 0 660860368 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=16127616 CPUtime=4727.6 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) R 28737 28739 28586 0 -1 4202560 1815417 0 0 0 471851 909 0 0 25 0 8 0 660860368 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=16127616 CPUtime=4727.09 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) R 28737 28739 28586 0 -1 4202560 637982 0 0 0 472210 499 0 0 25 0 8 0 660860368 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=16127616 CPUtime=4727.52 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) R 28737 28739 28586 0 -1 4202560 483044 0 0 0 472272 480 0 0 25 0 8 0 660860368 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=16127616 CPUtime=4959.43 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 8232 0 0 0 495940 3 0 0 25 0 8 0 660860368 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669280 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=16127616 CPUtime=4652.08 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) R 28737 28739 28586 0 -1 4202560 673326 0 0 0 464580 628 0 0 25 0 8 0 660860368 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=16127616 CPUtime=4729.28 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) R 28737 28739 28586 0 -1 4202560 174561 0 0 0 472831 97 0 0 25 0 8 0 660860368 16514678784 3918351 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37984
Current children cumulated vsize (KiB) 16127616

[startup+5022.3 s]
/proc/loadavg: 8.04 8.04 8.00 9/489 28904
/proc/meminfo: memFree=385408/32951124 swapFree=62248496/67111528
[pid=28739] ppid=28737 vsize=16258688 CPUtime=38462.6 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 3995343 0 0 0 3843487 2768 0 0 25 0 8 0 660860078 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669026 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 4064672 3945844 466 189 0 4059021 0
[pid=28739/tid=28741] ppid=28737 vsize=16258688 CPUtime=4789.19 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) R 28737 28739 28586 0 -1 4202560 96112 0 0 0 478808 111 0 0 25 0 8 0 660860368 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=16258688 CPUtime=4787.6 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) R 28737 28739 28586 0 -1 4202560 1842841 0 0 0 477838 922 0 0 25 0 8 0 660860368 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4632289 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=16258688 CPUtime=4786.92 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) R 28737 28739 28586 0 -1 4202560 637982 0 0 0 478193 499 0 0 25 0 8 0 660860368 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=16258688 CPUtime=4786.55 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) R 28737 28739 28586 0 -1 4202560 483044 0 0 0 478175 480 0 0 25 0 8 0 660860368 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=16258688 CPUtime=5019.43 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 8232 0 0 0 501940 3 0 0 25 0 8 0 660860368 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4399343 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=16258688 CPUtime=4711.79 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) R 28737 28739 28586 0 -1 4202560 673326 0 0 0 470551 628 0 0 25 0 8 0 660860368 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=16258688 CPUtime=4789.28 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) R 28737 28739 28586 0 -1 4202560 174561 0 0 0 478828 100 0 0 25 0 8 0 660860368 16648896512 3945844 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38462.6
Current children cumulated vsize (KiB) 16258688

[startup+5082.3 s]
/proc/loadavg: 8.05 8.04 8.00 9/489 28906
/proc/meminfo: memFree=265976/32951124 swapFree=62248496/67111528
[pid=28739] ppid=28737 vsize=16324224 CPUtime=38941.6 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 4026402 0 0 0 3891373 2783 0 0 25 0 8 0 660860078 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669131 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 4081056 3976588 466 189 0 4075405 0
[pid=28739/tid=28741] ppid=28737 vsize=16324224 CPUtime=4849.19 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) R 28737 28739 28586 0 -1 4202560 96330 0 0 0 484808 111 0 0 25 0 8 0 660860368 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=16324224 CPUtime=4847.6 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) R 28737 28739 28586 0 -1 4202560 1873682 0 0 0 483823 937 0 0 25 0 8 0 660860368 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=16324224 CPUtime=4846.92 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) R 28737 28739 28586 0 -1 4202560 637982 0 0 0 484193 499 0 0 25 0 8 0 660860368 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=16324224 CPUtime=4845.57 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) R 28737 28739 28586 0 -1 4202560 483044 0 0 0 484077 480 0 0 25 0 8 0 660860368 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=16324224 CPUtime=5079.44 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 8232 0 0 0 507941 3 0 0 25 0 8 0 660860368 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669280 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=16324224 CPUtime=4771.8 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) R 28737 28739 28586 0 -1 4202560 673326 0 0 0 476552 628 0 0 25 0 8 0 660860368 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=16324224 CPUtime=4849.28 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) R 28737 28739 28586 0 -1 4202560 174561 0 0 0 484828 100 0 0 25 0 8 0 660860368 16716005376 3976588 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38941.6
Current children cumulated vsize (KiB) 16324224



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 8.11 8.06 8.00 9/489 28907
/proc/meminfo: memFree=223260/32951124 swapFree=62248496/67111528
[pid=28739] ppid=28737 vsize=16389760 CPUtime=39083 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 4036325 0 0 0 3905510 2788 0 0 25 0 8 0 660860078 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4669557 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 4097440 3986511 466 189 0 4091789 0
[pid=28739/tid=28741] ppid=28737 vsize=16389760 CPUtime=4866.91 cores=0-7
/proc/28739/task/28741/stat : 28741 (claspmt) R 28737 28739 28586 0 -1 4202560 96330 0 0 0 486580 111 0 0 25 0 8 0 660860368 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28739/tid=28742] ppid=28737 vsize=16389760 CPUtime=4865.32 cores=0-7
/proc/28739/task/28742/stat : 28742 (claspmt) R 28737 28739 28586 0 -1 4202560 1883605 0 0 0 485590 942 0 0 25 0 8 0 660860368 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28739/tid=28743] ppid=28737 vsize=16389760 CPUtime=4864.64 cores=0-7
/proc/28739/task/28743/stat : 28743 (claspmt) R 28737 28739 28586 0 -1 4202560 637982 0 0 0 485965 499 0 0 25 0 8 0 660860368 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28739/tid=28744] ppid=28737 vsize=16389760 CPUtime=4862.98 cores=0-7
/proc/28739/task/28744/stat : 28744 (claspmt) R 28737 28739 28586 0 -1 4202560 483044 0 0 0 485818 480 0 0 25 0 8 0 660860368 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28739/tid=28745] ppid=28737 vsize=16389760 CPUtime=5097.16 cores=0-7
/proc/28739/task/28745/stat : 28745 (claspmt) R 28737 28739 28586 0 -1 4202560 8232 0 0 0 509713 3 0 0 25 0 8 0 660860368 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28739/tid=28746] ppid=28737 vsize=16389760 CPUtime=4789.51 cores=0-7
/proc/28739/task/28746/stat : 28746 (claspmt) R 28737 28739 28586 0 -1 4202560 673326 0 0 0 478323 628 0 0 25 0 8 0 660860368 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28739/tid=28747] ppid=28737 vsize=16389760 CPUtime=4867 cores=0-7
/proc/28739/task/28747/stat : 28747 (claspmt) R 28737 28739 28586 0 -1 4202560 174561 0 0 0 486600 100 0 0 25 0 8 0 660860368 16783114240 3986511 33554432000 4194304 4964800 140736063470288 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39083
Current children cumulated vsize (KiB) 16389760

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -28739

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 8.11 8.06 8.00 2/483 28908
/proc/meminfo: memFree=273280/32951124 swapFree=62248496/67111528
[pid=28739] ppid=28737 vsize=16302164 CPUtime=39083.7 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 4036336 0 0 0 3905578 2788 0 0 25 0 1 0 660860078 16693415936 3973706 33554432000 4194304 4964800 140736063470288 18446744073709551615 218905386187 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 4075541 3973706 476 189 0 4069890 0
Current children cumulated CPU time (s) 39083.7
Current children cumulated vsize (KiB) 16302164

[startup+5101.4 s]
/proc/loadavg: 8.11 8.06 8.00 2/483 28908
/proc/meminfo: memFree=273280/32951124 swapFree=62248496/67111528
[pid=28739] ppid=28737 vsize=16171092 CPUtime=39084.4 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 4036336 0 0 0 3905656 2789 0 0 25 0 1 0 660860078 16559198208 3947732 33554432000 4194304 4964800 140736063470288 18446744073709551615 218905379791 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 4042773 3947732 476 189 0 4037122 0
Current children cumulated CPU time (s) 39084.4
Current children cumulated vsize (KiB) 16171092

[startup+5101.82 s]
/proc/loadavg: 7.54 7.94 7.97 2/483 28908
/proc/meminfo: memFree=4408556/32951124 swapFree=62248496/67111528
[pid=28739] ppid=28737 vsize=9617492 CPUtime=39084.9 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4202496 4036336 0 0 0 3905672 2814 0 0 25 0 1 0 660860078 9848311808 2336050 33554432000 4194304 4964800 140736063470288 18446744073709551615 218905774215 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/28739/statm: 2404373 2336050 476 189 0 2398722 0
Current children cumulated CPU time (s) 39084.9
Current children cumulated vsize (KiB) 9617492

[startup+5102.2 s]
/proc/loadavg: 7.54 7.94 7.97 2/483 28908
/proc/meminfo: memFree=4408556/32951124 swapFree=62248496/67111528
[pid=28739] ppid=28737 vsize=0 CPUtime=39085.2 cores=0-7
/proc/28739/stat : 28739 (claspmt) R 28737 28739 28586 0 -1 4203524 4036336 0 0 0 3905674 2850 0 0 25 0 1 0 660860078 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/28739/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 39085.2
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5102.21
CPU time (s): 39085.3
CPU user time (s): 39056.7
CPU system time (s): 28.5187
CPU usage (%): 766.046
Max. virtual memory (cumulated for all children) (KiB): 16389760

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39056.7
system time used= 28.5187
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 4036336
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 37676
involuntary context switches= 181401

runsolver used 22.2436 second user time and 61.2147 second system time

The end

Launcher Data 

Begin job on node137 at 2011-05-15 22:50:24
IDJOB=3380799
IDBENCH=82394
IDSOLVER=1865
FILE ID=node137/3380799-1305492622
RUNJOBID= node137-1305491336-28600
PBS_JOBID= 13325413
Free space on /tmp= 73656 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.080-SAT.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380799-1305492622/watcher-3380799-1305492622 -o /tmp/evaluation-result-3380799-1305492622/solver-3380799-1305492622 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380799-1305492622.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 1ca09a2b911e29894c6603aee8d3e672
RANDOM SEED=2109525559

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      15049232 kB
Buffers:        320572 kB
Cached:        4560264 kB
SwapCached:      16844 kB
Active:        7957672 kB
Inactive:      9656188 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      15049232 kB
SwapTotal:    67111528 kB
SwapFree:     63082284 kB
Dirty:           51172 kB
Writeback:           0 kB
AnonPages:    12725764 kB
Mapped:          30836 kB
Slab:           142460 kB
PageTables:      68096 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 146924860 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73652 MiB
End job on node137 at 2011-05-16 00:15:26