Trace number 3321554

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 39476.6 5102.24

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.070-NOTKNOWN.cnf
MD5SUMf52ec4dacdcf65c3aedb61fb46ade01b
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1007.46
Satisfiable
(Un)Satisfiability was proved
Number of variables153284
Number of clauses2473656
Sum of the clauses size5647736
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 22387280
Number of clauses of size 36720
Number of clauses of size 431080
Number of clauses of size 513440
Number of clauses of size over 534440

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...359583/instance-3321554-1304359583.cnf
2.39/2.47	c Solving...
39474.26/5100.01	
39474.26/5100.01	*** INTERRUPTED! ***
39474.26/5100.01	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321554-1304359583/watcher-3321554-1304359583 -o /tmp/evaluation-result-3321554-1304359583/solver-3321554-1304359583 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321554-1304359583.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.95 8.31 8.18 1/219 14570
/proc/meminfo: memFree=19808224/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=40344 CPUtime=0 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 2170 0 0 0 0 0 0 0 21 0 1 0 547558822 41312256 2099 33554432000 4194304 4964704 140735886677920 18446744073709551615 4700332 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 10086 2106 376 189 0 4960 0

[startup+0.020157 s]
/proc/loadavg: 7.95 8.31 8.18 1/219 14570
/proc/meminfo: memFree=19808224/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=49332 CPUtime=0.01 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 6462 0 0 0 1 0 0 0 21 0 1 0 547558822 50515968 6138 33554432000 4194304 4964704 140735886677920 18446744073709551615 4679014 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 12333 6147 377 189 0 7207 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 49332

[startup+0.101129 s]
/proc/loadavg: 7.95 8.31 8.18 1/219 14570
/proc/meminfo: memFree=19808224/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=53288 CPUtime=0.09 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 9838 0 0 0 9 0 0 0 21 0 1 0 547558822 54566912 8399 33554432000 4194304 4964704 140735886677920 18446744073709551615 253640520389 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 13322 8400 389 189 0 8196 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 53288

[startup+0.30109 s]
/proc/loadavg: 7.95 8.31 8.18 1/219 14570
/proc/meminfo: memFree=19808224/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=86220 CPUtime=0.29 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 21535 0 0 0 27 2 0 0 22 0 1 0 547558822 88289280 16229 33554432000 4194304 4964704 140735886677920 18446744073709551615 253640520389 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 21555 16232 389 189 0 16429 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 86220

[startup+0.701014 s]
/proc/loadavg: 7.95 8.31 8.18 1/219 14570
/proc/meminfo: memFree=19808224/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=150196 CPUtime=0.69 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 44067 0 0 0 64 5 0 0 23 0 1 0 547558822 153800704 32100 33554432000 4194304 4964704 140735886677920 18446744073709551615 253640520020 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 37549 32101 399 189 0 32423 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 150196

[startup+1.50087 s]
/proc/loadavg: 7.95 8.31 8.18 2/220 14571
/proc/meminfo: memFree=19642924/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=191100 CPUtime=1.49 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 54081 0 0 0 143 6 0 0 25 0 1 0 547558822 195686400 42114 33554432000 4194304 4964704 140735886677920 18446744073709551615 4607554 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 47775 42114 403 189 0 42649 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 191100

[startup+3.10058 s]
/proc/loadavg: 7.95 8.31 8.18 2/220 14571
/proc/meminfo: memFree=19641560/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=599020 CPUtime=3.55 cores=0-7
/proc/14570/stat : 14570 (claspmt) S 14568 14570 13765 0 -1 4202496 91750 0 0 0 334 21 0 0 18 0 8 0 547558822 613396480 67843 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 149755 67843 443 189 0 144104 0
Current children cumulated CPU time (s) 3.55
Current children cumulated vsize (KiB) 599020

[startup+6.301 s]
/proc/loadavg: 7.40 8.19 8.15 3/227 14578
/proc/meminfo: memFree=19538956/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=599380 CPUtime=6.75 cores=0-7
/proc/14570/stat : 14570 (claspmt) S 14568 14570 13765 0 -1 4202496 91862 0 0 0 654 21 0 0 18 0 8 0 547558822 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 149845 67955 443 189 0 144194 0
[pid=14570/tid=14572] ppid=14568 vsize=599380 CPUtime=0.03 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) S 14568 14570 13765 0 -1 4202560 1308 0 0 0 3 0 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 1 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) S 14568 14570 13765 0 -1 4202560 7137 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 6 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) S 14568 14570 13765 0 -1 4202560 6961 0 0 0 7 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 7 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=599380 CPUtime=0.04 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) S 14568 14570 13765 0 -1 4202560 5915 0 0 0 2 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=599380 CPUtime=3.78 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5031 0 0 0 376 2 0 0 25 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) S 14568 14570 13765 0 -1 4202560 7193 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 2 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=599380 CPUtime=0.11 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) S 14568 14570 13765 0 -1 4202560 4150 0 0 0 10 1 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 6.75
Current children cumulated vsize (KiB) 599380

[startup+12.7008 s]
/proc/loadavg: 6.88 8.07 8.11 2/227 14578
/proc/meminfo: memFree=19540540/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=599380 CPUtime=13.15 cores=0-7
/proc/14570/stat : 14570 (claspmt) S 14568 14570 13765 0 -1 4202496 91862 0 0 0 1294 21 0 0 18 0 8 0 547558822 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 149845 67955 443 189 0 144194 0
[pid=14570/tid=14572] ppid=14568 vsize=599380 CPUtime=0.03 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) S 14568 14570 13765 0 -1 4202560 1308 0 0 0 3 0 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 1 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) S 14568 14570 13765 0 -1 4202560 7137 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 6 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) S 14568 14570 13765 0 -1 4202560 6961 0 0 0 7 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 7 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=599380 CPUtime=0.04 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) S 14568 14570 13765 0 -1 4202560 5915 0 0 0 2 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=599380 CPUtime=10.18 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5031 0 0 0 1016 2 0 0 25 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) S 14568 14570 13765 0 -1 4202560 7193 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 2 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=599380 CPUtime=0.11 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) S 14568 14570 13765 0 -1 4202560 4150 0 0 0 10 1 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 13.15
Current children cumulated vsize (KiB) 599380

[startup+25.5005 s]
/proc/loadavg: 5.58 7.72 7.99 2/227 14579
/proc/meminfo: memFree=19542432/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=599380 CPUtime=25.95 cores=0-7
/proc/14570/stat : 14570 (claspmt) S 14568 14570 13765 0 -1 4202496 91862 0 0 0 2574 21 0 0 18 0 8 0 547558822 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 149845 67955 443 189 0 144194 0
[pid=14570/tid=14572] ppid=14568 vsize=599380 CPUtime=0.03 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) S 14568 14570 13765 0 -1 4202560 1308 0 0 0 3 0 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 1 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) S 14568 14570 13765 0 -1 4202560 7137 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 6 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) S 14568 14570 13765 0 -1 4202560 6961 0 0 0 7 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 7 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=599380 CPUtime=0.04 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) S 14568 14570 13765 0 -1 4202560 5915 0 0 0 2 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=599380 CPUtime=22.98 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5031 0 0 0 2296 2 0 0 25 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) S 14568 14570 13765 0 -1 4202560 7193 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 2 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=599380 CPUtime=0.11 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) S 14568 14570 13765 0 -1 4202560 4150 0 0 0 10 1 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 25.95
Current children cumulated vsize (KiB) 599380

[startup+51.1008 s]
/proc/loadavg: 4.02 7.18 7.81 2/227 14580
/proc/meminfo: memFree=19541712/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=599380 CPUtime=51.55 cores=0-7
/proc/14570/stat : 14570 (claspmt) S 14568 14570 13765 0 -1 4202496 91862 0 0 0 5134 21 0 0 18 0 8 0 547558822 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 149845 67955 443 189 0 144194 0
[pid=14570/tid=14572] ppid=14568 vsize=599380 CPUtime=0.03 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) S 14568 14570 13765 0 -1 4202560 1308 0 0 0 3 0 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 1 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) S 14568 14570 13765 0 -1 4202560 7137 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 6 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) S 14568 14570 13765 0 -1 4202560 6961 0 0 0 7 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 7 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=599380 CPUtime=0.04 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) S 14568 14570 13765 0 -1 4202560 5915 0 0 0 2 2 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=599380 CPUtime=48.58 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5031 0 0 0 4856 2 0 0 25 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 4677956 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=599380 CPUtime=0.09 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) S 14568 14570 13765 0 -1 4202560 7193 0 0 0 6 3 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 2 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=599380 CPUtime=0.11 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) S 14568 14570 13765 0 -1 4202560 4150 0 0 0 10 1 0 0 18 0 8 0 547559069 613765120 67955 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 51.55
Current children cumulated vsize (KiB) 599380

[startup+102.307 s]
/proc/loadavg: 2.31 6.22 7.45 2/227 14581
/proc/meminfo: memFree=19541908/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=599920 CPUtime=102.76 cores=0-7
/proc/14570/stat : 14570 (claspmt) S 14568 14570 13765 0 -1 4202496 91954 0 0 0 10255 21 0 0 18 0 8 0 547558822 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 149980 68047 443 189 0 144329 0
[pid=14570/tid=14572] ppid=14568 vsize=599920 CPUtime=0.03 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) S 14568 14570 13765 0 -1 4202560 1308 0 0 0 3 0 0 0 18 0 8 0 547559069 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 1 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=599920 CPUtime=0.09 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) S 14568 14570 13765 0 -1 4202560 7137 0 0 0 6 3 0 0 18 0 8 0 547559069 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 6 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=599920 CPUtime=0.09 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) S 14568 14570 13765 0 -1 4202560 6961 0 0 0 7 2 0 0 18 0 8 0 547559069 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 7 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=599920 CPUtime=0.04 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) S 14568 14570 13765 0 -1 4202560 5915 0 0 0 2 2 0 0 18 0 8 0 547559069 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=599920 CPUtime=99.79 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5123 0 0 0 9977 2 0 0 25 0 8 0 547559069 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 4399297 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=599920 CPUtime=0.09 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) S 14568 14570 13765 0 -1 4202560 7193 0 0 0 6 3 0 0 18 0 8 0 547559069 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 2 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=599920 CPUtime=0.11 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) S 14568 14570 13765 0 -1 4202560 4150 0 0 0 10 1 0 0 18 0 8 0 547559069 614318080 68047 33554432000 4194304 4964704 140735886677920 18446744073709551615 253652675481 0 0 4096 18946 18446604453886837760 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 102.76
Current children cumulated vsize (KiB) 599920

[startup+162.301 s]
/proc/loadavg: 1.48 5.27 7.04 2/227 14582
/proc/meminfo: memFree=19541872/32951124 swapFree=67111364/67111528

################
# More data... #
################

[pid=14570/tid=14577] ppid=14568 vsize=15479164 CPUtime=4657.32 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) R 14568 14570 13765 0 -1 4202560 386912 0 0 0 465459 273 0 0 25 0 8 0 547559069 15850663936 3808471 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=15479164 CPUtime=4643.65 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) R 14568 14570 13765 0 -1 4202560 838533 0 0 0 463762 603 0 0 25 0 8 0 547559069 15850663936 3808471 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37415
Current children cumulated vsize (KiB) 15479164

[startup+4902.3 s]
/proc/loadavg: 8.03 8.03 8.00 9/227 14721
/proc/meminfo: memFree=4551952/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=15479164 CPUtime=37894.5 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3832802 0 0 0 3786909 2545 0 0 25 0 8 0 547558822 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669270 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 3869791 3808490 466 189 0 3864140 0
[pid=14570/tid=14572] ppid=14568 vsize=15479164 CPUtime=4717.98 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) R 14568 14570 13765 0 -1 4202560 174381 0 0 0 471631 167 0 0 25 0 8 0 547559069 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=15479164 CPUtime=4709.51 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) R 14568 14570 13765 0 -1 4202560 1107089 0 0 0 470488 463 0 0 25 0 8 0 547559069 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=15479164 CPUtime=4707.06 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) R 14568 14570 13765 0 -1 4202560 599375 0 0 0 470260 446 0 0 25 0 8 0 547559069 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4361330 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=15479164 CPUtime=4718.38 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) R 14568 14570 13765 0 -1 4202560 429733 0 0 0 471472 366 0 0 25 0 8 0 547559069 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=15479164 CPUtime=4899.88 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5162 0 0 0 489986 2 0 0 25 0 8 0 547559069 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4399264 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=15479164 CPUtime=4717.32 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) R 14568 14570 13765 0 -1 4202560 386912 0 0 0 471459 273 0 0 25 0 8 0 547559069 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4363263 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=15479164 CPUtime=4703.65 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) R 14568 14570 13765 0 -1 4202560 838533 0 0 0 469762 603 0 0 25 0 8 0 547559069 15850663936 3808490 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37894.5
Current children cumulated vsize (KiB) 15479164

[startup+4962.3 s]
/proc/loadavg: 8.11 8.05 8.00 9/227 14722
/proc/meminfo: memFree=4551688/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=15479164 CPUtime=38374.1 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3832847 0 0 0 3834865 2545 0 0 25 0 8 0 547558822 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670100 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 3869791 3808535 466 189 0 3864140 0
[pid=14570/tid=14572] ppid=14568 vsize=15479164 CPUtime=4777.98 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) R 14568 14570 13765 0 -1 4202560 174409 0 0 0 477631 167 0 0 25 0 8 0 547559069 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=15479164 CPUtime=4769.07 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) R 14568 14570 13765 0 -1 4202560 1107106 0 0 0 476444 463 0 0 25 0 8 0 547559069 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4313751 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=15479164 CPUtime=4767.06 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) R 14568 14570 13765 0 -1 4202560 599375 0 0 0 476260 446 0 0 25 0 8 0 547559069 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=15479164 CPUtime=4778.38 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) R 14568 14570 13765 0 -1 4202560 429733 0 0 0 477472 366 0 0 25 0 8 0 547559069 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=15479164 CPUtime=4959.88 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5162 0 0 0 495986 2 0 0 25 0 8 0 547559069 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=15479164 CPUtime=4777.32 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) R 14568 14570 13765 0 -1 4202560 386912 0 0 0 477459 273 0 0 25 0 8 0 547559069 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=15479164 CPUtime=4763.65 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) R 14568 14570 13765 0 -1 4202560 838533 0 0 0 475762 603 0 0 25 0 8 0 547559069 15850663936 3808535 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670038 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38374.1
Current children cumulated vsize (KiB) 15479164

[startup+5022.3 s]
/proc/loadavg: 8.04 8.04 8.00 9/227 14724
/proc/meminfo: memFree=4541616/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=15544700 CPUtime=38853.2 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3835304 0 0 0 3882771 2547 0 0 25 0 8 0 547558822 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669499 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 3886175 3810992 466 189 0 3880524 0
[pid=14570/tid=14572] ppid=14568 vsize=15544700 CPUtime=4837.71 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) R 14568 14570 13765 0 -1 4202560 174423 0 0 0 483604 167 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=15544700 CPUtime=4828.47 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) R 14568 14570 13765 0 -1 4202560 1107110 0 0 0 482383 464 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=15544700 CPUtime=4827.04 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) R 14568 14570 13765 0 -1 4202560 599375 0 0 0 482258 446 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=15544700 CPUtime=4838.39 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) R 14568 14570 13765 0 -1 4202560 432172 0 0 0 483471 368 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=15544700 CPUtime=5019.88 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5162 0 0 0 501986 2 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669296 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=15544700 CPUtime=4837.31 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) R 14568 14570 13765 0 -1 4202560 386912 0 0 0 483458 273 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4332576 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=15544700 CPUtime=4823.66 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) R 14568 14570 13765 0 -1 4202560 838533 0 0 0 481762 604 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38853.2
Current children cumulated vsize (KiB) 15544700

[startup+5082.3 s]
/proc/loadavg: 8.01 8.03 8.00 9/227 14726
/proc/meminfo: memFree=4541716/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=15544700 CPUtime=39332.7 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3835304 0 0 0 3930727 2547 0 0 25 0 8 0 547558822 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 3886175 3810992 466 189 0 3880524 0
[pid=14570/tid=14572] ppid=14568 vsize=15544700 CPUtime=4897.71 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) R 14568 14570 13765 0 -1 4202560 174423 0 0 0 489604 167 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=15544700 CPUtime=4888.03 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) R 14568 14570 13765 0 -1 4202560 1107110 0 0 0 488339 464 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4363370 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=15544700 CPUtime=4887.04 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) R 14568 14570 13765 0 -1 4202560 599375 0 0 0 488258 446 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=15544700 CPUtime=4898.39 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) R 14568 14570 13765 0 -1 4202560 432172 0 0 0 489471 368 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=15544700 CPUtime=5079.88 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5162 0 0 0 507986 2 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=15544700 CPUtime=4897.31 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) R 14568 14570 13765 0 -1 4202560 386912 0 0 0 489458 273 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4668928 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=15544700 CPUtime=4883.66 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) R 14568 14570 13765 0 -1 4202560 838533 0 0 0 487762 604 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39332.7
Current children cumulated vsize (KiB) 15544700



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 8.01 8.03 8.00 9/227 14726
/proc/meminfo: memFree=4541592/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=15544700 CPUtime=39474.3 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3835304 0 0 0 3944879 2547 0 0 25 0 8 0 547558822 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4668472 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 3886175 3810992 466 189 0 3880524 0
[pid=14570/tid=14572] ppid=14568 vsize=15544700 CPUtime=4915.41 cores=0-7
/proc/14570/task/14572/stat : 14572 (claspmt) R 14568 14570 13765 0 -1 4202560 174423 0 0 0 491374 167 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4659143 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=14570/tid=14573] ppid=14568 vsize=15544700 CPUtime=4905.6 cores=0-7
/proc/14570/task/14573/stat : 14573 (claspmt) R 14568 14570 13765 0 -1 4202560 1107110 0 0 0 490096 464 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=14570/tid=14574] ppid=14568 vsize=15544700 CPUtime=4904.75 cores=0-7
/proc/14570/task/14574/stat : 14574 (claspmt) R 14568 14570 13765 0 -1 4202560 599375 0 0 0 490029 446 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=14570/tid=14575] ppid=14568 vsize=15544700 CPUtime=4916.1 cores=0-7
/proc/14570/task/14575/stat : 14575 (claspmt) R 14568 14570 13765 0 -1 4202560 432172 0 0 0 491242 368 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=14570/tid=14576] ppid=14568 vsize=15544700 CPUtime=5097.59 cores=0-7
/proc/14570/task/14576/stat : 14576 (claspmt) R 14568 14570 13765 0 -1 4202560 5162 0 0 0 509757 2 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4669051 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=14570/tid=14577] ppid=14568 vsize=15544700 CPUtime=4915.02 cores=0-7
/proc/14570/task/14577/stat : 14577 (claspmt) R 14568 14570 13765 0 -1 4202560 386912 0 0 0 491229 273 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4361293 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=14570/tid=14578] ppid=14568 vsize=15544700 CPUtime=4901.36 cores=0-7
/proc/14570/task/14578/stat : 14578 (claspmt) R 14568 14570 13765 0 -1 4202560 838533 0 0 0 489532 604 0 0 25 0 8 0 547559069 15917772800 3810992 33554432000 4194304 4964704 140735886677920 18446744073709551615 4365682 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39474.3
Current children cumulated vsize (KiB) 15544700

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -14570

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 8.01 8.03 8.00 2/221 14727
/proc/meminfo: memFree=4560364/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=15525384 CPUtime=39474.9 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3835314 0 0 0 3944945 2547 0 0 25 0 1 0 547558822 15897993216 3806173 33554432000 4194304 4964704 140735886677920 18446744073709551615 253640516145 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 3881346 3806173 476 189 0 3875695 0
Current children cumulated CPU time (s) 39474.9
Current children cumulated vsize (KiB) 15525384

[startup+5101.4 s]
/proc/loadavg: 8.01 8.03 8.00 2/221 14727
/proc/meminfo: memFree=4560364/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=15459848 CPUtime=39475.7 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3835314 0 0 0 3945024 2547 0 0 25 0 1 0 547558822 15830884352 3803080 33554432000 4194304 4964704 140735886677920 18446744073709551615 253640514726 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 3864962 3803080 476 189 0 3859311 0
Current children cumulated CPU time (s) 39475.7
Current children cumulated vsize (KiB) 15459848

[startup+5101.81 s]
/proc/loadavg: 7.45 7.91 7.96 2/221 14727
/proc/meminfo: memFree=4836508/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=12510728 CPUtime=39476.1 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4202496 3835314 0 0 0 3945054 2558 0 0 25 0 1 0 547558822 12810985472 3048258 33554432000 4194304 4964704 140735886677920 18446744073709551615 253640902791 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/14570/statm: 3127682 3047746 476 189 0 3122031 0
Current children cumulated CPU time (s) 39476.1
Current children cumulated vsize (KiB) 12510728

[startup+5102.2 s]
/proc/loadavg: 7.45 7.91 7.96 2/221 14727
/proc/meminfo: memFree=4836508/32951124 swapFree=67111364/67111528
[pid=14570] ppid=14568 vsize=0 CPUtime=39476.5 cores=0-7
/proc/14570/stat : 14570 (claspmt) R 14568 14570 13765 0 -1 4203524 3835314 0 0 0 3945054 2597 0 0 25 0 1 0 547558822 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/14570/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 39476.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5102.24
CPU time (s): 39476.6
CPU user time (s): 39450.5
CPU system time (s): 26.011
CPU usage (%): 773.71
Max. virtual memory (cumulated for all children) (KiB): 15544700

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39450.5
system time used= 26.011
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3835314
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 39864
involuntary context switches= 181627

runsolver used 10.2164 second user time and 26.532 second system time

The end

Launcher Data 

Begin job on node106 at 2011-05-02 20:06:24
IDJOB=3321554
IDBENCH=82363
IDSOLVER=1711
FILE ID=node106/3321554-1304359583
RUNJOBID= node106-1304348663-13779
PBS_JOBID= 13178667
Free space on /tmp= 73392 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.070-NOTKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321554-1304359583/watcher-3321554-1304359583 -o /tmp/evaluation-result-3321554-1304359583/solver-3321554-1304359583 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321554-1304359583.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f52ec4dacdcf65c3aedb61fb46ade01b
RANDOM SEED=192520384

node106.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19808496 kB
Buffers:        319700 kB
Cached:        3604988 kB
SwapCached:         88 kB
Active:       10260900 kB
Inactive:      2622040 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19808496 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           44668 kB
Writeback:           0 kB
AnonPages:     8958480 kB
Mapped:          23028 kB
Slab:           168980 kB
PageTables:      25068 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 10339560 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73392 MiB
End job on node106 at 2011-05-02 21:31:27