Trace number 3381357

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 38981.6 5103.23

General information on the benchmark

NameSAT-Race-2010-CNF/hardware-verification/
velev/2dlx_ca_bp_f_liveness.cnf
MD5SUM99cd8de7f06ce8c087a5c7fb299dfb62
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables202253
Number of clauses4313014
Sum of the clauses size12562188
Maximum clause length651
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 23935640
Number of clauses of size 3199653
Number of clauses of size 410363
Number of clauses of size 55846
Number of clauses of size over 5161511

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...513717/instance-3381357-1305513717.cnf
1.29/1.36	c Solving...
38978.26/5100.02	
38978.26/5100.02	*** INTERRUPTED! ***
38978.26/5100.03	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381357-1305513717/watcher-3381357-1305513717 -o /tmp/evaluation-result-3381357-1305513717/solver-3381357-1305513717 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381357-1305513717.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.03 8.57 8.56 2/429 3959
/proc/meminfo: memFree=16472204/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=46560 CPUtime=0 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 1161 0 0 0 0 0 0 0 22 0 1 0 662969689 47677440 1092 33554432000 4194304 4964800 140736596414032 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/3959/statm: 11640 1100 376 189 0 6514 0

[startup+0.0794329 s]
/proc/loadavg: 8.03 8.57 8.56 2/429 3959
/proc/meminfo: memFree=16472204/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=64472 CPUtime=0.07 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 12785 0 0 0 6 1 0 0 22 0 1 0 662969689 66019328 11136 33554432000 4194304 4964800 140736596414032 18446744073709551615 228244348672 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/3959/statm: 16118 11137 388 189 0 10992 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 64472

[startup+0.100409 s]
/proc/loadavg: 8.03 8.57 8.56 2/429 3959
/proc/meminfo: memFree=16472204/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=66000 CPUtime=0.09 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 13211 0 0 0 8 1 0 0 22 0 1 0 662969689 67584000 11529 33554432000 4194304 4964800 140736596414032 18446744073709551615 4581897 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/3959/statm: 16500 11530 388 189 0 11374 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 66000

[startup+0.300393 s]
/proc/loadavg: 8.03 8.57 8.56 2/429 3959
/proc/meminfo: memFree=16472204/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=79956 CPUtime=0.29 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 16829 0 0 0 27 2 0 0 23 0 1 0 662969689 81874944 15022 33554432000 4194304 4964800 140736596414032 18446744073709551615 228244348672 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/3959/statm: 19989 15023 388 189 0 14863 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 79956

[startup+0.700303 s]
/proc/loadavg: 8.03 8.57 8.56 2/429 3959
/proc/meminfo: memFree=16472204/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=106572 CPUtime=0.69 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 23677 0 0 0 66 3 0 0 25 0 1 0 662969689 109129728 21591 33554432000 4194304 4964800 140736596414032 18446744073709551615 4567832 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/3959/statm: 26643 21592 388 189 0 21517 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 106572

[startup+1.50013 s]
/proc/loadavg: 8.03 8.57 8.56 2/430 3960
/proc/meminfo: memFree=16368540/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=738020 CPUtime=1.73 cores=0-7
/proc/3959/stat : 3959 (claspmt) S 3957 3959 3934 0 -1 4202496 86086 0 0 0 142 31 0 0 25 0 8 0 662969689 755732480 68686 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/3959/statm: 184505 68686 431 189 0 178854 0
Current children cumulated CPU time (s) 1.73
Current children cumulated vsize (KiB) 738020

[startup+3.10089 s]
/proc/loadavg: 7.79 8.51 8.54 2/437 3967
/proc/meminfo: memFree=16051308/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=759472 CPUtime=4.13 cores=0-7
/proc/3959/stat : 3959 (claspmt) S 3957 3959 3934 0 -1 4202496 124512 0 0 0 355 58 0 0 25 0 8 0 662969689 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/3959/statm: 189868 105811 437 189 0 184217 0
[pid=3959/tid=3961] ppid=3957 vsize=759472 CPUtime=0.14 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) S 3957 3959 3934 0 -1 4202560 10696 0 0 0 8 6 0 0 17 0 8 0 662969826 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 1 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) S 3957 3959 3934 0 -1 4202560 13149 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) S 3957 3959 3934 0 -1 4202560 13499 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 2 0 0 0
[pid=3959/tid=3964] ppid=3957 vsize=759472 CPUtime=0.13 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) S 3957 3959 3934 0 -1 4202560 10593 0 0 0 7 6 0 0 18 0 8 0 662969826 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 7 0 0 0
[pid=3959/tid=3965] ppid=3957 vsize=759472 CPUtime=1.48 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 13411 0 0 0 141 7 0 0 18 0 8 0 662969826 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 4342154 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=759472 CPUtime=0.27 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) S 3957 3959 3934 0 -1 4202560 13170 0 0 0 20 7 0 0 18 0 8 0 662969826 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 3 0 0 0
[pid=3959/tid=3967] ppid=3957 vsize=759472 CPUtime=0.29 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) S 3957 3959 3934 0 -1 4202560 11905 0 0 0 21 8 0 0 17 0 8 0 662969826 777699328 105811 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4.13
Current children cumulated vsize (KiB) 759472

[startup+6.30133 s]
/proc/loadavg: 7.79 8.51 8.54 2/437 3967
/proc/meminfo: memFree=16047852/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=759472 CPUtime=7.33 cores=0-7
/proc/3959/stat : 3959 (claspmt) S 3957 3959 3934 0 -1 4202496 124731 0 0 0 675 58 0 0 25 0 8 0 662969689 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/3959/statm: 189868 106030 437 189 0 184217 0
[pid=3959/tid=3961] ppid=3957 vsize=759472 CPUtime=0.14 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) S 3957 3959 3934 0 -1 4202560 10696 0 0 0 8 6 0 0 17 0 8 0 662969826 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 1 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) S 3957 3959 3934 0 -1 4202560 13149 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) S 3957 3959 3934 0 -1 4202560 13499 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 2 0 0 0
[pid=3959/tid=3964] ppid=3957 vsize=759472 CPUtime=0.13 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) S 3957 3959 3934 0 -1 4202560 10593 0 0 0 7 6 0 0 18 0 8 0 662969826 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 7 0 0 0
[pid=3959/tid=3965] ppid=3957 vsize=759472 CPUtime=4.68 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 13630 0 0 0 461 7 0 0 25 0 8 0 662969826 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=759472 CPUtime=0.27 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) S 3957 3959 3934 0 -1 4202560 13170 0 0 0 20 7 0 0 18 0 8 0 662969826 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 3 0 0 0
[pid=3959/tid=3967] ppid=3957 vsize=759472 CPUtime=0.29 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) S 3957 3959 3934 0 -1 4202560 11905 0 0 0 21 8 0 0 17 0 8 0 662969826 777699328 106030 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 7.33
Current children cumulated vsize (KiB) 759472

[startup+12.7012 s]
/proc/loadavg: 6.74 8.26 8.46 2/437 3968
/proc/meminfo: memFree=16050396/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=759472 CPUtime=13.73 cores=0-7
/proc/3959/stat : 3959 (claspmt) S 3957 3959 3934 0 -1 4202496 125167 0 0 0 1315 58 0 0 25 0 8 0 662969689 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/3959/statm: 189868 106466 437 189 0 184217 0
[pid=3959/tid=3961] ppid=3957 vsize=759472 CPUtime=0.14 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) S 3957 3959 3934 0 -1 4202560 10696 0 0 0 8 6 0 0 17 0 8 0 662969826 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 1 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) S 3957 3959 3934 0 -1 4202560 13149 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) S 3957 3959 3934 0 -1 4202560 13499 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 2 0 0 0
[pid=3959/tid=3964] ppid=3957 vsize=759472 CPUtime=0.13 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) S 3957 3959 3934 0 -1 4202560 10593 0 0 0 7 6 0 0 18 0 8 0 662969826 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 7 0 0 0
[pid=3959/tid=3965] ppid=3957 vsize=759472 CPUtime=11.08 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 14066 0 0 0 1101 7 0 0 25 0 8 0 662969826 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=759472 CPUtime=0.27 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) S 3957 3959 3934 0 -1 4202560 13170 0 0 0 20 7 0 0 18 0 8 0 662969826 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 3 0 0 0
[pid=3959/tid=3967] ppid=3957 vsize=759472 CPUtime=0.29 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) S 3957 3959 3934 0 -1 4202560 11905 0 0 0 21 8 0 0 17 0 8 0 662969826 777699328 106466 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 13.73
Current children cumulated vsize (KiB) 759472

[startup+25.501 s]
/proc/loadavg: 5.86 8.02 8.38 2/437 3968
/proc/meminfo: memFree=16050660/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=759472 CPUtime=26.53 cores=0-7
/proc/3959/stat : 3959 (claspmt) S 3957 3959 3934 0 -1 4202496 125459 0 0 0 2595 58 0 0 25 0 8 0 662969689 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/3959/statm: 189868 106758 437 189 0 184217 0
[pid=3959/tid=3961] ppid=3957 vsize=759472 CPUtime=0.14 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) S 3957 3959 3934 0 -1 4202560 10696 0 0 0 8 6 0 0 17 0 8 0 662969826 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 1 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) S 3957 3959 3934 0 -1 4202560 13149 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) S 3957 3959 3934 0 -1 4202560 13499 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 2 0 0 0
[pid=3959/tid=3964] ppid=3957 vsize=759472 CPUtime=0.13 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) S 3957 3959 3934 0 -1 4202560 10593 0 0 0 7 6 0 0 18 0 8 0 662969826 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 7 0 0 0
[pid=3959/tid=3965] ppid=3957 vsize=759472 CPUtime=23.88 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 14358 0 0 0 2381 7 0 0 25 0 8 0 662969826 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=759472 CPUtime=0.27 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) S 3957 3959 3934 0 -1 4202560 13170 0 0 0 20 7 0 0 18 0 8 0 662969826 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 3 0 0 0
[pid=3959/tid=3967] ppid=3957 vsize=759472 CPUtime=0.29 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) S 3957 3959 3934 0 -1 4202560 11905 0 0 0 21 8 0 0 17 0 8 0 662969826 777699328 106758 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 26.53
Current children cumulated vsize (KiB) 759472

[startup+51.1005 s]
/proc/loadavg: 4.20 7.45 8.19 2/437 3969
/proc/meminfo: memFree=16048448/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=759472 CPUtime=52.12 cores=0-7
/proc/3959/stat : 3959 (claspmt) S 3957 3959 3934 0 -1 4202496 125850 0 0 0 5154 58 0 0 25 0 8 0 662969689 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/3959/statm: 189868 107149 437 189 0 184217 0
[pid=3959/tid=3961] ppid=3957 vsize=759472 CPUtime=0.14 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) S 3957 3959 3934 0 -1 4202560 10696 0 0 0 8 6 0 0 17 0 8 0 662969826 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 1 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) S 3957 3959 3934 0 -1 4202560 13149 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) S 3957 3959 3934 0 -1 4202560 13499 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 2 0 0 0
[pid=3959/tid=3964] ppid=3957 vsize=759472 CPUtime=0.13 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) S 3957 3959 3934 0 -1 4202560 10593 0 0 0 7 6 0 0 18 0 8 0 662969826 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 7 0 0 0
[pid=3959/tid=3965] ppid=3957 vsize=759472 CPUtime=49.47 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 14749 0 0 0 4940 7 0 0 25 0 8 0 662969826 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=759472 CPUtime=0.27 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) S 3957 3959 3934 0 -1 4202560 13170 0 0 0 20 7 0 0 18 0 8 0 662969826 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 3 0 0 0
[pid=3959/tid=3967] ppid=3957 vsize=759472 CPUtime=0.29 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) S 3957 3959 3934 0 -1 4202560 11905 0 0 0 21 8 0 0 17 0 8 0 662969826 777699328 107149 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 52.12
Current children cumulated vsize (KiB) 759472

[startup+102.317 s]
/proc/loadavg: 2.34 6.38 7.77 2/437 3970
/proc/meminfo: memFree=16048088/32951124 swapFree=67082160/67111528
[pid=3959] ppid=3957 vsize=759472 CPUtime=103.34 cores=0-7
/proc/3959/stat : 3959 (claspmt) S 3957 3959 3934 0 -1 4202496 126017 0 0 0 10276 58 0 0 25 0 8 0 662969689 777699328 107316 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/3959/statm: 189868 107316 437 189 0 184217 0
[pid=3959/tid=3961] ppid=3957 vsize=759472 CPUtime=0.14 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) S 3957 3959 3934 0 -1 4202560 10696 0 0 0 8 6 0 0 17 0 8 0 662969826 777699328 107316 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 1 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) S 3957 3959 3934 0 -1 4202560 13149 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 107316 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=759472 CPUtime=0.19 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) S 3957 3959 3934 0 -1 4202560 13499 0 0 0 13 6 0 0 17 0 8 0 662969826 777699328 107316 33554432000 4194304 4964800 140736596414032 18446744073709551615 228256164761 0 0 4096 18946 18446604445569638592 0 0 -1 2 0 0 0

################
# More data... #
################

[pid=3959/tid=3966] ppid=3957 vsize=25240112 CPUtime=4593.76 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) R 3957 3959 3934 0 -1 4202560 341272 0 3 0 458940 436 0 0 25 0 8 0 662969826 25845874688 6232758 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 2 0 0 4
[pid=3959/tid=3967] ppid=3957 vsize=25240112 CPUtime=4577.03 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) R 3957 3959 3934 0 -1 4202560 2569740 0 7 0 456427 1276 0 0 25 0 8 0 662969826 25845874688 6232758 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 13
Current children cumulated CPU time (s) 36925.7
Current children cumulated vsize (KiB) 25240112

[startup+4902.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/437 4119
/proc/meminfo: memFree=95596/32951124 swapFree=64907828/67111528
[pid=3959] ppid=3957 vsize=25371184 CPUtime=37404.3 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 6306519 0 105 0 3735624 4802 0 0 25 0 8 0 662969689 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668900 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 6342796 6261332 460 189 0 6337145 0
[pid=3959/tid=3961] ppid=3957 vsize=25371184 CPUtime=4650.24 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) R 3957 3959 3934 0 -1 4202560 270755 0 0 0 464678 346 0 0 25 0 8 0 662969826 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=25371184 CPUtime=4620.51 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) R 3957 3959 3934 0 -1 4202560 1740262 0 3 0 460747 1304 0 0 25 0 8 0 662969826 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=25371184 CPUtime=4653.69 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) R 3957 3959 3934 0 -1 4202560 317541 0 80 0 464980 389 0 0 25 0 8 0 662969826 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 35
[pid=3959/tid=3964] ppid=3957 vsize=25371184 CPUtime=4639.52 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) R 3957 3959 3934 0 -1 4202560 630509 0 9 0 463335 617 0 0 25 0 8 0 662969826 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 4 0 0 14
[pid=3959/tid=3965] ppid=3957 vsize=25371184 CPUtime=4900.56 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 15148 0 0 0 490048 8 0 0 25 0 8 0 662969826 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4669136 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=25371184 CPUtime=4653.67 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) R 3957 3959 3934 0 -1 4202560 341322 0 3 0 464931 436 0 0 25 0 8 0 662969826 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 4
[pid=3959/tid=3967] ppid=3957 vsize=25371184 CPUtime=4636.16 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) R 3957 3959 3934 0 -1 4202560 2569740 0 8 0 462340 1276 0 0 25 0 8 0 662969826 25980092416 6261332 33554432000 4194304 4964800 140736596414032 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 7 0 0 15
Current children cumulated CPU time (s) 37404.3
Current children cumulated vsize (KiB) 25371184

[startup+4962.3 s]
/proc/loadavg: 8.22 8.08 8.02 10/438 4124
/proc/meminfo: memFree=92308/32951124 swapFree=64778404/67111528
[pid=3959] ppid=3957 vsize=25502256 CPUtime=37882.4 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 6337763 0 105 0 3783409 4829 0 0 25 0 8 0 662969689 26114310144 6292538 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668712 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 6375564 6292538 460 189 0 6369913 0
[pid=3959/tid=3961] ppid=3957 vsize=25502256 CPUtime=4709.73 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) R 3957 3959 3934 0 -1 4202560 270759 0 0 0 470627 346 0 0 25 0 8 0 662969826 26114310144 6292538 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=25502256 CPUtime=4680.06 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) R 3957 3959 3934 0 -1 4202560 1771387 0 3 0 466680 1326 0 0 25 0 8 0 662969826 26114310144 6292538 33554432000 4194304 4964800 140736596414032 18446744073709551615 4325336 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=25502256 CPUtime=4713.66 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) R 3957 3959 3934 0 -1 4202560 317581 0 80 0 470973 393 0 0 25 0 8 0 662969826 26114310144 6292538 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 35
[pid=3959/tid=3964] ppid=3957 vsize=25502256 CPUtime=4699.52 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) R 3957 3959 3934 0 -1 4202560 630557 0 9 0 469335 617 0 0 25 0 8 0 662969826 26114310144 6292538 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 14
[pid=3959/tid=3965] ppid=3957 vsize=25502256 CPUtime=4960.56 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 15148 0 0 0 496048 8 0 0 25 0 8 0 662969826 26114310144 6292544 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=25502256 CPUtime=4713.67 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) R 3957 3959 3934 0 -1 4202560 341349 0 3 0 470931 436 0 0 25 0 8 0 662969826 26114310144 6292544 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 0 0 0 4
[pid=3959/tid=3967] ppid=3957 vsize=25502256 CPUtime=4695.26 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) R 3957 3959 3934 0 -1 4202560 2569740 0 8 0 468250 1276 0 0 25 0 8 0 662969826 26114310144 6292544 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 15
Current children cumulated CPU time (s) 37882.4
Current children cumulated vsize (KiB) 25502256

[startup+5022.3 s]
/proc/loadavg: 8.71 8.24 8.07 10/438 4125
/proc/meminfo: memFree=93224/32951124 swapFree=64548404/67111528
[pid=3959] ppid=3957 vsize=25634900 CPUtime=38360.4 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 6379141 0 105 0 3831177 4865 0 0 25 0 8 0 662969689 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4653507 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 6408725 6333910 460 189 0 6403074 0
[pid=3959/tid=3961] ppid=3957 vsize=25634900 CPUtime=4768.86 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) R 3957 3959 3934 0 -1 4202560 270765 0 0 0 476540 346 0 0 25 0 8 0 662969826 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=25634900 CPUtime=4739.92 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) R 3957 3959 3934 0 -1 4202560 1811501 0 3 0 472637 1355 0 0 25 0 8 0 662969826 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=25634900 CPUtime=4773.51 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) R 3957 3959 3934 0 -1 4202560 317896 0 80 0 476951 400 0 0 25 0 8 0 662969826 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 6 0 0 35
[pid=3959/tid=3964] ppid=3957 vsize=25634900 CPUtime=4759.27 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) R 3957 3959 3934 0 -1 4202560 630598 0 9 0 475310 617 0 0 25 0 8 0 662969826 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4370224 0 0 4096 18946 0 0 0 -1 2 0 0 14
[pid=3959/tid=3965] ppid=3957 vsize=25634900 CPUtime=5020.57 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 15148 0 0 0 502049 8 0 0 25 0 8 0 662969826 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=25634900 CPUtime=4773.67 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) R 3957 3959 3934 0 -1 4202560 342250 0 3 0 476930 437 0 0 25 0 8 0 662969826 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 4
[pid=3959/tid=3967] ppid=3957 vsize=25634900 CPUtime=4754.72 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) R 3957 3959 3934 0 -1 4202560 2569741 0 8 0 474196 1276 0 0 25 0 8 0 662969826 26250137600 6333910 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 15
Current children cumulated CPU time (s) 38360.4
Current children cumulated vsize (KiB) 25634900

[startup+5082.3 s]
/proc/loadavg: 8.81 8.36 8.12 9/437 4127
/proc/meminfo: memFree=131416/32951124 swapFree=64202088/67111528
[pid=3959] ppid=3957 vsize=25968732 CPUtime=38836.9 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 6462499 0 105 0 3878757 4930 0 0 25 0 8 0 662969689 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4684054 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 6492183 6414183 460 189 0 6486532 0
[pid=3959/tid=3961] ppid=3957 vsize=25968732 CPUtime=4827.96 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) R 3957 3959 3934 0 -1 4202560 270766 0 0 0 482450 346 0 0 25 0 8 0 662969826 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=25968732 CPUtime=4799.92 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) R 3957 3959 3934 0 -1 4202560 1828610 0 3 0 478625 1367 0 0 25 0 8 0 662969826 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4313736 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=25968732 CPUtime=4833.02 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) R 3957 3959 3934 0 -1 4202560 317938 0 80 0 482899 403 0 0 25 0 8 0 662969826 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 35
[pid=3959/tid=3964] ppid=3957 vsize=25968732 CPUtime=4818.34 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) R 3957 3959 3934 0 -1 4202560 633705 0 9 0 481209 625 0 0 25 0 8 0 662969826 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 6 0 0 14
[pid=3959/tid=3965] ppid=3957 vsize=25968732 CPUtime=5080.57 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 15148 0 0 0 508049 8 0 0 25 0 8 0 662969826 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4342180 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=25968732 CPUtime=4833.67 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) R 3957 3959 3934 0 -1 4202560 348191 0 3 0 482924 443 0 0 25 0 8 0 662969826 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 4
[pid=3959/tid=3967] ppid=3957 vsize=25968732 CPUtime=4813.52 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) R 3957 3959 3934 0 -1 4202560 2626883 0 8 0 480041 1311 0 0 25 0 8 0 662969826 26591981568 6414183 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 15
Current children cumulated CPU time (s) 38836.9
Current children cumulated vsize (KiB) 25968732



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 8.63 8.34 8.12 9/437 4127
/proc/meminfo: memFree=132036/32951124 swapFree=64202088/67111528
[pid=3959] ppid=3957 vsize=25968732 CPUtime=38978.3 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 6462530 0 105 0 3892896 4930 0 0 25 0 8 0 662969689 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 6492183 6414213 460 189 0 6486532 0
[pid=3959/tid=3961] ppid=3957 vsize=25968732 CPUtime=4845.41 cores=0-7
/proc/3959/task/3961/stat : 3961 (claspmt) R 3957 3959 3934 0 -1 4202560 270768 0 0 0 484195 346 0 0 25 0 8 0 662969826 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3959/tid=3962] ppid=3957 vsize=25968732 CPUtime=4817.63 cores=0-7
/proc/3959/task/3962/stat : 3962 (claspmt) R 3957 3959 3934 0 -1 4202560 1828619 0 3 0 480396 1367 0 0 25 0 8 0 662969826 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=3959/tid=3963] ppid=3957 vsize=25968732 CPUtime=4850.73 cores=0-7
/proc/3959/task/3963/stat : 3963 (claspmt) R 3957 3959 3934 0 -1 4202560 317952 0 80 0 484670 403 0 0 25 0 8 0 662969826 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 35
[pid=3959/tid=3964] ppid=3957 vsize=25968732 CPUtime=4836.06 cores=0-7
/proc/3959/task/3964/stat : 3964 (claspmt) R 3957 3959 3934 0 -1 4202560 633705 0 9 0 482981 625 0 0 25 0 8 0 662969826 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 6 0 0 14
[pid=3959/tid=3965] ppid=3957 vsize=25968732 CPUtime=5098.28 cores=0-7
/proc/3959/task/3965/stat : 3965 (claspmt) R 3957 3959 3934 0 -1 4202560 15148 0 0 0 509820 8 0 0 25 0 8 0 662969826 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3959/tid=3966] ppid=3957 vsize=25968732 CPUtime=4851.38 cores=0-7
/proc/3959/task/3966/stat : 3966 (claspmt) R 3957 3959 3934 0 -1 4202560 348197 0 3 0 484695 443 0 0 25 0 8 0 662969826 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 0 0 0 4
[pid=3959/tid=3967] ppid=3957 vsize=25968732 CPUtime=4831.24 cores=0-7
/proc/3959/task/3967/stat : 3967 (claspmt) R 3957 3959 3934 0 -1 4202560 2626883 0 8 0 481813 1311 0 0 25 0 8 0 662969826 26591981568 6414213 33554432000 4194304 4964800 140736596414032 18446744073709551615 4668998 0 0 4096 18946 0 0 0 -1 3 0 0 15
Current children cumulated CPU time (s) 38978.3
Current children cumulated vsize (KiB) 25968732

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -3959

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 8.63 8.34 8.12 2/431 4128
/proc/meminfo: memFree=168292/32951124 swapFree=64202088/67111528
[pid=3959] ppid=3957 vsize=25933764 CPUtime=38979 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4202496 6462543 0 105 0 3892968 4930 0 0 25 0 1 0 662969689 26556174336 6404984 33554432000 4194304 4964800 140736596414032 18446744073709551615 228244005425 0 0 4096 18946 0 0 0 17 1 0 0 4
/proc/3959/statm: 6483441 6404984 470 189 0 6477790 0
Current children cumulated CPU time (s) 38979
Current children cumulated vsize (KiB) 25933764

[startup+5102.2 s]
/proc/loadavg: 8.02 8.22 8.08 2/431 4128
/proc/meminfo: memFree=693308/32951124 swapFree=64202088/67111528
[pid=3959] ppid=3957 vsize=0 CPUtime=38980.6 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4203524 6462543 0 105 0 3893107 4951 0 0 25 0 1 0 662969689 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 38980.6
Current children cumulated vsize (KiB) 0

[startup+5103 s]
/proc/loadavg: 8.02 8.22 8.08 2/430 4128
/proc/meminfo: memFree=19385440/32951124 swapFree=64205392/67111528
[pid=3959] ppid=3957 vsize=0 CPUtime=38981.4 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4203524 6462543 0 105 0 3893107 5031 0 0 25 0 1 0 662969689 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 38981.4
Current children cumulated vsize (KiB) 0

[startup+5103.2 s]
/proc/loadavg: 8.02 8.22 8.08 2/430 4128
/proc/meminfo: memFree=19385440/32951124 swapFree=64205392/67111528
[pid=3959] ppid=3957 vsize=0 CPUtime=38981.6 cores=0-7
/proc/3959/stat : 3959 (claspmt) R 3957 3959 3934 0 -1 4203524 6462543 0 105 0 3893107 5051 0 0 25 0 1 0 662969689 0 0 33554432000 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 4
/proc/3959/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 38981.6
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5103.23
CPU time (s): 38981.6
CPU user time (s): 38931.1
CPU system time (s): 50.5383
CPU usage (%): 763.862
Max. virtual memory (cumulated for all children) (KiB): 25968732

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 38931.1
system time used= 50.5383
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6462543
page faults= 105
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 104400
involuntary context switches= 178182

runsolver used 18.9751 second user time and 55.7255 second system time

The end

Launcher Data 

Begin job on node136 at 2011-05-16 04:41:57
IDJOB=3381357
IDBENCH=88165
IDSOLVER=1865
FILE ID=node136/3381357-1305513717
RUNJOBID= node136-1305513717-3948
PBS_JOBID= 13325540
Free space on /tmp= 72940 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT-Race-2010-CNF/hardware-verification/velev/2dlx_ca_bp_f_liveness.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381357-1305513717/watcher-3381357-1305513717 -o /tmp/evaluation-result-3381357-1305513717/solver-3381357-1305513717 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381357-1305513717.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 99cd8de7f06ce8c087a5c7fb299dfb62
RANDOM SEED=678408424

node136.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16472484 kB
Buffers:        332452 kB
Cached:        6093080 kB
SwapCached:       2732 kB
Active:       11758356 kB
Inactive:      4352060 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16472484 kB
SwapTotal:    67111528 kB
SwapFree:     67082160 kB
Dirty:           93716 kB
Writeback:           0 kB
AnonPages:     9683828 kB
Mapped:          34740 kB
Slab:           244012 kB
PageTables:      52084 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 122405744 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72936 MiB
End job on node136 at 2011-05-16 06:07:01