Trace number 3322516

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40000.5 5006.74

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
32/aes_32_5_keyfind_1.cnf
MD5SUM3f4ea4cf17e734bbe947db7c1dc9fde7
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark7602.69
Satisfiable
(Un)Satisfiability was proved
Number of variables1116
Number of clauses4312
Sum of the clauses size13688
Maximum clause length5
Minimum clause length1
Number of clauses of size 184
Number of clauses of size 2200
Number of clauses of size 33008
Number of clauses of size 4920
Number of clauses of size 5100
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...387635/instance-3322516-1304387635.cnf
0.00/0.01	c Solving...
40000.40/5006.71	
40000.40/5006.71	*** INTERRUPTED! ***
40000.40/5006.71	*** Shutting down threads...
40000.40/5006.73	s UNKNOWN
40000.40/5006.73	
40000.40/5006.73	c INTERRUPTED   : 1
40000.40/5006.73	c Models        : 0+    
40000.40/5006.73	c Time          : 5006.715s (Solving: 5006.71s 1st Model: 0.00s Unsat: 0.00s)
40000.40/5006.73	c CPU Time      : 40000.470s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322516-1304387635/watcher-3322516-1304387635 -o /tmp/evaluation-result-3322516-1304387635/solver-3322516-1304387635 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322516-1304387635.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.49 8.02 8.10 2/297 2619
/proc/meminfo: memFree=12044668/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=21568 CPUtime=0 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 725 0 0 0 0 0 0 0 25 0 1 0 550361396 22085632 637 33554432000 4194304 4964704 140733303643472 18446744073709551615 255376962245 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/2619/statm: 5392 637 404 189 0 266 0

[startup+0.0354941 s]
/proc/loadavg: 7.49 8.02 8.10 2/297 2619
/proc/meminfo: memFree=12044668/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=316844 CPUtime=0.11 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 1612 0 0 0 11 0 0 0 25 0 8 0 550361396 324448256 1474 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668632 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/2619/statm: 79211 1475 464 189 0 73560 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 316844

[startup+0.107307 s]
/proc/loadavg: 7.49 8.02 8.10 2/297 2619
/proc/meminfo: memFree=12044668/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=317108 CPUtime=0.63 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 1978 0 0 0 63 0 0 0 25 0 8 0 550361396 324718592 1840 33554432000 4194304 4964704 140733303643472 18446744073709551615 255377406446 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 79277 1840 464 189 0 73626 0
Current children cumulated CPU time (s) 0.63
Current children cumulated vsize (KiB) 317108

[startup+0.30041 s]
/proc/loadavg: 7.49 8.02 8.10 2/297 2619
/proc/meminfo: memFree=12044668/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=317364 CPUtime=2.17 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 2335 0 0 0 217 0 0 0 25 0 8 0 550361396 324980736 2197 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668725 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 79341 2197 464 189 0 73690 0
Current children cumulated CPU time (s) 2.17
Current children cumulated vsize (KiB) 317364

[startup+0.700336 s]
/proc/loadavg: 7.49 8.02 8.10 2/297 2619
/proc/meminfo: memFree=12044668/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=317636 CPUtime=5.37 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 2765 0 0 0 537 0 0 0 25 0 8 0 550361396 325259264 2627 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670314 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 79409 2627 464 189 0 73758 0
Current children cumulated CPU time (s) 5.37
Current children cumulated vsize (KiB) 317636

[startup+1.50119 s]
/proc/loadavg: 7.49 8.02 8.10 10/305 2627
/proc/meminfo: memFree=12034568/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=318480 CPUtime=11.77 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 3512 0 0 0 1176 1 0 0 25 0 8 0 550361396 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670323 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 79620 3374 464 189 0 73969 0
[pid=2619/tid=2621] ppid=2617 vsize=318480 CPUtime=1.43 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 191 0 0 0 143 0 0 0 25 0 8 0 550361397 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=318480 CPUtime=1.46 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 409 0 0 0 146 0 0 0 25 0 8 0 550361397 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=318480 CPUtime=1.48 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 342 0 0 0 148 0 0 0 20 0 8 0 550361397 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 255376968608 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=318480 CPUtime=1.48 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 319 0 0 0 148 0 0 0 20 0 8 0 550361397 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669200 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=318480 CPUtime=1.48 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 342 0 0 0 148 0 0 0 20 0 8 0 550361397 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=318480 CPUtime=1.47 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 384 0 0 0 147 0 0 0 20 0 8 0 550361397 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668618 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=318480 CPUtime=1.48 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 482 0 0 0 148 0 0 0 20 0 8 0 550361397 326123520 3374 33554432000 4194304 4964704 140733303643472 18446744073709551615 4660183 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.77
Current children cumulated vsize (KiB) 318480

[startup+3.10089 s]
/proc/loadavg: 7.49 8.02 8.10 9/305 2627
/proc/meminfo: memFree=12032088/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=319288 CPUtime=24.55 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 4522 0 0 0 2454 1 0 0 25 0 8 0 550361396 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 79822 4384 464 189 0 74171 0
[pid=2619/tid=2621] ppid=2617 vsize=319288 CPUtime=3.02 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 340 0 0 0 302 0 0 0 25 0 8 0 550361397 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670283 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=319288 CPUtime=3.06 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 679 0 0 0 306 0 0 0 25 0 8 0 550361397 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669472 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=319288 CPUtime=3.08 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 403 0 0 0 308 0 0 0 25 0 8 0 550361397 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668676 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=319288 CPUtime=3.08 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 474 0 0 0 308 0 0 0 25 0 8 0 550361397 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668609 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=319288 CPUtime=3.08 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 456 0 0 0 308 0 0 0 25 0 8 0 550361397 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668609 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=319288 CPUtime=3.07 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 460 0 0 0 307 0 0 0 25 0 8 0 550361397 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668635 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=319288 CPUtime=3.08 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 589 0 0 0 308 0 0 0 25 0 8 0 550361397 326950912 4384 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.55
Current children cumulated vsize (KiB) 319288

[startup+6.3003 s]
/proc/loadavg: 7.53 8.02 8.10 9/305 2627
/proc/meminfo: memFree=12026264/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=320352 CPUtime=50.12 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 5599 0 0 0 5010 2 0 0 25 0 8 0 550361396 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668668 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 80088 5461 464 189 0 74437 0
[pid=2619/tid=2621] ppid=2617 vsize=320352 CPUtime=6.22 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 495 0 0 0 622 0 0 0 25 0 8 0 550361397 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669399 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=320352 CPUtime=6.26 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 878 0 0 0 626 0 0 0 25 0 8 0 550361397 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=320352 CPUtime=6.28 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 518 0 0 0 628 0 0 0 25 0 8 0 550361397 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=320352 CPUtime=6.28 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 633 0 0 0 628 0 0 0 25 0 8 0 550361397 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=320352 CPUtime=6.28 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 551 0 0 0 628 0 0 0 25 0 8 0 550361397 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668632 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=320352 CPUtime=6.27 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 583 0 0 0 627 0 0 0 25 0 8 0 550361397 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=320352 CPUtime=6.27 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 695 0 0 0 627 0 0 0 25 0 8 0 550361397 328040448 5461 33554432000 4194304 4964704 140733303643472 18446744073709551615 4660345 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 50.12
Current children cumulated vsize (KiB) 320352

[startup+12.7011 s]
/proc/loadavg: 7.57 8.02 8.10 9/305 2627
/proc/meminfo: memFree=12020600/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=321724 CPUtime=101.26 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 6985 0 0 0 10122 4 0 0 25 0 8 0 550361396 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4364587 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 80431 6847 464 189 0 74780 0
[pid=2619/tid=2621] ppid=2617 vsize=321724 CPUtime=12.62 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 823 0 0 0 1262 0 0 0 25 0 8 0 550361397 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669006 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=321724 CPUtime=12.66 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 1079 0 0 0 1266 0 0 0 25 0 8 0 550361397 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4660085 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=321724 CPUtime=12.68 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 692 0 0 0 1268 0 0 0 25 0 8 0 550361397 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=321724 CPUtime=12.68 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 802 0 0 0 1268 0 0 0 25 0 8 0 550361397 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4320947 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=321724 CPUtime=12.68 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 644 0 0 0 1268 0 0 0 25 0 8 0 550361397 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=321724 CPUtime=12.67 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 690 0 0 0 1267 0 0 0 25 0 8 0 550361397 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4361321 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=321724 CPUtime=12.67 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 875 0 0 0 1267 0 0 0 25 0 8 0 550361397 329445376 6847 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 101.26
Current children cumulated vsize (KiB) 321724

[startup+25.5007 s]
/proc/loadavg: 7.66 8.02 8.10 9/305 2627
/proc/meminfo: memFree=12015004/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=322804 CPUtime=203.54 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 8705 0 0 0 20349 5 0 0 25 0 8 0 550361396 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 80701 8567 464 189 0 75050 0
[pid=2619/tid=2621] ppid=2617 vsize=322804 CPUtime=25.42 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 1135 0 0 0 2542 0 0 0 25 0 8 0 550361397 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=322804 CPUtime=25.46 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 1303 0 0 0 2546 0 0 0 25 0 8 0 550361397 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4321093 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=322804 CPUtime=25.47 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 769 0 0 0 2547 0 0 0 25 0 8 0 550361397 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=322804 CPUtime=25.48 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 1103 0 0 0 2548 0 0 0 25 0 8 0 550361397 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=322804 CPUtime=25.47 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 731 0 0 0 2547 0 0 0 25 0 8 0 550361397 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668618 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=322804 CPUtime=25.46 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 923 0 0 0 2546 0 0 0 25 0 8 0 550361397 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=322804 CPUtime=25.47 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 1211 0 0 0 2547 0 0 0 25 0 8 0 550361397 330551296 8567 33554432000 4194304 4964704 140733303643472 18446744073709551615 4653184 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 203.54
Current children cumulated vsize (KiB) 322804

[startup+51.101 s]
/proc/loadavg: 7.97 8.06 8.11 9/305 2628
/proc/meminfo: memFree=12006176/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=324616 CPUtime=408.09 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 10953 0 0 0 40801 8 0 0 25 0 8 0 550361396 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342174 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/2619/statm: 81154 10815 464 189 0 75503 0
[pid=2619/tid=2621] ppid=2617 vsize=324616 CPUtime=51.01 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 1481 0 0 0 5101 0 0 0 25 0 8 0 550361397 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=324616 CPUtime=51.06 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 1623 0 0 0 5105 1 0 0 25 0 8 0 550361397 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=324616 CPUtime=51.07 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 985 0 0 0 5107 0 0 0 25 0 8 0 550361397 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668601 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=324616 CPUtime=51.07 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 1540 0 0 0 5107 0 0 0 25 0 8 0 550361397 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670233 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=324616 CPUtime=51.07 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 953 0 0 0 5107 0 0 0 25 0 8 0 550361397 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=324616 CPUtime=51.07 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 1102 0 0 0 5106 1 0 0 25 0 8 0 550361397 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=324616 CPUtime=51.06 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 1533 0 0 0 5105 1 0 0 25 0 8 0 550361397 332406784 10815 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668632 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 408.09

################
# More data... #
################


[startup+4722.3 s]
/proc/loadavg: 8.12 8.16 8.11 9/305 3318
/proc/meminfo: memFree=11873160/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=353144 CPUtime=37727.9 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 43734 0 0 0 3772470 320 0 0 25 0 8 0 550361396 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2619/statm: 88286 43437 464 189 0 82635 0
[pid=2619/tid=2621] ppid=2617 vsize=353144 CPUtime=4710.76 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 9003 0 0 0 471028 48 0 0 25 0 8 0 550361397 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668668 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=353144 CPUtime=4719.85 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 5244 0 0 0 471945 40 0 0 25 0 8 0 550361397 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4363438 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=353144 CPUtime=4721.96 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 4819 0 0 0 472155 41 0 0 25 0 8 0 550361397 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4644854 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=353144 CPUtime=4721.82 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 5086 0 0 0 472142 40 0 0 25 0 8 0 550361397 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=353144 CPUtime=4721.78 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 4316 0 0 0 472144 34 0 0 25 0 8 0 550361397 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=353144 CPUtime=4703.66 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 5624 0 0 0 470324 42 0 0 25 0 8 0 550361397 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=353144 CPUtime=4715.79 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 4276 0 0 0 471546 33 0 0 25 0 8 0 550361397 361619456 43437 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669480 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37727.9
Current children cumulated vsize (KiB) 353144

[startup+4782.3 s]
/proc/loadavg: 8.18 8.17 8.11 9/305 3319
/proc/meminfo: memFree=11872760/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=353276 CPUtime=38207.3 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 43841 0 0 0 3820405 325 0 0 25 0 8 0 550361396 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2619/statm: 88319 43544 464 189 0 82668 0
[pid=2619/tid=2621] ppid=2617 vsize=353276 CPUtime=4770.75 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 9018 0 0 0 477026 49 0 0 25 0 8 0 550361397 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668601 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=353276 CPUtime=4779.84 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 5245 0 0 0 477944 40 0 0 25 0 8 0 550361397 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668484 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=353276 CPUtime=4781.97 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 4833 0 0 0 478155 42 0 0 25 0 8 0 550361397 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=353276 CPUtime=4781.83 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 5086 0 0 0 478142 41 0 0 25 0 8 0 550361397 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669112 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=353276 CPUtime=4781.78 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 4319 0 0 0 478144 34 0 0 25 0 8 0 550361397 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670252 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=353276 CPUtime=4763.09 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 5650 0 0 0 476266 43 0 0 25 0 8 0 550361397 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4320864 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=353276 CPUtime=4775.78 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 4300 0 0 0 477544 34 0 0 25 0 8 0 550361397 361754624 43544 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38207.3
Current children cumulated vsize (KiB) 353276

[startup+4842.3 s]
/proc/loadavg: 8.11 8.16 8.11 9/305 3321
/proc/meminfo: memFree=11872476/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=353276 CPUtime=38686.7 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 43972 0 0 0 3868344 329 0 0 25 0 8 0 550361396 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669477 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2619/statm: 88319 43675 464 189 0 82668 0
[pid=2619/tid=2621] ppid=2617 vsize=353276 CPUtime=4830.74 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 9072 0 0 0 483025 49 0 0 25 0 8 0 550361397 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=353276 CPUtime=4839.82 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 5245 0 0 0 483942 40 0 0 21 0 8 0 550361397 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668629 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=353276 CPUtime=4841.96 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 4851 0 0 0 484154 42 0 0 25 0 8 0 550361397 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670291 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=353276 CPUtime=4841.82 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 5088 0 0 0 484141 41 0 0 25 0 8 0 550361397 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4654508 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=353276 CPUtime=4841.77 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 4321 0 0 0 484143 34 0 0 25 0 8 0 550361397 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668622 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=353276 CPUtime=4822.56 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 5670 0 0 0 482212 44 0 0 21 0 8 0 550361397 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670250 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=353276 CPUtime=4835.78 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 4311 0 0 0 483543 35 0 0 25 0 8 0 550361397 361754624 43675 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38686.7
Current children cumulated vsize (KiB) 353276

[startup+4902.3 s]
/proc/loadavg: 8.19 8.17 8.11 9/305 3322
/proc/meminfo: memFree=11872204/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=353276 CPUtime=39166.2 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 44081 0 0 0 3916284 332 0 0 25 0 8 0 550361396 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2619/statm: 88319 43784 464 189 0 82668 0
[pid=2619/tid=2621] ppid=2617 vsize=353276 CPUtime=4890.74 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 9081 0 0 0 489024 50 0 0 25 0 8 0 550361397 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=353276 CPUtime=4899.81 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 5270 0 0 0 489940 41 0 0 25 0 8 0 550361397 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=353276 CPUtime=4901.96 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 4876 0 0 0 490154 42 0 0 25 0 8 0 550361397 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670281 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=353276 CPUtime=4901.82 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 5107 0 0 0 490140 42 0 0 25 0 8 0 550361397 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=353276 CPUtime=4901.78 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 4322 0 0 0 490143 35 0 0 25 0 8 0 550361397 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=353276 CPUtime=4882 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 5687 0 0 0 488156 44 0 0 25 0 8 0 550361397 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668609 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=353276 CPUtime=4895.78 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 4311 0 0 0 489543 35 0 0 25 0 8 0 550361397 361754624 43784 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668609 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39166.2
Current children cumulated vsize (KiB) 353276

[startup+4962.3 s]
/proc/loadavg: 8.22 8.18 8.12 9/305 3324
/proc/meminfo: memFree=11871676/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=353276 CPUtime=39645.6 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 44147 0 0 0 3964223 336 0 0 25 0 8 0 550361396 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2619/statm: 88319 43850 464 189 0 82668 0
[pid=2619/tid=2621] ppid=2617 vsize=353276 CPUtime=4950.74 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 9098 0 0 0 495023 51 0 0 25 0 8 0 550361397 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669432 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=353276 CPUtime=4959.8 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 5270 0 0 0 495939 41 0 0 25 0 8 0 550361397 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4660457 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=353276 CPUtime=4961.95 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 4893 0 0 0 496153 42 0 0 25 0 8 0 550361397 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=353276 CPUtime=4961.81 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 5121 0 0 0 496139 42 0 0 25 0 8 0 550361397 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=353276 CPUtime=4961.78 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 4323 0 0 0 496143 35 0 0 25 0 8 0 550361397 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=353276 CPUtime=4941.46 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 5688 0 0 0 494101 45 0 0 25 0 8 0 550361397 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4660183 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=353276 CPUtime=4955.77 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 4311 0 0 0 495542 35 0 0 25 0 8 0 550361397 361754624 43850 33554432000 4194304 4964704 140733303643472 18446744073709551615 4360553 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39645.6
Current children cumulated vsize (KiB) 353276



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.7 s]
/proc/loadavg: 8.18 8.17 8.11 9/305 3325
/proc/meminfo: memFree=11871272/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=353536 CPUtime=40000.4 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 44286 0 0 0 3999702 338 0 0 25 0 8 0 550361396 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670233 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2619/statm: 88384 43989 464 189 0 82733 0
[pid=2619/tid=2621] ppid=2617 vsize=353536 CPUtime=4995.15 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 9157 0 0 0 499464 51 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=353536 CPUtime=5004.21 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 5284 0 0 0 500379 42 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342186 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=353536 CPUtime=5006.37 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 4900 0 0 0 500594 43 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=353536 CPUtime=5006.23 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 5150 0 0 0 500580 43 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=353536 CPUtime=5006.18 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 4323 0 0 0 500583 35 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669477 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=353536 CPUtime=4985.45 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 5703 0 0 0 498500 45 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668709 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=353536 CPUtime=5000.17 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 4311 0 0 0 499982 35 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4321010 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 353536

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5006.7 s]
/proc/loadavg: 8.18 8.17 8.11 9/305 3325
/proc/meminfo: memFree=11871272/32951124 swapFree=67111352/67111528
[pid=2619] ppid=2617 vsize=353536 CPUtime=40000.4 cores=0-7
/proc/2619/stat : 2619 (claspmt) R 2617 2619 2525 0 -1 4202496 44286 0 0 0 3999702 338 0 0 25 0 8 0 550361396 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670233 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2619/statm: 88384 43989 464 189 0 82733 0
[pid=2619/tid=2621] ppid=2617 vsize=353536 CPUtime=4995.15 cores=0-7
/proc/2619/task/2621/stat : 2621 (claspmt) R 2617 2619 2525 0 -1 4202560 9157 0 0 0 499464 51 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2619/tid=2622] ppid=2617 vsize=353536 CPUtime=5004.21 cores=0-7
/proc/2619/task/2622/stat : 2622 (claspmt) R 2617 2619 2525 0 -1 4202560 5284 0 0 0 500379 42 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342186 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2619/tid=2623] ppid=2617 vsize=353536 CPUtime=5006.37 cores=0-7
/proc/2619/task/2623/stat : 2623 (claspmt) R 2617 2619 2525 0 -1 4202560 4900 0 0 0 500594 43 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2619/tid=2624] ppid=2617 vsize=353536 CPUtime=5006.23 cores=0-7
/proc/2619/task/2624/stat : 2624 (claspmt) R 2617 2619 2525 0 -1 4202560 5150 0 0 0 500580 43 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2619/tid=2625] ppid=2617 vsize=353536 CPUtime=5006.18 cores=0-7
/proc/2619/task/2625/stat : 2625 (claspmt) R 2617 2619 2525 0 -1 4202560 4323 0 0 0 500583 35 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4669477 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2619/tid=2626] ppid=2617 vsize=353536 CPUtime=4985.45 cores=0-7
/proc/2619/task/2626/stat : 2626 (claspmt) R 2617 2619 2525 0 -1 4202560 5703 0 0 0 498500 45 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4668709 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2619/tid=2627] ppid=2617 vsize=353536 CPUtime=5000.17 cores=0-7
/proc/2619/task/2627/stat : 2627 (claspmt) R 2617 2619 2525 0 -1 4202560 4311 0 0 0 499982 35 0 0 25 0 8 0 550361397 362020864 43989 33554432000 4194304 4964704 140733303643472 18446744073709551615 4321010 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 353536

Child status: 0
Real time (s): 5006.74
CPU time (s): 40000.5
CPU user time (s): 39997.1
CPU system time (s): 3.39248
CPU usage (%): 798.933
Max. virtual memory (cumulated for all children) (KiB): 353536

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39997.1
system time used= 3.39248
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 44297
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 450857
involuntary context switches= 412683

runsolver used 13.5819 second user time and 32.0301 second system time

The end

Launcher Data 

Begin job on node135 at 2011-05-03 03:53:55
IDJOB=3322516
IDBENCH=83430
IDSOLVER=1711
FILE ID=node135/3322516-1304387635
RUNJOBID= node135-1304387635-2539
PBS_JOBID= 13181817
Free space on /tmp= 72892 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/32/aes_32_5_keyfind_1.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322516-1304387635/watcher-3322516-1304387635 -o /tmp/evaluation-result-3322516-1304387635/solver-3322516-1304387635 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322516-1304387635.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 3f4ea4cf17e734bbe947db7c1dc9fde7
RANDOM SEED=1294976891

node135.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6640.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.78
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.851
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      12044692 kB
Buffers:       1005284 kB
Cached:        3888480 kB
SwapCached:        172 kB
Active:       16141372 kB
Inactive:      4181712 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      12044692 kB
SwapTotal:    67111528 kB
SwapFree:     67111352 kB
Dirty:             948 kB
Writeback:          16 kB
AnonPages:    15428604 kB
Mapped:          23820 kB
Slab:           472376 kB
PageTables:      46760 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 57058088 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73280 MiB
End job on node135 at 2011-05-03 05:17:24