Trace number 3322458

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40002 5009.11

General information on the benchmark

Namecrafted/manthey/sudoku-single-hard/
sudoku_81x81_unknown.cnf
MD5SUMcbd78921480e79d133eb06cb2172caa1
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables18217
Number of clauses284829
Sum of the clauses size622430
Maximum clause length28
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2275099
Number of clauses of size 3620
Number of clauses of size 4984
Number of clauses of size 51120
Number of clauses of size over 57006

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...385640/instance-3322458-1304385640.cnf
0.29/0.32	c Solving...
40000.72/5007.91	
40000.72/5007.91	*** INTERRUPTED! ***
40000.72/5007.91	*** Shutting down threads...
40001.90/5009.10	s UNKNOWN
40001.90/5009.10	
40001.90/5009.10	c INTERRUPTED   : 1
40001.90/5009.10	c Models        : 0+    
40001.90/5009.10	c Time          : 5007.921s (Solving: 5007.60s 1st Model: 0.00s Unsat: 0.00s)
40001.90/5009.10	c CPU Time      : 40000.820s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322458-1304385640/watcher-3322458-1304385640 -o /tmp/evaluation-result-3322458-1304385640/solver-3322458-1304385640 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322458-1304385640.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.77 7.76 7.92 1/323 3051
/proc/meminfo: memFree=14131460/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=24452 CPUtime=0 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1544 0 0 0 0 0 0 0 23 0 1 0 530979678 25038848 1331 33554432000 4194304 4964704 140735254202944 18446744073709551615 4599938 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/3051/statm: 6113 1331 386 189 0 987 0

[startup+0.0608249 s]
/proc/loadavg: 6.77 7.76 7.92 1/323 3051
/proc/meminfo: memFree=14131460/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=34020 CPUtime=0.05 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 4884 0 0 0 5 0 0 0 23 0 1 0 530979678 34836480 3521 33554432000 4194304 4964704 140735254202944 18446744073709551615 270338044613 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/3051/statm: 8505 3524 386 189 0 3379 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 34020

[startup+0.100817 s]
/proc/loadavg: 6.77 7.76 7.92 1/323 3051
/proc/meminfo: memFree=14131460/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=40152 CPUtime=0.09 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 6717 0 0 0 9 0 0 0 23 0 1 0 530979678 41115648 5177 33554432000 4194304 4964704 140735254202944 18446744073709551615 270338044613 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/3051/statm: 10038 5179 397 189 0 4912 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 40152

[startup+0.300778 s]
/proc/loadavg: 6.77 7.76 7.92 1/323 3051
/proc/meminfo: memFree=14131460/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=43744 CPUtime=0.29 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 7616 0 0 0 28 1 0 0 25 0 1 0 530979678 44793856 6076 33554432000 4194304 4964704 140735254202944 18446744073709551615 4646324 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/3051/statm: 10936 6076 403 189 0 5810 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 43744

[startup+1.68439 s]
/proc/loadavg: 6.77 7.76 7.92 1/323 3051
/proc/meminfo: memFree=14131460/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=472172 CPUtime=9.99 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 17590 0 0 0 992 7 0 0 25 0 8 0 530979678 483504128 14640 33554432000 4194304 4964704 140735254202944 18446744073709551615 270338488863 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 118043 14641 464 189 0 112392 0
Current children cumulated CPU time (s) 9.99
Current children cumulated vsize (KiB) 472172

[startup+1.70046 s]
/proc/loadavg: 6.77 7.76 7.92 1/323 3051
/proc/meminfo: memFree=14131460/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=472172 CPUtime=10.13 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 17670 0 0 0 1006 7 0 0 25 0 8 0 530979678 483504128 14720 33554432000 4194304 4964704 140735254202944 18446744073709551615 4669051 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 118043 14720 464 189 0 112392 0
Current children cumulated CPU time (s) 10.13
Current children cumulated vsize (KiB) 472172

[startup+3.11219 s]
/proc/loadavg: 6.77 7.76 7.92 9/331 3060
/proc/meminfo: memFree=14061840/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=472600 CPUtime=21.39 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 20844 0 0 0 2130 9 0 0 25 0 8 0 530979678 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 118150 17894 464 189 0 112499 0
[pid=3051/tid=3053] ppid=3049 vsize=472600 CPUtime=2.61 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 1120 0 0 0 261 0 0 0 19 0 8 0 530979711 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=472600 CPUtime=2.57 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 2086 0 0 0 257 0 0 0 20 0 8 0 530979711 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=472600 CPUtime=2.63 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 1793 0 0 0 262 1 0 0 20 0 8 0 530979711 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4342914 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=472600 CPUtime=2.61 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 2011 0 0 0 260 1 0 0 20 0 8 0 530979711 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=472600 CPUtime=2.73 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 1069 0 0 0 273 0 0 0 23 0 8 0 530979711 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4407824 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=472600 CPUtime=2.62 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 1700 0 0 0 262 0 0 0 20 0 8 0 530979711 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=472600 CPUtime=2.62 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 2571 0 0 0 260 2 0 0 20 0 8 0 530979711 483942400 17894 33554432000 4194304 4964704 140735254202944 18446744073709551615 4660088 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 21.39
Current children cumulated vsize (KiB) 472600

[startup+6.30056 s]
/proc/loadavg: 6.87 7.77 7.92 9/331 3060
/proc/meminfo: memFree=14026524/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=472996 CPUtime=46.92 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 29637 0 0 0 4676 16 0 0 25 0 8 0 530979678 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4668488 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 118249 26687 464 189 0 112598 0
[pid=3051/tid=3053] ppid=3049 vsize=472996 CPUtime=5.81 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 1851 0 0 0 580 1 0 0 25 0 8 0 530979711 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=472996 CPUtime=5.74 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 3458 0 0 0 573 1 0 0 25 0 8 0 530979711 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=472996 CPUtime=5.82 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 3095 0 0 0 581 1 0 0 25 0 8 0 530979711 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=472996 CPUtime=5.81 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 3092 0 0 0 580 1 0 0 25 0 8 0 530979711 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=472996 CPUtime=5.94 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 1866 0 0 0 593 1 0 0 25 0 8 0 530979711 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=472996 CPUtime=5.82 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 3022 0 0 0 580 2 0 0 25 0 8 0 530979711 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4342154 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=472996 CPUtime=5.82 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 4102 0 0 0 579 3 0 0 25 0 8 0 530979711 484347904 26687 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 46.92
Current children cumulated vsize (KiB) 472996

[startup+12.7013 s]
/proc/loadavg: 6.96 7.77 7.92 9/331 3060
/proc/meminfo: memFree=13965808/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=483108 CPUtime=98.01 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 47497 0 0 0 9773 28 0 0 25 0 8 0 530979678 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 120777 44547 466 189 0 115126 0
[pid=3051/tid=3053] ppid=3049 vsize=483108 CPUtime=12.2 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 3428 0 0 0 1217 3 0 0 25 0 8 0 530979711 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=483108 CPUtime=12.07 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 5953 0 0 0 1204 3 0 0 25 0 8 0 530979711 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 4378525 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=483108 CPUtime=12.21 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 4826 0 0 0 1219 2 0 0 25 0 8 0 530979711 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=483108 CPUtime=12.2 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 4879 0 0 0 1217 3 0 0 25 0 8 0 530979711 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=483108 CPUtime=12.32 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 4782 0 0 0 1230 2 0 0 25 0 8 0 530979711 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 4669997 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=483108 CPUtime=12.21 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 5296 0 0 0 1218 3 0 0 25 0 8 0 530979711 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=483108 CPUtime=12.2 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 7541 0 0 0 1215 5 0 0 25 0 8 0 530979711 494702592 44547 33554432000 4194304 4964704 140735254202944 18446744073709551615 270338044961 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 98.01
Current children cumulated vsize (KiB) 483108

[startup+25.5007 s]
/proc/loadavg: 7.19 7.78 7.92 9/331 3061
/proc/meminfo: memFree=13882524/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=495944 CPUtime=200.25 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 66108 0 0 0 19980 45 0 0 25 0 8 0 530979678 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4669872 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 123986 63158 466 189 0 118335 0
[pid=3051/tid=3053] ppid=3049 vsize=495944 CPUtime=25 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 5356 0 0 0 2495 5 0 0 25 0 8 0 530979711 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=495944 CPUtime=24.73 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 8558 0 0 0 2467 6 0 0 25 0 8 0 530979711 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=495944 CPUtime=25.01 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 7012 0 0 0 2496 5 0 0 23 0 8 0 530979711 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=495944 CPUtime=25 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 6461 0 0 0 2496 4 0 0 25 0 8 0 530979711 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=495944 CPUtime=25.13 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 7087 0 0 0 2508 5 0 0 25 0 8 0 530979711 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4631266 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=495944 CPUtime=24.99 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 7373 0 0 0 2494 5 0 0 25 0 8 0 530979711 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=495944 CPUtime=25.01 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 11697 0 0 0 2493 8 0 0 25 0 8 0 530979711 507846656 63158 33554432000 4194304 4964704 140735254202944 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 200.25
Current children cumulated vsize (KiB) 495944

[startup+51.1007 s]
/proc/loadavg: 7.46 7.80 7.92 9/331 3061
/proc/meminfo: memFree=13753000/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=513052 CPUtime=404.77 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 98486 0 0 0 40406 71 0 0 25 0 8 0 530979678 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4668931 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 128263 95536 466 189 0 122612 0
[pid=3051/tid=3053] ppid=3049 vsize=513052 CPUtime=50.6 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 8466 0 0 0 5052 8 0 0 25 0 8 0 530979711 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670260 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=513052 CPUtime=50.07 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 10888 0 0 0 4999 8 0 0 25 0 8 0 530979711 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=513052 CPUtime=50.6 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 9974 0 0 0 5053 7 0 0 25 0 8 0 530979711 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=513052 CPUtime=50.59 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 11286 0 0 0 5052 7 0 0 25 0 8 0 530979711 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=513052 CPUtime=50.72 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 10405 0 0 0 5065 7 0 0 25 0 8 0 530979711 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=513052 CPUtime=50.59 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 11542 0 0 0 5051 8 0 0 25 0 8 0 530979711 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=513052 CPUtime=50.6 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 19292 0 0 0 5048 12 0 0 25 0 8 0 530979711 525365248 95536 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 404.77
Current children cumulated vsize (KiB) 513052

[startup+102.301 s]
/proc/loadavg: 7.77 7.83 7.92 9/331 3063
/proc/meminfo: memFree=13627748/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=793572 CPUtime=813.78 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 129636 0 0 0 81279 99 0 0 25 0 8 0 530979678 812617728 126670 33554432000 4194304 4964704 140735254202944 18446744073709551615 4668527 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/3051/statm: 198393 126670 466 189 0 192742 0
[pid=3051/tid=3053] ppid=3049 vsize=793572 CPUtime=101.77 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 11371 0 0 0 10167 10 0 0 25 0 8 0 530979711 812617728 126670 33554432000 4194304 4964704 140735254202944 18446744073709551615 4668472 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=793572 CPUtime=100.73 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 13863 0 0 0 10063 10 0 0 25 0 8 0 530979711 812617728 126670 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670462 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=793572 CPUtime=101.81 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 13042 0 0 0 10171 10 0 0 25 0 8 0 530979711 812617728 126670 33554432000 4194304 4964704 140735254202944 18446744073709551615 4321097 0 0 4096 18946 0 0 0 -1 0 0 0 0

################
# More data... #
################

/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 87178 0 0 0 465960 93 0 0 25 0 8 0 530979711 4809310208 1117862 33554432000 4194304 4964704 140735254202944 18446744073709551615 4321013 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=4696592 CPUtime=4661.74 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 362900 0 0 0 465895 279 0 0 25 0 8 0 530979711 4809310208 1117862 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37239.6
Current children cumulated vsize (KiB) 4696592

[startup+4722.3 s]
/proc/loadavg: 8.06 8.04 8.00 9/331 3742
/proc/meminfo: memFree=9487936/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=4846580 CPUtime=37719 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1170284 0 0 0 3770733 1163 0 0 25 0 8 0 530979678 4962897920 1159931 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/3051/statm: 1211645 1159932 466 189 0 1205994 0
[pid=3051/tid=3053] ppid=3049 vsize=4846580 CPUtime=4707.51 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 48016 0 0 0 470695 56 0 0 25 0 8 0 530979711 4962897920 1159932 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=4846580 CPUtime=4710 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 85557 0 0 0 470916 84 0 0 25 0 8 0 530979711 4962897920 1159932 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=4846580 CPUtime=4720.11 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 101887 0 0 0 471904 107 0 0 25 0 8 0 530979711 4962897920 1159932 33554432000 4194304 4964704 140735254202944 18446744073709551615 4653264 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=4846580 CPUtime=4717.38 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 126345 0 0 0 471591 147 0 0 25 0 8 0 530979711 4962897920 1159932 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=4846580 CPUtime=4704.53 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 88095 0 0 0 470350 103 0 0 25 0 8 0 530979711 4962897920 1159932 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=4846580 CPUtime=4720.53 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 88084 0 0 0 471959 94 0 0 25 0 8 0 530979711 4962897920 1159932 33554432000 4194304 4964704 140735254202944 18446744073709551615 270338044961 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=4846580 CPUtime=4721.73 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 375184 0 0 0 471884 289 0 0 25 0 8 0 530979711 4962897920 1159932 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 37719
Current children cumulated vsize (KiB) 4846580

[startup+4782.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/331 3744
/proc/meminfo: memFree=9352888/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=5003120 CPUtime=38198.4 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1204369 0 0 0 3818635 1200 0 0 25 0 8 0 530979678 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670255 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/3051/statm: 1250780 1194016 466 189 0 1245129 0
[pid=3051/tid=3053] ppid=3049 vsize=5003120 CPUtime=4767.52 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 48019 0 0 0 476696 56 0 0 25 0 8 0 530979711 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=5003120 CPUtime=4770.01 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 86063 0 0 0 476916 85 0 0 25 0 8 0 530979711 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=5003120 CPUtime=4780.11 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 103838 0 0 0 477901 110 0 0 25 0 8 0 530979711 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4332375 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=5003120 CPUtime=4777.38 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 134998 0 0 0 477582 156 0 0 25 0 8 0 530979711 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4653154 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=5003120 CPUtime=4763.93 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 91186 0 0 0 476288 105 0 0 25 0 8 0 530979711 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=5003120 CPUtime=4780.52 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 92467 0 0 0 477955 97 0 0 25 0 8 0 530979711 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=5003120 CPUtime=4781.74 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 382672 0 0 0 477877 297 0 0 25 0 8 0 530979711 5123194880 1194016 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38198.4
Current children cumulated vsize (KiB) 5003120

[startup+4842.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/331 3745
/proc/meminfo: memFree=9277728/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=5142028 CPUtime=38677.7 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1222942 0 0 0 3866551 1219 0 0 25 0 8 0 530979678 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/3051/statm: 1285507 1212589 466 189 0 1279856 0
[pid=3051/tid=3053] ppid=3049 vsize=5142028 CPUtime=4827.52 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 48532 0 0 0 482695 57 0 0 25 0 8 0 530979711 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=5142028 CPUtime=4830.02 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 86856 0 0 0 482916 86 0 0 25 0 8 0 530979711 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=5142028 CPUtime=4840.11 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 106157 0 0 0 483898 113 0 0 25 0 8 0 530979711 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=5142028 CPUtime=4837.37 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 141361 0 0 0 483575 162 0 0 25 0 8 0 530979711 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=5142028 CPUtime=4823.29 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 91845 0 0 0 482223 106 0 0 25 0 8 0 530979711 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=5142028 CPUtime=4840.51 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 93417 0 0 0 483952 99 0 0 25 0 8 0 530979711 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=5142028 CPUtime=4841.75 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 389633 0 0 0 483872 303 0 0 25 0 8 0 530979711 5265436672 1212589 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 38677.7
Current children cumulated vsize (KiB) 5142028

[startup+4902.3 s]
/proc/loadavg: 8.07 8.06 8.00 9/331 3747
/proc/meminfo: memFree=9059856/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=5272320 CPUtime=39157.1 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1277603 0 0 0 3914432 1277 0 0 25 0 8 0 530979678 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/3051/statm: 1318080 1267250 466 189 0 1312429 0
[pid=3051/tid=3053] ppid=3049 vsize=5272320 CPUtime=4887.52 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 50966 0 0 0 488691 61 0 0 25 0 8 0 530979711 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=5272320 CPUtime=4890.01 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 90642 0 0 0 488911 90 0 0 25 0 8 0 530979711 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=5272320 CPUtime=4900.12 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 112797 0 0 0 489892 120 0 0 25 0 8 0 530979711 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=5272320 CPUtime=4897.38 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 148574 0 0 0 489569 169 0 0 25 0 8 0 530979711 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=5272320 CPUtime=4882.68 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 97429 0 0 0 488154 114 0 0 25 0 8 0 530979711 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=5272320 CPUtime=4900.5 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 97820 0 0 0 489946 104 0 0 25 0 8 0 530979711 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=5272320 CPUtime=4901.75 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 395956 0 0 0 489866 309 0 0 25 0 8 0 530979711 5398855680 1267250 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39157.1
Current children cumulated vsize (KiB) 5272320

[startup+4962.3 s]
/proc/loadavg: 8.06 8.06 8.00 9/331 3748
/proc/meminfo: memFree=9021776/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=5274248 CPUtime=39636.3 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1286431 0 0 0 3962343 1290 0 0 25 0 8 0 530979678 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4320928 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/3051/statm: 1318562 1276078 466 189 0 1312911 0
[pid=3051/tid=3053] ppid=3049 vsize=5274248 CPUtime=4947.4 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 51591 0 0 0 494677 63 0 0 25 0 8 0 530979711 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4654500 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=5274248 CPUtime=4950.02 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 91317 0 0 0 494911 91 0 0 25 0 8 0 530979711 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670314 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=5274248 CPUtime=4960.09 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 114052 0 0 0 495888 121 0 0 25 0 8 0 530979711 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=5274248 CPUtime=4957.38 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 148576 0 0 0 495569 169 0 0 25 0 8 0 530979711 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=5274248 CPUtime=4942.06 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 99046 0 0 0 494089 117 0 0 25 0 8 0 530979711 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=5274248 CPUtime=4960.49 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 99332 0 0 0 495942 107 0 0 25 0 8 0 530979711 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=5274248 CPUtime=4961.75 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 397612 0 0 0 495865 310 0 0 25 0 8 0 530979711 5400829952 1276078 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39636.3
Current children cumulated vsize (KiB) 5274248



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.9 s]
/proc/loadavg: 8.03 8.05 8.00 9/331 3750
/proc/meminfo: memFree=9021644/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=5274248 CPUtime=40000.7 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1286482 0 0 0 3998782 1290 0 0 25 0 8 0 530979678 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4668820 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/3051/statm: 1318562 1276129 466 189 0 1312911 0
[pid=3051/tid=3053] ppid=3049 vsize=5274248 CPUtime=4993.02 cores=0-7
/proc/3051/task/3053/stat : 3053 (claspmt) R 3049 3051 2955 0 -1 4202560 51591 0 0 0 499239 63 0 0 25 0 8 0 530979711 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=3051/tid=3055] ppid=3049 vsize=5274248 CPUtime=4995.63 cores=0-7
/proc/3051/task/3055/stat : 3055 (claspmt) R 3049 3051 2955 0 -1 4202560 91318 0 0 0 499472 91 0 0 25 0 8 0 530979711 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=3051/tid=3056] ppid=3049 vsize=5274248 CPUtime=5005.71 cores=0-7
/proc/3051/task/3056/stat : 3056 (claspmt) R 3049 3051 2955 0 -1 4202560 114079 0 0 0 500450 121 0 0 25 0 8 0 530979711 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=3051/tid=3057] ppid=3049 vsize=5274248 CPUtime=5003 cores=0-7
/proc/3051/task/3057/stat : 3057 (claspmt) R 3049 3051 2955 0 -1 4202560 148577 0 0 0 500131 169 0 0 25 0 8 0 530979711 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=3051/tid=3058] ppid=3049 vsize=5274248 CPUtime=4987.21 cores=0-7
/proc/3051/task/3058/stat : 3058 (claspmt) R 3049 3051 2955 0 -1 4202560 99046 0 0 0 498604 117 0 0 25 0 8 0 530979711 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=3051/tid=3059] ppid=3049 vsize=5274248 CPUtime=5006.09 cores=0-7
/proc/3051/task/3059/stat : 3059 (claspmt) R 3049 3051 2955 0 -1 4202560 99353 0 0 0 500502 107 0 0 25 0 8 0 530979711 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=3051/tid=3060] ppid=3049 vsize=5274248 CPUtime=5007.35 cores=0-7
/proc/3051/task/3060/stat : 3060 (claspmt) R 3049 3051 2955 0 -1 4202560 397612 0 0 0 500425 310 0 0 25 0 8 0 530979711 5400829952 1276129 33554432000 4194304 4964704 140735254202944 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 5274248

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5008.7 s]
/proc/loadavg: 8.03 8.05 8.00 2/325 3751
/proc/meminfo: memFree=9023180/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=5263764 CPUtime=40001.6 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1286492 0 0 0 3998869 1290 0 0 25 0 1 0 530979678 5390094336 1272864 33554432000 4194304 4964704 140735254202944 18446744073709551615 270338033234 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/3051/statm: 1315941 1272864 476 189 0 1310290 0
Current children cumulated CPU time (s) 40001.6
Current children cumulated vsize (KiB) 5263764

[startup+5009.1 s]
/proc/loadavg: 8.03 8.05 8.00 2/325 3751
/proc/meminfo: memFree=13971772/32951124 swapFree=67111356/67111528
[pid=3051] ppid=3049 vsize=461256 CPUtime=40002 cores=0-7
/proc/3051/stat : 3051 (claspmt) R 3049 3051 2955 0 -1 4202496 1286492 0 0 0 3998891 1309 0 0 25 0 1 0 530979678 472326144 8745 33554432000 4194304 4964704 140735254202944 18446744073709551615 270338427239 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/3051/statm: 115314 8745 476 189 0 109663 0
Current children cumulated CPU time (s) 40002
Current children cumulated vsize (KiB) 461256

Child status: 0
Real time (s): 5009.11
CPU time (s): 40002
CPU user time (s): 39988.9
CPU system time (s): 13.101
CPU usage (%): 798.585
Max. virtual memory (cumulated for all children) (KiB): 5274248

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39988.9
system time used= 13.101
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1286493
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 67191
involuntary context switches= 196536

runsolver used 14.0929 second user time and 38.6711 second system time

The end

Launcher Data 

Begin job on node143 at 2011-05-03 03:20:41
IDJOB=3322458
IDBENCH=83244
IDSOLVER=1711
FILE ID=node143/3322458-1304385640
RUNJOBID= node143-1304385640-2970
PBS_JOBID= 13181795
Free space on /tmp= 73532 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/manthey/sudoku-single-hard/sudoku_81x81_unknown.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322458-1304385640/watcher-3322458-1304385640 -o /tmp/evaluation-result-3322458-1304385640/solver-3322458-1304385640 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322458-1304385640.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= cbd78921480e79d133eb06cb2172caa1
RANDOM SEED=1750967529

node143.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      14131616 kB
Buffers:        330864 kB
Cached:        1941104 kB
SwapCached:        168 kB
Active:       16813804 kB
Inactive:      1655444 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      14131616 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            5200 kB
Writeback:          12 kB
AnonPages:    16197360 kB
Mapped:          27896 kB
Slab:           230444 kB
PageTables:      53024 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 69931408 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73532 MiB
End job on node143 at 2011-05-03 04:44:11