Trace number 3322701

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40000.4 5007.94

General information on the benchmark

NameSAT02/handmade/pehoushek/plainoldcnf/
5cnf_3900_3900_220.shuffled.cnf
MD5SUMf0b7e0675915478b15cab3695fd0fd62
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.033994
Satisfiable
(Un)Satisfiability was proved
Number of variables220
Number of clauses3432
Sum of the clauses size17000
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32
Number of clauses of size 4156
Number of clauses of size 53274
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...393058/instance-3322701-1304393058.cnf
0.00/0.00	c Solving...
40000.35/5007.91	
40000.35/5007.91	*** INTERRUPTED! ***
40000.35/5007.91	*** Shutting down threads...
40000.35/5007.93	s UNKNOWN
40000.35/5007.93	
40000.35/5007.93	c INTERRUPTED   : 1
40000.35/5007.93	c Models        : 0+    
40000.35/5007.93	c Time          : 5007.913s (Solving: 5007.91s 1st Model: 0.00s Unsat: 0.00s)
40000.35/5007.93	c CPU Time      : 40000.390s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322701-1304393058/watcher-3322701-1304393058 -o /tmp/evaluation-result-3322701-1304393058/solver-3322701-1304393058 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322701-1304393058.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.00 8.02 8.00 2/287 31111
/proc/meminfo: memFree=24869032/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=49308 CPUtime=0 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 765 0 0 0 0 0 0 0 25 0 7 0 531721666 50491392 692 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816292849 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/31111/statm: 12327 693 429 189 0 6676 0
[pid=31111/tid=31112] ppid=31109 vsize=49468 CPUtime=0 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) S 31109 31111 30990 0 -1 4202560 22 0 0 0 0 0 0 0 25 0 7 0 531721667 50655232 701 33554432000 4194304 4964704 140737433273888 18446744073709551615 220828052377 0 0 4096 18946 18446604451281016000 0 0 -1 2 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=49468 CPUtime=0 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 32 0 0 0 0 0 0 0 25 0 7 0 531721667 50655232 707 33554432000 4194304 4964704 140737433273888 18446744073709551615 220815933195 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=49468 CPUtime=0 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 7 0 531721667 50655232 712 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816292849 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=49468 CPUtime=0 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 7 0 531721667 50655232 721 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816292849 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=49468 CPUtime=0 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 7 0 531721667 50655232 721 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816292849 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=49468 CPUtime=0 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 0 0 0 0 0 0 0 0 25 0 7 0 531721667 50655232 728 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816292849 0 0 4096 18946 0 0 0 -1 2 0 0 0

[startup+0.055083 s]
/proc/loadavg: 8.00 8.02 8.00 2/287 31111
/proc/meminfo: memFree=24869032/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=382244 CPUtime=0.26 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 1802 0 0 0 25 1 0 0 25 0 8 0 531721666 391417856 1725 33554432000 4194304 4964704 140737433273888 18446744073709551615 4332338 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31111/statm: 95561 1725 463 189 0 89910 0
[pid=31111/tid=31112] ppid=31109 vsize=382244 CPUtime=0.02 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 99 0 0 0 2 0 0 0 25 0 8 0 531721667 391417856 1725 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=382244 CPUtime=0.02 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 129 0 0 0 2 0 0 0 18 0 8 0 531721667 391417856 1725 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=382244 CPUtime=0.03 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 153 0 0 0 3 0 0 0 18 0 8 0 531721667 391417856 1725 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=382244 CPUtime=0.03 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 154 0 0 0 3 0 0 0 18 0 8 0 531721667 391417856 1725 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=382244 CPUtime=0.03 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 155 0 0 0 3 0 0 0 18 0 8 0 531721667 391417856 1725 33554432000 4194304 4964704 140737433273888 18446744073709551615 4332000 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=382244 CPUtime=0.03 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 179 0 0 0 3 0 0 0 18 0 8 0 531721667 391417856 1725 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.26
Current children cumulated vsize (KiB) 382244

[startup+0.101075 s]
/proc/loadavg: 8.00 8.02 8.00 2/287 31111
/proc/meminfo: memFree=24869032/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=382244 CPUtime=0.63 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 1970 0 0 0 62 1 0 0 25 0 8 0 531721666 391417856 1893 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320942 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31111/statm: 95561 1893 463 189 0 89910 0
[pid=31111/tid=31112] ppid=31109 vsize=382244 CPUtime=0.06 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 121 0 0 0 6 0 0 0 25 0 8 0 531721667 391417856 1893 33554432000 4194304 4964704 140737433273888 18446744073709551615 4653140 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=382244 CPUtime=0.07 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 153 0 0 0 7 0 0 0 18 0 8 0 531721667 391417856 1893 33554432000 4194304 4964704 140737433273888 18446744073709551615 4669494 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=382244 CPUtime=0.08 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 179 0 0 0 8 0 0 0 18 0 8 0 531721667 391417856 1893 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=382244 CPUtime=0.08 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 172 0 0 0 8 0 0 0 18 0 8 0 531721667 391417856 1893 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=382244 CPUtime=0.08 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 164 0 0 0 8 0 0 0 18 0 8 0 531721667 391417856 1893 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=382244 CPUtime=0.08 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 202 0 0 0 8 0 0 0 18 0 8 0 531721667 391417856 1893 33554432000 4194304 4964704 140737433273888 18446744073709551615 4378523 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 0.63
Current children cumulated vsize (KiB) 382244

[startup+0.301039 s]
/proc/loadavg: 8.00 8.02 8.00 2/287 31111
/proc/meminfo: memFree=24869032/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=382772 CPUtime=2.22 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 2714 0 0 0 221 1 0 0 25 0 8 0 531721666 391958528 2637 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670339 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31111/statm: 95693 2638 463 189 0 90042 0
[pid=31111/tid=31112] ppid=31109 vsize=382772 CPUtime=0.26 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 198 0 0 0 26 0 0 0 25 0 8 0 531721667 391958528 2638 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=382772 CPUtime=0.27 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 251 0 0 0 27 0 0 0 18 0 8 0 531721667 391958528 2638 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=382772 CPUtime=0.28 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 257 0 0 0 28 0 0 0 18 0 8 0 531721667 391958528 2638 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321150 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=382772 CPUtime=0.28 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 261 0 0 0 28 0 0 0 18 0 8 0 531721667 391958528 2638 33554432000 4194304 4964704 140737433273888 18446744073709551615 220815897285 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=382772 CPUtime=0.28 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 218 0 0 0 28 0 0 0 18 0 8 0 531721667 391958528 2638 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=382772 CPUtime=0.27 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 352 0 0 0 27 0 0 0 18 0 8 0 531721667 391958528 2638 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2.22
Current children cumulated vsize (KiB) 382772

[startup+0.700965 s]
/proc/loadavg: 8.00 8.02 8.00 2/287 31111
/proc/meminfo: memFree=24869032/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=383740 CPUtime=5.41 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 4079 0 0 0 539 2 0 0 18 0 8 0 531721666 392949760 4002 33554432000 4194304 4964704 140737433273888 18446744073709551615 4684027 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31111/statm: 95935 4002 464 189 0 90284 0
[pid=31111/tid=31112] ppid=31109 vsize=383740 CPUtime=0.66 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 388 0 0 0 66 0 0 0 25 0 8 0 531721667 392949760 4002 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=383740 CPUtime=0.67 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 537 0 0 0 67 0 0 0 18 0 8 0 531721667 392949760 4002 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=383740 CPUtime=0.68 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 390 0 0 0 68 0 0 0 18 0 8 0 531721667 392949760 4002 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=383740 CPUtime=0.67 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 418 0 0 0 67 0 0 0 18 0 8 0 531721667 392949760 4002 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=383740 CPUtime=0.68 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 370 0 0 0 68 0 0 0 18 0 8 0 531721667 392949760 4002 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320947 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=383740 CPUtime=0.67 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 493 0 0 0 67 0 0 0 18 0 8 0 531721667 392949760 4002 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670271 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5.41
Current children cumulated vsize (KiB) 383740

[startup+1.50082 s]
/proc/loadavg: 8.00 8.02 8.00 10/295 31119
/proc/meminfo: memFree=24852856/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=384676 CPUtime=11.8 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 5322 0 0 0 1177 3 0 0 19 0 8 0 531721666 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31111/statm: 96169 5245 464 189 0 90518 0
[pid=31111/tid=31112] ppid=31109 vsize=384676 CPUtime=1.46 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 592 0 0 0 146 0 0 0 18 0 8 0 531721667 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670258 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=384676 CPUtime=1.46 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 661 0 0 0 146 0 0 0 20 0 8 0 531721667 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=384676 CPUtime=1.47 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 506 0 0 0 147 0 0 0 19 0 8 0 531721667 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670230 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=384676 CPUtime=1.47 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 571 0 0 0 147 0 0 0 19 0 8 0 531721667 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=384676 CPUtime=1.48 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 565 0 0 0 148 0 0 0 20 0 8 0 531721667 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=384676 CPUtime=1.47 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 680 0 0 0 147 0 0 0 19 0 8 0 531721667 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=384676 CPUtime=1.47 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 668 0 0 0 147 0 0 0 19 0 8 0 531721668 393908224 5245 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.8
Current children cumulated vsize (KiB) 384676

[startup+3.10052 s]
/proc/loadavg: 8.00 8.02 8.00 10/295 31119
/proc/meminfo: memFree=24847948/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=385456 CPUtime=24.57 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 6497 0 0 0 2452 5 0 0 22 0 8 0 531721666 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31111/statm: 96364 6420 464 189 0 90713 0
[pid=31111/tid=31112] ppid=31109 vsize=385456 CPUtime=3.06 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 746 0 0 0 306 0 0 0 20 0 8 0 531721667 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=385456 CPUtime=3.06 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 829 0 0 0 306 0 0 0 25 0 8 0 531721667 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=385456 CPUtime=3.07 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 627 0 0 0 307 0 0 0 18 0 8 0 531721667 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=385456 CPUtime=3.07 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 837 0 0 0 307 0 0 0 22 0 8 0 531721667 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=385456 CPUtime=3.07 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 659 0 0 0 307 0 0 0 25 0 8 0 531721667 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320934 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=385456 CPUtime=3.06 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 797 0 0 0 306 0 0 0 23 0 8 0 531721667 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=385456 CPUtime=3.07 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 804 0 0 0 307 0 0 0 23 0 8 0 531721668 394706944 6420 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670462 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.57
Current children cumulated vsize (KiB) 385456

[startup+6.30093 s]
/proc/loadavg: 8.00 8.02 8.00 9/295 31119
/proc/meminfo: memFree=24840120/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=386828 CPUtime=50.13 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 8268 0 0 0 5004 9 0 0 25 0 8 0 531721666 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670458 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/31111/statm: 96707 8191 464 189 0 91056 0
[pid=31111/tid=31112] ppid=31109 vsize=386828 CPUtime=6.25 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 975 0 0 0 625 0 0 0 25 0 8 0 531721667 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=386828 CPUtime=6.26 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 1104 0 0 0 625 1 0 0 25 0 8 0 531721667 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=386828 CPUtime=6.27 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 798 0 0 0 626 1 0 0 25 0 8 0 531721667 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=386828 CPUtime=6.27 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 1151 0 0 0 627 0 0 0 25 0 8 0 531721667 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 4338784 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=386828 CPUtime=6.27 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 845 0 0 0 626 1 0 0 20 0 8 0 531721667 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 4654517 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=386828 CPUtime=6.27 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 978 0 0 0 626 1 0 0 25 0 8 0 531721667 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 220816341535 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=386828 CPUtime=6.27 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 1011 0 0 0 626 1 0 0 25 0 8 0 531721668 396111872 8191 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320890 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 50.13
Current children cumulated vsize (KiB) 386828

[startup+12.7008 s]
/proc/loadavg: 8.00 8.02 8.00 9/295 31119
/proc/meminfo: memFree=24831480/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=388164 CPUtime=101.23 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 10355 0 0 0 10108 15 0 0 24 0 8 0 531721666 397479936 10278 33554432000 4194304 4964704 140737433273888 18446744073709551615 4334304 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/31111/statm: 97041 10278 464 189 0 91390 0
[pid=31111/tid=31112] ppid=31109 vsize=388164 CPUtime=12.65 cores=0-7

################
# More data... #
################


[startup+4722.31 s]
/proc/loadavg: 8.06 8.07 8.05 9/295 31260
/proc/meminfo: memFree=24679924/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=414232 CPUtime=37719 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 47327 0 0 0 3769797 2103 0 0 20 0 8 0 531721666 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31111/statm: 103558 47248 464 189 0 97907 0
[pid=31111/tid=31112] ppid=31109 vsize=414232 CPUtime=4707.51 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 5459 0 0 0 470453 298 0 0 25 0 8 0 531721667 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=414232 CPUtime=4714.75 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 7317 0 0 0 471213 262 0 0 25 0 8 0 531721667 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 220815895284 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=414232 CPUtime=4710.62 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 4486 0 0 0 470814 248 0 0 20 0 8 0 531721667 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=414232 CPUtime=4720.36 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 6858 0 0 0 471776 260 0 0 25 0 8 0 531721667 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=414232 CPUtime=4714.18 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 6747 0 0 0 471155 263 0 0 25 0 8 0 531721667 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 4658370 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=414232 CPUtime=4720 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 5172 0 0 0 471745 255 0 0 25 0 8 0 531721667 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=414232 CPUtime=4716.04 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 6739 0 0 0 471351 253 0 0 25 0 8 0 531721668 424173568 47248 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37719
Current children cumulated vsize (KiB) 414232

[startup+4782.3 s]
/proc/loadavg: 8.02 8.06 8.05 9/295 31261
/proc/meminfo: memFree=24679520/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=414400 CPUtime=38198.2 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 47433 0 0 0 3817694 2127 0 0 21 0 8 0 531721666 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31111/statm: 103600 47354 464 189 0 97949 0
[pid=31111/tid=31112] ppid=31109 vsize=414400 CPUtime=4767.47 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 5459 0 0 0 476446 301 0 0 25 0 8 0 531721667 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=414400 CPUtime=4774.7 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 7328 0 0 0 477205 265 0 0 25 0 8 0 531721667 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670255 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=414400 CPUtime=4770.59 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 4498 0 0 0 476808 251 0 0 25 0 8 0 531721667 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=414400 CPUtime=4780.34 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 6892 0 0 0 477771 263 0 0 25 0 8 0 531721667 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4660068 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=414400 CPUtime=4773.61 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 6771 0 0 0 477095 266 0 0 25 0 8 0 531721667 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=414400 CPUtime=4779.96 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 5174 0 0 0 477739 257 0 0 25 0 8 0 531721667 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=414400 CPUtime=4776.01 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 6750 0 0 0 477345 256 0 0 25 0 8 0 531721668 424345600 47354 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38198.2
Current children cumulated vsize (KiB) 414400

[startup+4842.3 s]
/proc/loadavg: 8.05 8.06 8.05 9/295 31263
/proc/meminfo: memFree=24679368/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=414400 CPUtime=38677.5 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 47487 0 0 0 3865594 2153 0 0 20 0 8 0 531721666 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670230 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31111/statm: 103600 47408 464 189 0 97949 0
[pid=31111/tid=31112] ppid=31109 vsize=414400 CPUtime=4827.45 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 5460 0 0 0 482440 305 0 0 25 0 8 0 531721667 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=414400 CPUtime=4834.67 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 7340 0 0 0 483198 269 0 0 19 0 8 0 531721667 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=414400 CPUtime=4830.41 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 4499 0 0 0 482787 254 0 0 25 0 8 0 531721667 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4660310 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=414400 CPUtime=4840.31 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 6905 0 0 0 483764 267 0 0 25 0 8 0 531721667 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=414400 CPUtime=4833.2 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 6784 0 0 0 483051 269 0 0 25 0 8 0 531721667 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=414400 CPUtime=4839.95 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 5185 0 0 0 483734 261 0 0 25 0 8 0 531721667 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=414400 CPUtime=4836 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 6751 0 0 0 483342 258 0 0 25 0 8 0 531721668 424345600 47408 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38677.5
Current children cumulated vsize (KiB) 414400

[startup+4902.3 s]
/proc/loadavg: 8.06 8.06 8.05 9/295 31265
/proc/meminfo: memFree=24679208/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=414400 CPUtime=39156.8 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 47561 0 0 0 3913499 2177 0 0 25 0 8 0 531721666 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321079 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31111/statm: 103600 47482 464 189 0 97949 0
[pid=31111/tid=31112] ppid=31109 vsize=414400 CPUtime=4887.42 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 5461 0 0 0 488434 308 0 0 25 0 8 0 531721667 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=414400 CPUtime=4894.65 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 7351 0 0 0 489193 272 0 0 18 0 8 0 531721667 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=414400 CPUtime=4889.85 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 4534 0 0 0 488728 257 0 0 19 0 8 0 531721667 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=414400 CPUtime=4900.28 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 6917 0 0 0 489759 269 0 0 19 0 8 0 531721667 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 4653121 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=414400 CPUtime=4893.18 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 6795 0 0 0 489046 272 0 0 25 0 8 0 531721667 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=414400 CPUtime=4899.92 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 5187 0 0 0 489729 263 0 0 25 0 8 0 531721667 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=414400 CPUtime=4895.98 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 6753 0 0 0 489338 260 0 0 25 0 8 0 531721668 424345600 47482 33554432000 4194304 4964704 140737433273888 18446744073709551615 220815934112 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39156.8
Current children cumulated vsize (KiB) 414400

[startup+4962.3 s]
/proc/loadavg: 8.02 8.05 8.04 9/295 31266
/proc/meminfo: memFree=24678688/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=414400 CPUtime=39636 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 47605 0 0 0 3961398 2205 0 0 25 0 8 0 531721666 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31111/statm: 103600 47526 464 189 0 97949 0
[pid=31111/tid=31112] ppid=31109 vsize=414400 CPUtime=4947.38 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 5461 0 0 0 494426 312 0 0 18 0 8 0 531721667 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=414400 CPUtime=4954.62 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 7362 0 0 0 495187 275 0 0 22 0 8 0 531721667 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=414400 CPUtime=4949.29 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 4535 0 0 0 494669 260 0 0 25 0 8 0 531721667 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670450 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=414400 CPUtime=4960.27 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 6919 0 0 0 495754 273 0 0 25 0 8 0 531721667 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321076 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=414400 CPUtime=4953.16 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 6808 0 0 0 495040 276 0 0 25 0 8 0 531721667 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=414400 CPUtime=4959.91 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 5188 0 0 0 495724 267 0 0 25 0 8 0 531721667 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=414400 CPUtime=4955.95 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 6767 0 0 0 495332 263 0 0 25 0 8 0 531721668 424345600 47526 33554432000 4194304 4964704 140737433273888 18446744073709551615 4660167 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39636
Current children cumulated vsize (KiB) 414400



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.9 s]
/proc/loadavg: 8.05 8.05 8.04 9/295 31267
/proc/meminfo: memFree=24678784/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=414400 CPUtime=40000.4 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 47622 0 0 0 3997807 2228 0 0 25 0 8 0 531721666 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31111/statm: 103600 47543 464 189 0 97949 0
[pid=31111/tid=31112] ppid=31109 vsize=414400 CPUtime=4992.97 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 5462 0 0 0 498982 315 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321042 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=414400 CPUtime=5000.21 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 7375 0 0 0 499744 277 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321082 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=414400 CPUtime=4994.47 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 4536 0 0 0 499184 263 0 0 22 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=414400 CPUtime=5005.85 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 6920 0 0 0 500310 275 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=414400 CPUtime=4998.76 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 6808 0 0 0 499596 280 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=414400 CPUtime=5005.5 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 5188 0 0 0 500279 271 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=414400 CPUtime=5001.55 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 6767 0 0 0 499889 266 0 0 25 0 8 0 531721668 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 414400

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.9 s]
/proc/loadavg: 8.05 8.05 8.04 9/295 31267
/proc/meminfo: memFree=24678784/32951124 swapFree=67105204/67111528
[pid=31111] ppid=31109 vsize=414400 CPUtime=40000.4 cores=0-7
/proc/31111/stat : 31111 (claspmt) R 31109 31111 30990 0 -1 4202496 47622 0 0 0 3997807 2228 0 0 25 0 8 0 531721666 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/31111/statm: 103600 47543 464 189 0 97949 0
[pid=31111/tid=31112] ppid=31109 vsize=414400 CPUtime=4992.97 cores=0-7
/proc/31111/task/31112/stat : 31112 (claspmt) R 31109 31111 30990 0 -1 4202560 5462 0 0 0 498982 315 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321042 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=31111/tid=31113] ppid=31109 vsize=414400 CPUtime=5000.21 cores=0-7
/proc/31111/task/31113/stat : 31113 (claspmt) R 31109 31111 30990 0 -1 4202560 7375 0 0 0 499744 277 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321082 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=31111/tid=31114] ppid=31109 vsize=414400 CPUtime=4994.47 cores=0-7
/proc/31111/task/31114/stat : 31114 (claspmt) R 31109 31111 30990 0 -1 4202560 4536 0 0 0 499184 263 0 0 22 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=31111/tid=31115] ppid=31109 vsize=414400 CPUtime=5005.85 cores=0-7
/proc/31111/task/31115/stat : 31115 (claspmt) R 31109 31111 30990 0 -1 4202560 6920 0 0 0 500310 275 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=31111/tid=31116] ppid=31109 vsize=414400 CPUtime=4998.76 cores=0-7
/proc/31111/task/31116/stat : 31116 (claspmt) R 31109 31111 30990 0 -1 4202560 6808 0 0 0 499596 280 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=31111/tid=31117] ppid=31109 vsize=414400 CPUtime=5005.5 cores=0-7
/proc/31111/task/31117/stat : 31117 (claspmt) R 31109 31111 30990 0 -1 4202560 5188 0 0 0 500279 271 0 0 25 0 8 0 531721667 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=31111/tid=31119] ppid=31109 vsize=414400 CPUtime=5001.55 cores=0-7
/proc/31111/task/31119/stat : 31119 (claspmt) R 31109 31111 30990 0 -1 4202560 6767 0 0 0 499889 266 0 0 25 0 8 0 531721668 424345600 47543 33554432000 4194304 4964704 140737433273888 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 414400

Child status: 0
Real time (s): 5007.94
CPU time (s): 40000.4
CPU user time (s): 39978.1
CPU system time (s): 22.2956
CPU usage (%): 798.74
Max. virtual memory (cumulated for all children) (KiB): 414400

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39978.1
system time used= 22.2956
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 47634
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3199002
involuntary context switches= 191418

runsolver used 13.358 second user time and 32.1771 second system time

The end

Launcher Data 

Begin job on node147 at 2011-05-03 05:24:18
IDJOB=3322701
IDBENCH=87453
IDSOLVER=1711
FILE ID=node147/3322701-1304393058
RUNJOBID= node147-1304391115-31004
PBS_JOBID= 13181841
Free space on /tmp= 73680 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT02/handmade/pehoushek/plainoldcnf/5cnf_3900_3900_220.shuffled.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322701-1304393058/watcher-3322701-1304393058 -o /tmp/evaluation-result-3322701-1304393058/solver-3322701-1304393058 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322701-1304393058.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= f0b7e0675915478b15cab3695fd0fd62
RANDOM SEED=1920180191

node147.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.836
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24869560 kB
Buffers:        221736 kB
Cached:         903980 kB
SwapCached:       3248 kB
Active:        6772936 kB
Inactive:      1001052 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24869560 kB
SwapTotal:    67111528 kB
SwapFree:     67105204 kB
Dirty:             788 kB
Writeback:           0 kB
AnonPages:     6647396 kB
Mapped:          24152 kB
Slab:           214344 kB
PageTables:      29556 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 44855920 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73676 MiB
End job on node147 at 2011-05-03 06:47:48