Trace number 3380673

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 35703.1 5100.47

General information on the benchmark

NameAPPLICATIONS/satComp09_BioInstances/
rbcl_xits_09_UNKNOWN.cnf
MD5SUM18ad1dee1e9d863458e75a20545ae68a
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1430
Number of clauses79453
Sum of the clauses size236162
Maximum clause length10
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22358
Number of clauses of size 377072
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 523

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...486646/instance-3380673-1305486646.cnf
0.59/0.69	c Solving...
35702.54/5100.02	
35702.54/5100.02	*** INTERRUPTED! ***
35702.54/5100.02	*** Shutting down threads...
35703.04/5100.46	s UNKNOWN
35703.04/5100.46	
35703.04/5100.46	c INTERRUPTED   : 1
35703.04/5100.46	c Models        : 0+    
35703.04/5100.46	c Time          : 5100.175s (Solving: 5099.48s 1st Model: 0.00s Unsat: 0.00s)
35703.04/5100.46	c CPU Time      : 35702.810s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380673-1305486646/watcher-3380673-1305486646 -o /tmp/evaluation-result-3380673-1305486646/solver-3380673-1305486646 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380673-1305486646.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.46 8.32 8.17 4/386 1636
/proc/meminfo: memFree=16245644/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=21320 CPUtime=0 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 625 0 0 0 0 0 0 0 23 0 1 0 641080096 21831680 545 33554432000 4194304 4964800 140735572599072 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1636/statm: 5330 546 386 189 0 204 0

[startup+0.112547 s]
/proc/loadavg: 8.46 8.32 8.17 4/386 1636
/proc/meminfo: memFree=16245644/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=27616 CPUtime=0.1 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 2500 0 0 0 10 0 0 0 24 0 1 0 641080096 28278784 2115 33554432000 4194304 4964800 140735572599072 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1636/statm: 6904 2115 399 189 0 1778 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 27616

[startup+0.200534 s]
/proc/loadavg: 8.46 8.32 8.17 4/386 1636
/proc/meminfo: memFree=16245644/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=27748 CPUtime=0.19 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 2540 0 0 0 19 0 0 0 24 0 1 0 641080096 28413952 2155 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095111877 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1636/statm: 6937 2155 400 189 0 1811 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 27748

[startup+0.300515 s]
/proc/loadavg: 8.46 8.32 8.17 4/386 1636
/proc/meminfo: memFree=16245644/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=28276 CPUtime=0.29 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 2669 0 0 0 29 0 0 0 25 0 1 0 641080096 28954624 2284 33554432000 4194304 4964800 140735572599072 18446744073709551615 4600389 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1636/statm: 7069 2284 400 189 0 1943 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 28276

[startup+0.700434 s]
/proc/loadavg: 8.46 8.32 8.17 4/386 1636
/proc/meminfo: memFree=16245644/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=259996 CPUtime=0.69 cores=0-7
/proc/1636/stat : 1636 (claspmt) S 1634 1636 1142 0 -1 4202496 3823 0 0 0 69 0 0 0 18 0 8 0 641080096 266235904 3237 33554432000 4194304 4964800 140735572599072 18446744073709551615 228107266969 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/1636/statm: 64999 3240 433 189 0 59348 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 259996

[startup+1.50126 s]
/proc/loadavg: 8.46 8.32 8.17 10/394 1644
/proc/meminfo: memFree=16222524/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=262472 CPUtime=6.12 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 9081 0 0 0 610 2 0 0 18 0 8 0 641080096 268771328 8485 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1636/statm: 65618 8485 464 189 0 59967 0
[pid=1636/tid=1638] ppid=1634 vsize=262472 CPUtime=0.74 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 555 0 0 0 74 0 0 0 25 0 8 0 641080166 268771328 8485 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=262472 CPUtime=0.76 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 1034 0 0 0 76 0 0 0 19 0 8 0 641080166 268771328 8485 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=262472 CPUtime=0.7 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 538 0 0 0 70 0 0 0 18 0 8 0 641080166 268771328 8486 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=262472 CPUtime=0.63 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 848 0 0 0 63 0 0 0 18 0 8 0 641080166 268771328 8486 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095112225 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=262472 CPUtime=0.79 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 845 0 0 0 79 0 0 0 19 0 8 0 641080166 268771328 8486 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670546 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=262472 CPUtime=0.51 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 646 0 0 0 51 0 0 0 18 0 8 0 641080166 268771328 8486 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=262472 CPUtime=0.52 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 706 0 0 0 52 0 0 0 18 0 8 0 641080166 268771328 8491 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627259 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 6.12
Current children cumulated vsize (KiB) 262472

[startup+3.10096 s]
/proc/loadavg: 8.46 8.32 8.17 10/394 1644
/proc/meminfo: memFree=16200576/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=331228 CPUtime=17.29 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 14453 0 0 0 1723 6 0 0 21 0 8 0 641080096 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668709 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1636/statm: 82807 13857 464 189 0 77156 0
[pid=1636/tid=1638] ppid=1634 vsize=331228 CPUtime=2.32 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 783 0 0 0 232 0 0 0 25 0 8 0 641080166 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=331228 CPUtime=2.35 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 2303 0 0 0 234 1 0 0 21 0 8 0 641080166 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=331228 CPUtime=2.27 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 1111 0 0 0 227 0 0 0 21 0 8 0 641080166 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=331228 CPUtime=1.63 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 1521 0 0 0 163 0 0 0 18 0 8 0 641080166 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=331228 CPUtime=2.39 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 2227 0 0 0 238 1 0 0 22 0 8 0 641080166 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=331228 CPUtime=1.85 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 1148 0 0 0 185 0 0 0 20 0 8 0 641080166 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4342609 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=331228 CPUtime=1.39 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 1133 0 0 0 139 0 0 0 18 0 8 0 641080166 339177472 13857 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 17.29
Current children cumulated vsize (KiB) 331228

[startup+6.30037 s]
/proc/loadavg: 8.50 8.33 8.17 10/394 1644
/proc/meminfo: memFree=16178376/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=334048 CPUtime=39.63 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 18080 0 0 0 3955 8 0 0 25 0 8 0 641080096 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1636/statm: 83512 17484 466 189 0 77861 0
[pid=1636/tid=1638] ppid=1634 vsize=334048 CPUtime=5.47 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 995 0 0 0 547 0 0 0 25 0 8 0 641080166 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670357 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=334048 CPUtime=5.56 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 3279 0 0 0 554 2 0 0 25 0 8 0 641080166 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=334048 CPUtime=3.97 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 1496 0 0 0 397 0 0 0 25 0 8 0 641080166 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=334048 CPUtime=4.22 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 1871 0 0 0 422 0 0 0 25 0 8 0 641080166 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=334048 CPUtime=5.58 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 2756 0 0 0 557 1 0 0 25 0 8 0 641080166 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4628014 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=334048 CPUtime=4.01 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 1378 0 0 0 401 0 0 0 25 0 8 0 641080166 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=334048 CPUtime=4.53 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 1797 0 0 0 453 0 0 0 25 0 8 0 641080166 342065152 17484 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 39.63
Current children cumulated vsize (KiB) 334048

[startup+12.7012 s]
/proc/loadavg: 8.54 8.34 8.18 10/394 1644
/proc/meminfo: memFree=16157948/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=337300 CPUtime=84.35 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 22953 0 0 0 8424 11 0 0 25 0 8 0 641080096 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668689 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1636/statm: 84325 22357 466 189 0 78674 0
[pid=1636/tid=1638] ppid=1634 vsize=337300 CPUtime=11.79 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 1187 0 0 0 1179 0 0 0 25 0 8 0 641080166 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320984 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=337300 CPUtime=11.96 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 5081 0 0 0 1193 3 0 0 25 0 8 0 641080166 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=337300 CPUtime=7.19 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 1675 0 0 0 719 0 0 0 25 0 8 0 641080166 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=337300 CPUtime=8.22 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 2549 0 0 0 822 0 0 0 25 0 8 0 641080166 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=337300 CPUtime=11.99 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 3261 0 0 0 1197 2 0 0 25 0 8 0 641080166 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670378 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=337300 CPUtime=9.59 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 1527 0 0 0 959 0 0 0 25 0 8 0 641080166 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627272 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=337300 CPUtime=10.93 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 2805 0 0 0 1092 1 0 0 25 0 8 0 641080166 345395200 22357 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 84.35
Current children cumulated vsize (KiB) 337300

[startup+25.5008 s]
/proc/loadavg: 8.64 8.37 8.19 10/394 1644
/proc/meminfo: memFree=16120800/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=343692 CPUtime=173.77 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 32407 0 0 0 17360 17 0 0 25 0 8 0 641080096 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095148642 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1636/statm: 85923 31805 466 189 0 80272 0
[pid=1636/tid=1638] ppid=1634 vsize=343692 CPUtime=24.42 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 1903 0 0 0 2441 1 0 0 25 0 8 0 641080166 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=343692 CPUtime=24.76 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 7176 0 0 0 2472 4 0 0 25 0 8 0 641080166 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=343692 CPUtime=13.99 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 1993 0 0 0 1398 1 0 0 25 0 8 0 641080166 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=343692 CPUtime=18.72 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 3421 0 0 0 1872 0 0 0 25 0 8 0 641080166 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=343692 CPUtime=24.78 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 5302 0 0 0 2475 3 0 0 25 0 8 0 641080166 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668725 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=343692 CPUtime=22.4 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 2607 0 0 0 2239 1 0 0 25 0 8 0 641080166 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670546 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=343692 CPUtime=19.23 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 4214 0 0 0 1921 2 0 0 25 0 8 0 641080166 351940608 31805 33554432000 4194304 4964800 140735572599072 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 173.77
Current children cumulated vsize (KiB) 343692

[startup+51.1011 s]
/proc/loadavg: 8.76 8.42 8.21 10/394 1645
/proc/meminfo: memFree=16075380/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=354048 CPUtime=352.56 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 42289 0 0 0 35231 25 0 0 18 0 8 0 641080096 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668681 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/1636/statm: 88512 41504 466 189 0 82861 0
[pid=1636/tid=1638] ppid=1634 vsize=354048 CPUtime=41.43 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 2544 0 0 0 4142 1 0 0 25 0 8 0 641080166 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=354048 CPUtime=50.36 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 10412 0 0 0 5030 6 0 0 25 0 8 0 641080166 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=354048 CPUtime=33.25 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 2700 0 0 0 3323 2 0 0 25 0 8 0 641080166 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320942 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=354048 CPUtime=37.13 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 4512 0 0 0 3712 1 0 0 25 0 8 0 641080166 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 4334301 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=354048 CPUtime=50.3 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 6771 0 0 0 5025 5 0 0 25 0 8 0 641080166 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=354048 CPUtime=46.85 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 3429 0 0 0 4684 1 0 0 25 0 8 0 641080166 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=354048 CPUtime=42.96 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 5790 0 0 0 4293 3 0 0 25 0 8 0 641080166 362545152 41504 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 352.56

################
# More data... #
################

/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 17611 0 0 0 397697 27 0 0 25 0 8 0 641080166 1279672320 262632 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668640 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=1249680 CPUtime=4181.18 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 35504 0 0 0 418059 59 0 0 25 0 8 0 641080166 1279672320 262632 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 33481.4
Current children cumulated vsize (KiB) 1249680

[startup+4842.31 s]
/proc/loadavg: 9.02 9.06 9.01 10/394 1788
/proc/meminfo: memFree=14567980/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1250640 CPUtime=33900.7 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 279415 0 0 0 3389761 313 0 0 25 0 8 0 641080096 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668689 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1636/statm: 312660 263001 466 189 0 307009 0
[pid=1636/tid=1638] ppid=1634 vsize=1250640 CPUtime=4261.25 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 33117 0 0 0 426098 27 0 0 25 0 8 0 641080166 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670417 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=1250640 CPUtime=4607.27 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 68239 0 0 0 460678 49 0 0 25 0 8 0 641080166 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=1250640 CPUtime=3970.74 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 14593 0 0 0 397051 23 0 0 25 0 8 0 641080166 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=1250640 CPUtime=4448.24 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 44875 0 0 0 444778 46 0 0 25 0 8 0 641080166 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=1250640 CPUtime=4536.39 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 43504 0 0 0 453595 44 0 0 25 0 8 0 641080166 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=1250640 CPUtime=4032.68 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 17612 0 0 0 403241 27 0 0 25 0 8 0 641080166 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=1250640 CPUtime=4241.19 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 35556 0 0 0 424060 59 0 0 25 0 8 0 641080166 1280655360 263001 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095148658 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 33900.7
Current children cumulated vsize (KiB) 1250640

[startup+4902.3 s]
/proc/loadavg: 9.00 9.05 9.01 10/394 1790
/proc/meminfo: memFree=14564220/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1250640 CPUtime=34319.9 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 280769 0 0 0 3431671 316 0 0 25 0 8 0 641080096 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670310 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1636/statm: 312660 264193 466 189 0 307009 0
[pid=1636/tid=1638] ppid=1634 vsize=1250640 CPUtime=4301.04 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 33335 0 0 0 430077 27 0 0 25 0 8 0 641080166 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=1250640 CPUtime=4667.26 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 68685 0 0 0 466676 50 0 0 25 0 8 0 641080166 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=1250640 CPUtime=4021.44 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 14593 0 0 0 402120 24 0 0 25 0 8 0 641080166 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=1250640 CPUtime=4508.21 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 44883 0 0 0 450775 46 0 0 25 0 8 0 641080166 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=1250640 CPUtime=4596.37 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 43620 0 0 0 459593 44 0 0 25 0 8 0 641080166 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=1250640 CPUtime=4082.79 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 17845 0 0 0 408252 27 0 0 25 0 8 0 641080166 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=1250640 CPUtime=4301.18 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 35828 0 0 0 430058 60 0 0 25 0 8 0 641080166 1280655360 264193 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 34319.9
Current children cumulated vsize (KiB) 1250640

[startup+4962.3 s]
/proc/loadavg: 9.00 9.04 9.00 10/394 1791
/proc/meminfo: memFree=14561088/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1250640 CPUtime=34739.1 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 281564 0 0 0 3473593 317 0 0 25 0 8 0 641080096 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627412 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1636/statm: 312660 264850 466 189 0 307009 0
[pid=1636/tid=1638] ppid=1634 vsize=1250640 CPUtime=4346.2 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 33603 0 0 0 434593 27 0 0 25 0 8 0 641080166 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=1250640 CPUtime=4727.26 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 69205 0 0 0 472676 50 0 0 25 0 8 0 641080166 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=1250640 CPUtime=4056.22 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 14594 0 0 0 405598 24 0 0 25 0 8 0 641080166 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=1250640 CPUtime=4568.21 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 44889 0 0 0 456775 46 0 0 25 0 8 0 641080166 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=1250640 CPUtime=4656.36 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 43620 0 0 0 465592 44 0 0 25 0 8 0 641080166 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=1250640 CPUtime=4130.98 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 17845 0 0 0 413071 27 0 0 25 0 8 0 641080166 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627873 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=1250640 CPUtime=4352.28 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 35828 0 0 0 435167 61 0 0 24 0 8 0 641080166 1280655360 264850 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 34739.1
Current children cumulated vsize (KiB) 1250640

[startup+5022.3 s]
/proc/loadavg: 9.07 9.04 9.00 10/394 1793
/proc/meminfo: memFree=14556356/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1250640 CPUtime=35159.6 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 282680 0 0 0 3515644 318 0 0 25 0 8 0 641080096 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1636/statm: 312660 265828 466 189 0 307009 0
[pid=1636/tid=1638] ppid=1634 vsize=1250640 CPUtime=4405.03 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 33797 0 0 0 440476 27 0 0 24 0 8 0 641080166 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=1250640 CPUtime=4787.26 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 70123 0 0 0 478676 50 0 0 25 0 8 0 641080166 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=1250640 CPUtime=4105.07 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 14594 0 0 0 410483 24 0 0 25 0 8 0 641080166 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=1250640 CPUtime=4620.11 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 44889 0 0 0 461965 46 0 0 25 0 8 0 641080166 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=1250640 CPUtime=4711.83 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 43621 0 0 0 471139 44 0 0 25 0 8 0 641080166 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=1250640 CPUtime=4169 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 17848 0 0 0 416872 28 0 0 25 0 8 0 641080166 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=1250640 CPUtime=4403.28 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 35828 0 0 0 440267 61 0 0 25 0 8 0 641080166 1280655360 265828 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 35159.6
Current children cumulated vsize (KiB) 1250640

[startup+5082.3 s]
/proc/loadavg: 9.09 9.05 9.00 10/394 1794
/proc/meminfo: memFree=14546792/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1250640 CPUtime=35578.8 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 285201 0 0 0 3557561 323 0 0 25 0 8 0 641080096 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1636/statm: 312660 268349 466 189 0 307009 0
[pid=1636/tid=1638] ppid=1634 vsize=1250640 CPUtime=4452.28 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 33926 0 0 0 445200 28 0 0 25 0 8 0 641080166 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668766 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=1250640 CPUtime=4847.25 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 71294 0 0 0 484674 51 0 0 25 0 8 0 641080166 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4320984 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=1250640 CPUtime=4152.88 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 14597 0 0 0 415264 24 0 0 25 0 8 0 641080166 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=1250640 CPUtime=4680.11 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 45892 0 0 0 467963 48 0 0 25 0 8 0 641080166 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670321 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=1250640 CPUtime=4771.82 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 43749 0 0 0 477138 44 0 0 25 0 8 0 641080166 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=1250640 CPUtime=4211.13 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 17931 0 0 0 421085 28 0 0 25 0 8 0 641080166 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=1250640 CPUtime=4445.33 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 35832 0 0 0 444472 61 0 0 25 0 8 0 641080166 1280655360 268349 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 35578.8
Current children cumulated vsize (KiB) 1250640



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.07 9.05 9.00 10/394 1795
/proc/meminfo: memFree=14546168/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1250640 CPUtime=35702.5 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 285329 0 0 0 3569931 323 0 0 25 0 8 0 641080096 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1636/statm: 312660 268477 466 189 0 307009 0
[pid=1636/tid=1638] ppid=1634 vsize=1250640 CPUtime=4469.76 cores=0-7
/proc/1636/task/1638/stat : 1638 (claspmt) R 1634 1636 1142 0 -1 4202560 33926 0 0 0 446948 28 0 0 25 0 8 0 641080166 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1636/tid=1639] ppid=1634 vsize=1250640 CPUtime=4864.97 cores=0-7
/proc/1636/task/1639/stat : 1639 (claspmt) R 1634 1636 1142 0 -1 4202560 71294 0 0 0 486446 51 0 0 25 0 8 0 641080166 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1636/tid=1640] ppid=1634 vsize=1250640 CPUtime=4168.19 cores=0-7
/proc/1636/task/1640/stat : 1640 (claspmt) R 1634 1636 1142 0 -1 4202560 14601 0 0 0 416795 24 0 0 25 0 8 0 641080166 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095556078 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1641] ppid=1634 vsize=1250640 CPUtime=4697.83 cores=0-7
/proc/1636/task/1641/stat : 1641 (claspmt) R 1634 1636 1142 0 -1 4202560 45990 0 0 0 469735 48 0 0 25 0 8 0 641080166 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1636/tid=1642] ppid=1634 vsize=1250640 CPUtime=4789.54 cores=0-7
/proc/1636/task/1642/stat : 1642 (claspmt) R 1634 1636 1142 0 -1 4202560 43775 0 0 0 478910 44 0 0 25 0 8 0 641080166 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 4627213 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1636/tid=1643] ppid=1634 vsize=1250640 CPUtime=4220.98 cores=0-7
/proc/1636/task/1643/stat : 1643 (claspmt) R 1634 1636 1142 0 -1 4202560 17931 0 0 0 422070 28 0 0 25 0 8 0 641080166 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1636/tid=1644] ppid=1634 vsize=1250640 CPUtime=4455.59 cores=0-7
/proc/1636/task/1644/stat : 1644 (claspmt) R 1634 1636 1142 0 -1 4202560 35832 0 0 0 445498 61 0 0 25 0 8 0 641080166 1280655360 268477 33554432000 4194304 4964800 140735572599072 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 35702.5
Current children cumulated vsize (KiB) 1250640

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.3 s]
/proc/loadavg: 9.07 9.05 9.00 10/394 1795
/proc/meminfo: memFree=14546168/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1248028 CPUtime=35702.9 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 285339 0 0 0 3569970 324 0 0 24 0 1 0 641080096 1277980672 267834 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095106251 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/1636/statm: 312007 267834 476 189 0 306356 0
Current children cumulated CPU time (s) 35702.9
Current children cumulated vsize (KiB) 1248028

[startup+5100.4 s]
/proc/loadavg: 9.07 9.05 9.00 10/394 1795
/proc/meminfo: memFree=14546168/32951124 swapFree=65980216/67111528
[pid=1636] ppid=1634 vsize=1243164 CPUtime=35703 cores=0-7
/proc/1636/stat : 1636 (claspmt) R 1634 1636 1142 0 -1 4202496 285339 0 0 0 3569980 324 0 0 25 0 1 0 641080096 1272999936 264408 33554432000 4194304 4964800 140735572599072 18446744073709551615 228095106981 0 0 4096 18946 0 0 0 17 3 0 0 0
/proc/1636/statm: 310791 264408 476 189 0 305140 0
Current children cumulated CPU time (s) 35703
Current children cumulated vsize (KiB) 1243164

Child status: 0
Real time (s): 5100.47
CPU time (s): 35703.1
CPU user time (s): 35699.8
CPU system time (s): 3.2935
CPU usage (%): 699.997
Max. virtual memory (cumulated for all children) (KiB): 1250640

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35699.8
system time used= 3.2935
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 285340
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 121803
involuntary context switches= 261158

runsolver used 17.9903 second user time and 46.9829 second system time

The end

Launcher Data 

Begin job on node142 at 2011-05-15 21:10:46
IDJOB=3380673
IDBENCH=71098
IDSOLVER=1865
FILE ID=node142/3380673-1305486646
RUNJOBID= node142-1305481201-1156
PBS_JOBID= 13325338
Free space on /tmp= 73540 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/APPLICATIONS/satComp09_BioInstances/rbcl_xits_09_UNKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380673-1305486646/watcher-3380673-1305486646 -o /tmp/evaluation-result-3380673-1305486646/solver-3380673-1305486646 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380673-1305486646.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 18ad1dee1e9d863458e75a20545ae68a
RANDOM SEED=1985380740

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16245676 kB
Buffers:        345436 kB
Cached:        4845672 kB
SwapCached:      11520 kB
Active:       13253380 kB
Inactive:      3095584 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16245676 kB
SwapTotal:    67111528 kB
SwapFree:     65980216 kB
Dirty:            1960 kB
Writeback:           0 kB
AnonPages:    11155392 kB
Mapped:          22744 kB
Slab:           238560 kB
PageTables:      48244 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 84411220 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73536 MiB
End job on node142 at 2011-05-15 22:35:48