Trace number 3322656

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 35669 5100.15

General information on the benchmark

NameSAT05/crafted/sabharwal05/counting/fphp/unsat/
harder/fphp-030-029.shuffled-as.sat05-1234.cnf
MD5SUMc12aa6a0ac1bca448bef51c167f3bbbb
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark17.0914
Satisfiable
(Un)Satisfiability was proved
Number of variables870
Number of clauses24825
Sum of the clauses size50460
Maximum clause length29
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 224795
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 530

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...392260/instance-3322656-1304392260.cnf
0.00/0.05	c Solving...
35668.73/5100.01	
35668.73/5100.01	*** INTERRUPTED! ***
35668.73/5100.01	*** Shutting down threads...
35668.92/5100.14	s UNKNOWN
35668.92/5100.14	
35668.92/5100.14	c INTERRUPTED   : 1
35668.92/5100.14	c Models        : 0+    
35668.92/5100.14	c Time          : 5100.014s (Solving: 5099.96s 1st Model: 0.00s Unsat: 0.00s)
35668.92/5100.14	c CPU Time      : 35668.820s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322656-1304392260/watcher-3322656-1304392260 -o /tmp/evaluation-result-3322656-1304392260/solver-3322656-1304392260 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322656-1304392260.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.06 9.08 9.03 3/298 622
/proc/meminfo: memFree=5258640/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=20872 CPUtime=0 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 448 0 0 0 0 0 0 0 25 0 1 0 283417258 21372928 377 33554432000 4194304 4964704 140735530265632 18446744073709551615 4242896 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/622/statm: 5218 380 324 189 0 92 0

[startup+0.0735681 s]
/proc/loadavg: 9.06 9.08 9.03 3/298 622
/proc/meminfo: memFree=5258640/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=186128 CPUtime=0.09 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 1452 0 0 0 9 0 0 0 25 0 8 0 283417258 190595072 1257 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320970 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 46532 1258 458 189 0 40881 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 186128

[startup+0.100555 s]
/proc/loadavg: 9.06 9.08 9.03 3/298 622
/proc/meminfo: memFree=5258640/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=317996 CPUtime=0.29 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 2594 0 0 0 28 1 0 0 18 0 8 0 283417258 325627904 2400 33554432000 4194304 4964704 140735530265632 18446744073709551615 268660323361 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 79598 2686 464 189 0 73947 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 317996

[startup+0.300495 s]
/proc/loadavg: 9.06 9.08 9.03 3/298 622
/proc/meminfo: memFree=5258640/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=518168 CPUtime=1.6 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 9214 0 0 0 156 4 0 0 18 0 8 0 283417258 530604032 9020 33554432000 4194304 4964704 140735530265632 18446744073709551615 4362287 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 129542 9020 464 189 0 123891 0
Current children cumulated CPU time (s) 1.6
Current children cumulated vsize (KiB) 518168

[startup+0.700427 s]
/proc/loadavg: 9.06 9.08 9.03 3/298 622
/proc/meminfo: memFree=5258640/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=519228 CPUtime=4.4 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 12748 0 0 0 433 7 0 0 18 0 8 0 283417258 531689472 12553 33554432000 4194304 4964704 140735530265632 18446744073709551615 4332324 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 129807 12553 464 189 0 124156 0
Current children cumulated CPU time (s) 4.4
Current children cumulated vsize (KiB) 519228

[startup+1.50129 s]
/proc/loadavg: 9.06 9.08 9.03 10/306 630
/proc/meminfo: memFree=5198220/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=523060 CPUtime=9.96 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 19686 0 0 0 986 10 0 0 19 0 8 0 283417258 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4658359 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 130765 19491 466 189 0 125114 0
[pid=622/tid=624] ppid=620 vsize=523060 CPUtime=1.29 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 2337 0 0 0 128 1 0 0 19 0 8 0 283417264 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=625] ppid=620 vsize=523060 CPUtime=1.41 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 3585 0 0 0 140 1 0 0 19 0 8 0 283417264 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=626] ppid=620 vsize=523060 CPUtime=1.41 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 2003 0 0 0 140 1 0 0 19 0 8 0 283417264 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4660156 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=523060 CPUtime=1.4 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 2915 0 0 0 139 1 0 0 18 0 8 0 283417264 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4362349 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=622/tid=628] ppid=620 vsize=523060 CPUtime=0.9 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 1733 0 0 0 89 1 0 0 25 0 8 0 283417264 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=629] ppid=620 vsize=523060 CPUtime=0.62 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 916 0 0 0 62 0 0 0 25 0 8 0 283417264 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4627943 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=630] ppid=620 vsize=523060 CPUtime=1.39 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 3395 0 0 0 138 1 0 0 19 0 8 0 283417264 535613440 19491 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 9.96
Current children cumulated vsize (KiB) 523060

[startup+3.101 s]
/proc/loadavg: 9.06 9.08 9.03 10/306 630
/proc/meminfo: memFree=5172924/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=525044 CPUtime=21.13 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 26116 0 0 0 2100 13 0 0 23 0 8 0 283417258 537645056 25921 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320944 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 131261 25921 466 189 0 125610 0
[pid=622/tid=624] ppid=620 vsize=525044 CPUtime=2.67 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 2952 0 0 0 266 1 0 0 22 0 8 0 283417264 537645056 25921 33554432000 4194304 4964704 140735530265632 18446744073709551615 4653504 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=625] ppid=620 vsize=525044 CPUtime=2.81 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 4846 0 0 0 279 2 0 0 19 0 8 0 283417264 537645056 25921 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=626] ppid=620 vsize=525044 CPUtime=3.01 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 2361 0 0 0 300 1 0 0 25 0 8 0 283417264 537645056 25921 33554432000 4194304 4964704 140735530265632 18446744073709551615 268660317387 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=525044 CPUtime=2.99 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 4036 0 0 0 297 2 0 0 20 0 8 0 283417264 537645056 25921 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=622/tid=628] ppid=620 vsize=525044 CPUtime=1.32 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 1806 0 0 0 131 1 0 0 25 0 8 0 283417264 537645056 25922 33554432000 4194304 4964704 140735530265632 18446744073709551615 268660767214 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=629] ppid=620 vsize=525044 CPUtime=2.22 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 1128 0 0 0 222 0 0 0 25 0 8 0 283417264 537645056 25922 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=630] ppid=620 vsize=525044 CPUtime=2.97 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 4800 0 0 0 296 1 0 0 23 0 8 0 283417264 537645056 25922 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 21.13
Current children cumulated vsize (KiB) 525044

[startup+6.30044 s]
/proc/loadavg: 9.06 9.08 9.03 10/306 630
/proc/meminfo: memFree=5121980/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=542688 CPUtime=43.44 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 39732 0 0 0 4327 17 0 0 25 0 8 0 283417258 555712512 39456 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 135672 39457 466 189 0 130021 0
[pid=622/tid=624] ppid=620 vsize=542688 CPUtime=5.86 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 5068 0 0 0 584 2 0 0 18 0 8 0 283417264 555712512 39457 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=622/tid=625] ppid=620 vsize=542688 CPUtime=5.78 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 9842 0 0 0 575 3 0 0 25 0 8 0 283417264 555712512 39457 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=626] ppid=620 vsize=542688 CPUtime=6.2 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 2927 0 0 0 619 1 0 0 25 0 8 0 283417264 555712512 39457 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=542688 CPUtime=5.85 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 5766 0 0 0 582 3 0 0 25 0 8 0 283417264 555712512 39457 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=628] ppid=620 vsize=542688 CPUtime=2.59 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 2361 0 0 0 258 1 0 0 24 0 8 0 283417264 555712512 39457 33554432000 4194304 4964704 140735530265632 18446744073709551615 4653140 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=629] ppid=620 vsize=542688 CPUtime=4.71 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 2198 0 0 0 471 0 0 0 25 0 8 0 283417264 555712512 39457 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320942 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=630] ppid=620 vsize=542688 CPUtime=6.14 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 6362 0 0 0 612 2 0 0 25 0 8 0 283417264 555712512 39457 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321088 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 43.44
Current children cumulated vsize (KiB) 542688

[startup+12.7003 s]
/proc/loadavg: 9.05 9.08 9.03 10/306 630
/proc/meminfo: memFree=5022032/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=557516 CPUtime=88.06 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 62029 0 0 0 8778 28 0 0 25 0 8 0 283417258 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 139379 61632 466 189 0 133728 0
[pid=622/tid=624] ppid=620 vsize=557516 CPUtime=12.23 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 8830 0 0 0 1220 3 0 0 25 0 8 0 283417264 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 4325321 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=622/tid=625] ppid=620 vsize=557516 CPUtime=10.56 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 12921 0 0 0 1052 4 0 0 24 0 8 0 283417264 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 268660767214 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=626] ppid=620 vsize=557516 CPUtime=12.43 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 5381 0 0 0 1241 2 0 0 20 0 8 0 283417264 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 4668484 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=557516 CPUtime=11.45 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 8368 0 0 0 1141 4 0 0 25 0 8 0 283417264 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 4362377 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=628] ppid=620 vsize=557516 CPUtime=8.18 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 5546 0 0 0 816 2 0 0 25 0 8 0 283417264 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 4362359 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=629] ppid=620 vsize=557516 CPUtime=9.88 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 4040 0 0 0 986 2 0 0 25 0 8 0 283417264 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 4653264 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=630] ppid=620 vsize=557516 CPUtime=10.64 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 8842 0 0 0 1061 3 0 0 25 0 8 0 283417264 570896384 61632 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 88.06
Current children cumulated vsize (KiB) 557516

[startup+25.5011 s]
/proc/loadavg: 9.12 9.09 9.03 10/306 631
/proc/meminfo: memFree=4954044/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=557592 CPUtime=177.44 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 76446 0 0 0 17706 38 0 0 23 0 8 0 283417258 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320966 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/622/statm: 139398 75879 466 189 0 133747 0
[pid=622/tid=624] ppid=620 vsize=557592 CPUtime=23.63 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 12718 0 0 0 2357 6 0 0 18 0 8 0 283417264 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4660059 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=622/tid=625] ppid=620 vsize=557592 CPUtime=21.69 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 15461 0 0 0 2164 5 0 0 25 0 8 0 283417264 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320976 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=626] ppid=620 vsize=557592 CPUtime=24.37 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 6963 0 0 0 2434 3 0 0 25 0 8 0 283417264 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=557592 CPUtime=20.58 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 9387 0 0 0 2053 5 0 0 25 0 8 0 283417264 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=628] ppid=620 vsize=557592 CPUtime=20.41 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 6655 0 0 0 2037 4 0 0 25 0 8 0 283417264 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670431 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=629] ppid=620 vsize=557592 CPUtime=22.68 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 5372 0 0 0 2265 3 0 0 25 0 8 0 283417264 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=630] ppid=620 vsize=557592 CPUtime=19.91 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 9848 0 0 0 1987 4 0 0 25 0 8 0 283417264 570974208 75879 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 177.44
Current children cumulated vsize (KiB) 557592

[startup+51.1006 s]
/proc/loadavg: 9.08 9.08 9.03 10/306 631
/proc/meminfo: memFree=4868456/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=557592 CPUtime=356.29 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 97001 0 0 0 35577 52 0 0 25 0 8 0 283417258 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4364587 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/622/statm: 139398 96330 466 189 0 133747 0
[pid=622/tid=624] ppid=620 vsize=557592 CPUtime=43.11 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 14668 0 0 0 4303 8 0 0 25 0 8 0 283417264 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=625] ppid=620 vsize=557592 CPUtime=43.83 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 19875 0 0 0 4376 7 0 0 25 0 8 0 283417264 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320976 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=622/tid=626] ppid=620 vsize=557592 CPUtime=49.98 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 9545 0 0 0 4993 5 0 0 25 0 8 0 283417264 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=557592 CPUtime=42.12 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 11133 0 0 0 4206 6 0 0 25 0 8 0 283417264 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=628] ppid=620 vsize=557592 CPUtime=43.25 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 9139 0 0 0 4319 6 0 0 25 0 8 0 283417264 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670215 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=629] ppid=620 vsize=557592 CPUtime=43.05 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 7026 0 0 0 4301 4 0 0 25 0 8 0 283417264 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=630] ppid=620 vsize=557592 CPUtime=41.88 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 13396 0 0 0 4181 7 0 0 22 0 8 0 283417264 570974208 96330 33554432000 4194304 4964704 140735530265632 18446744073709551615 4325338 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 356.29

################
# More data... #
################

[pid=622/tid=625] ppid=620 vsize=1975304 CPUtime=4374.41 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 117434 0 0 0 437351 90 0 0 25 0 8 0 283417264 2022711296 406297 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=626] ppid=620 vsize=1975304 CPUtime=4059 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 37784 0 0 0 405850 50 0 0 25 0 8 0 283417264 2022711296 406297 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=627] ppid=620 vsize=1975304 CPUtime=4378.98 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 35227 0 0 0 437850 48 0 0 25 0 8 0 283417264 2022711296 406297 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=628] ppid=620 vsize=1975304 CPUtime=4342.61 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 38846 0 0 0 434210 51 0 0 25 0 8 0 283417264 2022711296 406297 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=629] ppid=620 vsize=1975304 CPUtime=4221.92 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 29468 0 0 0 422142 50 0 0 25 0 8 0 283417264 2022711296 406297 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=630] ppid=620 vsize=1975304 CPUtime=4233.37 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 46298 0 0 0 423280 57 0 0 25 0 8 0 283417264 2022711296 406297 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 33448.2
Current children cumulated vsize (KiB) 1975304

[startup+4842.3 s]
/proc/loadavg: 9.05 9.08 9.01 10/306 772
/proc/meminfo: memFree=3003740/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=1975304 CPUtime=33867.6 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 422442 0 0 0 3386253 506 0 0 25 0 8 0 283417258 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/622/statm: 493826 406309 466 189 0 488175 0
[pid=622/tid=624] ppid=620 vsize=1975304 CPUtime=3742.77 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 77251 0 0 0 374180 97 0 0 25 0 8 0 283417264 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=625] ppid=620 vsize=1975304 CPUtime=4430.15 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 117434 0 0 0 442925 90 0 0 25 0 8 0 283417264 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=626] ppid=620 vsize=1975304 CPUtime=4103.54 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 37785 0 0 0 410304 50 0 0 25 0 8 0 283417264 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=1975304 CPUtime=4431.04 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 35238 0 0 0 443056 48 0 0 25 0 8 0 283417264 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=628] ppid=620 vsize=1975304 CPUtime=4402.16 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 38851 0 0 0 440165 51 0 0 25 0 8 0 283417264 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=629] ppid=620 vsize=1975304 CPUtime=4266.84 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 29468 0 0 0 426634 50 0 0 25 0 8 0 283417264 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321082 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=630] ppid=620 vsize=1975304 CPUtime=4287.27 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 46298 0 0 0 428670 57 0 0 25 0 8 0 283417264 2022711296 406309 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 33867.6
Current children cumulated vsize (KiB) 1975304

[startup+4902.3 s]
/proc/loadavg: 9.02 9.06 9.01 10/306 774
/proc/meminfo: memFree=2994016/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=1975304 CPUtime=34286.9 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 423045 0 0 0 3428187 508 0 0 25 0 8 0 283417258 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 7 0 0 0
/proc/622/statm: 493826 406911 466 189 0 488175 0
[pid=622/tid=624] ppid=620 vsize=1975304 CPUtime=3798.83 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 77313 0 0 0 379786 97 0 0 25 0 8 0 283417264 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670250 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=625] ppid=620 vsize=1975304 CPUtime=4481.91 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 117707 0 0 0 448101 90 0 0 25 0 8 0 283417264 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670263 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=626] ppid=620 vsize=1975304 CPUtime=4145.05 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 37787 0 0 0 414455 50 0 0 25 0 8 0 283417264 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=1975304 CPUtime=4488.82 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 35363 0 0 0 448834 48 0 0 25 0 8 0 283417264 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320963 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=628] ppid=620 vsize=1975304 CPUtime=4462.17 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 38867 0 0 0 446165 52 0 0 25 0 8 0 283417264 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670252 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=629] ppid=620 vsize=1975304 CPUtime=4304.67 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 29469 0 0 0 430416 51 0 0 25 0 8 0 283417264 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320976 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=630] ppid=620 vsize=1975304 CPUtime=4346.18 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 46298 0 0 0 434560 58 0 0 25 0 8 0 283417264 2022711296 406911 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 34286.9
Current children cumulated vsize (KiB) 1975304

[startup+4962.3 s]
/proc/loadavg: 9.00 9.05 9.00 10/306 775
/proc/meminfo: memFree=2985908/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=1975304 CPUtime=34706.3 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 423203 0 0 0 3470118 511 0 0 25 0 8 0 283417258 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 0
/proc/622/statm: 493826 407067 466 189 0 488175 0
[pid=622/tid=624] ppid=620 vsize=1975304 CPUtime=3856.42 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 77314 0 0 0 385544 98 0 0 25 0 8 0 283417264 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670252 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=625] ppid=620 vsize=1975304 CPUtime=4536.16 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 117780 0 0 0 453525 91 0 0 25 0 8 0 283417264 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=626] ppid=620 vsize=1975304 CPUtime=4198.85 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 37790 0 0 0 419834 51 0 0 25 0 8 0 283417264 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=1975304 CPUtime=4545.5 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 35363 0 0 0 454501 49 0 0 25 0 8 0 283417264 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 268660767214 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=628] ppid=620 vsize=1975304 CPUtime=4510.24 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 38945 0 0 0 450972 52 0 0 25 0 8 0 283417264 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=629] ppid=620 vsize=1975304 CPUtime=4350.66 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 29472 0 0 0 435015 51 0 0 25 0 8 0 283417264 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 4653184 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=630] ppid=620 vsize=1975304 CPUtime=4398.53 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 46298 0 0 0 439795 58 0 0 25 0 8 0 283417264 2022711296 407067 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 34706.3
Current children cumulated vsize (KiB) 1975304

[startup+5022.3 s]
/proc/loadavg: 9.06 9.05 9.00 10/306 777
/proc/meminfo: memFree=2977952/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=1975304 CPUtime=35125.6 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 423211 0 0 0 3512050 514 0 0 25 0 8 0 283417258 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/622/statm: 493826 407068 466 189 0 488175 0
[pid=622/tid=624] ppid=620 vsize=1975304 CPUtime=3912.71 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 77314 0 0 0 391172 99 0 0 25 0 8 0 283417264 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=625] ppid=620 vsize=1975304 CPUtime=4590.87 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 117780 0 0 0 458996 91 0 0 25 0 8 0 283417264 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=626] ppid=620 vsize=1975304 CPUtime=4253.24 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 37790 0 0 0 425273 51 0 0 25 0 8 0 283417264 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=627] ppid=620 vsize=1975304 CPUtime=4595.96 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 35371 0 0 0 459547 49 0 0 25 0 8 0 283417264 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=628] ppid=620 vsize=1975304 CPUtime=4568.62 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 38945 0 0 0 456810 52 0 0 25 0 8 0 283417264 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670248 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=629] ppid=620 vsize=1975304 CPUtime=4395.34 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 29472 0 0 0 439483 51 0 0 25 0 8 0 283417264 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320939 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=630] ppid=620 vsize=1975304 CPUtime=4449.16 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 46298 0 0 0 444858 58 0 0 25 0 8 0 283417264 2022711296 407068 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320864 0 0 4096 18946 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 35125.6
Current children cumulated vsize (KiB) 1975304

[startup+5082.3 s]
/proc/loadavg: 9.08 9.06 9.00 10/306 779
/proc/meminfo: memFree=2970088/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=1975304 CPUtime=35545 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 423231 0 0 0 3553983 517 0 0 25 0 8 0 283417258 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4332351 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/622/statm: 493826 407087 466 189 0 488175 0
[pid=622/tid=624] ppid=620 vsize=1975304 CPUtime=3968.88 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 77314 0 0 0 396789 99 0 0 25 0 8 0 283417264 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=625] ppid=620 vsize=1975304 CPUtime=4650.86 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 117796 0 0 0 464995 91 0 0 25 0 8 0 283417264 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=626] ppid=620 vsize=1975304 CPUtime=4307.84 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 37790 0 0 0 430732 52 0 0 25 0 8 0 283417264 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=622/tid=627] ppid=620 vsize=1975304 CPUtime=4655.95 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 35371 0 0 0 465546 49 0 0 25 0 8 0 283417264 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=628] ppid=620 vsize=1975304 CPUtime=4603.99 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 38945 0 0 0 460347 52 0 0 25 0 8 0 283417264 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=622/tid=629] ppid=620 vsize=1975304 CPUtime=4454.74 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 29475 0 0 0 445422 52 0 0 25 0 8 0 283417264 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=630] ppid=620 vsize=1975304 CPUtime=4501.41 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 46298 0 0 0 450082 59 0 0 24 0 8 0 283417264 2022711296 407087 33554432000 4194304 4964704 140735530265632 18446744073709551615 4627290 0 0 4096 18946 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 35545
Current children cumulated vsize (KiB) 1975304



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 9.06 9.05 9.00 10/306 779
/proc/meminfo: memFree=2967720/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=1975304 CPUtime=35668.7 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 423231 0 0 0 3566356 517 0 0 25 0 8 0 283417258 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/622/statm: 493826 407086 466 189 0 488175 0
[pid=622/tid=624] ppid=620 vsize=1975304 CPUtime=3985.6 cores=0-7
/proc/622/task/624/stat : 624 (claspmt) R 620 622 481 0 -1 4202560 77314 0 0 0 398461 99 0 0 25 0 8 0 283417264 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=625] ppid=620 vsize=1975304 CPUtime=4668.57 cores=0-7
/proc/622/task/625/stat : 625 (claspmt) R 620 622 481 0 -1 4202560 117796 0 0 0 466766 91 0 0 25 0 8 0 283417264 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321093 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=622/tid=626] ppid=620 vsize=1975304 CPUtime=4320.28 cores=0-7
/proc/622/task/626/stat : 626 (claspmt) R 620 622 481 0 -1 4202560 37790 0 0 0 431976 52 0 0 25 0 8 0 283417264 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4653275 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=622/tid=627] ppid=620 vsize=1975304 CPUtime=4673.66 cores=0-7
/proc/622/task/627/stat : 627 (claspmt) R 620 622 481 0 -1 4202560 35371 0 0 0 467317 49 0 0 25 0 8 0 283417264 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=622/tid=628] ppid=620 vsize=1975304 CPUtime=4621.12 cores=0-7
/proc/622/task/628/stat : 628 (claspmt) R 620 622 481 0 -1 4202560 38945 0 0 0 462060 52 0 0 25 0 8 0 283417264 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=622/tid=629] ppid=620 vsize=1975304 CPUtime=4472.27 cores=0-7
/proc/622/task/629/stat : 629 (claspmt) R 620 622 481 0 -1 4202560 29475 0 0 0 447175 52 0 0 25 0 8 0 283417264 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=622/tid=630] ppid=620 vsize=1975304 CPUtime=4511.72 cores=0-7
/proc/622/task/630/stat : 630 (claspmt) R 620 622 481 0 -1 4202560 46298 0 0 0 451113 59 0 0 25 0 8 0 283417264 2022711296 407086 33554432000 4194304 4964704 140735530265632 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 35668.7
Current children cumulated vsize (KiB) 1975304

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5100.12 s]
/proc/loadavg: 9.06 9.05 9.00 10/306 779
/proc/meminfo: memFree=2967720/32951124 swapFree=67111524/67111528
[pid=622] ppid=620 vsize=1105692 CPUtime=35668.9 cores=0-7
/proc/622/stat : 622 (claspmt) R 620 622 481 0 -1 4202496 423241 0 0 0 3566371 521 0 0 25 0 1 0 283417258 1132228608 185364 33554432000 4194304 4964704 140735530265632 18446744073709551615 268660705639 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/622/statm: 276423 185364 476 189 0 270772 0
Current children cumulated CPU time (s) 35668.9
Current children cumulated vsize (KiB) 1105692

Child status: 0
Real time (s): 5100.15
CPU time (s): 35669
CPU user time (s): 35663.7
CPU system time (s): 5.2412
CPU usage (%): 699.371
Max. virtual memory (cumulated for all children) (KiB): 1975304

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35663.7
system time used= 5.2412
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 423242
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 386861
involuntary context switches= 324530

runsolver used 14.8737 second user time and 36.0385 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-03 05:11:00
IDJOB=3322656
IDBENCH=85538
IDSOLVER=1711
FILE ID=node107/3322656-1304392260
RUNJOBID= node107-1304390059-495
PBS_JOBID= 13181829
Free space on /tmp= 72996 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT05/crafted/sabharwal05/counting/fphp/unsat/harder/fphp-030-029.shuffled-as.sat05-1234.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322656-1304392260/watcher-3322656-1304392260 -o /tmp/evaluation-result-3322656-1304392260/solver-3322656-1304392260 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322656-1304392260.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c12aa6a0ac1bca448bef51c167f3bbbb
RANDOM SEED=406341010

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:       5258920 kB
Buffers:        317948 kB
Cached:        3153436 kB
SwapCached:          4 kB
Active:       24028544 kB
Inactive:      3268632 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:       5258920 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            1180 kB
Writeback:           0 kB
AnonPages:    23825720 kB
Mapped:          27824 kB
Slab:           266224 kB
PageTables:      65544 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 64181512 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72996 MiB
End job on node107 at 2011-05-03 06:36:02