Trace number 3381372

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40002.1 5020.87

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m62-q1504.5-n60-p15-s80450670.cnf
MD5SUM78d8e2b8c0fd15b93f96e38e33701f17
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.365943
Satisfiable
(Un)Satisfiability was proved
Number of variables3510
Number of clauses974465
Sum of the clauses size1952320
Maximum clause length62
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2974405
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...514180/instance-3381372-1305514180.cnf
10.79/10.80	c Solving...
40000.51/5019.43	
40000.51/5019.43	*** INTERRUPTED! ***
40000.51/5019.43	*** Shutting down threads...
40002.00/5020.82	s UNKNOWN
40002.00/5020.82	
40002.00/5020.82	c INTERRUPTED   : 1
40002.00/5020.82	c Models        : 0+    
40002.00/5020.82	c Time          : 5019.440s (Solving: 5008.64s 1st Model: 0.00s Unsat: 0.00s)
40002.00/5020.82	c CPU Time      : 40000.650s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3381372-1305514180/watcher-3381372-1305514180 -o /tmp/evaluation-result-3381372-1305514180/solver-3381372-1305514180 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381372-1305514180.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.65 7.69 8.16 2/374 1762
/proc/meminfo: memFree=20461208/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=20872 CPUtime=0 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 486 0 0 0 0 0 0 0 24 0 1 0 663017659 21372928 416 33554432000 4194304 4964800 140736943735536 18446744073709551615 210646843936 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/1762/statm: 5218 416 353 189 0 92 0

[startup+0.108785 s]
/proc/loadavg: 6.65 7.69 8.16 2/374 1762
/proc/meminfo: memFree=20461208/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=42312 CPUtime=0.1 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 8140 0 0 0 9 1 0 0 25 0 1 0 663017659 43327488 5373 33554432000 4194304 4964800 140736943735536 18446744073709551615 210646807237 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1762/statm: 10578 5375 386 189 0 5452 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 42312

[startup+0.20077 s]
/proc/loadavg: 6.65 7.69 8.16 2/374 1762
/proc/meminfo: memFree=20461208/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=58112 CPUtime=0.19 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 13621 0 0 0 18 1 0 0 25 0 1 0 663017659 59506688 9486 33554432000 4194304 4964800 140736943735536 18446744073709551615 210646807237 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1762/statm: 14528 9489 386 189 0 9402 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 58112

[startup+0.300753 s]
/proc/loadavg: 6.65 7.69 8.16 2/374 1762
/proc/meminfo: memFree=20461208/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=78916 CPUtime=0.29 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 18759 0 0 0 27 2 0 0 25 0 1 0 663017659 80809984 14591 33554432000 4194304 4964800 140736943735536 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1762/statm: 19729 14591 397 189 0 14603 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 78916

[startup+0.700686 s]
/proc/loadavg: 6.65 7.69 8.16 2/374 1762
/proc/meminfo: memFree=20461208/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=79972 CPUtime=0.69 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 19167 0 0 0 67 2 0 0 25 0 1 0 663017659 81891328 14999 33554432000 4194304 4964800 140736943735536 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1762/statm: 19993 14999 399 189 0 14867 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 79972

[startup+1.50055 s]
/proc/loadavg: 6.65 7.69 8.16 2/375 1763
/proc/meminfo: memFree=20402220/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=79972 CPUtime=1.49 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 19167 0 0 0 147 2 0 0 25 0 1 0 663017659 81891328 14999 33554432000 4194304 4964800 140736943735536 18446744073709551615 4607585 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1762/statm: 19993 14999 399 189 0 14867 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 79972

[startup+3.10028 s]
/proc/loadavg: 6.65 7.69 8.16 2/375 1763
/proc/meminfo: memFree=20401724/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=79972 CPUtime=3.09 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 19167 0 0 0 307 2 0 0 25 0 1 0 663017659 81891328 14999 33554432000 4194304 4964800 140736943735536 18446744073709551615 4607585 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1762/statm: 19993 14999 399 189 0 14867 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 79972

[startup+6.30075 s]
/proc/loadavg: 6.28 7.60 8.12 2/375 1763
/proc/meminfo: memFree=20401860/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=80368 CPUtime=6.29 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 19289 0 0 0 627 2 0 0 25 0 1 0 663017659 82296832 15121 33554432000 4194304 4964800 140736943735536 18446744073709551615 4600360 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/1762/statm: 20092 15121 400 189 0 14966 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 80368

[startup+12.7006 s]
/proc/loadavg: 5.86 7.49 8.08 9/382 1770
/proc/meminfo: memFree=20354524/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=439916 CPUtime=21.04 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 39852 0 0 0 2090 14 0 0 21 0 8 0 663017659 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627443 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1762/statm: 109979 33670 464 189 0 104328 0
[pid=1762/tid=1764] ppid=1760 vsize=439916 CPUtime=1.19 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 1924 0 0 0 119 0 0 0 17 0 8 0 663018740 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=439916 CPUtime=1.19 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 2276 0 0 0 118 1 0 0 17 0 8 0 663018740 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627281 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=439916 CPUtime=1.19 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 2234 0 0 0 117 2 0 0 17 0 8 0 663018740 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4342157 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=439916 CPUtime=1.18 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 2855 0 0 0 117 1 0 0 17 0 8 0 663018740 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=439916 CPUtime=1.85 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 1796 0 0 0 185 0 0 0 20 0 8 0 663018740 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670407 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=439916 CPUtime=1.19 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 2297 0 0 0 119 0 0 0 17 0 8 0 663018740 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4332539 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=439916 CPUtime=1.19 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 2966 0 0 0 117 2 0 0 17 0 8 0 663018740 450473984 33670 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 21.04
Current children cumulated vsize (KiB) 439916

[startup+25.5005 s]
/proc/loadavg: 6.33 7.51 8.08 9/382 1771
/proc/meminfo: memFree=19992356/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=735352 CPUtime=123.1 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 124077 0 0 0 12249 61 0 0 25 0 8 0 663017659 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4334298 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/1762/statm: 183838 117895 466 189 0 178187 0
[pid=1762/tid=1764] ppid=1760 vsize=735352 CPUtime=13.95 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 11599 0 0 0 1390 5 0 0 25 0 8 0 663018740 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=735352 CPUtime=13.97 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 7566 0 0 0 1394 3 0 0 25 0 8 0 663018740 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=735352 CPUtime=13.96 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 10757 0 0 0 1390 6 0 0 25 0 8 0 663018740 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=735352 CPUtime=13.97 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 15199 0 0 0 1388 9 0 0 25 0 8 0 663018740 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320993 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=735352 CPUtime=14.61 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 13706 0 0 0 1454 7 0 0 25 0 8 0 663018740 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=735352 CPUtime=13.82 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 12526 0 0 0 1376 6 0 0 25 0 8 0 663018740 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=735352 CPUtime=13.98 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 15436 0 0 0 1389 9 0 0 25 0 8 0 663018740 753000448 117895 33554432000 4194304 4964800 140736943735536 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 123.1
Current children cumulated vsize (KiB) 735352

[startup+51.1012 s]
/proc/loadavg: 7.05 7.58 8.09 9/382 1772
/proc/meminfo: memFree=19560620/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=1039992 CPUtime=327.44 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 233495 0 0 0 32614 130 0 0 25 0 8 0 663017659 1064951808 227312 33554432000 4194304 4964800 140736943735536 18446744073709551615 4342186 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 259998 227312 466 189 0 254347 0
[pid=1762/tid=1764] ppid=1760 vsize=1039992 CPUtime=39.55 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 25716 0 0 0 3939 16 0 0 25 0 8 0 663018740 1064951808 227312 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=1039992 CPUtime=39.57 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 11346 0 0 0 3952 5 0 0 25 0 8 0 663018740 1064951808 227312 33554432000 4194304 4964800 140736943735536 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=1039992 CPUtime=39.55 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 20713 0 0 0 3943 12 0 0 25 0 8 0 663018740 1064951808 227313 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=1039992 CPUtime=39.56 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 32471 0 0 0 3937 19 0 0 25 0 8 0 663018740 1064951808 227313 33554432000 4194304 4964800 140736943735536 18446744073709551615 4687029 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=1039992 CPUtime=40.21 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 32362 0 0 0 3999 22 0 0 25 0 8 0 663018740 1064951808 227313 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=1039992 CPUtime=39.25 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 22174 0 0 0 3912 13 0 0 25 0 8 0 663018740 1064951808 227313 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=1039992 CPUtime=39.37 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 31134 0 0 0 3919 18 0 0 25 0 8 0 663018740 1064951808 227313 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 327.44
Current children cumulated vsize (KiB) 1039992

[startup+102.301 s]
/proc/loadavg: 7.58 7.64 8.09 9/382 1773
/proc/meminfo: memFree=18965040/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=1813964 CPUtime=736.3 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 380537 0 0 0 73399 231 0 0 25 0 8 0 663017659 1857499136 374272 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670335 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 453491 374272 466 189 0 447840 0
[pid=1762/tid=1764] ppid=1760 vsize=1813964 CPUtime=90.74 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 48103 0 0 0 9042 32 0 0 25 0 8 0 663018740 1857499136 374284 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=1813964 CPUtime=90.78 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 14548 0 0 0 9072 6 0 0 25 0 8 0 663018740 1857499136 374284 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=1813964 CPUtime=90.74 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 33871 0 0 0 9051 23 0 0 25 0 8 0 663018740 1857499136 374284 33554432000 4194304 4964800 140736943735536 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=1813964 CPUtime=90.76 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 54450 0 0 0 9040 36 0 0 25 0 8 0 663018740 1857499136 374284 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=1813964 CPUtime=91.4 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 43729 0 0 0 9111 29 0 0 25 0 8 0 663018740 1857499136 374284 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=1813964 CPUtime=90.43 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 40833 0 0 0 9019 24 0 0 25 0 8 0 663018740 1857499136 374284 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=1813964 CPUtime=89.91 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 58580 0 0 0 8953 38 0 0 25 0 8 0 663018740 1857499136 374284 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 736.3
Current children cumulated vsize (KiB) 1813964

[startup+162.3 s]
/proc/loadavg: 7.96 7.74 8.09 9/382 1774
/proc/meminfo: memFree=18412872/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=2285652 CPUtime=1215.37 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 518179 0 0 0 121201 336 0 0 25 0 8 0 663017659 2340507648 511850 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 571413 511850 466 189 0 565762 0
[pid=1762/tid=1764] ppid=1760 vsize=2285652 CPUtime=150.7 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 78762 0 0 0 15020 50 0 0 25 0 8 0 663018740 2340507648 511850 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=2285652 CPUtime=150.76 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 18885 0 0 0 15067 9 0 0 25 0 8 0 663018740 2340507648 511850 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320905 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=2285652 CPUtime=150.67 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 53438 0 0 0 15026 41 0 0 25 0 8 0 663018740 2340507648 511850 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=2285652 CPUtime=150.73 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 71171 0 0 0 15025 48 0 0 25 0 8 0 663018740 2340507648 511850 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=2285652 CPUtime=151.38 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 53916 0 0 0 15098 40 0 0 25 0 8 0 663018740 2340507648 511850 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=2285652 CPUtime=150.39 cores=0-7

################
# More data... #
################

[pid=1762/tid=1767] ppid=1760 vsize=8283432 CPUtime=4695.81 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 186364 0 0 0 469397 184 0 0 25 0 8 0 663018740 8482234368 1992152 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=8283432 CPUtime=4704.03 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 214574 0 0 0 470126 277 0 0 25 0 8 0 663018740 8482234368 1992152 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=8283432 CPUtime=4696.95 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 208035 0 0 0 469447 248 0 0 25 0 8 0 663018740 8482234368 1992152 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=8283432 CPUtime=4696.05 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 399633 0 0 0 469140 465 0 0 25 0 8 0 663018740 8482234368 1992152 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 37627.8
Current children cumulated vsize (KiB) 8283432

[startup+4782.3 s]
/proc/loadavg: 8.10 8.09 8.02 9/382 1922
/proc/meminfo: memFree=12470652/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=8288144 CPUtime=38106.8 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2007936 0 0 0 3808488 2196 0 0 25 0 8 0 663017659 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 2072036 1994502 466 189 0 2066385 0
[pid=1762/tid=1764] ppid=1760 vsize=8288144 CPUtime=4755.47 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 389227 0 0 0 475081 466 0 0 25 0 8 0 663018740 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=8288144 CPUtime=4769.02 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 80462 0 0 0 476808 94 0 0 25 0 8 0 663018740 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=8288144 CPUtime=4769.44 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 185768 0 0 0 476759 185 0 0 25 0 8 0 663018740 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=8288144 CPUtime=4755.04 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 186371 0 0 0 475320 184 0 0 25 0 8 0 663018740 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=8288144 CPUtime=4764.01 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 214847 0 0 0 476124 277 0 0 25 0 8 0 663018740 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=8288144 CPUtime=4756.89 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 208071 0 0 0 475440 249 0 0 25 0 8 0 663018740 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=8288144 CPUtime=4756.04 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 400054 0 0 0 475138 466 0 0 25 0 8 0 663018740 8487059456 1994502 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38106.8
Current children cumulated vsize (KiB) 8288144

[startup+4842.31 s]
/proc/loadavg: 8.07 8.08 8.02 9/382 1923
/proc/meminfo: memFree=12466436/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=8289236 CPUtime=38586.1 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2009043 0 0 0 3856415 2198 0 0 25 0 8 0 663017659 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 2072309 1995609 466 189 0 2066658 0
[pid=1762/tid=1764] ppid=1760 vsize=8289236 CPUtime=4815.49 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 389230 0 0 0 481083 466 0 0 25 0 8 0 663018740 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=8289236 CPUtime=4829.03 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 80513 0 0 0 482809 94 0 0 25 0 8 0 663018740 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=8289236 CPUtime=4829.44 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 186041 0 0 0 482759 185 0 0 25 0 8 0 663018740 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=8289236 CPUtime=4814.29 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 186386 0 0 0 481245 184 0 0 25 0 8 0 663018740 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=8289236 CPUtime=4824.01 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 214942 0 0 0 482124 277 0 0 25 0 8 0 663018740 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=8289236 CPUtime=4816.87 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 208289 0 0 0 481437 250 0 0 25 0 8 0 663018740 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=8289236 CPUtime=4816.06 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 400500 0 0 0 481139 467 0 0 25 0 8 0 663018740 8488177664 1995609 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627269 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38586.1
Current children cumulated vsize (KiB) 8289236

[startup+4902.3 s]
/proc/loadavg: 8.07 8.08 8.02 9/382 1925
/proc/meminfo: memFree=12391304/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=8428640 CPUtime=39065.2 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2028154 0 0 0 3904298 2223 0 0 25 0 8 0 663017659 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320955 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 2107160 2014720 466 189 0 2101509 0
[pid=1762/tid=1764] ppid=1760 vsize=8428640 CPUtime=4875.47 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 390389 0 0 0 487080 467 0 0 25 0 8 0 663018740 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=8428640 CPUtime=4889.02 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 80526 0 0 0 488808 94 0 0 25 0 8 0 663018740 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=8428640 CPUtime=4889.42 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 186093 0 0 0 488757 185 0 0 25 0 8 0 663018740 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=8428640 CPUtime=4873.52 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 187817 0 0 0 487168 184 0 0 25 0 8 0 663018740 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=8428640 CPUtime=4883.99 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 215705 0 0 0 488121 278 0 0 25 0 8 0 663018740 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=8428640 CPUtime=4876.82 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 208500 0 0 0 487432 250 0 0 25 0 8 0 663018740 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=8428640 CPUtime=4876.04 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 415979 0 0 0 487113 491 0 0 25 0 8 0 663018740 8630927360 2014720 33554432000 4194304 4964800 140736943735536 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39065.2
Current children cumulated vsize (KiB) 8428640

[startup+4962.3 s]
/proc/loadavg: 8.06 8.08 8.02 9/382 1926
/proc/meminfo: memFree=12318896/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=8559712 CPUtime=39544.4 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2045874 0 0 0 3952183 2256 0 0 25 0 8 0 663017659 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 2139928 2032440 466 189 0 2134277 0
[pid=1762/tid=1764] ppid=1760 vsize=8559712 CPUtime=4935.47 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 398766 0 0 0 493063 484 0 0 25 0 8 0 663018740 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=8559712 CPUtime=4949.02 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 80537 0 0 0 494808 94 0 0 25 0 8 0 663018740 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=8559712 CPUtime=4949.41 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 190177 0 0 0 494751 190 0 0 25 0 8 0 663018740 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=8559712 CPUtime=4932.75 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 187880 0 0 0 493091 184 0 0 25 0 8 0 663018740 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=8559712 CPUtime=4943.98 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 215949 0 0 0 494120 278 0 0 25 0 8 0 663018740 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=8559712 CPUtime=4936.79 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 208761 0 0 0 493428 251 0 0 25 0 8 0 663018740 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=8559712 CPUtime=4936.04 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 420659 0 0 0 493104 500 0 0 25 0 8 0 663018740 8765145088 2032440 33554432000 4194304 4964800 140736943735536 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39544.4
Current children cumulated vsize (KiB) 8559712



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5019.42 s]
/proc/loadavg: 8.06 8.08 8.02 9/382 1928
/proc/meminfo: memFree=12286292/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=8562656 CPUtime=40000.5 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2053927 0 0 0 3997783 2268 0 0 25 0 8 0 663017659 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/1762/statm: 2140664 2040493 466 189 0 2135013 0
[pid=1762/tid=1764] ppid=1760 vsize=8562656 CPUtime=4992.6 cores=0-7
/proc/1762/task/1764/stat : 1764 (claspmt) R 1760 1762 1641 0 -1 4202560 401645 0 0 0 498770 490 0 0 25 0 8 0 663018740 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=1762/tid=1765] ppid=1760 vsize=8562656 CPUtime=5006.15 cores=0-7
/proc/1762/task/1765/stat : 1765 (claspmt) R 1760 1762 1641 0 -1 4202560 80595 0 0 0 500521 94 0 0 25 0 8 0 663018740 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=1762/tid=1766] ppid=1760 vsize=8562656 CPUtime=5006.51 cores=0-7
/proc/1762/task/1766/stat : 1766 (claspmt) R 1760 1762 1641 0 -1 4202560 193365 0 0 0 500456 195 0 0 25 0 8 0 663018740 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=1762/tid=1767] ppid=1760 vsize=8562656 CPUtime=4989.14 cores=0-7
/proc/1762/task/1767/stat : 1767 (claspmt) R 1760 1762 1641 0 -1 4202560 187933 0 0 0 498729 185 0 0 25 0 8 0 663018740 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=1762/tid=1768] ppid=1760 vsize=8562656 CPUtime=5001.09 cores=0-7
/proc/1762/task/1768/stat : 1768 (claspmt) R 1760 1762 1641 0 -1 4202560 216193 0 0 0 499831 278 0 0 25 0 8 0 663018740 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=1762/tid=1769] ppid=1760 vsize=8562656 CPUtime=4993.89 cores=0-7
/proc/1762/task/1769/stat : 1769 (claspmt) R 1760 1762 1641 0 -1 4202560 209890 0 0 0 499136 253 0 0 25 0 8 0 663018740 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=1762/tid=1770] ppid=1760 vsize=8562656 CPUtime=4993.17 cores=0-7
/proc/1762/task/1770/stat : 1770 (claspmt) R 1760 1762 1641 0 -1 4202560 421158 0 0 0 498816 501 0 0 25 0 8 0 663018740 8768159744 2040493 33554432000 4194304 4964800 140736943735536 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 8562656

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5019.9 s]
/proc/loadavg: 8.06 8.08 8.02 9/382 1928
/proc/meminfo: memFree=12286292/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=8496680 CPUtime=40001.1 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2053937 0 0 0 3997842 2269 0 0 25 0 1 0 663017659 8700600320 2033728 33554432000 4194304 4964800 140736943735536 18446744073709551615 210646803438 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/1762/statm: 2124170 2033728 476 189 0 2118519 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 8496680

[startup+5020.3 s]
/proc/loadavg: 8.06 8.08 8.02 9/382 1928
/proc/meminfo: memFree=12286292/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=8431144 CPUtime=40001.5 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2053937 0 0 0 3997882 2269 0 0 25 0 1 0 663017659 8633491456 2025812 33554432000 4194304 4964800 140736943735536 18446744073709551615 4646607 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/1762/statm: 2107786 2025812 476 189 0 2102135 0
Current children cumulated CPU time (s) 40001.5
Current children cumulated vsize (KiB) 8431144

[startup+5020.7 s]
/proc/loadavg: 8.06 8.08 8.02 2/376 1929
/proc/meminfo: memFree=12383308/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=4421144 CPUtime=40001.9 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2053937 0 0 0 3997904 2286 0 0 25 0 1 0 663017659 4527251456 988875 33554432000 4194304 4964800 140736943735536 18446744073709551615 210647189639 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/1762/statm: 1105286 988875 476 189 0 1099635 0
Current children cumulated CPU time (s) 40001.9
Current children cumulated vsize (KiB) 4421144

[startup+5020.8 s]
/proc/loadavg: 8.06 8.08 8.02 2/376 1929
/proc/meminfo: memFree=12383308/32951132 swapFree=64495988/67111528
[pid=1762] ppid=1760 vsize=2061848 CPUtime=40002 cores=0-7
/proc/1762/stat : 1762 (claspmt) R 1760 1762 1641 0 -1 4202496 2053937 0 0 0 3997904 2296 0 0 25 0 1 0 663017659 2111332352 412487 33554432000 4194304 4964800 140736943735536 18446744073709551615 210647189863 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/1762/statm: 515462 411975 476 189 0 509811 0
Current children cumulated CPU time (s) 40002
Current children cumulated vsize (KiB) 2061848

Child status: 0
Real time (s): 5020.87
CPU time (s): 40002.1
CPU user time (s): 39979
CPU system time (s): 23.0435
CPU usage (%): 796.716
Max. virtual memory (cumulated for all children) (KiB): 8562656

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39979
system time used= 23.0435
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2053938
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 76802
involuntary context switches= 135656

runsolver used 17.9443 second user time and 45.956 second system time

The end

Launcher Data 

Begin job on node129 at 2011-05-16 04:49:40
IDJOB=3381372
IDBENCH=88208
IDSOLVER=1865
FILE ID=node129/3381372-1305514180
RUNJOBID= node129-1305514077-1655
PBS_JOBID= 13325555
Free space on /tmp= 73400 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m62-q1504.5-n60-p15-s80450670.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3381372-1305514180/watcher-3381372-1305514180 -o /tmp/evaluation-result-3381372-1305514180/solver-3381372-1305514180 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3381372-1305514180.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 78d8e2b8c0fd15b93f96e38e33701f17
RANDOM SEED=638929293

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      20461488 kB
Buffers:        284732 kB
Cached:        6563740 kB
SwapCached:      52676 kB
Active:        3599232 kB
Inactive:      8646352 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      20461488 kB
SwapTotal:    67111528 kB
SwapFree:     64495988 kB
Dirty:           13628 kB
Writeback:          16 kB
AnonPages:     5353572 kB
Mapped:          20328 kB
Slab:           136256 kB
PageTables:      40464 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 93358660 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73400 MiB
End job on node129 at 2011-05-16 06:13:22