Trace number 3472342

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 160005 5022.87

General information on the benchmark

NameAPPLICATIONS/satComp09_BioInstances/
ndhf_xits_19_UNKNOWN.cnf
MD5SUM4cf2a693df1b6e5a084b88e84ec5f2af
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark162.569
Satisfiable
(Un)Satisfiability was proved
Number of variables4020
Number of clauses466486
Sum of the clauses size1393798
Maximum clause length20
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 26187
Number of clauses of size 3460268
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 531

Solver Data 

0.00/0.01	c clasp version 2.0.0-RC2
0.00/0.01	c Reading from ...493138/instance-3472342-1307493138.cnf
12.99/13.05	c Solving...
160002.12/5020.03	
160002.12/5020.03	*** INTERRUPTED! ***
160002.12/5020.03	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472342-1307493138/watcher-3472342-1307493138 -o /tmp/evaluation-result-3472342-1307493138/solver-3472342-1307493138 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472342-1307493138.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 29.01 31.38 31.77 1/413 18975
/proc/meminfo: memFree=263119160/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=23592 CPUtime=0.03 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 1293 0 0 0 3 0 0 0 21 0 1 0 14700767 24158208 1101 270582939648 4194304 4964800 140737356936576 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 5898 1102 383 189 0 772 0

[startup+0.0466749 s]
/proc/loadavg: 29.01 31.38 31.77 1/413 18975
/proc/meminfo: memFree=263119160/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=24544 CPUtime=0.04 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 1605 0 0 0 4 0 0 0 21 0 1 0 14700767 25133056 1332 270582939648 4194304 4964800 140737356936576 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 6136 1333 383 189 0 1010 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 24544

[startup+0.100725 s]
/proc/loadavg: 29.01 31.38 31.77 1/413 18975
/proc/meminfo: memFree=263119160/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=28972 CPUtime=0.09 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 3013 0 0 0 9 0 0 0 21 0 1 0 14700767 29667328 2438 270582939648 4194304 4964800 140737356936576 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 7243 2439 383 189 0 2117 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28972

[startup+0.300686 s]
/proc/loadavg: 29.01 31.38 31.77 1/413 18975
/proc/meminfo: memFree=263119160/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=45660 CPUtime=0.29 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 8421 0 0 0 26 3 0 0 22 0 1 0 14700767 46755840 6561 270582939648 4194304 4964800 140737356936576 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 11415 6562 383 189 0 6289 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 45660

[startup+0.700701 s]
/proc/loadavg: 29.01 31.38 31.77 1/413 18975
/proc/meminfo: memFree=263119160/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=58436 CPUtime=0.69 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 11752 0 0 0 65 4 0 0 23 0 1 0 14700767 59838464 9854 270582939648 4194304 4964800 140737356936576 18446744073709551615 4607486 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 14609 9854 396 189 0 9483 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 58436

[startup+1.50079 s]
/proc/loadavg: 29.01 31.38 31.77 2/414 18976
/proc/meminfo: memFree=263080696/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=58436 CPUtime=1.49 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 11752 0 0 0 145 4 0 0 25 0 1 0 14700767 59838464 9854 270582939648 4194304 4964800 140737356936576 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 14609 9854 396 189 0 9483 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 58436

[startup+3.10081 s]
/proc/loadavg: 29.01 31.38 31.77 2/414 18976
/proc/meminfo: memFree=263080716/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=58436 CPUtime=3.09 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 11752 0 0 0 305 4 0 0 25 0 1 0 14700767 59838464 9854 270582939648 4194304 4964800 140737356936576 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 14609 9854 396 189 0 9483 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 58436

[startup+6.30092 s]
/proc/loadavg: 26.77 30.88 31.60 2/414 18978
/proc/meminfo: memFree=263078328/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=62308 CPUtime=6.29 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 12718 0 0 0 625 4 0 0 25 0 1 0 14700767 63803392 10820 270582939648 4194304 4964800 140737356936576 18446744073709551615 4600360 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 15577 10820 397 189 0 10451 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 62308

[startup+12.7011 s]
/proc/loadavg: 24.70 30.38 31.44 2/414 18978
/proc/meminfo: memFree=263075192/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=67324 CPUtime=12.69 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 13939 0 0 0 1265 4 0 0 25 0 1 0 14700767 68939776 12041 270582939648 4194304 4964800 140737356936576 18446744073709551615 4608127 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/18975/statm: 16831 12041 397 189 0 11705 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 67324

[startup+25.5025 s]
/proc/loadavg: 24.22 29.96 31.28 33/445 19009
/proc/meminfo: memFree=262336792/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=1679752 CPUtime=384.21 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 204021 0 0 0 37812 609 0 0 18 0 32 0 14700767 1720066048 200747 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668712 0 0 4096 18946 18446744073709551615 0 0 17 24 0 0 0
/proc/18975/statm: 419938 200748 463 189 0 414287 0
[pid=18975/tid=18979] ppid=18973 vsize=1679752 CPUtime=11.5 cores=0-31
/proc/18975/task/18979/stat : 18979 (claspmt) R 18973 18975 18891 0 -1 4202560 5301 0 0 0 1132 18 0 0 18 0 32 0 14702073 1720066048 200748 270582939648 4194304 4964800 140737356936576 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=18975/tid=18980] ppid=18973 vsize=1679752 CPUtime=11.55 cores=0-31
/proc/18975/task/18980/stat : 18980 (claspmt) R 18973 18975 18891 0 -1 4202560 8240 0 0 0 1130 25 0 0 18 0 32 0 14702073 1720066048 200748 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=18975/tid=18981] ppid=18973 vsize=1679752 CPUtime=11.59 cores=0-31
/proc/18975/task/18981/stat : 18981 (claspmt) R 18973 18975 18891 0 -1 4202560 5528 0 0 0 1144 15 0 0 20 0 32 0 14702073 1720066048 200748 270582939648 4194304 4964800 140737356936576 18446744073709551615 4660165 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=18975/tid=18982] ppid=18973 vsize=1679752 CPUtime=11.56 cores=0-31
/proc/18975/task/18982/stat : 18982 (claspmt) R 18973 18975 18891 0 -1 4202560 5869 0 0 0 1136 20 0 0 18 0 32 0 14702073 1720066048 200748 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=18975/tid=18983] ppid=18973 vsize=1679752 CPUtime=12.06 cores=0-31
/proc/18975/task/18983/stat : 18983 (claspmt) R 18973 18975 18891 0 -1 4202560 5076 0 0 0 1192 14 0 0 18 0 32 0 14702073 1720066048 200748 270582939648 4194304 4964800 140737356936576 18446744073709551615 4627269 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=18975/tid=18984] ppid=18973 vsize=1679752 CPUtime=11.57 cores=0-31
/proc/18975/task/18984/stat : 18984 (claspmt) R 18973 18975 18891 0 -1 4202560 5521 0 0 0 1141 16 0 0 19 0 32 0 14702073 1720066048 200748 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670368 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=18975/tid=18985] ppid=18973 vsize=1679752 CPUtime=11.49 cores=0-31
/proc/18975/task/18985/stat : 18985 (claspmt) R 18973 18975 18891 0 -1 4202560 6350 0 0 0 1128 21 0 0 20 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=18975/tid=18986] ppid=18973 vsize=1679752 CPUtime=11.53 cores=0-31
/proc/18975/task/18986/stat : 18986 (claspmt) R 18973 18975 18891 0 -1 4202560 5510 0 0 0 1134 19 0 0 18 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=18975/tid=18987] ppid=18973 vsize=1679752 CPUtime=11.54 cores=0-31
/proc/18975/task/18987/stat : 18987 (claspmt) R 18973 18975 18891 0 -1 4202560 5033 0 0 0 1138 16 0 0 19 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4627293 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=18975/tid=18988] ppid=18973 vsize=1679752 CPUtime=11.46 cores=0-31
/proc/18975/task/18988/stat : 18988 (claspmt) R 18973 18975 18891 0 -1 4202560 7974 0 0 0 1122 24 0 0 20 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=18975/tid=18989] ppid=18973 vsize=1679752 CPUtime=11.54 cores=0-31
/proc/18975/task/18989/stat : 18989 (claspmt) R 18973 18975 18891 0 -1 4202560 5550 0 0 0 1138 16 0 0 18 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=18975/tid=18990] ppid=18973 vsize=1679752 CPUtime=11.58 cores=0-31
/proc/18975/task/18990/stat : 18990 (claspmt) R 18973 18975 18891 0 -1 4202560 5462 0 0 0 1138 20 0 0 18 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=18975/tid=18991] ppid=18973 vsize=1679752 CPUtime=12.08 cores=0-31
/proc/18975/task/18991/stat : 18991 (claspmt) R 18973 18975 18891 0 -1 4202560 4991 0 0 0 1195 13 0 0 18 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=18975/tid=18992] ppid=18973 vsize=1679752 CPUtime=11.59 cores=0-31
/proc/18975/task/18992/stat : 18992 (claspmt) R 18973 18975 18891 0 -1 4202560 5638 0 0 0 1141 18 0 0 18 0 32 0 14702073 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=18975/tid=18993] ppid=18973 vsize=1679752 CPUtime=11.51 cores=0-31
/proc/18975/task/18993/stat : 18993 (claspmt) R 18973 18975 18891 0 -1 4202560 6218 0 0 0 1130 21 0 0 18 0 32 0 14702074 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=18975/tid=18994] ppid=18973 vsize=1679752 CPUtime=11.46 cores=0-31
/proc/18975/task/18994/stat : 18994 (claspmt) R 18973 18975 18891 0 -1 4202560 5396 0 0 0 1126 20 0 0 22 0 32 0 14702074 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668698 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=18975/tid=18995] ppid=18973 vsize=1679752 CPUtime=11.58 cores=0-31
/proc/18975/task/18995/stat : 18995 (claspmt) R 18973 18975 18891 0 -1 4202560 5267 0 0 0 1141 17 0 0 18 0 32 0 14702074 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4653161 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=18975/tid=18996] ppid=18973 vsize=1679752 CPUtime=11.47 cores=0-31
/proc/18975/task/18996/stat : 18996 (claspmt) R 18973 18975 18891 0 -1 4202560 8206 0 0 0 1121 26 0 0 18 0 32 0 14702074 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=18975/tid=18997] ppid=18973 vsize=1679752 CPUtime=11.58 cores=0-31
/proc/18975/task/18997/stat : 18997 (claspmt) R 18973 18975 18891 0 -1 4202560 5780 0 0 0 1140 18 0 0 20 0 32 0 14702074 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=18975/tid=18998] ppid=18973 vsize=1679752 CPUtime=11.58 cores=0-31
/proc/18975/task/18998/stat : 18998 (claspmt) R 18973 18975 18891 0 -1 4202560 5343 0 0 0 1141 17 0 0 18 0 32 0 14702074 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4653222 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=18975/tid=18999] ppid=18973 vsize=1679752 CPUtime=12 cores=0-31
/proc/18975/task/18999/stat : 18999 (claspmt) R 18973 18975 18891 0 -1 4202560 5616 0 0 0 1184 16 0 0 18 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668732 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=18975/tid=19000] ppid=18973 vsize=1679752 CPUtime=11.57 cores=0-31
/proc/18975/task/19000/stat : 19000 (claspmt) R 18973 18975 18891 0 -1 4202560 5178 0 0 0 1140 17 0 0 18 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=18975/tid=19001] ppid=18973 vsize=1679752 CPUtime=11.45 cores=0-31
/proc/18975/task/19001/stat : 19001 (claspmt) R 18973 18975 18891 0 -1 4202560 6410 0 0 0 1125 20 0 0 18 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=18975/tid=19002] ppid=18973 vsize=1679752 CPUtime=11.47 cores=0-31
/proc/18975/task/19002/stat : 19002 (claspmt) R 18973 18975 18891 0 -1 4202560 5092 0 0 0 1130 17 0 0 18 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=18975/tid=19003] ppid=18973 vsize=1679752 CPUtime=11.54 cores=0-31
/proc/18975/task/19003/stat : 19003 (claspmt) R 18973 18975 18891 0 -1 4202560 5155 0 0 0 1139 15 0 0 21 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=18975/tid=19004] ppid=18973 vsize=1679752 CPUtime=11.43 cores=0-31
/proc/18975/task/19004/stat : 19004 (claspmt) R 18973 18975 18891 0 -1 4202560 8074 0 0 0 1118 25 0 0 18 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4363438 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=18975/tid=19005] ppid=18973 vsize=1679752 CPUtime=11.56 cores=0-31
/proc/18975/task/19005/stat : 19005 (claspmt) R 18973 18975 18891 0 -1 4202560 5754 0 0 0 1141 15 0 0 24 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=18975/tid=19006] ppid=18973 vsize=1679752 CPUtime=11.54 cores=0-31
/proc/18975/task/19006/stat : 19006 (claspmt) R 18973 18975 18891 0 -1 4202560 6039 0 0 0 1136 18 0 0 18 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=18975/tid=19007] ppid=18973 vsize=1679752 CPUtime=11.86 cores=0-31
/proc/18975/task/19007/stat : 19007 (claspmt) R 18973 18975 18891 0 -1 4202560 5755 0 0 0 1172 14 0 0 20 0 32 0 14702075 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4653172 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=18975/tid=19008] ppid=18973 vsize=1679752 CPUtime=11.56 cores=0-31
/proc/18975/task/19008/stat : 19008 (claspmt) R 18973 18975 18891 0 -1 4202560 5463 0 0 0 1140 16 0 0 18 0 32 0 14702076 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=18975/tid=19009] ppid=18973 vsize=1679752 CPUtime=11.52 cores=0-31
/proc/18975/task/19009/stat : 19009 (claspmt) R 18973 18975 18891 0 -1 4202560 6292 0 0 0 1130 22 0 0 19 0 32 0 14702076 1720066048 200749 270582939648 4194304 4964800 140737356936576 18446744073709551615 4653199 0 0 4096 18946 0 0 0 -1 8 0 0 0
Current children cumulated CPU time (s) 384.21
Current children cumulated vsize (KiB) 1679752

[startup+51.1024 s]
/proc/loadavg: 26.87 30.13 31.30 33/445 19009
/proc/meminfo: memFree=261340812/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=2979992 CPUtime=1196.92 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 454666 0 0 0 118389 1303 0 0 18 0 32 0 14700767 3051511808 451389 270582939648 4194304 4964800 140737356936576 18446744073709551615 258650623117 0 0 4096 18946 18446744073709551615 0 0 17 24 0 0 0
/proc/18975/statm: 744998 451389 463 189 0 739347 0
[pid=18975/tid=18979] ppid=18973 vsize=2979992 CPUtime=36.93 cores=0-31
/proc/18975/task/18979/stat : 18979 (claspmt) R 18973 18975 18891 0 -1 4202560 12045 0 0 0 3656 37 0 0 18 0 32 0 14702073 3051511808 451389 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=18975/tid=18980] ppid=18973 vsize=2979992 CPUtime=36.85 cores=0-31
/proc/18975/task/18980/stat : 18980 (claspmt) R 18973 18975 18891 0 -1 4202560 20489 0 0 0 3628 57 0 0 18 0 32 0 14702073 3051511808 451390 270582939648 4194304 4964800 140737356936576 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=18975/tid=18981] ppid=18973 vsize=2979992 CPUtime=37.08 cores=0-31
/proc/18975/task/18981/stat : 18981 (claspmt) R 18973 18975 18891 0 -1 4202560 11533 0 0 0 3682 26 0 0 21 0 32 0 14702073 3051511808 451390 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=18975/tid=18982] ppid=18973 vsize=2979992 CPUtime=36.98 cores=0-31
/proc/18975/task/18982/stat : 18982 (claspmt) R 18973 18975 18891 0 -1 4202560 12281 0 0 0 3661 37 0 0 20 0 32 0 14702073 3051511808 451390 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668725 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=18975/tid=18983] ppid=18973 vsize=2979992 CPUtime=37.48 cores=0-31
/proc/18975/task/18983/stat : 18983 (claspmt) R 18973 18975 18891 0 -1 4202560 13450 0 0 0 3712 36 0 0 25 0 32 0 14702073 3051511808 451392 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=18975/tid=18984] ppid=18973 vsize=2979992 CPUtime=37.05 cores=0-31
/proc/18975/task/18984/stat : 18984 (claspmt) R 18973 18975 18891 0 -1 4202560 11642 0 0 0 3674 31 0 0 20 0 32 0 14702073 3051511808 451392 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=18975/tid=18985] ppid=18973 vsize=2979992 CPUtime=36.89 cores=0-31
/proc/18975/task/18985/stat : 18985 (claspmt) R 18973 18975 18891 0 -1 4202560 14168 0 0 0 3643 46 0 0 22 0 32 0 14702073 3051511808 451392 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=18975/tid=18986] ppid=18973 vsize=2979992 CPUtime=36.69 cores=0-31
/proc/18975/task/18986/stat : 18986 (claspmt) R 18973 18975 18891 0 -1 4202560 13273 0 0 0 3626 43 0 0 20 0 32 0 14702073 3051511808 451392 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=18975/tid=18987] ppid=18973 vsize=2979992 CPUtime=36.97 cores=0-31
/proc/18975/task/18987/stat : 18987 (claspmt) R 18973 18975 18891 0 -1 4202560 11896 0 0 0 3656 41 0 0 20 0 32 0 14702073 3051511808 451392 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=18975/tid=18988] ppid=18973 vsize=2979992 CPUtime=36.81 cores=0-31
/proc/18975/task/18988/stat : 18988 (claspmt) R 18973 18975 18891 0 -1 4202560 20407 0 0 0 3623 58 0 0 18 0 32 0 14702073 3051511808 451392 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=18975/tid=18989] ppid=18973 vsize=2979992 CPUtime=37.01 cores=0-31
/proc/18975/task/18989/stat : 18989 (claspmt) R 18973 18975 18891 0 -1 4202560 11744 0 0 0 3672 29 0 0 20 0 32 0 14702073 3051511808 451393 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=18975/tid=18990] ppid=18973 vsize=2979992 CPUtime=36.97 cores=0-31

################
# More data... #
################

/proc/18975/task/18981/stat : 18981 (claspmt) R 18973 18975 18891 0 -1 4202560 82818 0 0 0 494527 155 0 0 25 0 32 0 14702073 17254412288 3921145 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=18975/tid=18982] ppid=18973 vsize=16850012 CPUtime=4941.25 cores=0-31
/proc/18975/task/18982/stat : 18982 (claspmt) R 18973 18975 18891 0 -1 4202560 104145 0 0 0 493836 289 0 0 25 0 32 0 14702073 17254412288 3921145 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=18975/tid=18983] ppid=18973 vsize=16850012 CPUtime=4937.51 cores=0-31
/proc/18975/task/18983/stat : 18983 (claspmt) R 18973 18975 18891 0 -1 4202560 90166 0 0 0 493515 236 0 0 25 0 32 0 14702073 17254412288 3921145 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=18975/tid=18984] ppid=18973 vsize=16850012 CPUtime=4946.72 cores=0-31
/proc/18975/task/18984/stat : 18984 (claspmt) R 18973 18975 18891 0 -1 4202560 95547 0 0 0 494438 234 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=18975/tid=18985] ppid=18973 vsize=16850012 CPUtime=4918.18 cores=0-31
/proc/18975/task/18985/stat : 18985 (claspmt) R 18973 18975 18891 0 -1 4202560 170776 0 0 0 491188 630 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668725 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=18975/tid=18986] ppid=18973 vsize=16850012 CPUtime=4943.24 cores=0-31
/proc/18975/task/18986/stat : 18986 (claspmt) R 18973 18975 18891 0 -1 4202560 242837 0 0 0 493409 915 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4653547 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=18975/tid=18987] ppid=18973 vsize=16850012 CPUtime=4923.63 cores=0-31
/proc/18975/task/18987/stat : 18987 (claspmt) R 18973 18975 18891 0 -1 4202560 107500 0 0 0 492058 305 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=18975/tid=18988] ppid=18973 vsize=16850012 CPUtime=4945.26 cores=0-31
/proc/18975/task/18988/stat : 18988 (claspmt) R 18973 18975 18891 0 -1 4202560 115542 0 0 0 494235 291 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=18975/tid=18989] ppid=18973 vsize=16850012 CPUtime=4946.73 cores=0-31
/proc/18975/task/18989/stat : 18989 (claspmt) R 18973 18975 18891 0 -1 4202560 81566 0 0 0 494489 184 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=18975/tid=18990] ppid=18973 vsize=16850012 CPUtime=4946.55 cores=0-31
/proc/18975/task/18990/stat : 18990 (claspmt) R 18973 18975 18891 0 -1 4202560 90311 0 0 0 494421 234 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=18975/tid=18991] ppid=18973 vsize=16850012 CPUtime=4947.13 cores=0-31
/proc/18975/task/18991/stat : 18991 (claspmt) R 18973 18975 18891 0 -1 4202560 84331 0 0 0 494506 207 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=18975/tid=18992] ppid=18973 vsize=16850012 CPUtime=4946.62 cores=0-31
/proc/18975/task/18992/stat : 18992 (claspmt) R 18973 18975 18891 0 -1 4202560 80639 0 0 0 494476 186 0 0 25 0 32 0 14702073 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=18975/tid=18993] ppid=18973 vsize=16850012 CPUtime=4946.06 cores=0-31
/proc/18975/task/18993/stat : 18993 (claspmt) R 18973 18975 18891 0 -1 4202560 173813 0 0 0 493961 645 0 0 25 0 32 0 14702074 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=18975/tid=18994] ppid=18973 vsize=16850012 CPUtime=4935.74 cores=0-31
/proc/18975/task/18994/stat : 18994 (claspmt) R 18973 18975 18891 0 -1 4202560 291773 0 0 0 492460 1114 0 0 25 0 32 0 14702074 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=18975/tid=18995] ppid=18973 vsize=16850012 CPUtime=4942.51 cores=0-31
/proc/18975/task/18995/stat : 18995 (claspmt) R 18973 18975 18891 0 -1 4202560 120940 0 0 0 493888 363 0 0 25 0 32 0 14702074 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=18975/tid=18996] ppid=18973 vsize=16850012 CPUtime=4946.82 cores=0-31
/proc/18975/task/18996/stat : 18996 (claspmt) R 18973 18975 18891 0 -1 4202560 109088 0 0 0 494448 234 0 0 25 0 32 0 14702074 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=18975/tid=18997] ppid=18973 vsize=16850012 CPUtime=4946.74 cores=0-31
/proc/18975/task/18997/stat : 18997 (claspmt) R 18973 18975 18891 0 -1 4202560 86631 0 0 0 494481 193 0 0 25 0 32 0 14702074 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=18975/tid=18998] ppid=18973 vsize=16850012 CPUtime=4942.57 cores=0-31
/proc/18975/task/18998/stat : 18998 (claspmt) R 18973 18975 18891 0 -1 4202560 108310 0 0 0 493969 288 0 0 25 0 32 0 14702074 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=18975/tid=18999] ppid=18973 vsize=16850012 CPUtime=4943.04 cores=0-31
/proc/18975/task/18999/stat : 18999 (claspmt) R 18973 18975 18891 0 -1 4202560 99293 0 0 0 494040 264 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=18975/tid=19000] ppid=18973 vsize=16850012 CPUtime=4946.61 cores=0-31
/proc/18975/task/19000/stat : 19000 (claspmt) R 18973 18975 18891 0 -1 4202560 91174 0 0 0 494472 189 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=18975/tid=19001] ppid=18973 vsize=16850012 CPUtime=4927.88 cores=0-31
/proc/18975/task/19001/stat : 19001 (claspmt) R 18973 18975 18891 0 -1 4202560 158851 0 0 0 492219 569 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=18975/tid=19002] ppid=18973 vsize=16850012 CPUtime=4922.02 cores=0-31
/proc/18975/task/19002/stat : 19002 (claspmt) R 18973 18975 18891 0 -1 4202560 245596 0 0 0 491304 898 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4659391 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=18975/tid=19003] ppid=18973 vsize=16850012 CPUtime=4946.35 cores=0-31
/proc/18975/task/19003/stat : 19003 (claspmt) R 18973 18975 18891 0 -1 4202560 107086 0 0 0 494321 314 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=18975/tid=19004] ppid=18973 vsize=16850012 CPUtime=4946.76 cores=0-31
/proc/18975/task/19004/stat : 19004 (claspmt) R 18973 18975 18891 0 -1 4202560 113925 0 0 0 494428 248 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=18975/tid=19005] ppid=18973 vsize=16850012 CPUtime=4946.79 cores=0-31
/proc/18975/task/19005/stat : 19005 (claspmt) R 18973 18975 18891 0 -1 4202560 79573 0 0 0 494522 157 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=18975/tid=19006] ppid=18973 vsize=16850012 CPUtime=4946.13 cores=0-31
/proc/18975/task/19006/stat : 19006 (claspmt) R 18973 18975 18891 0 -1 4202560 94768 0 0 0 494364 249 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=18975/tid=19007] ppid=18973 vsize=16850012 CPUtime=4946.79 cores=0-31
/proc/18975/task/19007/stat : 19007 (claspmt) R 18973 18975 18891 0 -1 4202560 93863 0 0 0 494463 216 0 0 25 0 32 0 14702075 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=18975/tid=19008] ppid=18973 vsize=16850012 CPUtime=4938.74 cores=0-31
/proc/18975/task/19008/stat : 19008 (claspmt) R 18973 18975 18891 0 -1 4202560 91027 0 0 0 493668 206 0 0 25 0 32 0 14702076 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=18975/tid=19009] ppid=18973 vsize=16850012 CPUtime=4946.12 cores=0-31
/proc/18975/task/19009/stat : 19009 (claspmt) R 18973 18975 18891 0 -1 4202560 161049 0 0 0 494044 568 0 0 25 0 32 0 14702076 17254412288 3921146 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 8 0 0 0
Current children cumulated CPU time (s) 158157
Current children cumulated vsize (KiB) 16850012



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5020 s]
/proc/loadavg: 32.05 32.07 32.08 33/445 19156
/proc/meminfo: memFree=247313672/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=16851620 CPUtime=160002 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4202496 3998311 0 0 0 15988369 11843 0 0 25 0 32 0 14700767 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/18975/statm: 4212905 3945662 463 189 0 4207254 0
[pid=18975/tid=18979] ppid=18973 vsize=16851620 CPUtime=5003.82 cores=0-31
/proc/18975/task/18979/stat : 18979 (claspmt) R 18973 18975 18891 0 -1 4202560 104156 0 0 0 500071 311 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=18975/tid=18980] ppid=18973 vsize=16851620 CPUtime=5004.44 cores=0-31
/proc/18975/task/18980/stat : 18980 (claspmt) R 18973 18975 18891 0 -1 4202560 118301 0 0 0 500186 258 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=18975/tid=18981] ppid=18973 vsize=16851620 CPUtime=5004.53 cores=0-31
/proc/18975/task/18981/stat : 18981 (claspmt) R 18973 18975 18891 0 -1 4202560 83277 0 0 0 500297 156 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=18975/tid=18982] ppid=18973 vsize=16851620 CPUtime=4998.28 cores=0-31
/proc/18975/task/18982/stat : 18982 (claspmt) R 18973 18975 18891 0 -1 4202560 104478 0 0 0 499538 290 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670678 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=18975/tid=18983] ppid=18973 vsize=16851620 CPUtime=4995.22 cores=0-31
/proc/18975/task/18983/stat : 18983 (claspmt) R 18973 18975 18891 0 -1 4202560 90835 0 0 0 499285 237 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=18975/tid=18984] ppid=18973 vsize=16851620 CPUtime=5004.44 cores=0-31
/proc/18975/task/18984/stat : 18984 (claspmt) R 18973 18975 18891 0 -1 4202560 96868 0 0 0 500204 240 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=18975/tid=18985] ppid=18973 vsize=16851620 CPUtime=4975.88 cores=0-31
/proc/18975/task/18985/stat : 18985 (claspmt) R 18973 18975 18891 0 -1 4202560 172991 0 0 0 496947 641 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=18975/tid=18986] ppid=18973 vsize=16851620 CPUtime=5000.95 cores=0-31
/proc/18975/task/18986/stat : 18986 (claspmt) R 18973 18975 18891 0 -1 4202560 244721 0 0 0 499173 922 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=18975/tid=18987] ppid=18973 vsize=16851620 CPUtime=4981.35 cores=0-31
/proc/18975/task/18987/stat : 18987 (claspmt) R 18973 18975 18891 0 -1 4202560 107526 0 0 0 497830 305 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=18975/tid=18988] ppid=18973 vsize=16851620 CPUtime=5002.96 cores=0-31
/proc/18975/task/18988/stat : 18988 (claspmt) R 18973 18975 18891 0 -1 4202560 116646 0 0 0 500003 293 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4684009 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=18975/tid=18989] ppid=18973 vsize=16851620 CPUtime=5004.44 cores=0-31
/proc/18975/task/18989/stat : 18989 (claspmt) R 18973 18975 18891 0 -1 4202560 82308 0 0 0 500259 185 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=18975/tid=18990] ppid=18973 vsize=16851620 CPUtime=5004.25 cores=0-31
/proc/18975/task/18990/stat : 18990 (claspmt) R 18973 18975 18891 0 -1 4202560 90840 0 0 0 500189 236 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=18975/tid=18991] ppid=18973 vsize=16851620 CPUtime=5004.84 cores=0-31
/proc/18975/task/18991/stat : 18991 (claspmt) R 18973 18975 18891 0 -1 4202560 84336 0 0 0 500277 207 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=18975/tid=18992] ppid=18973 vsize=16851620 CPUtime=5004.32 cores=0-31
/proc/18975/task/18992/stat : 18992 (claspmt) R 18973 18975 18891 0 -1 4202560 80786 0 0 0 500246 186 0 0 25 0 32 0 14702073 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=18975/tid=18993] ppid=18973 vsize=16851620 CPUtime=5003.77 cores=0-31
/proc/18975/task/18993/stat : 18993 (claspmt) R 18973 18975 18891 0 -1 4202560 174688 0 0 0 499728 649 0 0 25 0 32 0 14702074 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=18975/tid=18994] ppid=18973 vsize=16851620 CPUtime=4993.44 cores=0-31
/proc/18975/task/18994/stat : 18994 (claspmt) R 18973 18975 18891 0 -1 4202560 291853 0 0 0 498230 1114 0 0 25 0 32 0 14702074 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=18975/tid=18995] ppid=18973 vsize=16851620 CPUtime=5000.22 cores=0-31
/proc/18975/task/18995/stat : 18995 (claspmt) R 18973 18975 18891 0 -1 4202560 121550 0 0 0 499657 365 0 0 25 0 32 0 14702074 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=18975/tid=18996] ppid=18973 vsize=16851620 CPUtime=5004.53 cores=0-31
/proc/18975/task/18996/stat : 18996 (claspmt) R 18973 18975 18891 0 -1 4202560 110670 0 0 0 500219 234 0 0 25 0 32 0 14702074 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=18975/tid=18997] ppid=18973 vsize=16851620 CPUtime=5004.44 cores=0-31
/proc/18975/task/18997/stat : 18997 (claspmt) R 18973 18975 18891 0 -1 4202560 87317 0 0 0 500250 194 0 0 25 0 32 0 14702074 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=18975/tid=18998] ppid=18973 vsize=16851620 CPUtime=5000.29 cores=0-31
/proc/18975/task/18998/stat : 18998 (claspmt) R 18973 18975 18891 0 -1 4202560 108420 0 0 0 499740 289 0 0 25 0 32 0 14702074 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=18975/tid=18999] ppid=18973 vsize=16851620 CPUtime=5000.75 cores=0-31
/proc/18975/task/18999/stat : 18999 (claspmt) R 18973 18975 18891 0 -1 4202560 99412 0 0 0 499811 264 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=18975/tid=19000] ppid=18973 vsize=16851620 CPUtime=5004.32 cores=0-31
/proc/18975/task/19000/stat : 19000 (claspmt) R 18973 18975 18891 0 -1 4202560 91214 0 0 0 500243 189 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=18975/tid=19001] ppid=18973 vsize=16851620 CPUtime=4984.62 cores=0-31
/proc/18975/task/19001/stat : 19001 (claspmt) R 18973 18975 18891 0 -1 4202560 158890 0 0 0 497893 569 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=18975/tid=19002] ppid=18973 vsize=16851620 CPUtime=4979.72 cores=0-31
/proc/18975/task/19002/stat : 19002 (claspmt) R 18973 18975 18891 0 -1 4202560 249220 0 0 0 497058 914 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=18975/tid=19003] ppid=18973 vsize=16851620 CPUtime=5004.02 cores=0-31
/proc/18975/task/19003/stat : 19003 (claspmt) R 18973 18975 18891 0 -1 4202560 107628 0 0 0 500085 317 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=18975/tid=19004] ppid=18973 vsize=16851620 CPUtime=5004.46 cores=0-31
/proc/18975/task/19004/stat : 19004 (claspmt) R 18973 18975 18891 0 -1 4202560 115042 0 0 0 500198 248 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=18975/tid=19005] ppid=18973 vsize=16851620 CPUtime=5004.5 cores=0-31
/proc/18975/task/19005/stat : 19005 (claspmt) R 18973 18975 18891 0 -1 4202560 80247 0 0 0 500291 159 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=18975/tid=19006] ppid=18973 vsize=16851620 CPUtime=5003.84 cores=0-31
/proc/18975/task/19006/stat : 19006 (claspmt) R 18973 18975 18891 0 -1 4202560 95949 0 0 0 500131 253 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=18975/tid=19007] ppid=18973 vsize=16851620 CPUtime=5004.5 cores=0-31
/proc/18975/task/19007/stat : 19007 (claspmt) R 18973 18975 18891 0 -1 4202560 93867 0 0 0 500234 216 0 0 25 0 32 0 14702075 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=18975/tid=19008] ppid=18973 vsize=16851620 CPUtime=4996.45 cores=0-31
/proc/18975/task/19008/stat : 19008 (claspmt) R 18973 18975 18891 0 -1 4202560 91194 0 0 0 499439 206 0 0 25 0 32 0 14702076 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=18975/tid=19009] ppid=18973 vsize=16851620 CPUtime=5003.82 cores=0-31
/proc/18975/task/19009/stat : 19009 (claspmt) R 18973 18975 18891 0 -1 4202560 164119 0 0 0 499801 581 0 0 25 0 32 0 14702076 17256058880 3945662 270582939648 4194304 4964800 140737356936576 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 8 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 16851620

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -18975

[startup+5022.3 s]
/proc/loadavg: 32.05 32.07 32.08 2/415 19157
/proc/meminfo: memFree=247315028/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=0 CPUtime=160005 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4203524 3998322 0 0 0 15988599 11870 0 0 25 0 1 0 14700767 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/18975/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

Solver just ended. Dumping a history of the last processes samples

[startup+5022.63 s]
/proc/loadavg: 32.05 32.07 32.08 2/414 19157
/proc/meminfo: memFree=255451680/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=0 CPUtime=160005 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4203524 3998322 0 0 0 15988599 11903 0 0 25 0 1 0 14700767 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/18975/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

[startup+5022.8 s]
/proc/loadavg: 32.05 32.07 32.08 2/414 19157
/proc/meminfo: memFree=255451680/264257624 swapFree=67111528/67111528
[pid=18975] ppid=18973 vsize=0 CPUtime=160005 cores=0-31
/proc/18975/stat : 18975 (claspmt) R 18973 18975 18891 0 -1 4203524 3998322 0 0 0 15988599 11920 0 0 25 0 1 0 14700767 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/18975/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5022.87
CPU time (s): 160005
CPU user time (s): 159886
CPU system time (s): 119.274
CPU usage (%): 3185.53
Max. virtual memory (cumulated for all children) (KiB): 16851620

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159886
system time used= 119.274
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3998322
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1085108
involuntary context switches= 374042

runsolver used 27.0919 second user time and 114.6 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-08 02:32:19
IDJOB=3472342
IDBENCH=71095
IDSOLVER=1961
FILE ID=node200/3472342-1307493138
RUNJOBID= node200-1307493133-18905
PBS_JOBID= 13509202
Free space on /tmp= 61736 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/APPLICATIONS/satComp09_BioInstances/ndhf_xits_19_UNKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472342-1307493138/watcher-3472342-1307493138 -o /tmp/evaluation-result-3472342-1307493138/solver-3472342-1307493138 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472342-1307493138.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= 4cf2a693df1b6e5a084b88e84ec5f2af
RANDOM SEED=1545319284

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      263119556 kB
Buffers:        152616 kB
Cached:         719776 kB
SwapCached:          0 kB
Active:         189192 kB
Inactive:       746780 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      263119556 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            8820 kB
Writeback:           0 kB
AnonPages:       63708 kB
Mapped:          14532 kB
Slab:           120776 kB
PageTables:       4288 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   166188 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61736 MiB
End job on node200 at 2011-06-08 03:56:02