Trace number 3472525

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 160004 5009.75

General information on the benchmark

NameSAT05/industrial/grieu05/vmpc/
vmpc_36.renamed-as.sat05-1922.cnf
MD5SUMc9c90d9af49f4c08c7262ecb9a0b2a3d
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark18891.4
Satisfiable
(Un)Satisfiability was proved
Number of variables1296
Number of clauses230544
Sum of the clauses size603180
Maximum clause length36
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 290828
Number of clauses of size 3139644
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 572

Solver Data 

0.00/0.01	c clasp version 2.0.0-RC2
0.00/0.01	c Reading from ...824962/instance-3472525-1307824962.cnf
1.19/1.25	c Solving...
160001.73/5007.43	
160001.73/5007.43	*** INTERRUPTED! ***
160001.73/5007.43	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 988)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472525-1307824962/watcher-3472525-1307824962 -o /tmp/evaluation-result-3472525-1307824962/solver-3472525-1307824962 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472525-1307824962.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 32.00 30.45 30.06 2/413 12525
/proc/meminfo: memFree=261944636/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=22832 CPUtime=0.02 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 1093 0 0 0 2 0 0 0 18 0 1 0 47883098 23379968 924 270582939648 4194304 4964800 140736293276144 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 8 0 0 0
/proc/12525/statm: 5708 925 383 189 0 582 0

[startup+0.0512499 s]
/proc/loadavg: 32.00 30.45 30.06 2/413 12525
/proc/meminfo: memFree=261944636/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=24048 CPUtime=0.04 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 1488 0 0 0 4 0 0 0 18 0 1 0 47883098 24625152 1238 270582939648 4194304 4964800 140736293276144 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 8 0 0 0
/proc/12525/statm: 6012 1239 383 189 0 886 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 24048

[startup+0.101288 s]
/proc/loadavg: 32.00 30.45 30.06 2/413 12525
/proc/meminfo: memFree=261944636/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=27816 CPUtime=0.09 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 2705 0 0 0 9 0 0 0 18 0 1 0 47883098 28483584 2153 270582939648 4194304 4964800 140736293276144 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 8 0 0 0
/proc/12525/statm: 6954 2154 383 189 0 1828 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27816

[startup+0.301255 s]
/proc/loadavg: 32.00 30.45 30.06 2/413 12525
/proc/meminfo: memFree=261944636/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=38716 CPUtime=0.29 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 5902 0 0 0 28 1 0 0 18 0 1 0 47883098 39645184 4662 270582939648 4194304 4964800 140736293276144 18446744073709551615 4607536 0 0 4096 18946 0 0 0 17 8 0 0 0
/proc/12525/statm: 9679 4662 396 189 0 4553 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 38716

[startup+0.700289 s]
/proc/loadavg: 32.00 30.45 30.06 2/413 12525
/proc/meminfo: memFree=261944636/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=38716 CPUtime=0.69 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 5903 0 0 0 68 1 0 0 18 0 1 0 47883098 39645184 4663 270582939648 4194304 4964800 140736293276144 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 8 0 0 0
/proc/12525/statm: 9679 4663 396 189 0 4553 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 38716

[startup+1.53124 s]
/proc/loadavg: 29.52 29.96 29.90 2/414 12526
/proc/meminfo: memFree=261926904/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=822048 CPUtime=3 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 7167 0 0 0 296 4 0 0 18 0 32 0 47883098 841777152 5391 270582939648 4194304 4964800 140736293276144 18446744073709551615 4627263 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12525/statm: 205512 5392 455 189 0 199861 0
Current children cumulated CPU time (s) 3
Current children cumulated vsize (KiB) 822048

[startup+3.10123 s]
/proc/loadavg: 29.52 29.96 29.90 33/445 12557
/proc/meminfo: memFree=261910440/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=822444 CPUtime=52.94 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 18959 0 0 0 5246 48 0 0 18 0 32 0 47883098 842182656 17183 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668748 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12525/statm: 205611 17183 461 189 0 199960 0
[pid=12525/tid=12527] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12527/stat : 12527 (claspmt) R 12523 12525 12495 0 -1 4202560 305 0 0 0 159 0 0 0 19 0 32 0 47883224 842182656 17183 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668763 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=12525/tid=12528] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12528/stat : 12528 (claspmt) R 12523 12525 12495 0 -1 4202560 597 0 0 0 157 2 0 0 17 0 32 0 47883224 842182656 17183 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=12525/tid=12529] ppid=12523 vsize=822444 CPUtime=1.56 cores=0-31
/proc/12525/task/12529/stat : 12529 (claspmt) R 12523 12525 12495 0 -1 4202560 265 0 0 0 156 0 0 0 18 0 32 0 47883224 842182656 17183 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=12525/tid=12530] ppid=12523 vsize=822444 CPUtime=1.6 cores=0-31
/proc/12525/task/12530/stat : 12530 (claspmt) R 12523 12525 12495 0 -1 4202560 403 0 0 0 159 1 0 0 18 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12525/tid=12531] ppid=12523 vsize=822444 CPUtime=1.69 cores=0-31
/proc/12525/task/12531/stat : 12531 (claspmt) R 12523 12525 12495 0 -1 4202560 33 0 0 0 169 0 0 0 21 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=12525/tid=12532] ppid=12523 vsize=822444 CPUtime=1.58 cores=0-31
/proc/12525/task/12532/stat : 12532 (claspmt) R 12523 12525 12495 0 -1 4202560 466 0 0 0 157 1 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=12525/tid=12533] ppid=12523 vsize=822444 CPUtime=1.58 cores=0-31
/proc/12525/task/12533/stat : 12533 (claspmt) R 12523 12525 12495 0 -1 4202560 789 0 0 0 156 2 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12525/tid=12534] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12534/stat : 12534 (claspmt) R 12523 12525 12495 0 -1 4202560 387 0 0 0 158 1 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=12525/tid=12535] ppid=12523 vsize=822444 CPUtime=1.55 cores=0-31
/proc/12525/task/12535/stat : 12535 (claspmt) R 12523 12525 12495 0 -1 4202560 292 0 0 0 154 1 0 0 18 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=12525/tid=12536] ppid=12523 vsize=822444 CPUtime=1.58 cores=0-31
/proc/12525/task/12536/stat : 12536 (claspmt) R 12523 12525 12495 0 -1 4202560 665 0 0 0 157 1 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=12525/tid=12537] ppid=12523 vsize=822444 CPUtime=1.6 cores=0-31
/proc/12525/task/12537/stat : 12537 (claspmt) R 12523 12525 12495 0 -1 4202560 287 0 0 0 159 1 0 0 20 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=12525/tid=12538] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12538/stat : 12538 (claspmt) R 12523 12525 12495 0 -1 4202560 391 0 0 0 158 1 0 0 20 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=12525/tid=12539] ppid=12523 vsize=822444 CPUtime=1.8 cores=0-31
/proc/12525/task/12539/stat : 12539 (claspmt) R 12523 12525 12495 0 -1 4202560 33 0 0 0 180 0 0 0 25 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668698 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=12525/tid=12540] ppid=12523 vsize=822444 CPUtime=1.58 cores=0-31
/proc/12525/task/12540/stat : 12540 (claspmt) R 12523 12525 12495 0 -1 4202560 539 0 0 0 157 1 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=12525/tid=12541] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12541/stat : 12541 (claspmt) R 12523 12525 12495 0 -1 4202560 798 0 0 0 156 3 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668732 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=12525/tid=12542] ppid=12523 vsize=822444 CPUtime=1.6 cores=0-31
/proc/12525/task/12542/stat : 12542 (claspmt) R 12523 12525 12495 0 -1 4202560 287 0 0 0 160 0 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670147 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=12525/tid=12543] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12543/stat : 12543 (claspmt) R 12523 12525 12495 0 -1 4202560 351 0 0 0 158 1 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=12525/tid=12544] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12544/stat : 12544 (claspmt) R 12523 12525 12495 0 -1 4202560 677 0 0 0 157 2 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668698 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=12525/tid=12545] ppid=12523 vsize=822444 CPUtime=1.55 cores=0-31
/proc/12525/task/12545/stat : 12545 (claspmt) R 12523 12525 12495 0 -1 4202560 260 0 0 0 155 0 0 0 18 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=12525/tid=12546] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12546/stat : 12546 (claspmt) R 12523 12525 12495 0 -1 4202560 372 0 0 0 158 1 0 0 20 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668709 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=12525/tid=12547] ppid=12523 vsize=822444 CPUtime=1.81 cores=0-31
/proc/12525/task/12547/stat : 12547 (claspmt) R 12523 12525 12495 0 -1 4202560 37 0 0 0 181 0 0 0 20 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=12525/tid=12548] ppid=12523 vsize=822444 CPUtime=1.58 cores=0-31
/proc/12525/task/12548/stat : 12548 (claspmt) R 12523 12525 12495 0 -1 4202560 504 0 0 0 157 1 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=12525/tid=12549] ppid=12523 vsize=822444 CPUtime=1.57 cores=0-31
/proc/12525/task/12549/stat : 12549 (claspmt) R 12523 12525 12495 0 -1 4202560 749 0 0 0 155 2 0 0 17 0 32 0 47883224 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=12525/tid=12550] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12550/stat : 12550 (claspmt) R 12523 12525 12495 0 -1 4202560 318 0 0 0 158 1 0 0 20 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12525/tid=12551] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12551/stat : 12551 (claspmt) R 12523 12525 12495 0 -1 4202560 298 0 0 0 159 0 0 0 17 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4660137 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=12525/tid=12552] ppid=12523 vsize=822444 CPUtime=1.57 cores=0-31
/proc/12525/task/12552/stat : 12552 (claspmt) R 12523 12525 12495 0 -1 4202560 581 0 0 0 156 1 0 0 17 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668777 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12525/tid=12553] ppid=12523 vsize=822444 CPUtime=1.51 cores=0-31
/proc/12525/task/12553/stat : 12553 (claspmt) R 12523 12525 12495 0 -1 4202560 237 0 0 0 150 1 0 0 18 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12525/tid=12554] ppid=12523 vsize=822444 CPUtime=1.59 cores=0-31
/proc/12525/task/12554/stat : 12554 (claspmt) R 12523 12525 12495 0 -1 4202560 383 0 0 0 158 1 0 0 17 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12525/tid=12555] ppid=12523 vsize=822444 CPUtime=1.83 cores=0-31
/proc/12525/task/12555/stat : 12555 (claspmt) R 12523 12525 12495 0 -1 4202560 42 0 0 0 183 0 0 0 20 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=12525/tid=12556] ppid=12523 vsize=822444 CPUtime=1.57 cores=0-31
/proc/12525/task/12556/stat : 12556 (claspmt) R 12523 12525 12495 0 -1 4202560 577 0 0 0 156 1 0 0 19 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=12525/tid=12557] ppid=12523 vsize=822444 CPUtime=1.58 cores=0-31
/proc/12525/task/12557/stat : 12557 (claspmt) R 12523 12525 12495 0 -1 4202560 724 0 0 0 155 3 0 0 18 0 32 0 47883225 842182656 17184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 24 0 0 0
Current children cumulated CPU time (s) 52.94
Current children cumulated vsize (KiB) 822444

[startup+6.30134 s]
/proc/loadavg: 29.52 29.96 29.90 33/445 12557
/proc/meminfo: memFree=261825624/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=829920 CPUtime=155.01 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 35960 0 0 0 15403 98 0 0 19 0 32 0 47883098 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668734 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12525/statm: 207480 34184 461 189 0 201829 0
[pid=12525/tid=12527] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12527/stat : 12527 (claspmt) R 12523 12525 12495 0 -1 4202560 652 0 0 0 477 1 0 0 25 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668805 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=12525/tid=12528] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12528/stat : 12528 (claspmt) R 12523 12525 12495 0 -1 4202560 1515 0 0 0 472 6 0 0 18 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=12525/tid=12529] ppid=12523 vsize=829920 CPUtime=4.76 cores=0-31
/proc/12525/task/12529/stat : 12529 (claspmt) R 12523 12525 12495 0 -1 4202560 714 0 0 0 474 2 0 0 18 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=12525/tid=12530] ppid=12523 vsize=829920 CPUtime=4.79 cores=0-31
/proc/12525/task/12530/stat : 12530 (claspmt) R 12523 12525 12495 0 -1 4202560 1033 0 0 0 477 2 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670160 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12525/tid=12531] ppid=12523 vsize=829920 CPUtime=4.89 cores=0-31
/proc/12525/task/12531/stat : 12531 (claspmt) R 12523 12525 12495 0 -1 4202560 140 0 0 0 489 0 0 0 25 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=12525/tid=12532] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12532/stat : 12532 (claspmt) R 12523 12525 12495 0 -1 4202560 854 0 0 0 475 3 0 0 20 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321076 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=12525/tid=12533] ppid=12523 vsize=829920 CPUtime=4.77 cores=0-31
/proc/12525/task/12533/stat : 12533 (claspmt) R 12523 12525 12495 0 -1 4202560 1855 0 0 0 471 6 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668763 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12525/tid=12534] ppid=12523 vsize=829920 CPUtime=4.79 cores=0-31
/proc/12525/task/12534/stat : 12534 (claspmt) R 12523 12525 12495 0 -1 4202560 723 0 0 0 476 3 0 0 20 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=12525/tid=12535] ppid=12523 vsize=829920 CPUtime=4.65 cores=0-31
/proc/12525/task/12535/stat : 12535 (claspmt) R 12523 12525 12495 0 -1 4202560 667 0 0 0 463 2 0 0 18 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668709 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=12525/tid=12536] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12536/stat : 12536 (claspmt) R 12523 12525 12495 0 -1 4202560 1681 0 0 0 474 4 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=12525/tid=12537] ppid=12523 vsize=829920 CPUtime=4.79 cores=0-31
/proc/12525/task/12537/stat : 12537 (claspmt) R 12523 12525 12495 0 -1 4202560 676 0 0 0 478 1 0 0 23 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668737 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=12525/tid=12538] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12538/stat : 12538 (claspmt) R 12523 12525 12495 0 -1 4202560 1040 0 0 0 475 3 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668789 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=12525/tid=12539] ppid=12523 vsize=829920 CPUtime=5 cores=0-31
/proc/12525/task/12539/stat : 12539 (claspmt) R 12523 12525 12495 0 -1 4202560 123 0 0 0 500 0 0 0 25 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=12525/tid=12540] ppid=12523 vsize=829920 CPUtime=4.77 cores=0-31
/proc/12525/task/12540/stat : 12540 (claspmt) R 12523 12525 12495 0 -1 4202560 984 0 0 0 475 2 0 0 25 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4660461 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=12525/tid=12541] ppid=12523 vsize=829920 CPUtime=4.77 cores=0-31
/proc/12525/task/12541/stat : 12541 (claspmt) R 12523 12525 12495 0 -1 4202560 1749 0 0 0 471 6 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=12525/tid=12542] ppid=12523 vsize=829920 CPUtime=4.79 cores=0-31
/proc/12525/task/12542/stat : 12542 (claspmt) R 12523 12525 12495 0 -1 4202560 673 0 0 0 478 1 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=12525/tid=12543] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12543/stat : 12543 (claspmt) R 12523 12525 12495 0 -1 4202560 726 0 0 0 476 2 0 0 25 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668732 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=12525/tid=12544] ppid=12523 vsize=829920 CPUtime=4.79 cores=0-31
/proc/12525/task/12544/stat : 12544 (claspmt) R 12523 12525 12495 0 -1 4202560 1589 0 0 0 474 5 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=12525/tid=12545] ppid=12523 vsize=829920 CPUtime=4.75 cores=0-31
/proc/12525/task/12545/stat : 12545 (claspmt) R 12523 12525 12495 0 -1 4202560 703 0 0 0 473 2 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668770 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=12525/tid=12546] ppid=12523 vsize=829920 CPUtime=4.79 cores=0-31
/proc/12525/task/12546/stat : 12546 (claspmt) R 12523 12525 12495 0 -1 4202560 986 0 0 0 477 2 0 0 20 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668698 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=12525/tid=12547] ppid=12523 vsize=829920 CPUtime=5.01 cores=0-31
/proc/12525/task/12547/stat : 12547 (claspmt) R 12523 12525 12495 0 -1 4202560 108 0 0 0 501 0 0 0 25 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=12525/tid=12548] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12548/stat : 12548 (claspmt) R 12523 12525 12495 0 -1 4202560 929 0 0 0 475 3 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=12525/tid=12549] ppid=12523 vsize=829920 CPUtime=4.76 cores=0-31
/proc/12525/task/12549/stat : 12549 (claspmt) R 12523 12525 12495 0 -1 4202560 1763 0 0 0 471 5 0 0 19 0 32 0 47883224 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4365674 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=12525/tid=12550] ppid=12523 vsize=829920 CPUtime=4.79 cores=0-31
/proc/12525/task/12550/stat : 12550 (claspmt) R 12523 12525 12495 0 -1 4202560 666 0 0 0 477 2 0 0 25 0 32 0 47883225 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12525/tid=12551] ppid=12523 vsize=829920 CPUtime=4.78 cores=0-31
/proc/12525/task/12551/stat : 12551 (claspmt) R 12523 12525 12495 0 -1 4202560 694 0 0 0 476 2 0 0 18 0 32 0 47883225 849838080 34184 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668709 0 0 4096 18946 0 0 0 -1 29 0 0 0

################
# More data... #
################

/proc/12525/task/12538/stat : 12538 (claspmt) R 12523 12525 12495 0 -1 4202560 30083 0 0 0 495887 71 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=12525/tid=12539] ppid=12523 vsize=4959312 CPUtime=4959.75 cores=0-31
/proc/12525/task/12539/stat : 12539 (claspmt) R 12523 12525 12495 0 -1 4202560 30447 0 0 0 495905 70 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=12525/tid=12540] ppid=12523 vsize=4959312 CPUtime=4959.43 cores=0-31
/proc/12525/task/12540/stat : 12540 (claspmt) R 12523 12525 12495 0 -1 4202560 25034 0 0 0 495877 66 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=12525/tid=12541] ppid=12523 vsize=4959312 CPUtime=4954.37 cores=0-31
/proc/12525/task/12541/stat : 12541 (claspmt) R 12523 12525 12495 0 -1 4202560 50693 0 0 0 495235 202 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=12525/tid=12542] ppid=12523 vsize=4959312 CPUtime=4959.48 cores=0-31
/proc/12525/task/12542/stat : 12542 (claspmt) R 12523 12525 12495 0 -1 4202560 25703 0 0 0 495843 105 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=12525/tid=12543] ppid=12523 vsize=4959312 CPUtime=4954.75 cores=0-31
/proc/12525/task/12543/stat : 12543 (claspmt) R 12523 12525 12495 0 -1 4202560 30420 0 0 0 495401 74 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=12525/tid=12544] ppid=12523 vsize=4959312 CPUtime=4959.58 cores=0-31
/proc/12525/task/12544/stat : 12544 (claspmt) R 12523 12525 12495 0 -1 4202560 66623 0 0 0 495831 127 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=12525/tid=12545] ppid=12523 vsize=4959312 CPUtime=4952.3 cores=0-31
/proc/12525/task/12545/stat : 12545 (claspmt) R 12523 12525 12495 0 -1 4202560 21632 0 0 0 495165 65 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=12525/tid=12546] ppid=12523 vsize=4959312 CPUtime=4938.99 cores=0-31
/proc/12525/task/12546/stat : 12546 (claspmt) R 12523 12525 12495 0 -1 4202560 31970 0 0 0 493827 72 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=12525/tid=12547] ppid=12523 vsize=4959312 CPUtime=4959.84 cores=0-31
/proc/12525/task/12547/stat : 12547 (claspmt) R 12523 12525 12495 0 -1 4202560 30478 0 0 0 495916 68 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=12525/tid=12548] ppid=12523 vsize=4959312 CPUtime=4936.76 cores=0-31
/proc/12525/task/12548/stat : 12548 (claspmt) R 12523 12525 12495 0 -1 4202560 25370 0 0 0 493610 66 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=12525/tid=12549] ppid=12523 vsize=4959312 CPUtime=4959.42 cores=0-31
/proc/12525/task/12549/stat : 12549 (claspmt) R 12523 12525 12495 0 -1 4202560 52567 0 0 0 495743 199 0 0 25 0 32 0 47883224 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12525/tid=12550] ppid=12523 vsize=4959312 CPUtime=4955.82 cores=0-31
/proc/12525/task/12550/stat : 12550 (claspmt) R 12523 12525 12495 0 -1 4202560 24944 0 0 0 495484 98 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668805 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=12525/tid=12551] ppid=12523 vsize=4959312 CPUtime=4955.86 cores=0-31
/proc/12525/task/12551/stat : 12551 (claspmt) R 12523 12525 12495 0 -1 4202560 31365 0 0 0 495515 71 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=12525/tid=12552] ppid=12523 vsize=4959312 CPUtime=4958.49 cores=0-31
/proc/12525/task/12552/stat : 12552 (claspmt) R 12523 12525 12495 0 -1 4202560 65342 0 0 0 495724 125 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12525/tid=12553] ppid=12523 vsize=4959312 CPUtime=4941.35 cores=0-31
/proc/12525/task/12553/stat : 12553 (claspmt) R 12523 12525 12495 0 -1 4202560 22216 0 0 0 494071 64 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=12525/tid=12554] ppid=12523 vsize=4959312 CPUtime=4959.56 cores=0-31
/proc/12525/task/12554/stat : 12554 (claspmt) R 12523 12525 12495 0 -1 4202560 31144 0 0 0 495890 66 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12525/tid=12555] ppid=12523 vsize=4959312 CPUtime=4955.5 cores=0-31
/proc/12525/task/12555/stat : 12555 (claspmt) R 12523 12525 12495 0 -1 4202560 30644 0 0 0 495482 68 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12525/tid=12556] ppid=12523 vsize=4959312 CPUtime=4959.39 cores=0-31
/proc/12525/task/12556/stat : 12556 (claspmt) R 12523 12525 12495 0 -1 4202560 25710 0 0 0 495872 67 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=12525/tid=12557] ppid=12523 vsize=4959312 CPUtime=4959.46 cores=0-31
/proc/12525/task/12557/stat : 12557 (claspmt) R 12523 12525 12495 0 -1 4202560 51659 0 0 0 495769 177 0 0 25 0 32 0 47883225 5078335488 1113420 270582939648 4194304 4964800 140736293276144 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 13 0 0 0
Current children cumulated CPU time (s) 158560
Current children cumulated vsize (KiB) 4959312



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.4 s]
/proc/loadavg: 32.04 32.03 32.00 33/445 12719
/proc/meminfo: memFree=257476540/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=4960676 CPUtime=160002 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 1143780 0 0 0 15997049 3125 0 0 25 0 32 0 47883098 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668822 0 0 4096 18946 18446744073709551615 0 0 17 8 0 0 0
/proc/12525/statm: 1240169 1116505 463 189 0 1234518 0
[pid=12525/tid=12527] ppid=12523 vsize=4960676 CPUtime=5001.95 cores=0-31
/proc/12525/task/12527/stat : 12527 (claspmt) R 12523 12525 12495 0 -1 4202560 31894 0 0 0 500075 120 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321002 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=12525/tid=12528] ppid=12523 vsize=4960676 CPUtime=5004.71 cores=0-31
/proc/12525/task/12528/stat : 12528 (claspmt) R 12523 12525 12495 0 -1 4202560 67969 0 0 0 500332 139 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=12525/tid=12529] ppid=12523 vsize=4960676 CPUtime=4983.69 cores=0-31
/proc/12525/task/12529/stat : 12529 (claspmt) R 12523 12525 12495 0 -1 4202560 22726 0 0 0 498304 65 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=12525/tid=12530] ppid=12523 vsize=4960676 CPUtime=5004.66 cores=0-31
/proc/12525/task/12530/stat : 12530 (claspmt) R 12523 12525 12495 0 -1 4202560 31404 0 0 0 500393 73 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12525/tid=12531] ppid=12523 vsize=4960676 CPUtime=5004.84 cores=0-31
/proc/12525/task/12531/stat : 12531 (claspmt) R 12523 12525 12495 0 -1 4202560 28951 0 0 0 500421 63 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=12525/tid=12532] ppid=12523 vsize=4960676 CPUtime=5003.78 cores=0-31
/proc/12525/task/12532/stat : 12532 (claspmt) R 12523 12525 12495 0 -1 4202560 23969 0 0 0 500307 71 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=12525/tid=12533] ppid=12523 vsize=4960676 CPUtime=4988.92 cores=0-31
/proc/12525/task/12533/stat : 12533 (claspmt) R 12523 12525 12495 0 -1 4202560 52977 0 0 0 498693 199 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12525/tid=12534] ppid=12523 vsize=4960676 CPUtime=5001.74 cores=0-31
/proc/12525/task/12534/stat : 12534 (claspmt) R 12523 12525 12495 0 -1 4202560 26285 0 0 0 500062 112 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=12525/tid=12535] ppid=12523 vsize=4960676 CPUtime=5002.55 cores=0-31
/proc/12525/task/12535/stat : 12535 (claspmt) R 12523 12525 12495 0 -1 4202560 30414 0 0 0 500187 68 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=12525/tid=12536] ppid=12523 vsize=4960676 CPUtime=5004.61 cores=0-31
/proc/12525/task/12536/stat : 12536 (claspmt) R 12523 12525 12495 0 -1 4202560 66398 0 0 0 500349 112 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=12525/tid=12537] ppid=12523 vsize=4960676 CPUtime=4994.95 cores=0-31
/proc/12525/task/12537/stat : 12537 (claspmt) R 12523 12525 12495 0 -1 4202560 22880 0 0 0 499431 64 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=12525/tid=12538] ppid=12523 vsize=4960676 CPUtime=5004.69 cores=0-31
/proc/12525/task/12538/stat : 12538 (claspmt) R 12523 12525 12495 0 -1 4202560 30205 0 0 0 500398 71 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=12525/tid=12539] ppid=12523 vsize=4960676 CPUtime=5004.86 cores=0-31
/proc/12525/task/12539/stat : 12539 (claspmt) R 12523 12525 12495 0 -1 4202560 30447 0 0 0 500416 70 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4325336 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=12525/tid=12540] ppid=12523 vsize=4960676 CPUtime=5004.54 cores=0-31
/proc/12525/task/12540/stat : 12540 (claspmt) R 12523 12525 12495 0 -1 4202560 25047 0 0 0 500388 66 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=12525/tid=12541] ppid=12523 vsize=4960676 CPUtime=4999.49 cores=0-31
/proc/12525/task/12541/stat : 12541 (claspmt) R 12523 12525 12495 0 -1 4202560 50843 0 0 0 499746 203 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=12525/tid=12542] ppid=12523 vsize=4960676 CPUtime=5004.59 cores=0-31
/proc/12525/task/12542/stat : 12542 (claspmt) R 12523 12525 12495 0 -1 4202560 25736 0 0 0 500354 105 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668805 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=12525/tid=12543] ppid=12523 vsize=4960676 CPUtime=4999.86 cores=0-31
/proc/12525/task/12543/stat : 12543 (claspmt) R 12523 12525 12495 0 -1 4202560 30492 0 0 0 499912 74 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=12525/tid=12544] ppid=12523 vsize=4960676 CPUtime=5004.69 cores=0-31
/proc/12525/task/12544/stat : 12544 (claspmt) R 12523 12525 12495 0 -1 4202560 66676 0 0 0 500342 127 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=12525/tid=12545] ppid=12523 vsize=4960676 CPUtime=4997.41 cores=0-31
/proc/12525/task/12545/stat : 12545 (claspmt) R 12523 12525 12495 0 -1 4202560 21654 0 0 0 499676 65 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=12525/tid=12546] ppid=12523 vsize=4960676 CPUtime=4984.1 cores=0-31
/proc/12525/task/12546/stat : 12546 (claspmt) R 12523 12525 12495 0 -1 4202560 32493 0 0 0 498338 72 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=12525/tid=12547] ppid=12523 vsize=4960676 CPUtime=5004.95 cores=0-31
/proc/12525/task/12547/stat : 12547 (claspmt) R 12523 12525 12495 0 -1 4202560 30524 0 0 0 500427 68 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=12525/tid=12548] ppid=12523 vsize=4960676 CPUtime=4981.88 cores=0-31
/proc/12525/task/12548/stat : 12548 (claspmt) R 12523 12525 12495 0 -1 4202560 25388 0 0 0 498122 66 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=12525/tid=12549] ppid=12523 vsize=4960676 CPUtime=5004.53 cores=0-31
/proc/12525/task/12549/stat : 12549 (claspmt) R 12523 12525 12495 0 -1 4202560 52590 0 0 0 500254 199 0 0 25 0 32 0 47883224 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12525/tid=12550] ppid=12523 vsize=4960676 CPUtime=4999.62 cores=0-31
/proc/12525/task/12550/stat : 12550 (claspmt) R 12523 12525 12495 0 -1 4202560 24966 0 0 0 499864 98 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=12525/tid=12551] ppid=12523 vsize=4960676 CPUtime=5000.97 cores=0-31
/proc/12525/task/12551/stat : 12551 (claspmt) R 12523 12525 12495 0 -1 4202560 31378 0 0 0 500026 71 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670343 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=12525/tid=12552] ppid=12523 vsize=4960676 CPUtime=5003.6 cores=0-31
/proc/12525/task/12552/stat : 12552 (claspmt) R 12523 12525 12495 0 -1 4202560 65606 0 0 0 500235 125 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4321010 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12525/tid=12553] ppid=12523 vsize=4960676 CPUtime=4986.46 cores=0-31
/proc/12525/task/12553/stat : 12553 (claspmt) R 12523 12525 12495 0 -1 4202560 22368 0 0 0 498582 64 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=12525/tid=12554] ppid=12523 vsize=4960676 CPUtime=5004.67 cores=0-31
/proc/12525/task/12554/stat : 12554 (claspmt) R 12523 12525 12495 0 -1 4202560 31568 0 0 0 500401 66 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12525/tid=12555] ppid=12523 vsize=4960676 CPUtime=5000.62 cores=0-31
/proc/12525/task/12555/stat : 12555 (claspmt) R 12523 12525 12495 0 -1 4202560 30659 0 0 0 499994 68 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12525/tid=12556] ppid=12523 vsize=4960676 CPUtime=5004.5 cores=0-31
/proc/12525/task/12556/stat : 12556 (claspmt) R 12523 12525 12495 0 -1 4202560 25892 0 0 0 500383 67 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=12525/tid=12557] ppid=12523 vsize=4960676 CPUtime=5004.56 cores=0-31
/proc/12525/task/12557/stat : 12557 (claspmt) R 12523 12525 12495 0 -1 4202560 51742 0 0 0 500279 177 0 0 25 0 32 0 47883225 5079732224 1116505 270582939648 4194304 4964800 140736293276144 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 13 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 4960676

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -12525

Solver just ended. Dumping a history of the last processes samples

[startup+5007.8 s]
/proc/loadavg: 32.04 32.03 32.00 33/445 12719
/proc/meminfo: memFree=257476540/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=4870312 CPUtime=160002 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 1143793 0 0 0 15997123 3125 0 0 25 0 1 0 47883098 4987199488 1116432 270582939648 4194304 4964800 140736293276144 18446744073709551615 258650610907 0 0 4096 18946 0 0 0 17 8 0 0 0
/proc/12525/statm: 1217578 1116432 474 189 0 1211927 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 4870312

[startup+5008.6 s]
/proc/loadavg: 32.04 32.03 32.00 2/415 12720
/proc/meminfo: memFree=257484440/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=4848424 CPUtime=160003 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4202496 1143793 0 0 0 15997203 3125 0 0 25 0 1 0 47883098 4964786176 1112011 270582939648 4194304 4964800 140736293276144 18446744073709551615 258650610907 0 0 4096 18946 0 0 0 17 8 0 0 0
/proc/12525/statm: 1212106 1112011 474 189 0 1206455 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 4848424

[startup+5009.43 s]
/proc/loadavg: 32.04 32.03 32.00 2/415 12720
/proc/meminfo: memFree=257500808/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=0 CPUtime=160004 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4203524 1143793 0 0 0 15997286 3126 0 0 25 0 1 0 47883098 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 8 0 0 0
/proc/12525/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 0

[startup+5009.6 s]
/proc/loadavg: 32.04 32.03 32.00 2/415 12720
/proc/meminfo: memFree=257500808/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=0 CPUtime=160004 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4203524 1143793 0 0 0 15997286 3142 0 0 25 0 1 0 47883098 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 8 0 0 0
/proc/12525/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 0

[startup+5009.7 s]
/proc/loadavg: 32.04 32.03 32.00 2/415 12720
/proc/meminfo: memFree=257500808/264257624 swapFree=67111528/67111528
[pid=12525] ppid=12523 vsize=0 CPUtime=160004 cores=0-31
/proc/12525/stat : 12525 (claspmt) R 12523 12525 12495 0 -1 4203524 1143793 0 0 0 15997286 3152 0 0 25 0 1 0 47883098 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 8 0 0 0
/proc/12525/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5009.75
CPU time (s): 160004
CPU user time (s): 159973
CPU system time (s): 31.5792
CPU usage (%): 3193.86
Max. virtual memory (cumulated for all children) (KiB): 4960676

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159973
system time used= 31.5792
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1143793
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 340741
involuntary context switches= 366498

runsolver used 26.7219 second user time and 114.543 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-11 22:42:42
IDJOB=3472525
IDBENCH=85991
IDSOLVER=1961
FILE ID=node200/3472525-1307824962
RUNJOBID= node200-1307824962-12509
PBS_JOBID= 13551031
Free space on /tmp= 61740 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT05/industrial/grieu05/vmpc/vmpc_36.renamed-as.sat05-1922.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472525-1307824962/watcher-3472525-1307824962 -o /tmp/evaluation-result-3472525-1307824962/solver-3472525-1307824962 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472525-1307824962.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= c9c90d9af49f4c08c7262ecb9a0b2a3d
RANDOM SEED=294266041

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      261944932 kB
Buffers:        297752 kB
Cached:        1687740 kB
SwapCached:          0 kB
Active:         328020 kB
Inactive:      1720968 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      261944932 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            4088 kB
Writeback:           0 kB
AnonPages:       63372 kB
Mapped:          14480 kB
Slab:           185884 kB
PageTables:       4324 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   159216 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61736 MiB
End job on node200 at 2011-06-12 00:06:12