Trace number 3472269

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 160005 5097.46

General information on the benchmark

Nameindustrial/palacios/hard/
sortnet-8-ipc5-h19-sat.cnf
MD5SUM409f4401e9e8b60ee665ac5567a13e41
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark98394.4
Satisfiable
(Un)Satisfiability was proved
Number of variables359341
Number of clauses1254773
Sum of the clauses size3046189
Maximum clause length57
Minimum clause length1
Number of clauses of size 1256
Number of clauses of size 2896240
Number of clauses of size 3346435
Number of clauses of size 4830
Number of clauses of size 5571
Number of clauses of size over 510441

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...367982/instance-3472269-1307367982.cnf
7.72/7.76	c Solving...
160002.69/5095.15	
160002.69/5095.15	*** INTERRUPTED! ***
160002.69/5095.15	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472269-1307367982/watcher-3472269-1307367982 -o /tmp/evaluation-result-3472269-1307367982/solver-3472269-1307367982 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472269-1307367982.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 29.89 31.79 32.02 1/413 9653
/proc/meminfo: memFree=263674956/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=66732 CPUtime=0.03 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 4758 0 0 0 2 1 0 0 22 0 1 0 2185222 68333568 4688 270582939648 4194304 4964800 140735947022976 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 16683 4696 373 189 0 11557 0

[startup+0.089777 s]
/proc/loadavg: 29.89 31.79 32.02 1/413 9653
/proc/meminfo: memFree=263674956/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=87896 CPUtime=0.08 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 15237 0 0 0 5 3 0 0 22 0 1 0 2185222 90005504 14288 270582939648 4194304 4964800 140735947022976 18446744073709551615 4679091 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 21974 14299 375 189 0 16848 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 87896

[startup+0.100753 s]
/proc/loadavg: 29.89 31.79 32.02 1/413 9653
/proc/meminfo: memFree=263674956/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=87896 CPUtime=0.09 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 18447 0 0 0 6 3 0 0 22 0 1 0 2185222 90005504 16214 270582939648 4194304 4964800 140735947022976 18446744073709551615 4679091 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 21974 16223 374 189 0 16848 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 87896

[startup+0.300767 s]
/proc/loadavg: 29.89 31.79 32.02 1/413 9653
/proc/meminfo: memFree=263674956/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=118604 CPUtime=0.29 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 29849 0 0 0 24 5 0 0 23 0 1 0 2185222 121450496 24287 270582939648 4194304 4964800 140735947022976 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 29651 24289 386 189 0 24525 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 118604

[startup+0.70077 s]
/proc/loadavg: 29.89 31.79 32.02 1/413 9653
/proc/meminfo: memFree=263674956/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=185764 CPUtime=0.69 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 52952 0 0 0 60 9 0 0 25 0 1 0 2185222 190222336 40356 270582939648 4194304 4964800 140735947022976 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 46441 40356 396 189 0 41315 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 185764

[startup+1.5008 s]
/proc/loadavg: 29.89 31.79 32.02 2/414 9654
/proc/meminfo: memFree=263489836/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=216068 CPUtime=1.49 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 60072 0 0 0 138 11 0 0 25 0 1 0 2185222 221253632 47476 270582939648 4194304 4964800 140735947022976 18446744073709551615 4600585 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 54017 47476 401 189 0 48891 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 216068

[startup+3.10086 s]
/proc/loadavg: 29.89 31.79 32.02 2/414 9654
/proc/meminfo: memFree=263470104/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=251612 CPUtime=3.09 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 68911 0 0 0 296 13 0 0 25 0 1 0 2185222 257650688 56315 270582939648 4194304 4964800 140735947022976 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 62903 56315 401 189 0 57777 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 251612

[startup+6.30099 s]
/proc/loadavg: 27.58 31.28 31.85 2/414 9654
/proc/meminfo: memFree=263438160/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=265792 CPUtime=6.29 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 72446 0 0 0 615 14 0 0 25 0 1 0 2185222 272171008 59850 270582939648 4194304 4964800 140735947022976 18446744073709551615 4601745 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/9653/statm: 66448 59851 401 189 0 61322 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 265792

[startup+12.7022 s]
/proc/loadavg: 27.77 31.26 31.84 6/445 9685
/proc/meminfo: memFree=261903152/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=2915272 CPUtime=58.07 cores=0-31
/proc/9653/stat : 9653 (claspmt) S 9651 9653 9627 0 -1 4202496 552621 0 0 0 1917 3890 0 0 18 0 32 0 2185222 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 21 0 0 0
/proc/9653/statm: 728818 445655 435 189 0 723167 0
[pid=9653/tid=9655] ppid=9651 vsize=2915272 CPUtime=1.54 cores=0-31
/proc/9653/task/9655/stat : 9655 (claspmt) S 9651 9653 9627 0 -1 4202560 13598 0 0 0 21 133 0 0 16 0 32 0 2185999 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709295783392 0 0 -1 1 0 0 0
[pid=9653/tid=9656] ppid=9651 vsize=2915272 CPUtime=1.61 cores=0-31
/proc/9653/task/9656/stat : 9656 (claspmt) S 9651 9653 9627 0 -1 4202560 16269 0 0 0 28 133 0 0 16 0 32 0 2185999 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709305703904 0 0 -1 2 0 0 0
[pid=9653/tid=9657] ppid=9651 vsize=2915272 CPUtime=1.51 cores=0-31
/proc/9653/task/9657/stat : 9657 (claspmt) S 9651 9653 9627 0 -1 4202560 17377 0 0 0 26 125 0 0 16 0 32 0 2185999 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709618220512 0 0 -1 26 0 0 0
[pid=9653/tid=9658] ppid=9651 vsize=2915272 CPUtime=1.43 cores=0-31
/proc/9653/task/9658/stat : 9658 (claspmt) S 9651 9653 9627 0 -1 4202560 13767 0 0 0 17 126 0 0 15 0 32 0 2185999 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604711319248352 0 0 -1 4 0 0 0
[pid=9653/tid=9659] ppid=9651 vsize=2915272 CPUtime=2.41 cores=0-31
/proc/9653/task/9659/stat : 9659 (claspmt) R 9651 9653 9627 0 -1 4202560 15260 0 0 0 108 133 0 0 17 0 32 0 2185999 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670363 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=9653/tid=9660] ppid=9651 vsize=2915272 CPUtime=1.6 cores=0-31
/proc/9653/task/9660/stat : 9660 (claspmt) S 9651 9653 9627 0 -1 4202560 17255 0 0 0 29 131 0 0 16 0 32 0 2185999 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569123290592 0 0 -1 5 0 0 0
[pid=9653/tid=9661] ppid=9651 vsize=2915272 CPUtime=1.6 cores=0-31
/proc/9653/task/9661/stat : 9661 (claspmt) S 9651 9653 9627 0 -1 4202560 15705 0 0 0 30 130 0 0 16 0 32 0 2185999 2985238528 445655 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709295709664 0 0 -1 14 0 0 0
[pid=9653/tid=9662] ppid=9651 vsize=2915272 CPUtime=1.19 cores=0-31
/proc/9653/task/9662/stat : 9662 (claspmt) S 9651 9653 9627 0 -1 4202560 14316 0 0 0 21 98 0 0 15 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569004367328 0 0 -1 27 0 0 0
[pid=9653/tid=9663] ppid=9651 vsize=2915272 CPUtime=1.47 cores=0-31
/proc/9653/task/9663/stat : 9663 (claspmt) S 9651 9653 9627 0 -1 4202560 13947 0 0 0 18 129 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604712308006368 0 0 -1 6 0 0 0
[pid=9653/tid=9664] ppid=9651 vsize=2915272 CPUtime=1.54 cores=0-31
/proc/9653/task/9664/stat : 9664 (claspmt) S 9651 9653 9627 0 -1 4202560 16252 0 0 0 25 129 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569067372000 0 0 -1 30 0 0 0
[pid=9653/tid=9665] ppid=9651 vsize=2915272 CPUtime=1.56 cores=0-31
/proc/9653/task/9665/stat : 9665 (claspmt) S 9651 9653 9627 0 -1 4202560 17138 0 0 0 25 131 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709793553888 0 0 -1 10 0 0 0
[pid=9653/tid=9666] ppid=9651 vsize=2915272 CPUtime=1.49 cores=0-31
/proc/9653/task/9666/stat : 9666 (claspmt) S 9651 9653 9627 0 -1 4202560 13323 0 0 0 20 129 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709664841184 0 0 -1 3 0 0 0
[pid=9653/tid=9667] ppid=9651 vsize=2915272 CPUtime=2.34 cores=0-31
/proc/9653/task/9667/stat : 9667 (claspmt) R 9651 9653 9627 0 -1 4202560 14288 0 0 0 96 138 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=9653/tid=9668] ppid=9651 vsize=2915272 CPUtime=1.58 cores=0-31
/proc/9653/task/9668/stat : 9668 (claspmt) S 9651 9653 9627 0 -1 4202560 17680 0 0 0 30 128 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709618228704 0 0 -1 9 0 0 0
[pid=9653/tid=9669] ppid=9651 vsize=2915272 CPUtime=1.62 cores=0-31
/proc/9653/task/9669/stat : 9669 (claspmt) S 9651 9653 9627 0 -1 4202560 15759 0 0 0 30 132 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604505213320672 0 0 -1 16 0 0 0
[pid=9653/tid=9670] ppid=9651 vsize=2915272 CPUtime=1.39 cores=0-31
/proc/9653/task/9670/stat : 9670 (claspmt) S 9651 9653 9627 0 -1 4202560 13976 0 0 0 20 119 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569102400992 0 0 -1 7 0 0 0
[pid=9653/tid=9671] ppid=9651 vsize=2915272 CPUtime=1.43 cores=0-31
/proc/9653/task/9671/stat : 9671 (claspmt) S 9651 9653 9627 0 -1 4202560 13963 0 0 0 20 123 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604504221973984 0 0 -1 13 0 0 0
[pid=9653/tid=9672] ppid=9651 vsize=2915272 CPUtime=1.48 cores=0-31
/proc/9653/task/9672/stat : 9672 (claspmt) S 9651 9653 9627 0 -1 4202560 17865 0 0 0 31 117 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604505607552480 0 0 -1 23 0 0 0
[pid=9653/tid=9673] ppid=9651 vsize=2915272 CPUtime=1.55 cores=0-31
/proc/9653/task/9673/stat : 9673 (claspmt) S 9651 9653 9627 0 -1 4202560 17480 0 0 0 25 130 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709643042272 0 0 -1 20 0 0 0
[pid=9653/tid=9674] ppid=9651 vsize=2915272 CPUtime=1.31 cores=0-31
/proc/9653/task/9674/stat : 9674 (claspmt) S 9651 9653 9627 0 -1 4202560 13923 0 0 0 19 112 0 0 15 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709689261536 0 0 -1 7 0 0 0
[pid=9653/tid=9675] ppid=9651 vsize=2915272 CPUtime=2.37 cores=0-31
/proc/9653/task/9675/stat : 9675 (claspmt) R 9651 9653 9627 0 -1 4202560 15263 0 0 0 110 127 0 0 16 0 32 0 2185999 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=9653/tid=9676] ppid=9651 vsize=2915272 CPUtime=1.57 cores=0-31
/proc/9653/task/9676/stat : 9676 (claspmt) S 9651 9653 9627 0 -1 4202560 17040 0 0 0 28 129 0 0 16 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569088769504 0 0 -1 29 0 0 0
[pid=9653/tid=9677] ppid=9651 vsize=2915272 CPUtime=1.61 cores=0-31
/proc/9653/task/9677/stat : 9677 (claspmt) S 9651 9653 9627 0 -1 4202560 15879 0 0 0 37 124 0 0 16 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709845548512 0 0 -1 17 0 0 0
[pid=9653/tid=9678] ppid=9651 vsize=2915272 CPUtime=1.42 cores=0-31
/proc/9653/task/9678/stat : 9678 (claspmt) S 9651 9653 9627 0 -1 4202560 13991 0 0 0 16 126 0 0 15 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709295865312 0 0 -1 24 0 0 0
[pid=9653/tid=9679] ppid=9651 vsize=2915272 CPUtime=1.35 cores=0-31
/proc/9653/task/9679/stat : 9679 (claspmt) S 9651 9653 9627 0 -1 4202560 13732 0 0 0 18 117 0 0 15 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709305408992 0 0 -1 25 0 0 0
[pid=9653/tid=9680] ppid=9651 vsize=2915272 CPUtime=1.55 cores=0-31
/proc/9653/task/9680/stat : 9680 (claspmt) S 9651 9653 9627 0 -1 4202560 17472 0 0 0 22 133 0 0 16 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709305499104 0 0 -1 8 0 0 0
[pid=9653/tid=9681] ppid=9651 vsize=2915272 CPUtime=1.56 cores=0-31
/proc/9653/task/9681/stat : 9681 (claspmt) S 9651 9653 9627 0 -1 4202560 17550 0 0 0 27 129 0 0 16 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604505630596576 0 0 -1 18 0 0 0
[pid=9653/tid=9682] ppid=9651 vsize=2915272 CPUtime=1.45 cores=0-31
/proc/9653/task/9682/stat : 9682 (claspmt) S 9651 9653 9627 0 -1 4202560 14074 0 0 0 21 124 0 0 16 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604710535331296 0 0 -1 22 0 0 0
[pid=9653/tid=9683] ppid=9651 vsize=2915272 CPUtime=2.64 cores=0-31
/proc/9653/task/9683/stat : 9683 (claspmt) R 9651 9653 9627 0 -1 4202560 14352 0 0 0 150 114 0 0 16 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 4342157 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=9653/tid=9684] ppid=9651 vsize=2915272 CPUtime=1.18 cores=0-31
/proc/9653/task/9684/stat : 9684 (claspmt) S 9651 9653 9627 0 -1 4202560 12820 0 0 0 40 78 0 0 18 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569102564832 0 0 -1 11 0 0 0
[pid=9653/tid=9685] ppid=9651 vsize=2915272 CPUtime=1.61 cores=0-31
/proc/9653/task/9685/stat : 9685 (claspmt) S 9651 9653 9627 0 -1 4202560 15629 0 0 0 31 130 0 0 16 0 32 0 2186000 2985238528 445656 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709312339424 0 0 -1 21 0 0 0
Current children cumulated CPU time (s) 58.07
Current children cumulated vsize (KiB) 2915272

[startup+25.5017 s]
/proc/loadavg: 22.50 29.92 31.39 5/445 9685
/proc/meminfo: memFree=261895564/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=2915272 CPUtime=109.25 cores=0-31
/proc/9653/stat : 9653 (claspmt) S 9651 9653 9627 0 -1 4202496 553194 0 0 0 7034 3891 0 0 18 0 32 0 2185222 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 21 0 0 0
/proc/9653/statm: 728818 446228 435 189 0 723167 0
[pid=9653/tid=9655] ppid=9651 vsize=2915272 CPUtime=1.54 cores=0-31
/proc/9653/task/9655/stat : 9655 (claspmt) S 9651 9653 9627 0 -1 4202560 13598 0 0 0 21 133 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709295783392 0 0 -1 1 0 0 0
[pid=9653/tid=9656] ppid=9651 vsize=2915272 CPUtime=1.61 cores=0-31
/proc/9653/task/9656/stat : 9656 (claspmt) S 9651 9653 9627 0 -1 4202560 16269 0 0 0 28 133 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709305703904 0 0 -1 2 0 0 0
[pid=9653/tid=9657] ppid=9651 vsize=2915272 CPUtime=1.51 cores=0-31
/proc/9653/task/9657/stat : 9657 (claspmt) S 9651 9653 9627 0 -1 4202560 17377 0 0 0 26 125 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709618220512 0 0 -1 26 0 0 0
[pid=9653/tid=9658] ppid=9651 vsize=2915272 CPUtime=1.43 cores=0-31
/proc/9653/task/9658/stat : 9658 (claspmt) S 9651 9653 9627 0 -1 4202560 13767 0 0 0 17 126 0 0 15 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604711319248352 0 0 -1 4 0 0 0
[pid=9653/tid=9659] ppid=9651 vsize=2915272 CPUtime=15.2 cores=0-31
/proc/9653/task/9659/stat : 9659 (claspmt) R 9651 9653 9627 0 -1 4202560 15406 0 0 0 1387 133 0 0 25 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=9653/tid=9660] ppid=9651 vsize=2915272 CPUtime=1.6 cores=0-31
/proc/9653/task/9660/stat : 9660 (claspmt) S 9651 9653 9627 0 -1 4202560 17255 0 0 0 29 131 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569123290592 0 0 -1 5 0 0 0
[pid=9653/tid=9661] ppid=9651 vsize=2915272 CPUtime=1.6 cores=0-31
/proc/9653/task/9661/stat : 9661 (claspmt) S 9651 9653 9627 0 -1 4202560 15705 0 0 0 30 130 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709295709664 0 0 -1 14 0 0 0
[pid=9653/tid=9662] ppid=9651 vsize=2915272 CPUtime=1.19 cores=0-31
/proc/9653/task/9662/stat : 9662 (claspmt) S 9651 9653 9627 0 -1 4202560 14316 0 0 0 21 98 0 0 15 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569004367328 0 0 -1 27 0 0 0
[pid=9653/tid=9663] ppid=9651 vsize=2915272 CPUtime=1.47 cores=0-31
/proc/9653/task/9663/stat : 9663 (claspmt) S 9651 9653 9627 0 -1 4202560 13947 0 0 0 18 129 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604712308006368 0 0 -1 6 0 0 0
[pid=9653/tid=9664] ppid=9651 vsize=2915272 CPUtime=1.54 cores=0-31
/proc/9653/task/9664/stat : 9664 (claspmt) S 9651 9653 9627 0 -1 4202560 16252 0 0 0 25 129 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569067372000 0 0 -1 30 0 0 0
[pid=9653/tid=9665] ppid=9651 vsize=2915272 CPUtime=1.56 cores=0-31
/proc/9653/task/9665/stat : 9665 (claspmt) S 9651 9653 9627 0 -1 4202560 17138 0 0 0 25 131 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709793553888 0 0 -1 10 0 0 0
[pid=9653/tid=9666] ppid=9651 vsize=2915272 CPUtime=1.49 cores=0-31
/proc/9653/task/9666/stat : 9666 (claspmt) S 9651 9653 9627 0 -1 4202560 13323 0 0 0 20 129 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709664841184 0 0 -1 3 0 0 0
[pid=9653/tid=9667] ppid=9651 vsize=2915272 CPUtime=15.14 cores=0-31
/proc/9653/task/9667/stat : 9667 (claspmt) R 9651 9653 9627 0 -1 4202560 14510 0 0 0 1375 139 0 0 25 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=9653/tid=9668] ppid=9651 vsize=2915272 CPUtime=1.58 cores=0-31
/proc/9653/task/9668/stat : 9668 (claspmt) S 9651 9653 9627 0 -1 4202560 17680 0 0 0 30 128 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604709618228704 0 0 -1 9 0 0 0
[pid=9653/tid=9669] ppid=9651 vsize=2915272 CPUtime=1.62 cores=0-31
/proc/9653/task/9669/stat : 9669 (claspmt) S 9651 9653 9627 0 -1 4202560 15759 0 0 0 30 132 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604505213320672 0 0 -1 16 0 0 0
[pid=9653/tid=9670] ppid=9651 vsize=2915272 CPUtime=1.39 cores=0-31
/proc/9653/task/9670/stat : 9670 (claspmt) S 9651 9653 9627 0 -1 4202560 13976 0 0 0 20 119 0 0 16 0 32 0 2185999 2985238528 446228 270582939648 4194304 4964800 140735947022976 18446744073709551615 258662772457 0 0 4096 18946 18446604569102400992 0 0 -1 7 0 0 0

################
# More data... #
################

[pid=9653/tid=9662] ppid=9651 vsize=6677372 CPUtime=4980.74 cores=0-31
/proc/9653/task/9662/stat : 9662 (claspmt) R 9651 9653 9627 0 -1 4202560 20507 0 0 0 497944 130 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=9653/tid=9663] ppid=9651 vsize=6677372 CPUtime=4980.98 cores=0-31
/proc/9653/task/9663/stat : 9663 (claspmt) R 9651 9653 9627 0 -1 4202560 20475 0 0 0 497939 159 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4660444 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=9653/tid=9664] ppid=9651 vsize=6677372 CPUtime=4981.05 cores=0-31
/proc/9653/task/9664/stat : 9664 (claspmt) R 9651 9653 9627 0 -1 4202560 86383 0 0 0 497674 431 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=9653/tid=9665] ppid=9651 vsize=6677372 CPUtime=4981.06 cores=0-31
/proc/9653/task/9665/stat : 9665 (claspmt) R 9651 9653 9627 0 -1 4202560 32188 0 0 0 497920 186 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=9653/tid=9666] ppid=9651 vsize=6677372 CPUtime=4981 cores=0-31
/proc/9653/task/9666/stat : 9666 (claspmt) R 9651 9653 9627 0 -1 4202560 58325 0 0 0 497791 309 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=9653/tid=9667] ppid=9651 vsize=6677372 CPUtime=5070.6 cores=0-31
/proc/9653/task/9667/stat : 9667 (claspmt) R 9651 9653 9627 0 -1 4202560 16379 0 0 0 506919 141 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=9653/tid=9668] ppid=9651 vsize=6677372 CPUtime=4981.04 cores=0-31
/proc/9653/task/9668/stat : 9668 (claspmt) R 9651 9653 9627 0 -1 4202560 33549 0 0 0 497916 188 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=9653/tid=9669] ppid=9651 vsize=6677372 CPUtime=4981.14 cores=0-31
/proc/9653/task/9669/stat : 9669 (claspmt) R 9651 9653 9627 0 -1 4202560 98143 0 0 0 497677 437 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=9653/tid=9670] ppid=9651 vsize=6677372 CPUtime=4980.92 cores=0-31
/proc/9653/task/9670/stat : 9670 (claspmt) R 9651 9653 9627 0 -1 4202560 24606 0 0 0 497920 172 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=9653/tid=9671] ppid=9651 vsize=6677372 CPUtime=4980.94 cores=0-31
/proc/9653/task/9671/stat : 9671 (claspmt) R 9651 9653 9627 0 -1 4202560 22149 0 0 0 497935 159 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=9653/tid=9672] ppid=9651 vsize=6677372 CPUtime=4981 cores=0-31
/proc/9653/task/9672/stat : 9672 (claspmt) R 9651 9653 9627 0 -1 4202560 95148 0 0 0 497736 364 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4342144 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=9653/tid=9673] ppid=9651 vsize=6677372 CPUtime=4980.53 cores=0-31
/proc/9653/task/9673/stat : 9673 (claspmt) R 9651 9653 9627 0 -1 4202560 32157 0 0 0 497875 178 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=9653/tid=9674] ppid=9651 vsize=6677372 CPUtime=4980.83 cores=0-31
/proc/9653/task/9674/stat : 9674 (claspmt) R 9651 9653 9627 0 -1 4202560 55116 0 0 0 497789 294 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=9653/tid=9675] ppid=9651 vsize=6677372 CPUtime=5068.22 cores=0-31
/proc/9653/task/9675/stat : 9675 (claspmt) R 9651 9653 9627 0 -1 4202560 17956 0 0 0 506636 186 0 0 25 0 32 0 2185999 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=9653/tid=9676] ppid=9651 vsize=6677372 CPUtime=4981.07 cores=0-31
/proc/9653/task/9676/stat : 9676 (claspmt) R 9651 9653 9627 0 -1 4202560 34687 0 0 0 497911 196 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=9653/tid=9677] ppid=9651 vsize=6677372 CPUtime=4981.12 cores=0-31
/proc/9653/task/9677/stat : 9677 (claspmt) R 9651 9653 9627 0 -1 4202560 107464 0 0 0 497678 434 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4669014 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=9653/tid=9678] ppid=9651 vsize=6677372 CPUtime=4980.75 cores=0-31
/proc/9653/task/9678/stat : 9678 (claspmt) R 9651 9653 9627 0 -1 4202560 22109 0 0 0 497917 158 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=9653/tid=9679] ppid=9651 vsize=6677372 CPUtime=4980.88 cores=0-31
/proc/9653/task/9679/stat : 9679 (claspmt) R 9651 9653 9627 0 -1 4202560 20311 0 0 0 497942 146 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=9653/tid=9680] ppid=9651 vsize=6677372 CPUtime=4979.37 cores=0-31
/proc/9653/task/9680/stat : 9680 (claspmt) R 9651 9653 9627 0 -1 4202560 88624 0 0 0 497544 393 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4660444 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=9653/tid=9681] ppid=9651 vsize=6677372 CPUtime=4981.06 cores=0-31
/proc/9653/task/9681/stat : 9681 (claspmt) R 9651 9653 9627 0 -1 4202560 37153 0 0 0 497899 207 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=9653/tid=9682] ppid=9651 vsize=6677372 CPUtime=4980.95 cores=0-31
/proc/9653/task/9682/stat : 9682 (claspmt) R 9651 9653 9627 0 -1 4202560 49893 0 0 0 497826 269 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=9653/tid=9683] ppid=9651 vsize=6677372 CPUtime=5063.8 cores=0-31
/proc/9653/task/9683/stat : 9683 (claspmt) R 9651 9653 9627 0 -1 4202560 17154 0 0 0 506261 119 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=9653/tid=9684] ppid=9651 vsize=6677372 CPUtime=4956.24 cores=0-31
/proc/9653/task/9684/stat : 9684 (claspmt) R 9651 9653 9627 0 -1 4202560 33700 0 0 0 495457 167 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=9653/tid=9685] ppid=9651 vsize=6677372 CPUtime=4865.07 cores=0-31
/proc/9653/task/9685/stat : 9685 (claspmt) R 9651 9653 9627 0 -1 4202560 48853 0 0 0 486234 273 0 0 25 0 32 0 2186000 6837628928 1394322 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 31 0 0 0
Current children cumulated CPU time (s) 159592
Current children cumulated vsize (KiB) 6677372



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5095.1 s]
/proc/loadavg: 32.18 32.16 32.08 34/445 9842
/proc/meminfo: memFree=258089520/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=6677372 CPUtime=160003 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 1502889 0 0 0 15992576 7692 0 0 25 0 32 0 2185222 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/9653/statm: 1669343 1395365 463 189 0 1663692 0
[pid=9653/tid=9655] ppid=9651 vsize=6677372 CPUtime=4993.68 cores=0-31
/proc/9653/task/9655/stat : 9655 (claspmt) R 9651 9653 9627 0 -1 4202560 21010 0 0 0 499201 167 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=9653/tid=9656] ppid=9651 vsize=6677372 CPUtime=4993.95 cores=0-31
/proc/9653/task/9656/stat : 9656 (claspmt) R 9651 9653 9627 0 -1 4202560 77708 0 0 0 498993 402 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=9653/tid=9657] ppid=9651 vsize=6677372 CPUtime=4993.87 cores=0-31
/proc/9653/task/9657/stat : 9657 (claspmt) R 9651 9653 9627 0 -1 4202560 30753 0 0 0 499211 176 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4363459 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=9653/tid=9658] ppid=9651 vsize=6677372 CPUtime=4993.8 cores=0-31
/proc/9653/task/9658/stat : 9658 (claspmt) R 9651 9653 9627 0 -1 4202560 56996 0 0 0 499073 307 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=9653/tid=9659] ppid=9651 vsize=6677372 CPUtime=5071.72 cores=0-31
/proc/9653/task/9659/stat : 9659 (claspmt) R 9651 9653 9627 0 -1 4202560 20061 0 0 0 507025 147 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=9653/tid=9660] ppid=9651 vsize=6677372 CPUtime=4993.96 cores=0-31
/proc/9653/task/9660/stat : 9660 (claspmt) R 9651 9653 9627 0 -1 4202560 35124 0 0 0 499196 200 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=9653/tid=9661] ppid=9651 vsize=6677372 CPUtime=4993.94 cores=0-31
/proc/9653/task/9661/stat : 9661 (claspmt) R 9651 9653 9627 0 -1 4202560 98493 0 0 0 498906 488 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=9653/tid=9662] ppid=9651 vsize=6677372 CPUtime=4993.58 cores=0-31
/proc/9653/task/9662/stat : 9662 (claspmt) R 9651 9653 9627 0 -1 4202560 20507 0 0 0 499228 130 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=9653/tid=9663] ppid=9651 vsize=6677372 CPUtime=4993.82 cores=0-31
/proc/9653/task/9663/stat : 9663 (claspmt) R 9651 9653 9627 0 -1 4202560 20475 0 0 0 499223 159 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=9653/tid=9664] ppid=9651 vsize=6677372 CPUtime=4993.88 cores=0-31
/proc/9653/task/9664/stat : 9664 (claspmt) R 9651 9653 9627 0 -1 4202560 86386 0 0 0 498957 431 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=9653/tid=9665] ppid=9651 vsize=6677372 CPUtime=4993.9 cores=0-31
/proc/9653/task/9665/stat : 9665 (claspmt) R 9651 9653 9627 0 -1 4202560 32221 0 0 0 499204 186 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=9653/tid=9666] ppid=9651 vsize=6677372 CPUtime=4993.84 cores=0-31
/proc/9653/task/9666/stat : 9666 (claspmt) R 9651 9653 9627 0 -1 4202560 58325 0 0 0 499075 309 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668984 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=9653/tid=9667] ppid=9651 vsize=6677372 CPUtime=5083.44 cores=0-31
/proc/9653/task/9667/stat : 9667 (claspmt) R 9651 9653 9627 0 -1 4202560 16379 0 0 0 508203 141 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=9653/tid=9668] ppid=9651 vsize=6677372 CPUtime=4993.87 cores=0-31
/proc/9653/task/9668/stat : 9668 (claspmt) R 9651 9653 9627 0 -1 4202560 33559 0 0 0 499198 189 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=9653/tid=9669] ppid=9651 vsize=6677372 CPUtime=4993.98 cores=0-31
/proc/9653/task/9669/stat : 9669 (claspmt) R 9651 9653 9627 0 -1 4202560 98143 0 0 0 498961 437 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=9653/tid=9670] ppid=9651 vsize=6677372 CPUtime=4993.76 cores=0-31
/proc/9653/task/9670/stat : 9670 (claspmt) R 9651 9653 9627 0 -1 4202560 24606 0 0 0 499204 172 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=9653/tid=9671] ppid=9651 vsize=6677372 CPUtime=4993.78 cores=0-31
/proc/9653/task/9671/stat : 9671 (claspmt) R 9651 9653 9627 0 -1 4202560 22149 0 0 0 499219 159 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=9653/tid=9672] ppid=9651 vsize=6677372 CPUtime=4993.84 cores=0-31
/proc/9653/task/9672/stat : 9672 (claspmt) R 9651 9653 9627 0 -1 4202560 95148 0 0 0 499020 364 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=9653/tid=9673] ppid=9651 vsize=6677372 CPUtime=4993.37 cores=0-31
/proc/9653/task/9673/stat : 9673 (claspmt) R 9651 9653 9627 0 -1 4202560 32157 0 0 0 499159 178 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=9653/tid=9674] ppid=9651 vsize=6677372 CPUtime=4993.67 cores=0-31
/proc/9653/task/9674/stat : 9674 (claspmt) R 9651 9653 9627 0 -1 4202560 55170 0 0 0 499073 294 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=9653/tid=9675] ppid=9651 vsize=6677372 CPUtime=5081.06 cores=0-31
/proc/9653/task/9675/stat : 9675 (claspmt) R 9651 9653 9627 0 -1 4202560 18166 0 0 0 507919 187 0 0 25 0 32 0 2185999 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=9653/tid=9676] ppid=9651 vsize=6677372 CPUtime=4993.91 cores=0-31
/proc/9653/task/9676/stat : 9676 (claspmt) R 9651 9653 9627 0 -1 4202560 34702 0 0 0 499195 196 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=9653/tid=9677] ppid=9651 vsize=6677372 CPUtime=4993.97 cores=0-31
/proc/9653/task/9677/stat : 9677 (claspmt) R 9651 9653 9627 0 -1 4202560 107590 0 0 0 498962 435 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=9653/tid=9678] ppid=9651 vsize=6677372 CPUtime=4993.59 cores=0-31
/proc/9653/task/9678/stat : 9678 (claspmt) R 9651 9653 9627 0 -1 4202560 22109 0 0 0 499201 158 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=9653/tid=9679] ppid=9651 vsize=6677372 CPUtime=4993.72 cores=0-31
/proc/9653/task/9679/stat : 9679 (claspmt) R 9651 9653 9627 0 -1 4202560 20311 0 0 0 499226 146 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=9653/tid=9680] ppid=9651 vsize=6677372 CPUtime=4992.21 cores=0-31
/proc/9653/task/9680/stat : 9680 (claspmt) R 9651 9653 9627 0 -1 4202560 88630 0 0 0 498828 393 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=9653/tid=9681] ppid=9651 vsize=6677372 CPUtime=4993.9 cores=0-31
/proc/9653/task/9681/stat : 9681 (claspmt) R 9651 9653 9627 0 -1 4202560 37153 0 0 0 499183 207 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=9653/tid=9682] ppid=9651 vsize=6677372 CPUtime=4993.79 cores=0-31
/proc/9653/task/9682/stat : 9682 (claspmt) R 9651 9653 9627 0 -1 4202560 49902 0 0 0 499109 270 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4678036 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=9653/tid=9683] ppid=9651 vsize=6677372 CPUtime=5076.63 cores=0-31
/proc/9653/task/9683/stat : 9683 (claspmt) R 9651 9653 9627 0 -1 4202560 17277 0 0 0 507544 119 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=9653/tid=9684] ppid=9651 vsize=6677372 CPUtime=4969.08 cores=0-31
/proc/9653/task/9684/stat : 9684 (claspmt) R 9651 9653 9627 0 -1 4202560 33709 0 0 0 496741 167 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=9653/tid=9685] ppid=9651 vsize=6677372 CPUtime=4877.52 cores=0-31
/proc/9653/task/9685/stat : 9685 (claspmt) R 9651 9653 9627 0 -1 4202560 48854 0 0 0 487479 273 0 0 25 0 32 0 2186000 6837628928 1395365 270582939648 4194304 4964800 140735947022976 18446744073709551615 4668777 0 0 4096 18946 0 0 0 -1 31 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 6677372

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -9653

Solver just ended. Dumping a history of the last processes samples

[startup+5095.8 s]
/proc/loadavg: 32.18 32.16 32.08 34/445 9842
/proc/meminfo: memFree=258089520/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=6541672 CPUtime=160004 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 1502903 0 0 0 15992688 7692 0 0 25 0 1 0 2185222 6698672128 1383960 270582939648 4194304 4964800 140735947022976 18446744073709551615 258650605132 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/9653/statm: 1635418 1383960 474 189 0 1629767 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 6541672

[startup+5096.6 s]
/proc/loadavg: 32.18 32.16 32.08 2/415 9843
/proc/meminfo: memFree=258151016/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=6473960 CPUtime=160005 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 1502903 0 0 0 15992768 7693 0 0 25 0 1 0 2185222 6629335040 1367022 270582939648 4194304 4964800 140735947022976 18446744073709551615 4628834 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/9653/statm: 1618490 1367022 474 189 0 1612839 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 6473960

[startup+5097 s]
/proc/loadavg: 32.18 32.16 32.08 2/415 9843
/proc/meminfo: memFree=258151016/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=6457032 CPUtime=160005 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4202496 1502903 0 0 0 15992808 7693 0 0 25 0 1 0 2185222 6612000768 1362790 270582939648 4194304 4964800 140735947022976 18446744073709551615 258650612746 0 0 4096 18946 0 0 0 17 0 0 0 0
/proc/9653/statm: 1614258 1362790 474 189 0 1608607 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 6457032

[startup+5097.43 s]
/proc/loadavg: 32.18 32.16 32.08 2/414 9843
/proc/meminfo: memFree=260642552/264257624 swapFree=67111528/67111528
[pid=9653] ppid=9651 vsize=0 CPUtime=160005 cores=0-31
/proc/9653/stat : 9653 (claspmt) R 9651 9653 9627 0 -1 4203524 1502903 0 0 0 15992823 7721 0 0 25 0 1 0 2185222 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 0 0 0 0
/proc/9653/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5097.46
CPU time (s): 160005
CPU user time (s): 159928
CPU system time (s): 77.2383
CPU usage (%): 3138.93
Max. virtual memory (cumulated for all children) (KiB): 6677372

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159928
system time used= 77.2383
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1502903
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 412865
involuntary context switches= 376469

runsolver used 25.7311 second user time and 118.167 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-06 15:46:23
IDJOB=3472269
IDBENCH=20410
IDSOLVER=1961
FILE ID=node200/3472269-1307367982
RUNJOBID= node200-1307367982-9641
PBS_JOBID= 13509162
Free space on /tmp= 61720 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT07/industrial/palacios/hard/sortnet-8-ipc5-h19-sat.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472269-1307367982/watcher-3472269-1307367982 -o /tmp/evaluation-result-3472269-1307367982/solver-3472269-1307367982 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472269-1307367982.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= 409f4401e9e8b60ee665ac5567a13e41
RANDOM SEED=589145622

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      263675352 kB
Buffers:         35012 kB
Cached:         340584 kB
SwapCached:          0 kB
Active:         161992 kB
Inactive:       276668 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      263675352 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           23860 kB
Writeback:           0 kB
AnonPages:       63112 kB
Mapped:          14472 kB
Slab:            64512 kB
PageTables:       4404 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   160156 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61720 MiB
End job on node200 at 2011-06-06 17:11:21