Trace number 3472343

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 160006 5009.94

General information on the benchmark

NameAPPLICATIONS/satComp09_BioInstances/
rbcl_xits_09_UNKNOWN.cnf
MD5SUM18ad1dee1e9d863458e75a20545ae68a
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables1430
Number of clauses79453
Sum of the clauses size236162
Maximum clause length10
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22358
Number of clauses of size 377072
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 523

Solver Data 

0.00/0.01	c clasp version 2.0.0-RC2
0.00/0.01	c Reading from ...498167/instance-3472343-1307498167.cnf
1.19/1.22	c Solving...
160002.20/5007.73	
160002.20/5007.73	*** INTERRUPTED! ***
160002.20/5007.73	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472343-1307498167/watcher-3472343-1307498167 -o /tmp/evaluation-result-3472343-1307498167/solver-3472343-1307498167 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472343-1307498167.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 27.19 31.03 31.74 1/413 19205
/proc/meminfo: memFree=263127112/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=23260 CPUtime=0.02 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 1183 0 0 0 2 0 0 0 21 0 1 0 15203579 23818240 1012 270582939648 4194304 4964800 140737373750432 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/19205/statm: 5815 1014 383 189 0 689 0

[startup+0.0647721 s]
/proc/loadavg: 27.19 31.03 31.74 1/413 19205
/proc/meminfo: memFree=263127112/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=25912 CPUtime=0.05 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 2054 0 0 0 5 0 0 0 21 0 1 0 15203579 26533888 1668 270582939648 4194304 4964800 140737373750432 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/19205/statm: 6478 1668 394 189 0 1352 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 25912

[startup+0.100772 s]
/proc/loadavg: 27.19 31.03 31.74 1/413 19205
/proc/meminfo: memFree=263127112/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=27640 CPUtime=0.09 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 2502 0 0 0 9 0 0 0 21 0 1 0 15203579 28303360 2116 270582939648 4194304 4964800 140737373750432 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/19205/statm: 6910 2116 396 189 0 1784 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27640

[startup+0.300746 s]
/proc/loadavg: 27.19 31.03 31.74 1/413 19205
/proc/meminfo: memFree=263127112/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=27640 CPUtime=0.29 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 2502 0 0 0 29 0 0 0 22 0 1 0 15203579 28303360 2116 270582939648 4194304 4964800 140737373750432 18446744073709551615 4607590 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/19205/statm: 6910 2116 396 189 0 1784 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 27640

[startup+0.700745 s]
/proc/loadavg: 27.19 31.03 31.74 1/413 19205
/proc/meminfo: memFree=263127112/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=28300 CPUtime=0.69 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 2656 0 0 0 69 0 0 0 23 0 1 0 15203579 28979200 2270 270582939648 4194304 4964800 140737373750432 18446744073709551615 4600357 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/19205/statm: 7075 2270 397 189 0 1949 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 28300

[startup+1.50066 s]
/proc/loadavg: 27.19 31.03 31.74 2/414 19206
/proc/meminfo: memFree=263118408/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=1081132 CPUtime=3.93 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 12762 0 0 0 359 34 0 0 17 0 32 0 15203579 1107079168 11950 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 10 0 0 0
/proc/19205/statm: 270283 11951 458 189 0 264632 0
Current children cumulated CPU time (s) 3.93
Current children cumulated vsize (KiB) 1081132

[startup+3.10172 s]
/proc/loadavg: 27.19 31.03 31.74 33/445 19239
/proc/meminfo: memFree=263053092/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=1088460 CPUtime=54.43 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 35555 0 0 0 5343 100 0 0 18 0 32 0 15203579 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4660236 0 0 4096 18946 18446744073709551615 0 0 17 10 0 0 0
/proc/19205/statm: 272115 34743 461 189 0 266464 0
[pid=19205/tid=19209] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19209/stat : 19209 (claspmt) R 19203 19205 19179 0 -1 4202560 891 0 0 0 162 2 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627335 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=19205/tid=19210] ppid=19203 vsize=1088460 CPUtime=1.63 cores=0-31
/proc/19205/task/19210/stat : 19210 (claspmt) R 19203 19205 19179 0 -1 4202560 1584 0 0 0 159 4 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19205/tid=19211] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19211/stat : 19211 (claspmt) R 19203 19205 19179 0 -1 4202560 855 0 0 0 162 2 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668998 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19205/tid=19212] ppid=19203 vsize=1088460 CPUtime=1.65 cores=0-31
/proc/19205/task/19212/stat : 19212 (claspmt) R 19203 19205 19179 0 -1 4202560 804 0 0 0 164 1 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627636 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=19205/tid=19213] ppid=19203 vsize=1088460 CPUtime=1.76 cores=0-31
/proc/19205/task/19213/stat : 19213 (claspmt) R 19203 19205 19179 0 -1 4202560 1194 0 0 0 174 2 0 0 19 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19205/tid=19214] ppid=19203 vsize=1088460 CPUtime=1.66 cores=0-31
/proc/19205/task/19214/stat : 19214 (claspmt) R 19203 19205 19179 0 -1 4202560 843 0 0 0 164 2 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627465 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=19205/tid=19215] ppid=19203 vsize=1088460 CPUtime=1.63 cores=0-31
/proc/19205/task/19215/stat : 19215 (claspmt) R 19203 19205 19179 0 -1 4202560 989 0 0 0 161 2 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=19205/tid=19216] ppid=19203 vsize=1088460 CPUtime=1.65 cores=0-31
/proc/19205/task/19216/stat : 19216 (claspmt) R 19203 19205 19179 0 -1 4202560 807 0 0 0 163 2 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627290 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=19205/tid=19217] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19217/stat : 19217 (claspmt) R 19203 19205 19179 0 -1 4202560 934 0 0 0 162 2 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19205/tid=19218] ppid=19203 vsize=1088460 CPUtime=1.62 cores=0-31
/proc/19205/task/19218/stat : 19218 (claspmt) R 19203 19205 19179 0 -1 4202560 1606 0 0 0 159 3 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668675 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=19205/tid=19219] ppid=19203 vsize=1088460 CPUtime=1.66 cores=0-31
/proc/19205/task/19219/stat : 19219 (claspmt) R 19203 19205 19179 0 -1 4202560 857 0 0 0 164 2 0 0 18 0 32 0 15203702 1114583040 34743 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=19205/tid=19220] ppid=19203 vsize=1088460 CPUtime=1.66 cores=0-31
/proc/19205/task/19220/stat : 19220 (claspmt) R 19203 19205 19179 0 -1 4202560 838 0 0 0 163 3 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=19205/tid=19221] ppid=19203 vsize=1088460 CPUtime=1.76 cores=0-31
/proc/19205/task/19221/stat : 19221 (claspmt) R 19203 19205 19179 0 -1 4202560 1182 0 0 0 174 2 0 0 19 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19205/tid=19222] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19222/stat : 19222 (claspmt) R 19203 19205 19179 0 -1 4202560 820 0 0 0 162 2 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668732 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=19205/tid=19223] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19223/stat : 19223 (claspmt) R 19203 19205 19179 0 -1 4202560 982 0 0 0 162 2 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670330 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=19205/tid=19224] ppid=19203 vsize=1088460 CPUtime=1.66 cores=0-31
/proc/19205/task/19224/stat : 19224 (claspmt) R 19203 19205 19179 0 -1 4202560 804 0 0 0 164 2 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4660419 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19205/tid=19225] ppid=19203 vsize=1088460 CPUtime=1.65 cores=0-31
/proc/19205/task/19225/stat : 19225 (claspmt) R 19203 19205 19179 0 -1 4202560 858 0 0 0 162 3 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668698 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=19205/tid=19226] ppid=19203 vsize=1088460 CPUtime=1.63 cores=0-31
/proc/19205/task/19226/stat : 19226 (claspmt) R 19203 19205 19179 0 -1 4202560 1628 0 0 0 161 2 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 258650652667 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=19205/tid=19227] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19227/stat : 19227 (claspmt) R 19203 19205 19179 0 -1 4202560 894 0 0 0 162 2 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=19205/tid=19228] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19228/stat : 19228 (claspmt) R 19203 19205 19179 0 -1 4202560 913 0 0 0 161 3 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668681 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=19205/tid=19229] ppid=19203 vsize=1088460 CPUtime=1.74 cores=0-31
/proc/19205/task/19229/stat : 19229 (claspmt) R 19203 19205 19179 0 -1 4202560 1481 0 0 0 170 4 0 0 18 0 32 0 15203702 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=19205/tid=19230] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19230/stat : 19230 (claspmt) R 19203 19205 19179 0 -1 4202560 847 0 0 0 161 3 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670335 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=19205/tid=19231] ppid=19203 vsize=1088460 CPUtime=1.65 cores=0-31
/proc/19205/task/19231/stat : 19231 (claspmt) R 19203 19205 19179 0 -1 4202560 979 0 0 0 162 3 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4675752 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=19205/tid=19232] ppid=19203 vsize=1088460 CPUtime=1.65 cores=0-31
/proc/19205/task/19232/stat : 19232 (claspmt) R 19203 19205 19179 0 -1 4202560 823 0 0 0 163 2 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=19205/tid=19233] ppid=19203 vsize=1088460 CPUtime=1.65 cores=0-31
/proc/19205/task/19233/stat : 19233 (claspmt) R 19203 19205 19179 0 -1 4202560 943 0 0 0 162 3 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=19205/tid=19234] ppid=19203 vsize=1088460 CPUtime=1.63 cores=0-31
/proc/19205/task/19234/stat : 19234 (claspmt) R 19203 19205 19179 0 -1 4202560 1341 0 0 0 159 4 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=19205/tid=19235] ppid=19203 vsize=1088460 CPUtime=1.62 cores=0-31
/proc/19205/task/19235/stat : 19235 (claspmt) R 19203 19205 19179 0 -1 4202560 897 0 0 0 159 3 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19205/tid=19236] ppid=19203 vsize=1088460 CPUtime=1.6 cores=0-31
/proc/19205/task/19236/stat : 19236 (claspmt) R 19203 19205 19179 0 -1 4202560 961 0 0 0 157 3 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=19205/tid=19237] ppid=19203 vsize=1088460 CPUtime=1.7 cores=0-31
/proc/19205/task/19237/stat : 19237 (claspmt) R 19203 19205 19179 0 -1 4202560 1014 0 0 0 168 2 0 0 18 0 32 0 15203703 1114583040 34746 270582939648 4194304 4964800 140737373750432 18446744073709551615 4321029 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=19205/tid=19238] ppid=19203 vsize=1088460 CPUtime=1.63 cores=0-31
/proc/19205/task/19238/stat : 19238 (claspmt) R 19203 19205 19179 0 -1 4202560 856 0 0 0 160 3 0 0 18 0 32 0 15203703 1114583040 34749 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=19205/tid=19239] ppid=19203 vsize=1088460 CPUtime=1.64 cores=0-31
/proc/19205/task/19239/stat : 19239 (claspmt) R 19203 19205 19179 0 -1 4202560 1048 0 0 0 160 4 0 0 18 0 32 0 15203703 1114583040 34749 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627257 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 54.43
Current children cumulated vsize (KiB) 1088460

[startup+6.30184 s]
/proc/loadavg: 27.58 31.05 31.74 33/445 19239
/proc/meminfo: memFree=262901384/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=1095920 CPUtime=156.44 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 60654 0 0 0 15488 156 0 0 20 0 32 0 15203579 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668712 0 0 4096 18946 18446744073709551615 0 0 17 10 0 0 0
/proc/19205/statm: 273980 59842 461 189 0 268329 0
[pid=19205/tid=19209] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19209/stat : 19209 (claspmt) R 19203 19205 19179 0 -1 4202560 1297 0 0 0 480 3 0 0 25 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=19205/tid=19210] ppid=19203 vsize=1095920 CPUtime=4.82 cores=0-31
/proc/19205/task/19210/stat : 19210 (claspmt) R 19203 19205 19179 0 -1 4202560 2975 0 0 0 475 7 0 0 19 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19205/tid=19211] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19211/stat : 19211 (claspmt) R 19203 19205 19179 0 -1 4202560 1451 0 0 0 479 4 0 0 25 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19205/tid=19212] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19212/stat : 19212 (claspmt) R 19203 19205 19179 0 -1 4202560 1547 0 0 0 481 2 0 0 19 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 258651061007 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=19205/tid=19213] ppid=19203 vsize=1095920 CPUtime=4.94 cores=0-31
/proc/19205/task/19213/stat : 19213 (claspmt) R 19203 19205 19179 0 -1 4202560 2409 0 0 0 490 4 0 0 21 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19205/tid=19214] ppid=19203 vsize=1095920 CPUtime=4.84 cores=0-31
/proc/19205/task/19214/stat : 19214 (claspmt) R 19203 19205 19179 0 -1 4202560 1681 0 0 0 481 3 0 0 20 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=19205/tid=19215] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19215/stat : 19215 (claspmt) R 19203 19205 19179 0 -1 4202560 1705 0 0 0 479 4 0 0 25 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627810 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=19205/tid=19216] ppid=19203 vsize=1095920 CPUtime=4.84 cores=0-31
/proc/19205/task/19216/stat : 19216 (claspmt) R 19203 19205 19179 0 -1 4202560 1279 0 0 0 481 3 0 0 20 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=19205/tid=19217] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19217/stat : 19217 (claspmt) R 19203 19205 19179 0 -1 4202560 1376 0 0 0 480 3 0 0 20 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19205/tid=19218] ppid=19203 vsize=1095920 CPUtime=4.81 cores=0-31
/proc/19205/task/19218/stat : 19218 (claspmt) R 19203 19205 19179 0 -1 4202560 3071 0 0 0 476 5 0 0 23 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=19205/tid=19219] ppid=19203 vsize=1095920 CPUtime=4.85 cores=0-31
/proc/19205/task/19219/stat : 19219 (claspmt) R 19203 19205 19179 0 -1 4202560 1448 0 0 0 482 3 0 0 21 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=19205/tid=19220] ppid=19203 vsize=1095920 CPUtime=4.86 cores=0-31
/proc/19205/task/19220/stat : 19220 (claspmt) R 19203 19205 19179 0 -1 4202560 1564 0 0 0 481 5 0 0 24 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=19205/tid=19221] ppid=19203 vsize=1095920 CPUtime=4.95 cores=0-31
/proc/19205/task/19221/stat : 19221 (claspmt) R 19203 19205 19179 0 -1 4202560 2294 0 0 0 491 4 0 0 23 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4642509 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19205/tid=19222] ppid=19203 vsize=1095920 CPUtime=4.84 cores=0-31
/proc/19205/task/19222/stat : 19222 (claspmt) R 19203 19205 19179 0 -1 4202560 1391 0 0 0 480 4 0 0 20 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 258651061007 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=19205/tid=19223] ppid=19203 vsize=1095920 CPUtime=4.84 cores=0-31
/proc/19205/task/19223/stat : 19223 (claspmt) R 19203 19205 19179 0 -1 4202560 1730 0 0 0 479 5 0 0 20 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=19205/tid=19224] ppid=19203 vsize=1095920 CPUtime=4.85 cores=0-31
/proc/19205/task/19224/stat : 19224 (claspmt) R 19203 19205 19179 0 -1 4202560 1097 0 0 0 482 3 0 0 18 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668763 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19205/tid=19225] ppid=19203 vsize=1095920 CPUtime=4.79 cores=0-31
/proc/19205/task/19225/stat : 19225 (claspmt) R 19203 19205 19179 0 -1 4202560 1344 0 0 0 475 4 0 0 20 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=19205/tid=19226] ppid=19203 vsize=1095920 CPUtime=4.82 cores=0-31
/proc/19205/task/19226/stat : 19226 (claspmt) R 19203 19205 19179 0 -1 4202560 3112 0 0 0 477 5 0 0 20 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=19205/tid=19227] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19227/stat : 19227 (claspmt) R 19203 19205 19179 0 -1 4202560 1714 0 0 0 480 3 0 0 19 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=19205/tid=19228] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19228/stat : 19228 (claspmt) R 19203 19205 19179 0 -1 4202560 1822 0 0 0 477 6 0 0 21 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4321059 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=19205/tid=19229] ppid=19203 vsize=1095920 CPUtime=4.94 cores=0-31
/proc/19205/task/19229/stat : 19229 (claspmt) R 19203 19205 19179 0 -1 4202560 2538 0 0 0 488 6 0 0 21 0 32 0 15203702 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670273 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=19205/tid=19230] ppid=19203 vsize=1095920 CPUtime=4.84 cores=0-31
/proc/19205/task/19230/stat : 19230 (claspmt) R 19203 19205 19179 0 -1 4202560 1349 0 0 0 480 4 0 0 23 0 32 0 15203703 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=19205/tid=19231] ppid=19203 vsize=1095920 CPUtime=4.83 cores=0-31
/proc/19205/task/19231/stat : 19231 (claspmt) R 19203 19205 19179 0 -1 4202560 1738 0 0 0 478 5 0 0 25 0 32 0 15203703 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668725 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=19205/tid=19232] ppid=19203 vsize=1095920 CPUtime=4.84 cores=0-31
/proc/19205/task/19232/stat : 19232 (claspmt) R 19203 19205 19179 0 -1 4202560 1431 0 0 0 481 3 0 0 19 0 32 0 15203703 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4342213 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=19205/tid=19233] ppid=19203 vsize=1095920 CPUtime=4.85 cores=0-31
/proc/19205/task/19233/stat : 19233 (claspmt) R 19203 19205 19179 0 -1 4202560 1250 0 0 0 481 4 0 0 20 0 32 0 15203703 1122222080 59842 270582939648 4194304 4964800 140737373750432 18446744073709551615 4653299 0 0 4096 18946 0 0 0 -1 29 0 0 0

################
# More data... #
################

/proc/19205/task/19223/stat : 19223 (claspmt) R 19203 19205 19179 0 -1 4202560 28519 0 0 0 495400 139 0 0 25 0 32 0 15203702 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=19205/tid=19224] ppid=19203 vsize=4100460 CPUtime=4958.89 cores=0-31
/proc/19205/task/19224/stat : 19224 (claspmt) R 19203 19205 19179 0 -1 4202560 7954 0 0 0 495820 69 0 0 25 0 32 0 15203702 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19205/tid=19225] ppid=19203 vsize=4100460 CPUtime=4957.59 cores=0-31
/proc/19205/task/19225/stat : 19225 (claspmt) R 19203 19205 19179 0 -1 4202560 32095 0 0 0 495703 56 0 0 25 0 32 0 15203702 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=19205/tid=19226] ppid=19203 vsize=4100460 CPUtime=4959.53 cores=0-31
/proc/19205/task/19226/stat : 19226 (claspmt) R 19203 19205 19179 0 -1 4202560 61445 0 0 0 495860 93 0 0 25 0 32 0 15203702 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19205/tid=19227] ppid=19203 vsize=4100460 CPUtime=4959.17 cores=0-31
/proc/19205/task/19227/stat : 19227 (claspmt) R 19203 19205 19179 0 -1 4202560 16172 0 0 0 495856 61 0 0 25 0 32 0 15203702 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=19205/tid=19228] ppid=19203 vsize=4100460 CPUtime=4959.4 cores=0-31
/proc/19205/task/19228/stat : 19228 (claspmt) R 19203 19205 19179 0 -1 4202560 35319 0 0 0 495845 95 0 0 25 0 32 0 15203702 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=19205/tid=19229] ppid=19203 vsize=4100460 CPUtime=4959.45 cores=0-31
/proc/19205/task/19229/stat : 19229 (claspmt) R 19203 19205 19179 0 -1 4202560 29608 0 0 0 495872 73 0 0 25 0 32 0 15203702 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19205/tid=19230] ppid=19203 vsize=4100460 CPUtime=4959.07 cores=0-31
/proc/19205/task/19230/stat : 19230 (claspmt) R 19203 19205 19179 0 -1 4202560 15179 0 0 0 495841 66 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=19205/tid=19231] ppid=19203 vsize=4100460 CPUtime=4958.27 cores=0-31
/proc/19205/task/19231/stat : 19231 (claspmt) R 19203 19205 19179 0 -1 4202560 29285 0 0 0 495690 137 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668698 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=19205/tid=19232] ppid=19203 vsize=4100460 CPUtime=4937.65 cores=0-31
/proc/19205/task/19232/stat : 19232 (claspmt) R 19203 19205 19179 0 -1 4202560 8997 0 0 0 493696 69 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=19205/tid=19233] ppid=19203 vsize=4100460 CPUtime=4959.17 cores=0-31
/proc/19205/task/19233/stat : 19233 (claspmt) R 19203 19205 19179 0 -1 4202560 30971 0 0 0 495856 61 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=19205/tid=19234] ppid=19203 vsize=4100460 CPUtime=4952.62 cores=0-31
/proc/19205/task/19234/stat : 19234 (claspmt) R 19203 19205 19179 0 -1 4202560 56632 0 0 0 495167 95 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=19205/tid=19235] ppid=19203 vsize=4100460 CPUtime=4949.18 cores=0-31
/proc/19205/task/19235/stat : 19235 (claspmt) R 19203 19205 19179 0 -1 4202560 17438 0 0 0 494854 64 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19205/tid=19236] ppid=19203 vsize=4100460 CPUtime=4958.41 cores=0-31
/proc/19205/task/19236/stat : 19236 (claspmt) R 19203 19205 19179 0 -1 4202560 31960 0 0 0 495762 79 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=19205/tid=19237] ppid=19203 vsize=4100460 CPUtime=4946.45 cores=0-31
/proc/19205/task/19237/stat : 19237 (claspmt) R 19203 19205 19179 0 -1 4202560 31938 0 0 0 494562 83 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=19205/tid=19238] ppid=19203 vsize=4100460 CPUtime=4959 cores=0-31
/proc/19205/task/19238/stat : 19238 (claspmt) R 19203 19205 19179 0 -1 4202560 15350 0 0 0 495828 72 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670423 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=19205/tid=19239] ppid=19203 vsize=4100460 CPUtime=4938.72 cores=0-31
/proc/19205/task/19239/stat : 19239 (claspmt) R 19203 19205 19179 0 -1 4202560 29021 0 0 0 493727 145 0 0 25 0 32 0 15203703 4198871040 828213 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 11 0 0 0
Current children cumulated CPU time (s) 158551
Current children cumulated vsize (KiB) 4100460



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.7 s]
/proc/loadavg: 32.13 32.09 32.03 33/445 19860
/proc/meminfo: memFree=259802188/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=4100460 CPUtime=160002 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 889687 0 0 0 15997531 2689 0 0 25 0 32 0 15203579 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4668734 0 0 4096 18946 18446744073709551615 0 0 17 14 0 0 0
/proc/19205/statm: 1025115 830440 463 189 0 1019464 0
[pid=19205/tid=19209] ppid=19203 vsize=4100460 CPUtime=5002.21 cores=0-31
/proc/19205/task/19209/stat : 19209 (claspmt) R 19203 19205 19179 0 -1 4202560 33559 0 0 0 500111 110 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=19205/tid=19210] ppid=19203 vsize=4100460 CPUtime=5004.62 cores=0-31
/proc/19205/task/19210/stat : 19210 (claspmt) R 19203 19205 19179 0 -1 4202560 55993 0 0 0 500371 91 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=19205/tid=19211] ppid=19203 vsize=4100460 CPUtime=4975.26 cores=0-31
/proc/19205/task/19211/stat : 19211 (claspmt) R 19203 19205 19179 0 -1 4202560 16051 0 0 0 497462 64 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=19205/tid=19212] ppid=19203 vsize=4100460 CPUtime=4990.96 cores=0-31
/proc/19205/task/19212/stat : 19212 (claspmt) R 19203 19205 19179 0 -1 4202560 32973 0 0 0 499002 94 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=19205/tid=19213] ppid=19203 vsize=4100460 CPUtime=5004.95 cores=0-31
/proc/19205/task/19213/stat : 19213 (claspmt) R 19203 19205 19179 0 -1 4202560 30097 0 0 0 500415 80 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=19205/tid=19214] ppid=19203 vsize=4100460 CPUtime=5004.27 cores=0-31
/proc/19205/task/19214/stat : 19214 (claspmt) R 19203 19205 19179 0 -1 4202560 14792 0 0 0 500363 64 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=19205/tid=19215] ppid=19203 vsize=4100460 CPUtime=4993.19 cores=0-31
/proc/19205/task/19215/stat : 19215 (claspmt) R 19203 19205 19179 0 -1 4202560 22645 0 0 0 499193 126 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=19205/tid=19216] ppid=19203 vsize=4100460 CPUtime=5004.32 cores=0-31
/proc/19205/task/19216/stat : 19216 (claspmt) R 19203 19205 19179 0 -1 4202560 9541 0 0 0 500368 64 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 258650653530 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=19205/tid=19217] ppid=19203 vsize=4100460 CPUtime=5004.57 cores=0-31
/proc/19205/task/19217/stat : 19217 (claspmt) R 19203 19205 19179 0 -1 4202560 31626 0 0 0 500404 53 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=19205/tid=19218] ppid=19203 vsize=4100460 CPUtime=5004.37 cores=0-31
/proc/19205/task/19218/stat : 19218 (claspmt) R 19203 19205 19179 0 -1 4202560 53924 0 0 0 500350 87 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=19205/tid=19219] ppid=19203 vsize=4100460 CPUtime=5004.62 cores=0-31
/proc/19205/task/19219/stat : 19219 (claspmt) R 19203 19205 19179 0 -1 4202560 18773 0 0 0 500394 68 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4644896 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=19205/tid=19220] ppid=19203 vsize=4100460 CPUtime=5004.8 cores=0-31
/proc/19205/task/19220/stat : 19220 (claspmt) R 19203 19205 19179 0 -1 4202560 33582 0 0 0 500395 85 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=19205/tid=19221] ppid=19203 vsize=4100460 CPUtime=5004.85 cores=0-31
/proc/19205/task/19221/stat : 19221 (claspmt) R 19203 19205 19179 0 -1 4202560 26595 0 0 0 500409 76 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=19205/tid=19222] ppid=19203 vsize=4100460 CPUtime=5003.91 cores=0-31
/proc/19205/task/19222/stat : 19222 (claspmt) R 19203 19205 19179 0 -1 4202560 17156 0 0 0 500326 65 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4659848 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=19205/tid=19223] ppid=19203 vsize=4100460 CPUtime=5000.79 cores=0-31
/proc/19205/task/19223/stat : 19223 (claspmt) R 19203 19205 19179 0 -1 4202560 28519 0 0 0 499940 139 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4320966 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=19205/tid=19224] ppid=19203 vsize=4100460 CPUtime=5004.29 cores=0-31
/proc/19205/task/19224/stat : 19224 (claspmt) R 19203 19205 19179 0 -1 4202560 7987 0 0 0 500360 69 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4321016 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=19205/tid=19225] ppid=19203 vsize=4100460 CPUtime=5003 cores=0-31
/proc/19205/task/19225/stat : 19225 (claspmt) R 19203 19205 19179 0 -1 4202560 32376 0 0 0 500243 57 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=19205/tid=19226] ppid=19203 vsize=4100460 CPUtime=5004.94 cores=0-31
/proc/19205/task/19226/stat : 19226 (claspmt) R 19203 19205 19179 0 -1 4202560 61445 0 0 0 500401 93 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=19205/tid=19227] ppid=19203 vsize=4100460 CPUtime=5004.58 cores=0-31
/proc/19205/task/19227/stat : 19227 (claspmt) R 19203 19205 19179 0 -1 4202560 16221 0 0 0 500396 62 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=19205/tid=19228] ppid=19203 vsize=4100460 CPUtime=5004.8 cores=0-31
/proc/19205/task/19228/stat : 19228 (claspmt) R 19203 19205 19179 0 -1 4202560 35322 0 0 0 500385 95 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=19205/tid=19229] ppid=19203 vsize=4100460 CPUtime=5004.87 cores=0-31
/proc/19205/task/19229/stat : 19229 (claspmt) R 19203 19205 19179 0 -1 4202560 29990 0 0 0 500414 73 0 0 25 0 32 0 15203702 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=19205/tid=19230] ppid=19203 vsize=4100460 CPUtime=5004.47 cores=0-31
/proc/19205/task/19230/stat : 19230 (claspmt) R 19203 19205 19179 0 -1 4202560 15233 0 0 0 500381 66 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=19205/tid=19231] ppid=19203 vsize=4100460 CPUtime=5003.68 cores=0-31
/proc/19205/task/19231/stat : 19231 (claspmt) R 19203 19205 19179 0 -1 4202560 29384 0 0 0 500231 137 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670295 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=19205/tid=19232] ppid=19203 vsize=4100460 CPUtime=4983.06 cores=0-31
/proc/19205/task/19232/stat : 19232 (claspmt) R 19203 19205 19179 0 -1 4202560 9032 0 0 0 498236 70 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=19205/tid=19233] ppid=19203 vsize=4100460 CPUtime=5004.59 cores=0-31
/proc/19205/task/19233/stat : 19233 (claspmt) R 19203 19205 19179 0 -1 4202560 31141 0 0 0 500397 62 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 258650653514 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=19205/tid=19234] ppid=19203 vsize=4100460 CPUtime=4996.74 cores=0-31
/proc/19205/task/19234/stat : 19234 (claspmt) R 19203 19205 19179 0 -1 4202560 56645 0 0 0 499579 95 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=19205/tid=19235] ppid=19203 vsize=4100460 CPUtime=4994.58 cores=0-31
/proc/19205/task/19235/stat : 19235 (claspmt) R 19203 19205 19179 0 -1 4202560 17438 0 0 0 499394 64 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=19205/tid=19236] ppid=19203 vsize=4100460 CPUtime=5003.8 cores=0-31
/proc/19205/task/19236/stat : 19236 (claspmt) R 19203 19205 19179 0 -1 4202560 31962 0 0 0 500300 80 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=19205/tid=19237] ppid=19203 vsize=4100460 CPUtime=4991.87 cores=0-31
/proc/19205/task/19237/stat : 19237 (claspmt) R 19203 19205 19179 0 -1 4202560 31980 0 0 0 499103 84 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=19205/tid=19238] ppid=19203 vsize=4100460 CPUtime=5004.42 cores=0-31
/proc/19205/task/19238/stat : 19238 (claspmt) R 19203 19205 19179 0 -1 4202560 15350 0 0 0 500369 73 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=19205/tid=19239] ppid=19203 vsize=4100460 CPUtime=4984.12 cores=0-31
/proc/19205/task/19239/stat : 19239 (claspmt) R 19203 19205 19179 0 -1 4202560 29021 0 0 0 498267 145 0 0 25 0 32 0 15203703 4198871040 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 258651061007 0 0 4096 18946 0 0 0 -1 11 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 4100460

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -19205

Solver just ended. Dumping a history of the last processes samples

[startup+5007.83 s]
/proc/loadavg: 32.13 32.09 32.03 33/445 19860
/proc/meminfo: memFree=259802188/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=4096360 CPUtime=160003 cores=0-31
/proc/19205/stat : 19205 (claspmt) S 19203 19205 19179 0 -1 4202496 889689 0 0 0 15997592 2690 0 0 25 0 3 0 15203579 4194672640 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 258662759221 0 0 4096 18946 18446744073709551615 0 0 17 14 0 0 0
/proc/19205/statm: 1024090 830440 465 189 0 1018439 0
[pid=19205/tid=19219] ppid=19203 vsize=4096360 CPUtime=5004.72 cores=0-31
/proc/19205/task/19219/stat : 19219 (claspmt) R 19203 19205 19179 0 -1 4202560 18773 0 0 0 500404 68 0 0 25 0 3 0 15203702 4194672640 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 258650653542 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=19205/tid=19233] ppid=19203 vsize=4096360 CPUtime=5004.69 cores=0-31
/proc/19205/task/19233/stat : 19233 (claspmt) R 19203 19205 19179 0 -1 4202560 31141 0 0 0 500407 62 0 0 25 0 3 0 15203703 4194672640 830440 270582939648 4194304 4964800 140737373750432 18446744073709551615 4644876 0 0 4096 18946 0 0 0 -1 29 0 0 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 4096360

[startup+5008.6 s]
/proc/loadavg: 32.13 32.09 32.03 3/417 19861
/proc/meminfo: memFree=259802520/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=4096360 CPUtime=160004 cores=0-31
/proc/19205/stat : 19205 (claspmt) S 19203 19205 19179 0 -1 4202496 889689 0 0 0 15997730 2690 0 0 25 0 2 0 15203579 4194672640 830302 270582939648 4194304 4964800 140737373750432 18446744073709551615 258662759221 0 0 4096 18946 18446744073709551615 0 0 17 14 0 0 0
/proc/19205/statm: 1024090 830302 465 189 0 1018439 0
[pid=19205/tid=19219] ppid=19203 vsize=4096360 CPUtime=5005.5 cores=0-31
/proc/19205/task/19219/stat : 19219 (claspmt) R 19203 19205 19179 0 -1 4202560 18773 0 0 0 500481 69 0 0 25 0 2 0 15203702 4194672640 830302 270582939648 4194304 4964800 140737373750432 18446744073709551615 4644900 0 0 4096 18946 0 0 0 -1 22 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 4096360

[startup+5009.43 s]
/proc/loadavg: 32.13 32.09 32.03 2/415 19861
/proc/meminfo: memFree=259807000/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=4000352 CPUtime=160005 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4202496 889698 0 0 0 15997813 2690 0 0 18 0 1 0 15203579 4096360448 828284 270582939648 4194304 4964800 140737373750432 18446744073709551615 258650612305 0 0 4096 18946 0 0 0 17 14 0 0 0
/proc/19205/statm: 1000088 828284 474 189 0 994437 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 4000352

[startup+5009.8 s]
/proc/loadavg: 32.13 32.09 32.03 2/415 19861
/proc/meminfo: memFree=259807000/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=0 CPUtime=160005 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4203524 889698 0 0 0 15997843 2697 0 0 19 0 1 0 15203579 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 14 0 0 0
/proc/19205/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

[startup+5009.9 s]
/proc/loadavg: 32.13 32.09 32.03 2/415 19861
/proc/meminfo: memFree=259807000/264257624 swapFree=67111528/67111528
[pid=19205] ppid=19203 vsize=0 CPUtime=160006 cores=0-31
/proc/19205/stat : 19205 (claspmt) R 19203 19205 19179 0 -1 4203524 889698 0 0 0 15997843 2707 0 0 19 0 1 0 15203579 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 14 0 0 0
/proc/19205/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160006
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5009.94
CPU time (s): 160006
CPU user time (s): 159978
CPU system time (s): 27.1189
CPU usage (%): 3193.76
Max. virtual memory (cumulated for all children) (KiB): 4100460

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159978
system time used= 27.1189
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 889698
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 718507
involuntary context switches= 392742

runsolver used 29.7565 second user time and 111.585 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-08 03:56:07
IDJOB=3472343
IDBENCH=71098
IDSOLVER=1961
FILE ID=node200/3472343-1307498167
RUNJOBID= node200-1307498166-19193
PBS_JOBID= 13509203
Free space on /tmp= 61744 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/APPLICATIONS/satComp09_BioInstances/rbcl_xits_09_UNKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472343-1307498167/watcher-3472343-1307498167 -o /tmp/evaluation-result-3472343-1307498167/solver-3472343-1307498167 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472343-1307498167.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= 18ad1dee1e9d863458e75a20545ae68a
RANDOM SEED=1234609466

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      263127392 kB
Buffers:        155620 kB
Cached:         713576 kB
SwapCached:          0 kB
Active:         183492 kB
Inactive:       748996 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      263127392 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            2444 kB
Writeback:           0 kB
AnonPages:       63908 kB
Mapped:          14404 kB
Slab:           120360 kB
PageTables:       4336 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   165008 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61740 MiB
End job on node200 at 2011-06-08 05:19:37