Trace number 3472375

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 21869.7 5102.42

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
transport-transport-city-sequential-35nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.050-SAT.cnf
MD5SUM48f03e5fbe31fc85b9a464480fc8aa5a
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark5148.61
Satisfiable
(Un)Satisfiability was proved
Number of variables1575206
Number of clauses8464620
Sum of the clauses size20118920
Maximum clause length982
Minimum clause length1
Number of clauses of size 1720
Number of clauses of size 28393300
Number of clauses of size 3400
Number of clauses of size 41600
Number of clauses of size 54000
Number of clauses of size over 564600

Solver Data 

0.00/0.01	c clasp version 2.0.0-RC2
0.00/0.01	c Reading from ...564451/instance-3472375-1307564451.cnf
3.59/3.60	c Solving...
21866.98/5100.02	
21866.98/5100.02	*** INTERRUPTED! ***
21866.98/5100.02	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472375-1307564451/watcher-3472375-1307564451 -o /tmp/evaluation-result-3472375-1307564451/solver-3472375-1307564451 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472375-1307564451.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 21.51 29.14 24.53 1/413 23861
/proc/meminfo: memFree=262563908/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=220860 CPUtime=0.03 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 1221 0 0 0 3 0 0 0 23 0 1 0 21832724 226160640 1150 270582939648 4194304 4964800 140736515963104 18446744073709551615 4686341 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/23861/statm: 55215 1151 372 189 0 50089 0

[startup+0.114373 s]
/proc/loadavg: 21.51 29.14 24.53 1/413 23861
/proc/meminfo: memFree=262563908/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=220860 CPUtime=0.1 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 7281 0 0 0 8 2 0 0 23 0 1 0 21832724 226160640 7210 270582939648 4194304 4964800 140736515963104 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/23861/statm: 55215 7215 373 189 0 50089 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 220860

[startup+0.201351 s]
/proc/loadavg: 21.51 29.14 24.53 1/413 23861
/proc/meminfo: memFree=262563908/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=220860 CPUtime=0.19 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 14318 0 0 0 15 4 0 0 24 0 1 0 21832724 226160640 14247 270582939648 4194304 4964800 140736515963104 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/23861/statm: 55215 14251 373 189 0 50089 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 220860

[startup+0.301357 s]
/proc/loadavg: 21.51 29.14 24.53 1/413 23861
/proc/meminfo: memFree=262563908/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=220860 CPUtime=0.29 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 22436 0 0 0 23 6 0 0 25 0 1 0 21832724 226160640 22365 270582939648 4194304 4964800 140736515963104 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/23861/statm: 55215 22370 373 189 0 50089 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 220860

[startup+0.700392 s]
/proc/loadavg: 21.51 29.14 24.53 1/413 23861
/proc/meminfo: memFree=262563908/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=313164 CPUtime=0.69 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 61651 0 0 0 52 17 0 0 25 0 1 0 21832724 320679936 58971 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/23861/statm: 78291 58981 374 189 0 73165 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 313164

[startup+1.5004 s]
/proc/loadavg: 19.86 28.68 24.41 2/414 23862
/proc/meminfo: memFree=262253864/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=352080 CPUtime=1.49 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 94170 0 0 0 123 26 0 0 25 0 1 0 21832724 360529920 81793 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650955760 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/23861/statm: 88020 81793 387 189 0 82894 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 352080

[startup+3.10045 s]
/proc/loadavg: 19.86 28.68 24.41 2/414 23862
/proc/meminfo: memFree=262208356/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=416968 CPUtime=3.09 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 110546 0 0 0 273 36 0 0 25 0 1 0 21832724 426975232 98011 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650955760 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/23861/statm: 104242 98011 387 189 0 99116 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 416968

[startup+6.30315 s]
/proc/loadavg: 20.76 28.72 24.44 32/445 23893
/proc/meminfo: memFree=260879556/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=3548200 CPUtime=85.82 cores=0-31
/proc/23861/stat : 23861 (claspmt) S 23859 23861 23835 0 -1 4202496 732561 0 0 0 483 8099 0 0 18 0 32 0 21832724 3633356800 511255 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/23861/statm: 887050 511256 424 189 0 881399 0
[pid=23861/tid=23863] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23863/stat : 23863 (claspmt) R 23859 23861 23835 0 -1 4202560 19435 0 0 0 4 261 0 0 22 0 32 0 21833086 3633356800 511266 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=23861/tid=23864] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23864/stat : 23864 (claspmt) R 23859 23861 23835 0 -1 4202560 28764 0 0 0 7 258 0 0 22 0 32 0 21833086 3633356800 511266 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=23861/tid=23865] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23865/stat : 23865 (claspmt) R 23859 23861 23835 0 -1 4202560 22697 0 0 0 7 258 0 0 22 0 32 0 21833086 3633356800 511266 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=23861/tid=23866] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23866/stat : 23866 (claspmt) R 23859 23861 23835 0 -1 4202560 18329 0 0 0 5 260 0 0 22 0 32 0 21833086 3633356800 511276 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23861/tid=23867] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23867/stat : 23867 (claspmt) R 23859 23861 23835 0 -1 4202560 20199 0 0 0 3 262 0 0 22 0 32 0 21833086 3633356800 511276 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=23861/tid=23868] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23868/stat : 23868 (claspmt) R 23859 23861 23835 0 -1 4202560 9014 0 0 0 3 263 0 0 22 0 32 0 21833086 3633356800 511276 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=23861/tid=23869] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23869/stat : 23869 (claspmt) R 23859 23861 23835 0 -1 4202560 17704 0 0 0 4 262 0 0 22 0 32 0 21833086 3633356800 511276 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=23861/tid=23870] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23870/stat : 23870 (claspmt) R 23859 23861 23835 0 -1 4202560 17467 0 0 0 5 260 0 0 22 0 32 0 21833086 3633356800 511276 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=23861/tid=23871] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23871/stat : 23871 (claspmt) R 23859 23861 23835 0 -1 4202560 18937 0 0 0 4 262 0 0 22 0 32 0 21833086 3633356800 511276 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=23861/tid=23872] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23872/stat : 23872 (claspmt) R 23859 23861 23835 0 -1 4202560 20279 0 0 0 8 257 0 0 22 0 32 0 21833086 3633356800 511288 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23861/tid=23873] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23873/stat : 23873 (claspmt) R 23859 23861 23835 0 -1 4202560 18134 0 0 0 5 261 0 0 22 0 32 0 21833086 3633356800 511288 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=23861/tid=23874] ppid=23859 vsize=3548200 CPUtime=2.64 cores=0-31
/proc/23861/task/23874/stat : 23874 (claspmt) R 23859 23861 23835 0 -1 4202560 19419 0 0 0 3 261 0 0 22 0 32 0 21833086 3633356800 511288 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=23861/tid=23875] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23875/stat : 23875 (claspmt) R 23859 23861 23835 0 -1 4202560 21702 0 0 0 6 259 0 0 22 0 32 0 21833086 3633356800 511300 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=23861/tid=23876] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23876/stat : 23876 (claspmt) R 23859 23861 23835 0 -1 4202560 20457 0 0 0 6 259 0 0 22 0 32 0 21833086 3633356800 511301 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=23861/tid=23877] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23877/stat : 23877 (claspmt) R 23859 23861 23835 0 -1 4202560 18337 0 0 0 7 259 0 0 22 0 32 0 21833086 3633356800 511301 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=23861/tid=23878] ppid=23859 vsize=3548200 CPUtime=2.61 cores=0-31
/proc/23861/task/23878/stat : 23878 (claspmt) R 23859 23861 23835 0 -1 4202560 20839 0 0 0 5 256 0 0 22 0 32 0 21833086 3633356800 511310 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=23861/tid=23879] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23879/stat : 23879 (claspmt) R 23859 23861 23835 0 -1 4202560 12759 0 0 0 4 262 0 0 22 0 32 0 21833086 3633356800 511311 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23861/tid=23880] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23880/stat : 23880 (claspmt) R 23859 23861 23835 0 -1 4202560 12374 0 0 0 2 263 0 0 22 0 32 0 21833086 3633356800 511311 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23861/tid=23881] ppid=23859 vsize=3548200 CPUtime=2.64 cores=0-31
/proc/23861/task/23881/stat : 23881 (claspmt) R 23859 23861 23835 0 -1 4202560 20119 0 0 0 4 260 0 0 22 0 32 0 21833086 3633356800 511318 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=23861/tid=23882] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23882/stat : 23882 (claspmt) R 23859 23861 23835 0 -1 4202560 19169 0 0 0 7 259 0 0 22 0 32 0 21833086 3633356800 511323 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=23861/tid=23883] ppid=23859 vsize=3548200 CPUtime=2.58 cores=0-31
/proc/23861/task/23883/stat : 23883 (claspmt) R 23859 23861 23835 0 -1 4202560 12139 0 0 0 4 254 0 0 22 0 32 0 21833086 3633356800 511323 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23861/tid=23884] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23884/stat : 23884 (claspmt) R 23859 23861 23835 0 -1 4202560 17966 0 0 0 7 258 0 0 25 0 32 0 21833086 3633356800 511331 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679151 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23861/tid=23885] ppid=23859 vsize=3548200 CPUtime=2.62 cores=0-31
/proc/23861/task/23885/stat : 23885 (claspmt) R 23859 23861 23835 0 -1 4202560 20964 0 0 0 5 257 0 0 25 0 32 0 21833086 3633356800 511335 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=23861/tid=23886] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23886/stat : 23886 (claspmt) R 23859 23861 23835 0 -1 4202560 20129 0 0 0 4 262 0 0 22 0 32 0 21833086 3633356800 511335 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=23861/tid=23887] ppid=23859 vsize=3548200 CPUtime=2.66 cores=0-31
/proc/23861/task/23887/stat : 23887 (claspmt) R 23859 23861 23835 0 -1 4202560 19251 0 0 0 6 260 0 0 22 0 32 0 21833086 3633356800 511342 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679091 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=23861/tid=23888] ppid=23859 vsize=3548200 CPUtime=2.58 cores=0-31
/proc/23861/task/23888/stat : 23888 (claspmt) R 23859 23861 23835 0 -1 4202560 11763 0 0 0 3 255 0 0 22 0 32 0 21833086 3633356800 511347 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=23861/tid=23889] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23889/stat : 23889 (claspmt) R 23859 23861 23835 0 -1 4202560 21034 0 0 0 4 261 0 0 25 0 32 0 21833086 3633356800 511347 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=23861/tid=23890] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23890/stat : 23890 (claspmt) R 23859 23861 23835 0 -1 4202560 27929 0 0 0 6 259 0 0 22 0 32 0 21833087 3633356800 511352 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679091 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=23861/tid=23891] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23891/stat : 23891 (claspmt) R 23859 23861 23835 0 -1 4202560 20839 0 0 0 2 263 0 0 25 0 32 0 21833087 3633356800 511362 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=23861/tid=23892] ppid=23859 vsize=3548200 CPUtime=2.65 cores=0-31
/proc/23861/task/23892/stat : 23892 (claspmt) R 23859 23861 23835 0 -1 4202560 18064 0 0 0 3 262 0 0 25 0 32 0 21833087 3633356800 511362 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679094 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=23861/tid=23893] ppid=23859 vsize=3548200 CPUtime=2.56 cores=0-31
/proc/23861/task/23893/stat : 23893 (claspmt) R 23859 23861 23835 0 -1 4202560 20852 0 0 0 6 250 0 0 22 0 32 0 21833087 3633356800 511363 270582939648 4194304 4964800 140736515963104 18446744073709551615 4679106 0 0 4096 18946 0 0 0 -1 15 0 0 0
Current children cumulated CPU time (s) 85.82
Current children cumulated vsize (KiB) 3548200

[startup+12.7027 s]
/proc/loadavg: 21.42 28.72 24.47 5/445 23893
/proc/meminfo: memFree=258095156/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=5462008 CPUtime=177.22 cores=0-31
/proc/23861/stat : 23861 (claspmt) S 23859 23861 23835 0 -1 4202496 1607213 0 0 0 1677 16045 0 0 18 0 32 0 21832724 5593096192 1169807 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/23861/statm: 1365502 1169820 430 189 0 1359851 0
[pid=23861/tid=23863] ppid=23859 vsize=5462008 CPUtime=5.43 cores=0-31
/proc/23861/task/23863/stat : 23863 (claspmt) S 23859 23861 23835 0 -1 4202560 41678 0 0 0 21 522 0 0 18 0 32 0 21833086 5593096192 1169827 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604572692020704 0 0 -1 22 0 0 0
[pid=23861/tid=23864] ppid=23859 vsize=5462008 CPUtime=4.95 cores=0-31
/proc/23861/task/23864/stat : 23864 (claspmt) S 23859 23861 23835 0 -1 4202560 61982 0 0 0 111 384 0 0 18 0 32 0 21833086 5593096192 1169830 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604505458646496 0 0 -1 18 0 0 0
[pid=23861/tid=23865] ppid=23859 vsize=5462008 CPUtime=5.99 cores=0-31
/proc/23861/task/23865/stat : 23865 (claspmt) D 23859 23861 23835 0 -1 4202560 62639 0 0 0 102 497 0 0 18 0 32 0 21833086 5593096192 1169839 270582939648 4194304 4964800 140736515963104 18446744073709551615 4375558 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=23861/tid=23866] ppid=23859 vsize=5462008 CPUtime=5.4 cores=0-31
/proc/23861/task/23866/stat : 23866 (claspmt) S 23859 23861 23835 0 -1 4202560 40582 0 0 0 23 517 0 0 18 0 32 0 21833086 5593096192 1169848 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604498551954912 0 0 -1 5 0 0 0
[pid=23861/tid=23867] ppid=23859 vsize=5462008 CPUtime=5.67 cores=0-31
/proc/23861/task/23867/stat : 23867 (claspmt) D 23859 23861 23835 0 -1 4202560 49551 0 0 0 35 532 0 0 18 0 32 0 21833086 5593096192 1169852 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=23861/tid=23868] ppid=23859 vsize=5462008 CPUtime=5.62 cores=0-31
/proc/23861/task/23868/stat : 23868 (claspmt) D 23859 23861 23835 0 -1 4202560 39873 0 0 0 18 544 0 0 18 0 32 0 21833086 5593096192 1169858 270582939648 4194304 4964800 140736515963104 18446744073709551615 4378056 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=23861/tid=23869] ppid=23859 vsize=5462008 CPUtime=5.78 cores=0-31
/proc/23861/task/23869/stat : 23869 (claspmt) R 23859 23861 23835 0 -1 4202560 48651 0 0 0 29 549 0 0 18 0 32 0 21833086 5593096192 1169864 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=23861/tid=23870] ppid=23859 vsize=5462008 CPUtime=5.51 cores=0-31
/proc/23861/task/23870/stat : 23870 (claspmt) S 23859 23861 23835 0 -1 4202560 41106 0 0 0 23 528 0 0 18 0 32 0 21833086 5593096192 1169874 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604563661209056 0 0 -1 30 0 0 0
[pid=23861/tid=23871] ppid=23859 vsize=5462008 CPUtime=5.44 cores=0-31
/proc/23861/task/23871/stat : 23871 (claspmt) S 23859 23861 23835 0 -1 4202560 41280 0 0 0 20 524 0 0 18 0 32 0 21833086 5593096192 1169878 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604567099948512 0 0 -1 14 0 0 0
[pid=23861/tid=23872] ppid=23859 vsize=5462008 CPUtime=6.14 cores=0-31
/proc/23861/task/23872/stat : 23872 (claspmt) D 23859 23861 23835 0 -1 4202560 53423 0 0 0 83 531 0 0 18 0 32 0 21833086 5593096192 1169882 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650653381 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=23861/tid=23873] ppid=23859 vsize=5462008 CPUtime=5.88 cores=0-31
/proc/23861/task/23873/stat : 23873 (claspmt) D 23859 23861 23835 0 -1 4202560 49160 0 0 0 50 538 0 0 18 0 32 0 21833086 5593096192 1169883 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650653381 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=23861/tid=23874] ppid=23859 vsize=5462008 CPUtime=5.33 cores=0-31
/proc/23861/task/23874/stat : 23874 (claspmt) S 23859 23861 23835 0 -1 4202560 41462 0 0 0 21 512 0 0 18 0 32 0 21833086 5593096192 1169894 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604492929318368 0 0 -1 13 0 0 0
[pid=23861/tid=23875] ppid=23859 vsize=5462008 CPUtime=5.62 cores=0-31
/proc/23861/task/23875/stat : 23875 (claspmt) R 23859 23861 23835 0 -1 4202560 50851 0 0 0 41 521 0 0 18 0 32 0 21833086 5593096192 1169902 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=23861/tid=23876] ppid=23859 vsize=5462008 CPUtime=5.97 cores=0-31
/proc/23861/task/23876/stat : 23876 (claspmt) D 23859 23861 23835 0 -1 4202560 54279 0 0 0 64 533 0 0 18 0 32 0 21833086 5593096192 1169909 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650653280 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=23861/tid=23877] ppid=23859 vsize=5462008 CPUtime=6.03 cores=0-31
/proc/23861/task/23877/stat : 23877 (claspmt) R 23859 23861 23835 0 -1 4202560 48508 0 0 0 60 543 0 0 18 0 32 0 21833086 5593096192 1169912 270582939648 4194304 4964800 140736515963104 18446744073709551615 4342186 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=23861/tid=23878] ppid=23859 vsize=5462008 CPUtime=5.17 cores=0-31
/proc/23861/task/23878/stat : 23878 (claspmt) S 23859 23861 23835 0 -1 4202560 40621 0 0 0 22 495 0 0 18 0 32 0 21833086 5593096192 1169918 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604498593545696 0 0 -1 23 0 0 0
[pid=23861/tid=23879] ppid=23859 vsize=5462008 CPUtime=5.35 cores=0-31
/proc/23861/task/23879/stat : 23879 (claspmt) S 23859 23861 23835 0 -1 4202560 40645 0 0 0 29 506 0 0 18 0 32 0 21833086 5593096192 1169928 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604561904172512 0 0 -1 2 0 0 0
[pid=23861/tid=23880] ppid=23859 vsize=5462008 CPUtime=5.72 cores=0-31
/proc/23861/task/23880/stat : 23880 (claspmt) R 23859 23861 23835 0 -1 4202560 42760 0 0 0 18 554 0 0 18 0 32 0 21833086 5593096192 1169936 270582939648 4194304 4964800 140736515963104 18446744073709551615 4687029 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23861/tid=23881] ppid=23859 vsize=5462008 CPUtime=6.06 cores=0-31
/proc/23861/task/23881/stat : 23881 (claspmt) R 23859 23861 23835 0 -1 4202560 52923 0 0 0 71 535 0 0 18 0 32 0 21833086 5593096192 1169939 270582939648 4194304 4964800 140736515963104 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=23861/tid=23882] ppid=23859 vsize=5462008 CPUtime=5.43 cores=0-31
/proc/23861/task/23882/stat : 23882 (claspmt) S 23859 23861 23835 0 -1 4202560 41719 0 0 0 26 517 0 0 18 0 32 0 21833086 5593096192 1169942 270582939648 4194304 4964800 140736515963104 18446744073709551615 258662772457 0 0 4096 18946 18446604573612080608 0 0 -1 12 0 0 0
[pid=23861/tid=23883] ppid=23859 vsize=5462008 CPUtime=5.62 cores=0-31

################
# More data... #
################

/proc/23861/task/23874/stat : 23874 (claspmt) R 23859 23861 23835 0 -1 4202560 54727 0 0 0 3711 539 0 0 25 0 32 0 21833086 8551727104 1842365 270582939648 4194304 4964800 140736515963104 18446744073709551615 4370264 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=23861/tid=23875] ppid=23859 vsize=8351296 CPUtime=5015.2 cores=0-31
/proc/23861/task/23875/stat : 23875 (claspmt) R 23859 23861 23835 0 -1 4202560 67174 0 0 0 500958 562 0 0 25 0 32 0 21833086 8551727104 1842365 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=23861/tid=23876] ppid=23859 vsize=8351296 CPUtime=43.11 cores=0-31
/proc/23861/task/23876/stat : 23876 (claspmt) R 23859 23861 23835 0 -1 4202560 73749 0 0 0 3736 575 0 0 20 0 32 0 21833086 8551727104 1842365 270582939648 4194304 4964800 140736515963104 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=23861/tid=23877] ppid=23859 vsize=8351296 CPUtime=43.57 cores=0-31
/proc/23861/task/23877/stat : 23877 (claspmt) R 23859 23861 23835 0 -1 4202560 62415 0 0 0 3774 583 0 0 20 0 32 0 21833086 8551727104 1842365 270582939648 4194304 4964800 140736515963104 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23861/tid=23878] ppid=23859 vsize=8351296 CPUtime=42.36 cores=0-31
/proc/23861/task/23878/stat : 23878 (claspmt) R 23859 23861 23835 0 -1 4202560 51083 0 0 0 3718 518 0 0 25 0 32 0 21833086 8551727104 1842365 270582939648 4194304 4964800 140736515963104 18446744073709551615 4370381 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=23861/tid=23879] ppid=23859 vsize=8351296 CPUtime=42.65 cores=0-31
/proc/23861/task/23879/stat : 23879 (claspmt) R 23859 23861 23835 0 -1 4202560 50538 0 0 0 3738 527 0 0 25 0 32 0 21833086 8551727104 1842367 270582939648 4194304 4964800 140736515963104 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=23861/tid=23880] ppid=23859 vsize=8351296 CPUtime=43.34 cores=0-31
/proc/23861/task/23880/stat : 23880 (claspmt) R 23859 23861 23835 0 -1 4202560 93036 0 0 0 3716 618 0 0 20 0 32 0 21833086 8551727104 1842367 270582939648 4194304 4964800 140736515963104 18446744073709551615 4363459 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=23861/tid=23881] ppid=23859 vsize=8351296 CPUtime=42.9 cores=0-31
/proc/23861/task/23881/stat : 23881 (claspmt) R 23859 23861 23835 0 -1 4202560 82363 0 0 0 3697 593 0 0 20 0 32 0 21833086 8551727104 1842368 270582939648 4194304 4964800 140736515963104 18446744073709551615 4361114 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=23861/tid=23882] ppid=23859 vsize=8351296 CPUtime=42.64 cores=0-31
/proc/23861/task/23882/stat : 23882 (claspmt) R 23859 23861 23835 0 -1 4202560 53508 0 0 0 3718 546 0 0 25 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=23861/tid=23883] ppid=23859 vsize=8351296 CPUtime=5072.82 cores=0-31
/proc/23861/task/23883/stat : 23883 (claspmt) R 23859 23861 23835 0 -1 4202560 67250 0 0 0 506688 594 0 0 25 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668576 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23861/tid=23884] ppid=23859 vsize=8351296 CPUtime=42.79 cores=0-31
/proc/23861/task/23884/stat : 23884 (claspmt) R 23859 23861 23835 0 -1 4202560 73804 0 0 0 3700 579 0 0 25 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4673998 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=23861/tid=23885] ppid=23859 vsize=8351296 CPUtime=43.25 cores=0-31
/proc/23861/task/23885/stat : 23885 (claspmt) R 23859 23861 23835 0 -1 4202560 68289 0 0 0 3738 587 0 0 25 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=23861/tid=23886] ppid=23859 vsize=8351296 CPUtime=42.07 cores=0-31
/proc/23861/task/23886/stat : 23886 (claspmt) R 23859 23861 23835 0 -1 4202560 54053 0 0 0 3685 522 0 0 25 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4370364 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=23861/tid=23887] ppid=23859 vsize=8351296 CPUtime=42.38 cores=0-31
/proc/23861/task/23887/stat : 23887 (claspmt) R 23859 23861 23835 0 -1 4202560 49834 0 0 0 3715 523 0 0 25 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=23861/tid=23888] ppid=23859 vsize=8351296 CPUtime=42.74 cores=0-31
/proc/23861/task/23888/stat : 23888 (claspmt) R 23859 23861 23835 0 -1 4202560 93378 0 0 0 3674 600 0 0 19 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=23861/tid=23889] ppid=23859 vsize=8351296 CPUtime=43.02 cores=0-31
/proc/23861/task/23889/stat : 23889 (claspmt) R 23859 23861 23835 0 -1 4202560 87558 0 0 0 3736 566 0 0 18 0 32 0 21833086 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4669017 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23861/tid=23890] ppid=23859 vsize=8351296 CPUtime=41.16 cores=0-31
/proc/23861/task/23890/stat : 23890 (claspmt) R 23859 23861 23835 0 -1 4202560 51533 0 0 0 3726 390 0 0 19 0 32 0 21833087 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=23861/tid=23891] ppid=23859 vsize=8351296 CPUtime=5049.64 cores=0-31
/proc/23861/task/23891/stat : 23891 (claspmt) R 23859 23861 23835 0 -1 4202560 66970 0 0 0 504397 567 0 0 25 0 32 0 21833087 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=23861/tid=23892] ppid=23859 vsize=8351296 CPUtime=43 cores=0-31
/proc/23861/task/23892/stat : 23892 (claspmt) R 23859 23861 23835 0 -1 4202560 73317 0 0 0 3716 584 0 0 18 0 32 0 21833087 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23861/tid=23893] ppid=23859 vsize=8351296 CPUtime=43.98 cores=0-31
/proc/23861/task/23893/stat : 23893 (claspmt) R 23859 23861 23835 0 -1 4202560 68318 0 0 0 3817 581 0 0 25 0 32 0 21833087 8551727104 1842370 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 23 0 0 0
Current children cumulated CPU time (s) 21301.2
Current children cumulated vsize (KiB) 8351296



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 20.10 8.24 5.49 33/445 24044
/proc/meminfo: memFree=254549832/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=8768488 CPUtime=21867 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 2465160 0 0 0 2169034 17664 0 0 25 0 32 0 21832724 8978931712 2002215 270582939648 4194304 4964800 140736515963104 18446744073709551615 4370295 0 0 4096 18946 18446744073709551615 0 0 17 12 0 0 0
/proc/23861/statm: 2192122 2002215 455 189 0 2186471 0
[pid=23861/tid=23863] ppid=23859 vsize=8768488 CPUtime=60.23 cores=0-31
/proc/23861/task/23863/stat : 23863 (claspmt) R 23859 23861 23835 0 -1 4202560 54514 0 0 0 5474 549 0 0 25 0 32 0 21833086 8978931712 2002215 270582939648 4194304 4964800 140736515963104 18446744073709551615 4370242 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=23861/tid=23864] ppid=23859 vsize=8768488 CPUtime=59.16 cores=0-31
/proc/23861/task/23864/stat : 23864 (claspmt) R 23859 23861 23835 0 -1 4202560 118363 0 0 0 5460 456 0 0 19 0 32 0 21833086 8978931712 2002215 270582939648 4194304 4964800 140736515963104 18446744073709551615 4361356 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=23861/tid=23865] ppid=23859 vsize=8768488 CPUtime=60.16 cores=0-31
/proc/23861/task/23865/stat : 23865 (claspmt) R 23859 23861 23835 0 -1 4202560 108092 0 0 0 5438 578 0 0 18 0 32 0 21833086 8978931712 2002215 270582939648 4194304 4964800 140736515963104 18446744073709551615 4632355 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=23861/tid=23866] ppid=23859 vsize=8768488 CPUtime=60.43 cores=0-31
/proc/23861/task/23866/stat : 23866 (claspmt) R 23859 23861 23835 0 -1 4202560 53982 0 0 0 5497 546 0 0 25 0 32 0 21833086 8978931712 2002219 270582939648 4194304 4964800 140736515963104 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=23861/tid=23867] ppid=23859 vsize=8768488 CPUtime=4986.47 cores=0-31
/proc/23861/task/23867/stat : 23867 (claspmt) R 23859 23861 23835 0 -1 4202560 68553 0 0 0 498066 581 0 0 25 0 32 0 21833086 8978931712 2002219 270582939648 4194304 4964800 140736515963104 18446744073709551615 4651371 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=23861/tid=23868] ppid=23859 vsize=8768488 CPUtime=60.24 cores=0-31
/proc/23861/task/23868/stat : 23868 (claspmt) R 23859 23861 23835 0 -1 4202560 69720 0 0 0 5427 597 0 0 25 0 32 0 21833086 8978931712 2002219 270582939648 4194304 4964800 140736515963104 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=23861/tid=23869] ppid=23859 vsize=8768488 CPUtime=61.3 cores=0-31
/proc/23861/task/23869/stat : 23869 (claspmt) R 23859 23861 23835 0 -1 4202560 68054 0 0 0 5525 605 0 0 25 0 32 0 21833086 8978931712 2002230 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23861/tid=23870] ppid=23859 vsize=8768488 CPUtime=60.36 cores=0-31
/proc/23861/task/23870/stat : 23870 (claspmt) R 23859 23861 23835 0 -1 4202560 57640 0 0 0 5476 560 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4627615 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=23861/tid=23871] ppid=23859 vsize=8768488 CPUtime=60.1 cores=0-31
/proc/23861/task/23871/stat : 23871 (claspmt) R 23859 23861 23835 0 -1 4202560 57067 0 0 0 5454 556 0 0 19 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=23861/tid=23872] ppid=23859 vsize=8768488 CPUtime=60.38 cores=0-31
/proc/23861/task/23872/stat : 23872 (claspmt) R 23859 23861 23835 0 -1 4202560 118311 0 0 0 5415 623 0 0 20 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650653116 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23861/tid=23873] ppid=23859 vsize=8768488 CPUtime=60.63 cores=0-31
/proc/23861/task/23873/stat : 23873 (claspmt) R 23859 23861 23835 0 -1 4202560 105766 0 0 0 5437 626 0 0 18 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4660408 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=23861/tid=23874] ppid=23859 vsize=8768488 CPUtime=60.19 cores=0-31
/proc/23861/task/23874/stat : 23874 (claspmt) R 23859 23861 23835 0 -1 4202560 57711 0 0 0 5474 545 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=23861/tid=23875] ppid=23859 vsize=8768488 CPUtime=5032.91 cores=0-31
/proc/23861/task/23875/stat : 23875 (claspmt) R 23859 23861 23835 0 -1 4202560 67174 0 0 0 502729 562 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=23861/tid=23876] ppid=23859 vsize=8768488 CPUtime=60.82 cores=0-31
/proc/23861/task/23876/stat : 23876 (claspmt) R 23859 23861 23835 0 -1 4202560 74982 0 0 0 5505 577 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=23861/tid=23877] ppid=23859 vsize=8768488 CPUtime=61.28 cores=0-31
/proc/23861/task/23877/stat : 23877 (claspmt) R 23859 23861 23835 0 -1 4202560 66970 0 0 0 5536 592 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23861/tid=23878] ppid=23859 vsize=8768488 CPUtime=60.06 cores=0-31
/proc/23861/task/23878/stat : 23878 (claspmt) R 23859 23861 23835 0 -1 4202560 54299 0 0 0 5480 526 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=23861/tid=23879] ppid=23859 vsize=8768488 CPUtime=60.35 cores=0-31
/proc/23861/task/23879/stat : 23879 (claspmt) R 23859 23861 23835 0 -1 4202560 56261 0 0 0 5499 536 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=23861/tid=23880] ppid=23859 vsize=8768488 CPUtime=61.02 cores=0-31
/proc/23861/task/23880/stat : 23880 (claspmt) R 23859 23861 23835 0 -1 4202560 109459 0 0 0 5467 635 0 0 18 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4362864 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=23861/tid=23881] ppid=23859 vsize=8768488 CPUtime=60.57 cores=0-31
/proc/23861/task/23881/stat : 23881 (claspmt) R 23859 23861 23835 0 -1 4202560 93030 0 0 0 5443 614 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4343193 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=23861/tid=23882] ppid=23859 vsize=8768488 CPUtime=60.34 cores=0-31
/proc/23861/task/23882/stat : 23882 (claspmt) R 23859 23861 23835 0 -1 4202560 56683 0 0 0 5481 553 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4380234 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=23861/tid=23883] ppid=23859 vsize=8768488 CPUtime=5090.54 cores=0-31
/proc/23861/task/23883/stat : 23883 (claspmt) R 23859 23861 23835 0 -1 4202560 67250 0 0 0 508460 594 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23861/tid=23884] ppid=23859 vsize=8768488 CPUtime=60.5 cores=0-31
/proc/23861/task/23884/stat : 23884 (claspmt) R 23859 23861 23835 0 -1 4202560 74484 0 0 0 5469 581 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=23861/tid=23885] ppid=23859 vsize=8768488 CPUtime=60.43 cores=0-31
/proc/23861/task/23885/stat : 23885 (claspmt) R 23859 23861 23835 0 -1 4202560 71687 0 0 0 5448 595 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4663308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23861/tid=23886] ppid=23859 vsize=8768488 CPUtime=59.78 cores=0-31
/proc/23861/task/23886/stat : 23886 (claspmt) R 23859 23861 23835 0 -1 4202560 57050 0 0 0 5449 529 0 0 18 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=23861/tid=23887] ppid=23859 vsize=8768488 CPUtime=60.09 cores=0-31
/proc/23861/task/23887/stat : 23887 (claspmt) R 23859 23861 23835 0 -1 4202560 53043 0 0 0 5478 531 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=23861/tid=23888] ppid=23859 vsize=8768488 CPUtime=60.43 cores=0-31
/proc/23861/task/23888/stat : 23888 (claspmt) R 23859 23861 23835 0 -1 4202560 103502 0 0 0 5430 613 0 0 20 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4363687 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=23861/tid=23889] ppid=23859 vsize=8768488 CPUtime=60.69 cores=0-31
/proc/23861/task/23889/stat : 23889 (claspmt) R 23859 23861 23835 0 -1 4202560 98479 0 0 0 5484 585 0 0 25 0 32 0 21833086 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4660835 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23861/tid=23890] ppid=23859 vsize=8768488 CPUtime=58.87 cores=0-31
/proc/23861/task/23890/stat : 23890 (claspmt) R 23859 23861 23835 0 -1 4202560 55395 0 0 0 5489 398 0 0 25 0 32 0 21833087 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=23861/tid=23891] ppid=23859 vsize=8768488 CPUtime=5067.36 cores=0-31
/proc/23861/task/23891/stat : 23891 (claspmt) R 23859 23861 23835 0 -1 4202560 66970 0 0 0 506169 567 0 0 25 0 32 0 21833087 8978931712 2002233 270582939648 4194304 4964800 140736515963104 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=23861/tid=23892] ppid=23859 vsize=8768488 CPUtime=60.72 cores=0-31
/proc/23861/task/23892/stat : 23892 (claspmt) R 23859 23861 23835 0 -1 4202560 74077 0 0 0 5486 586 0 0 25 0 32 0 21833087 8978931712 2002234 270582939648 4194304 4964800 140736515963104 18446744073709551615 4678338 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23861/tid=23893] ppid=23859 vsize=8768488 CPUtime=61.69 cores=0-31
/proc/23861/task/23893/stat : 23893 (claspmt) R 23859 23861 23835 0 -1 4202560 70732 0 0 0 5584 585 0 0 25 0 32 0 21833087 8978931712 2002235 270582939648 4194304 4964800 140736515963104 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 23 0 0 0
Current children cumulated CPU time (s) 21867
Current children cumulated vsize (KiB) 8768488

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -23861

Solver just ended. Dumping a history of the last processes samples

[startup+5100.7 s]
/proc/loadavg: 18.57 8.12 5.47 2/415 24045
/proc/meminfo: memFree=254879168/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=8347540 CPUtime=21868 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 2465307 0 0 0 2169136 17667 0 0 25 0 1 0 21832724 8547880960 1919629 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650604748 0 0 4096 18946 0 0 0 17 12 0 0 0
/proc/23861/statm: 2086885 1919629 466 189 0 2081234 0
Current children cumulated CPU time (s) 21868
Current children cumulated vsize (KiB) 8347540

[startup+5101.5 s]
/proc/loadavg: 18.57 8.12 5.47 2/415 24045
/proc/meminfo: memFree=254879168/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=8000752 CPUtime=21868.8 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 2465307 0 0 0 2169210 17670 0 0 25 0 1 0 21832724 8192770048 1833573 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650611751 0 0 4096 18946 0 0 0 17 12 0 0 0
/proc/23861/statm: 2000188 1833573 466 189 0 1994537 0
Current children cumulated CPU time (s) 21868.8
Current children cumulated vsize (KiB) 8000752

[startup+5101.93 s]
/proc/loadavg: 18.57 8.12 5.47 2/415 24045
/proc/meminfo: memFree=255479080/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=7637692 CPUtime=21869.2 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4202496 2465307 0 0 0 2169251 17672 0 0 25 0 1 0 21832724 7820996608 1743417 270582939648 4194304 4964800 140736515963104 18446744073709551615 258650611643 0 0 4096 18946 0 0 0 17 12 0 0 0
/proc/23861/statm: 1909423 1743417 466 189 0 1903772 0
Current children cumulated CPU time (s) 21869.2
Current children cumulated vsize (KiB) 7637692

[startup+5102.3 s]
/proc/loadavg: 18.57 8.12 5.47 2/415 24045
/proc/meminfo: memFree=255479080/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=0 CPUtime=21869.6 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4203524 2465307 0 0 0 2169261 17699 0 0 25 0 1 0 21832724 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 12 0 0 0
/proc/23861/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 21869.6
Current children cumulated vsize (KiB) 0

[startup+5102.4 s]
/proc/loadavg: 18.57 8.12 5.47 2/415 24045
/proc/meminfo: memFree=255479080/264257624 swapFree=67111528/67111528
[pid=23861] ppid=23859 vsize=0 CPUtime=21869.7 cores=0-31
/proc/23861/stat : 23861 (claspmt) R 23859 23861 23835 0 -1 4203524 2465307 0 0 0 2169261 17709 0 0 25 0 1 0 21832724 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 12 0 0 0
/proc/23861/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 21869.7
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5102.42
CPU time (s): 21869.7
CPU user time (s): 21692.6
CPU system time (s): 177.111
CPU usage (%): 428.615
Max. virtual memory (cumulated for all children) (KiB): 8768488

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 21692.6
system time used= 177.111
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2465307
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 882063
involuntary context switches= 36889

runsolver used 27.1359 second user time and 112.384 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-08 22:20:58
IDJOB=3472375
IDBENCH=82388
IDSOLVER=1961
FILE ID=node200/3472375-1307564451
RUNJOBID= node200-1307564451-23849
PBS_JOBID= 13547639
Free space on /tmp= 61576 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/transport-transport-city-sequential-35nodes-1000size-4degree-100mindistance-4trucks-14packages-2008seed.050-SAT.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472375-1307564451/watcher-3472375-1307564451 -o /tmp/evaluation-result-3472375-1307564451/solver-3472375-1307564451 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472375-1307564451.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= 48f03e5fbe31fc85b9a464480fc8aa5a
RANDOM SEED=587292424

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      262564552 kB
Buffers:        186640 kB
Cached:        1240188 kB
SwapCached:          0 kB
Active:         375808 kB
Inactive:      1114820 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      262564552 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:          171632 kB
Writeback:           0 kB
AnonPages:       63604 kB
Mapped:          14472 kB
Slab:           123536 kB
PageTables:       4404 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   172720 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61576 MiB
End job on node200 at 2011-06-08 23:46:01