Trace number 3472361

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 158327 5103.36

General information on the benchmark

Nameapplication/rintanen/SATPlanning/
sokoban-sequential-p145-microban-sequential.060-NOTKNOWN.cnf
MD5SUMe44b288003a3eaf34673a2c0fb8c6f3e
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark18152.8
Satisfiable
(Un)Satisfiability was proved
Number of variables131484
Number of clauses2120376
Sum of the clauses size4841016
Maximum clause length78
Minimum clause length1
Number of clauses of size 1696
Number of clauses of size 22046240
Number of clauses of size 35760
Number of clauses of size 426640
Number of clauses of size 511520
Number of clauses of size over 529520

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...517122/instance-3472361-1307517122.cnf
2.59/2.62	c Solving...
158322.77/5100.03	
158322.77/5100.03	*** INTERRUPTED! ***
158322.77/5100.03	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472361-1307517122/watcher-3472361-1307517122 -o /tmp/evaluation-result-3472361-1307517122/solver-3472361-1307517122 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472361-1307517122.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 12.50 9.94 19.42 1/413 21383
/proc/meminfo: memFree=262935980/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=38092 CPUtime=0.02 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 4775 0 0 0 1 1 0 0 22 0 1 0 17099230 39006208 4704 270582939648 4194304 4964800 140736672323968 18446744073709551615 4680298 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/21383/statm: 9523 4723 374 189 0 4397 0

[startup+0.0700121 s]
/proc/loadavg: 12.50 9.94 19.42 1/413 21383
/proc/meminfo: memFree=262935980/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=46968 CPUtime=0.06 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 7929 0 0 0 4 2 0 0 22 0 1 0 17099230 48095232 6831 270582939648 4194304 4964800 140736672323968 18446744073709551615 258650955760 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/21383/statm: 11742 6831 386 189 0 6616 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 46968

[startup+0.101012 s]
/proc/loadavg: 12.50 9.94 19.42 1/413 21383
/proc/meminfo: memFree=262935980/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=47508 CPUtime=0.09 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 8070 0 0 0 7 2 0 0 22 0 1 0 17099230 48648192 6972 270582939648 4194304 4964800 140736672323968 18446744073709551615 258650955760 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/21383/statm: 11877 6972 386 189 0 6751 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 47508

[startup+0.301022 s]
/proc/loadavg: 12.50 9.94 19.42 1/413 21383
/proc/meminfo: memFree=262935980/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=71448 CPUtime=0.29 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 16531 0 0 0 25 4 0 0 23 0 1 0 17099230 73162752 12763 270582939648 4194304 4964800 140736672323968 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/21383/statm: 17862 12765 386 189 0 12736 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 71448

[startup+0.701043 s]
/proc/loadavg: 12.50 9.94 19.42 1/413 21383
/proc/meminfo: memFree=262935980/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=129372 CPUtime=0.69 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 37769 0 0 0 61 8 0 0 25 0 1 0 17099230 132476928 26190 270582939648 4194304 4964800 140736672323968 18446744073709551615 258650653371 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/21383/statm: 32343 26203 396 189 0 27217 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 129372

[startup+1.50106 s]
/proc/loadavg: 12.50 9.94 19.42 2/414 21384
/proc/meminfo: memFree=262801664/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=171952 CPUtime=1.49 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 47817 0 0 0 138 11 0 0 25 0 1 0 17099230 176078848 36238 270582939648 4194304 4964800 140736672323968 18446744073709551615 4608728 0 0 4096 18946 0 0 0 17 20 0 0 0
/proc/21383/statm: 42988 36238 400 189 0 37862 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 171952

[startup+3.10103 s]
/proc/loadavg: 12.50 9.94 19.42 2/414 21384
/proc/meminfo: memFree=262791180/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=953224 CPUtime=13.31 cores=0-31
/proc/21383/stat : 21383 (claspmt) S 21381 21383 21283 0 -1 4202496 135725 0 0 0 315 1016 0 0 18 0 32 0 17099230 976101376 92494 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 8 0 0 0
/proc/21383/statm: 238306 92510 439 189 0 232655 0
Current children cumulated CPU time (s) 13.31
Current children cumulated vsize (KiB) 953224

[startup+6.30121 s]
/proc/loadavg: 13.50 10.19 19.45 5/445 21415
/proc/meminfo: memFree=262297900/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=2031564 CPUtime=27.4 cores=0-31
/proc/21383/stat : 21383 (claspmt) S 21381 21383 21283 0 -1 4202496 206104 0 0 0 1528 1212 0 0 18 0 32 0 17099230 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 8 0 0 0
/proc/21383/statm: 507891 160408 440 189 0 502240 0
[pid=21383/tid=21385] ppid=21381 vsize=2031564 CPUtime=0.07 cores=0-31
/proc/21383/task/21385/stat : 21385 (claspmt) S 21381 21383 21283 0 -1 4202560 303 0 0 0 6 1 0 0 25 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604575293038048 0 0 -1 1 0 0 0
[pid=21383/tid=21386] ppid=21381 vsize=2031564 CPUtime=0.6 cores=0-31
/proc/21383/task/21386/stat : 21386 (claspmt) S 21381 21383 21283 0 -1 4202560 6165 0 0 0 15 45 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604504014347744 0 0 -1 2 0 0 0
[pid=21383/tid=21387] ppid=21381 vsize=2031564 CPUtime=0.46 cores=0-31
/proc/21383/task/21387/stat : 21387 (claspmt) S 21381 21383 21283 0 -1 4202560 5902 0 0 0 14 32 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604505810927072 0 0 -1 19 0 0 0
[pid=21383/tid=21388] ppid=21381 vsize=2031564 CPUtime=0.43 cores=0-31
/proc/21383/task/21388/stat : 21388 (claspmt) S 21381 21383 21283 0 -1 4202560 5026 0 0 0 5 38 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604499972177376 0 0 -1 23 0 0 0
[pid=21383/tid=21389] ppid=21381 vsize=2031564 CPUtime=2.82 cores=0-31
/proc/21383/task/21389/stat : 21389 (claspmt) R 21381 21383 21283 0 -1 4202560 6128 0 0 0 237 45 0 0 21 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=21383/tid=21390] ppid=21381 vsize=2031564 CPUtime=0.56 cores=0-31
/proc/21383/task/21390/stat : 21390 (claspmt) S 21381 21383 21283 0 -1 4202560 5409 0 0 0 15 41 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604498631843296 0 0 -1 29 0 0 0
[pid=21383/tid=21391] ppid=21381 vsize=2031564 CPUtime=0.59 cores=0-31
/proc/21383/task/21391/stat : 21391 (claspmt) S 21381 21383 21283 0 -1 4202560 4696 0 0 0 17 42 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604506045251040 0 0 -1 20 0 0 0
[pid=21383/tid=21392] ppid=21381 vsize=2031564 CPUtime=0.46 cores=0-31
/proc/21383/task/21392/stat : 21392 (claspmt) S 21381 21383 21283 0 -1 4202560 4870 0 0 0 6 40 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501699452384 0 0 -1 21 0 0 0
[pid=21383/tid=21393] ppid=21381 vsize=2031564 CPUtime=0.44 cores=0-31
/proc/21383/task/21393/stat : 21393 (claspmt) S 21381 21383 21283 0 -1 4202560 4511 0 0 0 5 39 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604506042416608 0 0 -1 14 0 0 0
[pid=21383/tid=21394] ppid=21381 vsize=2031564 CPUtime=0.52 cores=0-31
/proc/21383/task/21394/stat : 21394 (claspmt) S 21381 21383 21283 0 -1 4202560 6119 0 0 0 15 37 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501448089056 0 0 -1 7 0 0 0
[pid=21383/tid=21395] ppid=21381 vsize=2031564 CPUtime=0.58 cores=0-31
/proc/21383/task/21395/stat : 21395 (claspmt) S 21381 21383 21283 0 -1 4202560 5900 0 0 0 17 41 0 0 18 0 32 0 17099492 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604498631835104 0 0 -1 5 0 0 0
[pid=21383/tid=21396] ppid=21381 vsize=2031564 CPUtime=0.44 cores=0-31
/proc/21383/task/21396/stat : 21396 (claspmt) S 21381 21383 21283 0 -1 4202560 4705 0 0 0 5 39 0 0 18 0 32 0 17099493 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501521112544 0 0 -1 30 0 0 0
[pid=21383/tid=21397] ppid=21381 vsize=2031564 CPUtime=2.8 cores=0-31
/proc/21383/task/21397/stat : 21397 (claspmt) R 21381 21383 21283 0 -1 4202560 6163 0 0 0 236 44 0 0 21 0 32 0 17099493 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670688 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=21383/tid=21398] ppid=21381 vsize=2031564 CPUtime=0.58 cores=0-31
/proc/21383/task/21398/stat : 21398 (claspmt) S 21381 21383 21283 0 -1 4202560 5353 0 0 0 16 42 0 0 18 0 32 0 17099493 2080321536 160408 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604499302809056 0 0 -1 26 0 0 0
[pid=21383/tid=21399] ppid=21381 vsize=2031564 CPUtime=0.56 cores=0-31
/proc/21383/task/21399/stat : 21399 (claspmt) S 21381 21383 21283 0 -1 4202560 5143 0 0 0 19 37 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604503090576864 0 0 -1 15 0 0 0
[pid=21383/tid=21400] ppid=21381 vsize=2031564 CPUtime=0.28 cores=0-31
/proc/21383/task/21400/stat : 21400 (claspmt) S 21381 21383 21283 0 -1 4202560 2724 0 0 0 5 23 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501521128928 0 0 -1 9 0 0 0
[pid=21383/tid=21401] ppid=21381 vsize=2031564 CPUtime=0.41 cores=0-31
/proc/21383/task/21401/stat : 21401 (claspmt) S 21381 21383 21283 0 -1 4202560 3457 0 0 0 5 36 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501699460576 0 0 -1 8 0 0 0
[pid=21383/tid=21402] ppid=21381 vsize=2031564 CPUtime=0.53 cores=0-31
/proc/21383/task/21402/stat : 21402 (claspmt) S 21381 21383 21283 0 -1 4202560 4990 0 0 0 14 39 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501448080864 0 0 -1 12 0 0 0
[pid=21383/tid=21403] ppid=21381 vsize=2031564 CPUtime=0.53 cores=0-31
/proc/21383/task/21403/stat : 21403 (claspmt) S 21381 21383 21283 0 -1 4202560 5906 0 0 0 18 35 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604504014355936 0 0 -1 11 0 0 0
[pid=21383/tid=21404] ppid=21381 vsize=2031564 CPUtime=0.46 cores=0-31
/proc/21383/task/21404/stat : 21404 (claspmt) S 21381 21383 21283 0 -1 4202560 4644 0 0 0 4 42 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501699444192 0 0 -1 28 0 0 0
[pid=21383/tid=21405] ppid=21381 vsize=2031564 CPUtime=2.78 cores=0-31
/proc/21383/task/21405/stat : 21405 (claspmt) R 21381 21383 21283 0 -1 4202560 6190 0 0 0 233 45 0 0 21 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 4399345 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=21383/tid=21406] ppid=21381 vsize=2031564 CPUtime=0.58 cores=0-31
/proc/21383/task/21406/stat : 21406 (claspmt) S 21381 21383 21283 0 -1 4202560 6062 0 0 0 16 42 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604506045226464 0 0 -1 6 0 0 0
[pid=21383/tid=21407] ppid=21381 vsize=2031564 CPUtime=0.62 cores=0-31
/proc/21383/task/21407/stat : 21407 (claspmt) S 21381 21383 21283 0 -1 4202560 6156 0 0 0 18 44 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604505693175264 0 0 -1 31 0 0 0
[pid=21383/tid=21408] ppid=21381 vsize=2031564 CPUtime=0.44 cores=0-31
/proc/21383/task/21408/stat : 21408 (claspmt) S 21381 21383 21283 0 -1 4202560 4205 0 0 0 5 39 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604497960041952 0 0 -1 13 0 0 0
[pid=21383/tid=21409] ppid=21381 vsize=2031564 CPUtime=0.47 cores=0-31
/proc/21383/task/21409/stat : 21409 (claspmt) S 21381 21383 21283 0 -1 4202560 4661 0 0 0 6 41 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501448072672 0 0 -1 4 0 0 0
[pid=21383/tid=21410] ppid=21381 vsize=2031564 CPUtime=0.53 cores=0-31
/proc/21383/task/21410/stat : 21410 (claspmt) S 21381 21383 21283 0 -1 4202560 6150 0 0 0 15 38 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604504014331360 0 0 -1 1 0 0 0
[pid=21383/tid=21411] ppid=21381 vsize=2031564 CPUtime=0.57 cores=0-31
/proc/21383/task/21411/stat : 21411 (claspmt) S 21381 21383 21283 0 -1 4202560 6050 0 0 0 17 40 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604504547442144 0 0 -1 10 0 0 0
[pid=21383/tid=21412] ppid=21381 vsize=2031564 CPUtime=0.41 cores=0-31
/proc/21383/task/21412/stat : 21412 (claspmt) S 21381 21383 21283 0 -1 4202560 4205 0 0 0 5 36 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604499302800864 0 0 -1 17 0 0 0
[pid=21383/tid=21413] ppid=21381 vsize=2031564 CPUtime=2.75 cores=0-31
/proc/21383/task/21413/stat : 21413 (claspmt) R 21381 21383 21283 0 -1 4202560 5443 0 0 0 241 34 0 0 21 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 4678036 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=21383/tid=21414] ppid=21381 vsize=2031564 CPUtime=0.57 cores=0-31
/proc/21383/task/21414/stat : 21414 (claspmt) S 21381 21383 21283 0 -1 4202560 4952 0 0 0 14 43 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501521120736 0 0 -1 24 0 0 0
[pid=21383/tid=21415] ppid=21381 vsize=2031564 CPUtime=0.61 cores=0-31
/proc/21383/task/21415/stat : 21415 (claspmt) S 21381 21383 21283 0 -1 4202560 5983 0 0 0 18 43 0 0 18 0 32 0 17099493 2080321536 160409 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604503090585056 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 27.4
Current children cumulated vsize (KiB) 2031564

[startup+12.7014 s]
/proc/loadavg: 12.74 10.09 19.37 5/445 21415
/proc/meminfo: memFree=262294856/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=2031564 CPUtime=52.99 cores=0-31
/proc/21383/stat : 21383 (claspmt) S 21381 21383 21283 0 -1 4202496 206654 0 0 0 4086 1213 0 0 18 0 32 0 17099230 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 8 0 0 0
/proc/21383/statm: 507891 160958 440 189 0 502240 0
[pid=21383/tid=21385] ppid=21381 vsize=2031564 CPUtime=0.07 cores=0-31
/proc/21383/task/21385/stat : 21385 (claspmt) S 21381 21383 21283 0 -1 4202560 303 0 0 0 6 1 0 0 25 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604575293038048 0 0 -1 1 0 0 0
[pid=21383/tid=21386] ppid=21381 vsize=2031564 CPUtime=0.6 cores=0-31
/proc/21383/task/21386/stat : 21386 (claspmt) S 21381 21383 21283 0 -1 4202560 6165 0 0 0 15 45 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604504014347744 0 0 -1 2 0 0 0
[pid=21383/tid=21387] ppid=21381 vsize=2031564 CPUtime=0.46 cores=0-31
/proc/21383/task/21387/stat : 21387 (claspmt) S 21381 21383 21283 0 -1 4202560 5902 0 0 0 14 32 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604505810927072 0 0 -1 19 0 0 0
[pid=21383/tid=21388] ppid=21381 vsize=2031564 CPUtime=0.43 cores=0-31
/proc/21383/task/21388/stat : 21388 (claspmt) S 21381 21383 21283 0 -1 4202560 5026 0 0 0 5 38 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604499972177376 0 0 -1 23 0 0 0
[pid=21383/tid=21389] ppid=21381 vsize=2031564 CPUtime=9.23 cores=0-31
/proc/21383/task/21389/stat : 21389 (claspmt) R 21381 21383 21283 0 -1 4202560 6274 0 0 0 877 46 0 0 25 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 4668998 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=21383/tid=21390] ppid=21381 vsize=2031564 CPUtime=0.56 cores=0-31
/proc/21383/task/21390/stat : 21390 (claspmt) S 21381 21383 21283 0 -1 4202560 5409 0 0 0 15 41 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604498631843296 0 0 -1 29 0 0 0
[pid=21383/tid=21391] ppid=21381 vsize=2031564 CPUtime=0.59 cores=0-31
/proc/21383/task/21391/stat : 21391 (claspmt) S 21381 21383 21283 0 -1 4202560 4696 0 0 0 17 42 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604506045251040 0 0 -1 20 0 0 0
[pid=21383/tid=21392] ppid=21381 vsize=2031564 CPUtime=0.46 cores=0-31
/proc/21383/task/21392/stat : 21392 (claspmt) S 21381 21383 21283 0 -1 4202560 4870 0 0 0 6 40 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501699452384 0 0 -1 21 0 0 0
[pid=21383/tid=21393] ppid=21381 vsize=2031564 CPUtime=0.44 cores=0-31
/proc/21383/task/21393/stat : 21393 (claspmt) S 21381 21383 21283 0 -1 4202560 4511 0 0 0 5 39 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604506042416608 0 0 -1 14 0 0 0
[pid=21383/tid=21394] ppid=21381 vsize=2031564 CPUtime=0.52 cores=0-31
/proc/21383/task/21394/stat : 21394 (claspmt) S 21381 21383 21283 0 -1 4202560 6119 0 0 0 15 37 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501448089056 0 0 -1 7 0 0 0
[pid=21383/tid=21395] ppid=21381 vsize=2031564 CPUtime=0.58 cores=0-31
/proc/21383/task/21395/stat : 21395 (claspmt) S 21381 21383 21283 0 -1 4202560 5900 0 0 0 17 41 0 0 18 0 32 0 17099492 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604498631835104 0 0 -1 5 0 0 0
[pid=21383/tid=21396] ppid=21381 vsize=2031564 CPUtime=0.44 cores=0-31
/proc/21383/task/21396/stat : 21396 (claspmt) S 21381 21383 21283 0 -1 4202560 4705 0 0 0 5 39 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501521112544 0 0 -1 30 0 0 0
[pid=21383/tid=21397] ppid=21381 vsize=2031564 CPUtime=9.2 cores=0-31
/proc/21383/task/21397/stat : 21397 (claspmt) R 21381 21383 21283 0 -1 4202560 6301 0 0 0 876 44 0 0 25 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670688 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=21383/tid=21398] ppid=21381 vsize=2031564 CPUtime=0.58 cores=0-31
/proc/21383/task/21398/stat : 21398 (claspmt) S 21381 21383 21283 0 -1 4202560 5353 0 0 0 16 42 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604499302809056 0 0 -1 26 0 0 0
[pid=21383/tid=21399] ppid=21381 vsize=2031564 CPUtime=0.56 cores=0-31
/proc/21383/task/21399/stat : 21399 (claspmt) S 21381 21383 21283 0 -1 4202560 5143 0 0 0 19 37 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604503090576864 0 0 -1 15 0 0 0
[pid=21383/tid=21400] ppid=21381 vsize=2031564 CPUtime=0.28 cores=0-31
/proc/21383/task/21400/stat : 21400 (claspmt) S 21381 21383 21283 0 -1 4202560 2724 0 0 0 5 23 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501521128928 0 0 -1 9 0 0 0
[pid=21383/tid=21401] ppid=21381 vsize=2031564 CPUtime=0.41 cores=0-31
/proc/21383/task/21401/stat : 21401 (claspmt) S 21381 21383 21283 0 -1 4202560 3457 0 0 0 5 36 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501699460576 0 0 -1 8 0 0 0
[pid=21383/tid=21402] ppid=21381 vsize=2031564 CPUtime=0.53 cores=0-31
/proc/21383/task/21402/stat : 21402 (claspmt) S 21381 21383 21283 0 -1 4202560 4990 0 0 0 14 39 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501448080864 0 0 -1 12 0 0 0
[pid=21383/tid=21403] ppid=21381 vsize=2031564 CPUtime=0.53 cores=0-31
/proc/21383/task/21403/stat : 21403 (claspmt) S 21381 21383 21283 0 -1 4202560 5906 0 0 0 18 35 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604504014355936 0 0 -1 11 0 0 0
[pid=21383/tid=21404] ppid=21381 vsize=2031564 CPUtime=0.46 cores=0-31
/proc/21383/task/21404/stat : 21404 (claspmt) S 21381 21383 21283 0 -1 4202560 4644 0 0 0 4 42 0 0 18 0 32 0 17099493 2080321536 160958 270582939648 4194304 4964800 140736672323968 18446744073709551615 258662772457 0 0 4096 18946 18446604501699444192 0 0 -1 28 0 0 0
[pid=21383/tid=21405] ppid=21381 vsize=2031564 CPUtime=9.19 cores=0-31

################
# More data... #
################

/proc/21383/task/21396/stat : 21396 (claspmt) R 21381 21383 21283 0 -1 4202560 171565 0 0 0 490498 387 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=21383/tid=21397] ppid=21381 vsize=28759136 CPUtime=5076.07 cores=0-31
/proc/21383/task/21397/stat : 21397 (claspmt) R 21381 21383 21283 0 -1 4202560 6801 0 0 0 507562 45 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=21383/tid=21398] ppid=21381 vsize=28759136 CPUtime=4913.58 cores=0-31
/proc/21383/task/21398/stat : 21398 (claspmt) R 21381 21383 21283 0 -1 4202560 220384 0 0 0 490897 461 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=21383/tid=21399] ppid=21381 vsize=28759136 CPUtime=4907.32 cores=0-31
/proc/21383/task/21399/stat : 21399 (claspmt) R 21381 21383 21283 0 -1 4202560 397597 0 0 0 489905 827 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=21383/tid=21400] ppid=21381 vsize=28759136 CPUtime=4913.4 cores=0-31
/proc/21383/task/21400/stat : 21400 (claspmt) R 21381 21383 21283 0 -1 4202560 67899 0 0 0 491116 224 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21383/tid=21401] ppid=21381 vsize=28759136 CPUtime=4912.46 cores=0-31
/proc/21383/task/21401/stat : 21401 (claspmt) R 21381 21383 21283 0 -1 4202560 105180 0 0 0 490902 344 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21383/tid=21402] ppid=21381 vsize=28759136 CPUtime=4912.49 cores=0-31
/proc/21383/task/21402/stat : 21402 (claspmt) R 21381 21383 21283 0 -1 4202560 567479 0 0 0 490514 735 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=21383/tid=21403] ppid=21381 vsize=28759136 CPUtime=4913.79 cores=0-31
/proc/21383/task/21403/stat : 21403 (claspmt) R 21381 21383 21283 0 -1 4202560 157549 0 0 0 491075 304 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4660384 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21383/tid=21404] ppid=21381 vsize=28759136 CPUtime=4913.91 cores=0-31
/proc/21383/task/21404/stat : 21404 (claspmt) R 21381 21383 21283 0 -1 4202560 219881 0 0 0 490936 455 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21383/tid=21405] ppid=21381 vsize=28759136 CPUtime=5077.17 cores=0-31
/proc/21383/task/21405/stat : 21405 (claspmt) R 21381 21383 21283 0 -1 4202560 6733 0 0 0 507670 47 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669520 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=21383/tid=21406] ppid=21381 vsize=28759136 CPUtime=4913.61 cores=0-31
/proc/21383/task/21406/stat : 21406 (claspmt) R 21381 21383 21283 0 -1 4202560 184561 0 0 0 490974 387 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=21383/tid=21407] ppid=21381 vsize=28759136 CPUtime=4907.91 cores=0-31
/proc/21383/task/21407/stat : 21407 (claspmt) R 21381 21383 21283 0 -1 4202560 503397 0 0 0 489788 1003 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=21383/tid=21408] ppid=21381 vsize=28759136 CPUtime=4856.78 cores=0-31
/proc/21383/task/21408/stat : 21408 (claspmt) R 21381 21383 21283 0 -1 4202560 81078 0 0 0 485419 259 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=21383/tid=21409] ppid=21381 vsize=28759136 CPUtime=4914.01 cores=0-31
/proc/21383/task/21409/stat : 21409 (claspmt) R 21381 21383 21283 0 -1 4202560 73003 0 0 0 491161 240 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=21383/tid=21410] ppid=21381 vsize=28759136 CPUtime=4913.58 cores=0-31
/proc/21383/task/21410/stat : 21410 (claspmt) R 21381 21383 21283 0 -1 4202560 731924 0 0 0 490440 918 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=21383/tid=21411] ppid=21381 vsize=28759136 CPUtime=4897.72 cores=0-31
/proc/21383/task/21411/stat : 21411 (claspmt) R 21381 21383 21283 0 -1 4202560 126076 0 0 0 489484 288 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4660447 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=21383/tid=21412] ppid=21381 vsize=28759136 CPUtime=4911.55 cores=0-31
/proc/21383/task/21412/stat : 21412 (claspmt) R 21381 21383 21283 0 -1 4202560 131279 0 0 0 490857 298 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21383/tid=21413] ppid=21381 vsize=28759136 CPUtime=5077.8 cores=0-31
/proc/21383/task/21413/stat : 21413 (claspmt) R 21381 21383 21283 0 -1 4202560 5986 0 0 0 507745 35 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4678045 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=21383/tid=21414] ppid=21381 vsize=28759136 CPUtime=4913.83 cores=0-31
/proc/21383/task/21414/stat : 21414 (claspmt) R 21381 21383 21283 0 -1 4202560 161212 0 0 0 491041 342 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=21383/tid=21415] ppid=21381 vsize=28759136 CPUtime=4901.58 cores=0-31
/proc/21383/task/21415/stat : 21415 (claspmt) R 21381 21383 21283 0 -1 4202560 340274 0 0 0 489433 725 0 0 25 0 32 0 17099493 29449355264 6914628 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 8 0 0 0
Current children cumulated CPU time (s) 157756
Current children cumulated vsize (KiB) 28759136



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 32.14 32.13 32.02 33/445 21586
/proc/meminfo: memFree=235225836/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=28759136 CPUtime=158323 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 6966288 0 0 0 15819125 13151 0 0 25 0 32 0 17099230 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/21383/statm: 7189784 6914632 463 189 0 7184133 0
[pid=21383/tid=21385] ppid=21381 vsize=28759136 CPUtime=4931.39 cores=0-31
/proc/21383/task/21385/stat : 21385 (claspmt) R 21381 21383 21283 0 -1 4202560 82872 0 0 0 492916 223 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=21383/tid=21386] ppid=21381 vsize=28759136 CPUtime=4928.01 cores=0-31
/proc/21383/task/21386/stat : 21386 (claspmt) R 21381 21383 21283 0 -1 4202560 684742 0 0 0 491966 835 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=21383/tid=21387] ppid=21381 vsize=28759136 CPUtime=4918.48 cores=0-31
/proc/21383/task/21387/stat : 21387 (claspmt) R 21381 21383 21283 0 -1 4202560 159686 0 0 0 491506 342 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=21383/tid=21388] ppid=21381 vsize=28759136 CPUtime=4927.98 cores=0-31
/proc/21383/task/21388/stat : 21388 (claspmt) R 21381 21383 21283 0 -1 4202560 149211 0 0 0 492460 338 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=21383/tid=21389] ppid=21381 vsize=28759136 CPUtime=5094.74 cores=0-31
/proc/21383/task/21389/stat : 21389 (claspmt) R 21381 21383 21283 0 -1 4202560 6632 0 0 0 509428 46 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=21383/tid=21390] ppid=21381 vsize=28759136 CPUtime=4925.85 cores=0-31
/proc/21383/task/21390/stat : 21390 (claspmt) R 21381 21383 21283 0 -1 4202560 151986 0 0 0 492257 328 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=21383/tid=21391] ppid=21381 vsize=28759136 CPUtime=4927.98 cores=0-31
/proc/21383/task/21391/stat : 21391 (claspmt) R 21381 21383 21283 0 -1 4202560 309053 0 0 0 492154 644 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=21383/tid=21392] ppid=21381 vsize=28759136 CPUtime=4931.95 cores=0-31
/proc/21383/task/21392/stat : 21392 (claspmt) R 21381 21383 21283 0 -1 4202560 80834 0 0 0 492929 266 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4660495 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=21383/tid=21393] ppid=21381 vsize=28759136 CPUtime=4931.72 cores=0-31
/proc/21383/task/21393/stat : 21393 (claspmt) R 21381 21383 21283 0 -1 4202560 79518 0 0 0 492898 274 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=21383/tid=21394] ppid=21381 vsize=28759136 CPUtime=4929.84 cores=0-31
/proc/21383/task/21394/stat : 21394 (claspmt) R 21381 21383 21283 0 -1 4202560 720544 0 0 0 492092 892 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=21383/tid=21395] ppid=21381 vsize=28759136 CPUtime=4929.4 cores=0-31
/proc/21383/task/21395/stat : 21395 (claspmt) R 21381 21383 21283 0 -1 4202560 146816 0 0 0 492593 347 0 0 25 0 32 0 17099492 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=21383/tid=21396] ppid=21381 vsize=28759136 CPUtime=4926.57 cores=0-31
/proc/21383/task/21396/stat : 21396 (claspmt) R 21381 21383 21283 0 -1 4202560 171565 0 0 0 492270 387 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 19 0 0 0
[pid=21383/tid=21397] ppid=21381 vsize=28759136 CPUtime=5093.78 cores=0-31
/proc/21383/task/21397/stat : 21397 (claspmt) R 21381 21383 21283 0 -1 4202560 6801 0 0 0 509333 45 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=21383/tid=21398] ppid=21381 vsize=28759136 CPUtime=4931.29 cores=0-31
/proc/21383/task/21398/stat : 21398 (claspmt) R 21381 21383 21283 0 -1 4202560 220384 0 0 0 492668 461 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=21383/tid=21399] ppid=21381 vsize=28759136 CPUtime=4925.04 cores=0-31
/proc/21383/task/21399/stat : 21399 (claspmt) R 21381 21383 21283 0 -1 4202560 397597 0 0 0 491677 827 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=21383/tid=21400] ppid=21381 vsize=28759136 CPUtime=4931.12 cores=0-31
/proc/21383/task/21400/stat : 21400 (claspmt) R 21381 21383 21283 0 -1 4202560 67899 0 0 0 492888 224 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=21383/tid=21401] ppid=21381 vsize=28759136 CPUtime=4930.18 cores=0-31
/proc/21383/task/21401/stat : 21401 (claspmt) R 21381 21383 21283 0 -1 4202560 105180 0 0 0 492674 344 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=21383/tid=21402] ppid=21381 vsize=28759136 CPUtime=4930.21 cores=0-31
/proc/21383/task/21402/stat : 21402 (claspmt) R 21381 21383 21283 0 -1 4202560 567479 0 0 0 492286 735 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=21383/tid=21403] ppid=21381 vsize=28759136 CPUtime=4931.51 cores=0-31
/proc/21383/task/21403/stat : 21403 (claspmt) R 21381 21383 21283 0 -1 4202560 157553 0 0 0 492847 304 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=21383/tid=21404] ppid=21381 vsize=28759136 CPUtime=4931.63 cores=0-31
/proc/21383/task/21404/stat : 21404 (claspmt) R 21381 21383 21283 0 -1 4202560 219881 0 0 0 492708 455 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=21383/tid=21405] ppid=21381 vsize=28759136 CPUtime=5094.89 cores=0-31
/proc/21383/task/21405/stat : 21405 (claspmt) R 21381 21383 21283 0 -1 4202560 6733 0 0 0 509442 47 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=21383/tid=21406] ppid=21381 vsize=28759136 CPUtime=4931.33 cores=0-31
/proc/21383/task/21406/stat : 21406 (claspmt) R 21381 21383 21283 0 -1 4202560 184561 0 0 0 492746 387 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=21383/tid=21407] ppid=21381 vsize=28759136 CPUtime=4925.63 cores=0-31
/proc/21383/task/21407/stat : 21407 (claspmt) R 21381 21383 21283 0 -1 4202560 503397 0 0 0 491560 1003 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=21383/tid=21408] ppid=21381 vsize=28759136 CPUtime=4873.98 cores=0-31
/proc/21383/task/21408/stat : 21408 (claspmt) R 21381 21383 21283 0 -1 4202560 81078 0 0 0 487139 259 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4627360 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=21383/tid=21409] ppid=21381 vsize=28759136 CPUtime=4931.73 cores=0-31
/proc/21383/task/21409/stat : 21409 (claspmt) R 21381 21383 21283 0 -1 4202560 73003 0 0 0 492933 240 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=21383/tid=21410] ppid=21381 vsize=28759136 CPUtime=4931.3 cores=0-31
/proc/21383/task/21410/stat : 21410 (claspmt) R 21381 21383 21283 0 -1 4202560 731924 0 0 0 492212 918 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669523 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=21383/tid=21411] ppid=21381 vsize=28759136 CPUtime=4915.44 cores=0-31
/proc/21383/task/21411/stat : 21411 (claspmt) R 21381 21383 21283 0 -1 4202560 126076 0 0 0 491256 288 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=21383/tid=21412] ppid=21381 vsize=28759136 CPUtime=4929.26 cores=0-31
/proc/21383/task/21412/stat : 21412 (claspmt) R 21381 21383 21283 0 -1 4202560 131279 0 0 0 492628 298 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=21383/tid=21413] ppid=21381 vsize=28759136 CPUtime=5095.52 cores=0-31
/proc/21383/task/21413/stat : 21413 (claspmt) R 21381 21383 21283 0 -1 4202560 5986 0 0 0 509517 35 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=21383/tid=21414] ppid=21381 vsize=28759136 CPUtime=4931.55 cores=0-31
/proc/21383/task/21414/stat : 21414 (claspmt) R 21381 21383 21283 0 -1 4202560 161212 0 0 0 492813 342 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=21383/tid=21415] ppid=21381 vsize=28759136 CPUtime=4919.3 cores=0-31
/proc/21383/task/21415/stat : 21415 (claspmt) R 21381 21383 21283 0 -1 4202560 340274 0 0 0 491204 726 0 0 25 0 32 0 17099493 29449355264 6914632 270582939648 4194304 4964800 140736672323968 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 8 0 0 0
Current children cumulated CPU time (s) 158323
Current children cumulated vsize (KiB) 28759136

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -21383

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 32.14 32.13 32.02 2/415 21587
/proc/meminfo: memFree=235226076/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=28668936 CPUtime=158324 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4202496 6966300 0 0 0 15819236 13151 0 0 25 0 1 0 17099230 29356990464 6914600 270582939648 4194304 4964800 140736672323968 18446744073709551615 4627081 0 0 4096 18946 0 0 0 17 28 0 0 0
/proc/21383/statm: 7167234 6914600 474 189 0 7161583 0
Current children cumulated CPU time (s) 158324
Current children cumulated vsize (KiB) 28668936

[startup+5102.2 s]
/proc/loadavg: 32.14 32.13 32.02 2/415 21587
/proc/meminfo: memFree=235226076/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=0 CPUtime=158325 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4203524 6966300 0 0 0 15819375 13168 0 0 25 0 1 0 17099230 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/21383/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 158325
Current children cumulated vsize (KiB) 0

[startup+5103 s]
/proc/loadavg: 32.14 32.13 32.02 2/414 21587
/proc/meminfo: memFree=249880364/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=0 CPUtime=158326 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4203524 6966300 0 0 0 15819375 13248 0 0 25 0 1 0 17099230 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/21383/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 158326
Current children cumulated vsize (KiB) 0

[startup+5103.2 s]
/proc/loadavg: 32.14 32.13 32.02 2/414 21587
/proc/meminfo: memFree=249880364/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=0 CPUtime=158326 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4203524 6966300 0 0 0 15819375 13268 0 0 25 0 1 0 17099230 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/21383/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 158326
Current children cumulated vsize (KiB) 0

[startup+5103.3 s]
/proc/loadavg: 32.14 32.13 32.02 2/414 21587
/proc/meminfo: memFree=249880364/264257624 swapFree=67111528/67111528
[pid=21383] ppid=21381 vsize=0 CPUtime=158327 cores=0-31
/proc/21383/stat : 21383 (claspmt) R 21381 21383 21283 0 -1 4203524 6966300 0 0 0 15819375 13278 0 0 25 0 1 0 17099230 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 28 0 0 0
/proc/21383/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 158327
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5103.36
CPU time (s): 158327
CPU user time (s): 158194
CPU system time (s): 132.84
CPU usage (%): 3102.4
Max. virtual memory (cumulated for all children) (KiB): 28759136

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 158194
system time used= 132.84
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6966300
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 391207
involuntary context switches= 382452

runsolver used 27.9038 second user time and 115.366 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-08 09:12:04
IDJOB=3472361
IDBENCH=82366
IDSOLVER=1961
FILE ID=node200/3472361-1307517122
RUNJOBID= node200-1307516531-21297
PBS_JOBID= 13509207
Free space on /tmp= 61708 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT11/application/rintanen/SATPlanning/sokoban-sequential-p145-microban-sequential.060-NOTKNOWN.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472361-1307517122/watcher-3472361-1307517122 -o /tmp/evaluation-result-3472361-1307517122/solver-3472361-1307517122 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472361-1307517122.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= e44b288003a3eaf34673a2c0fb8c6f3e
RANDOM SEED=1869531144

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      262936376 kB
Buffers:        168068 kB
Cached:         891808 kB
SwapCached:          0 kB
Active:         227160 kB
Inactive:       896508 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      262936376 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           37312 kB
Writeback:           0 kB
AnonPages:       63540 kB
Mapped:          14532 kB
Slab:           121888 kB
PageTables:       4312 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   164908 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61708 MiB
End job on node200 at 2011-06-08 10:37:07