Trace number 3472308

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 160005 5025.93

General information on the benchmark

NameAPPLICATIONS/
bitverif/maxxor/maxxor128.cnf
MD5SUM065c126f0d55807310a398728b44423f
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark148113
Satisfiable
(Un)Satisfiability was proved
Number of variables200440
Number of clauses599015
Sum of the clauses size1397699
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2399342
Number of clauses of size 3199671
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...448933/instance-3472308-1307448933.cnf
2.78/2.89	c Solving...
160001.83/5023.23	
160001.83/5023.23	*** INTERRUPTED! ***
160001.83/5023.23	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472308-1307448933/watcher-3472308-1307448933 -o /tmp/evaluation-result-3472308-1307448933/solver-3472308-1307448933 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472308-1307448933.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 29.50 31.53 31.86 1/413 15345
/proc/meminfo: memFree=263355200/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=46324 CPUtime=0.02 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 4850 0 0 0 1 1 0 0 18 0 1 0 10280273 47435776 4780 270582939648 4194304 4964800 140736028740832 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 16 0 0 0
/proc/15345/statm: 11581 4788 373 189 0 6455 0

[startup+0.101031 s]
/proc/loadavg: 29.50 31.53 31.86 1/413 15345
/proc/meminfo: memFree=263355200/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=65620 CPUtime=0.09 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 13492 0 0 0 7 2 0 0 18 0 1 0 10280273 67194880 11383 270582939648 4194304 4964800 140736028740832 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 16 0 0 0
/proc/15345/statm: 16405 11385 383 189 0 11279 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 65620

[startup+0.201028 s]
/proc/loadavg: 29.50 31.53 31.86 1/413 15345
/proc/meminfo: memFree=263355200/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=78876 CPUtime=0.19 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 18004 0 0 0 16 3 0 0 18 0 1 0 10280273 80769024 14610 270582939648 4194304 4964800 140736028740832 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 16 0 0 0
/proc/15345/statm: 19719 14612 383 189 0 14593 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 78876

[startup+0.301032 s]
/proc/loadavg: 29.50 31.53 31.86 1/413 15345
/proc/meminfo: memFree=263355200/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=98488 CPUtime=0.29 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 25762 0 0 0 25 4 0 0 18 0 1 0 10280273 100851712 19450 270582939648 4194304 4964800 140736028740832 18446744073709551615 4618467 0 0 4096 18946 0 0 0 17 16 0 0 0
/proc/15345/statm: 24622 19453 396 189 0 19496 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 98488

[startup+0.701066 s]
/proc/loadavg: 29.50 31.53 31.86 1/413 15345
/proc/meminfo: memFree=263355200/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=127300 CPUtime=0.69 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 32814 0 0 0 63 6 0 0 18 0 1 0 10280273 130355200 26502 270582939648 4194304 4964800 140736028740832 18446744073709551615 4615763 0 0 4096 18946 0 0 0 17 16 0 0 0
/proc/15345/statm: 31825 26502 401 189 0 26699 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 127300

[startup+1.5011 s]
/proc/loadavg: 27.22 31.02 31.70 2/414 15346
/proc/meminfo: memFree=263234384/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=151928 CPUtime=1.49 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 38971 0 0 0 142 7 0 0 20 0 1 0 10280273 155574272 32659 270582939648 4194304 4964800 140736028740832 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 16 0 0 0
/proc/15345/statm: 37982 32659 401 189 0 32856 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 151928

[startup+3.10829 s]
/proc/loadavg: 27.22 31.02 31.70 2/414 15346
/proc/meminfo: memFree=263221860/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=843980 CPUtime=8.64 cores=0-31
/proc/15345/stat : 15345 (claspmt) S 15343 15345 15319 0 -1 4202496 91203 0 0 0 302 562 0 0 18 0 32 0 10280273 864235520 68658 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/15345/statm: 210995 68671 428 189 0 205344 0
Current children cumulated CPU time (s) 8.64
Current children cumulated vsize (KiB) 843980

[startup+6.30219 s]
/proc/loadavg: 27.22 31.02 31.70 5/445 15377
/proc/meminfo: memFree=262572188/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=2062812 CPUtime=32.3 cores=0-31
/proc/15345/stat : 15345 (claspmt) S 15343 15345 15319 0 -1 4202496 254487 0 0 0 1325 1905 0 0 18 0 32 0 10280273 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/15345/statm: 515703 195987 435 189 0 510052 0
[pid=15345/tid=15347] ppid=15343 vsize=2062812 CPUtime=0.71 cores=0-31
/proc/15345/task/15347/stat : 15347 (claspmt) S 15343 15345 15319 0 -1 4202560 5253 0 0 0 6 65 0 0 18 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499571899872 0 0 -1 1 0 0 0
[pid=15345/tid=15348] ppid=15343 vsize=2062812 CPUtime=0.76 cores=0-31
/proc/15345/task/15348/stat : 15348 (claspmt) S 15343 15345 15319 0 -1 4202560 7814 0 0 0 11 65 0 0 18 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604505862987232 0 0 -1 14 0 0 0
[pid=15345/tid=15349] ppid=15343 vsize=2062812 CPUtime=0.7 cores=0-31
/proc/15345/task/15349/stat : 15349 (claspmt) S 15343 15345 15319 0 -1 4202560 7696 0 0 0 12 58 0 0 18 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499119152608 0 0 -1 19 0 0 0
[pid=15345/tid=15350] ppid=15343 vsize=2062812 CPUtime=0.62 cores=0-31
/proc/15345/task/15350/stat : 15350 (claspmt) S 15343 15345 15319 0 -1 4202560 5905 0 0 0 6 56 0 0 18 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498475113952 0 0 -1 11 0 0 0
[pid=15345/tid=15351] ppid=15343 vsize=2062812 CPUtime=2.67 cores=0-31
/proc/15345/task/15351/stat : 15351 (claspmt) R 15343 15345 15319 0 -1 4202560 6852 0 0 0 207 60 0 0 21 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=15345/tid=15352] ppid=15343 vsize=2062812 CPUtime=0.7 cores=0-31
/proc/15345/task/15352/stat : 15352 (claspmt) S 15343 15345 15319 0 -1 4202560 7818 0 0 0 9 61 0 0 18 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604500988517856 0 0 -1 7 0 0 0
[pid=15345/tid=15353] ppid=15343 vsize=2062812 CPUtime=0.76 cores=0-31
/proc/15345/task/15353/stat : 15353 (claspmt) S 15343 15345 15319 0 -1 4202560 7397 0 0 0 13 63 0 0 18 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604501659753952 0 0 -1 27 0 0 0
[pid=15345/tid=15354] ppid=15343 vsize=2062812 CPUtime=0.62 cores=0-31
/proc/15345/task/15354/stat : 15354 (claspmt) S 15343 15345 15319 0 -1 4202560 5888 0 0 0 3 59 0 0 18 0 32 0 10280562 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498979421664 0 0 -1 24 0 0 0
[pid=15345/tid=15355] ppid=15343 vsize=2062812 CPUtime=0.67 cores=0-31
/proc/15345/task/15355/stat : 15355 (claspmt) S 15343 15345 15319 0 -1 4202560 6178 0 0 0 4 63 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499295165920 0 0 -1 28 0 0 0
[pid=15345/tid=15356] ppid=15343 vsize=2062812 CPUtime=0.73 cores=0-31
/proc/15345/task/15356/stat : 15356 (claspmt) S 15343 15345 15319 0 -1 4202560 7744 0 0 0 10 63 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499571891680 0 0 -1 3 0 0 0
[pid=15345/tid=15357] ppid=15343 vsize=2062812 CPUtime=0.75 cores=0-31
/proc/15345/task/15357/stat : 15357 (claspmt) S 15343 15345 15319 0 -1 4202560 6389 0 0 0 10 65 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498546892256 0 0 -1 5 0 0 0
[pid=15345/tid=15358] ppid=15343 vsize=2062812 CPUtime=0.66 cores=0-31
/proc/15345/task/15358/stat : 15358 (claspmt) S 15343 15345 15319 0 -1 4202560 5481 0 0 0 7 59 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604573910941152 0 0 -1 6 0 0 0
[pid=15345/tid=15359] ppid=15343 vsize=2062812 CPUtime=2.57 cores=0-31
/proc/15345/task/15359/stat : 15359 (claspmt) R 15343 15345 15319 0 -1 4202560 6992 0 0 0 191 66 0 0 20 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=15345/tid=15360] ppid=15343 vsize=2062812 CPUtime=0.65 cores=0-31
/proc/15345/task/15360/stat : 15360 (claspmt) S 15343 15345 15319 0 -1 4202560 8450 0 0 0 11 54 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604502147718624 0 0 -1 10 0 0 0
[pid=15345/tid=15361] ppid=15343 vsize=2062812 CPUtime=0.77 cores=0-31
/proc/15345/task/15361/stat : 15361 (claspmt) S 15343 15345 15319 0 -1 4202560 7571 0 0 0 12 65 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499079011808 0 0 -1 4 0 0 0
[pid=15345/tid=15362] ppid=15343 vsize=2062812 CPUtime=0.67 cores=0-31
/proc/15345/task/15362/stat : 15362 (claspmt) S 15343 15345 15319 0 -1 4202560 6118 0 0 0 5 62 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498383912416 0 0 -1 15 0 0 0
[pid=15345/tid=15363] ppid=15343 vsize=2062812 CPUtime=0.66 cores=0-31
/proc/15345/task/15363/stat : 15363 (claspmt) S 15343 15345 15319 0 -1 4202560 5993 0 0 0 4 62 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604504799870432 0 0 -1 20 0 0 0
[pid=15345/tid=15364] ppid=15343 vsize=2062812 CPUtime=0.71 cores=0-31
/proc/15345/task/15364/stat : 15364 (claspmt) S 15343 15345 15319 0 -1 4202560 8082 0 0 0 9 62 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604505456147936 0 0 -1 8 0 0 0
[pid=15345/tid=15365] ppid=15343 vsize=2062812 CPUtime=0.72 cores=0-31
/proc/15345/task/15365/stat : 15365 (claspmt) S 15343 15345 15319 0 -1 4202560 7649 0 0 0 10 62 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499295092192 0 0 -1 29 0 0 0
[pid=15345/tid=15366] ppid=15343 vsize=2062812 CPUtime=0.65 cores=0-31
/proc/15345/task/15366/stat : 15366 (claspmt) S 15343 15345 15319 0 -1 4202560 5874 0 0 0 5 60 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499483598304 0 0 -1 2 0 0 0
[pid=15345/tid=15367] ppid=15343 vsize=2062812 CPUtime=2.59 cores=0-31
/proc/15345/task/15367/stat : 15367 (claspmt) R 15343 15345 15319 0 -1 4202560 7281 0 0 0 194 65 0 0 20 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=15345/tid=15368] ppid=15343 vsize=2062812 CPUtime=0.72 cores=0-31
/proc/15345/task/15368/stat : 15368 (claspmt) S 15343 15345 15319 0 -1 4202560 7874 0 0 0 11 61 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499294838240 0 0 -1 22 0 0 0
[pid=15345/tid=15369] ppid=15343 vsize=2062812 CPUtime=0.76 cores=0-31
/proc/15345/task/15369/stat : 15369 (claspmt) S 15343 15345 15319 0 -1 4202560 7386 0 0 0 14 62 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499071270368 0 0 -1 13 0 0 0
[pid=15345/tid=15370] ppid=15343 vsize=2062812 CPUtime=0.6 cores=0-31
/proc/15345/task/15370/stat : 15370 (claspmt) S 15343 15345 15319 0 -1 4202560 5845 0 0 0 7 53 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604506175987168 0 0 -1 27 0 0 0
[pid=15345/tid=15371] ppid=15343 vsize=2062812 CPUtime=0.64 cores=0-31
/proc/15345/task/15371/stat : 15371 (claspmt) S 15343 15345 15319 0 -1 4202560 5800 0 0 0 4 60 0 0 18 0 32 0 10280563 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604501367225824 0 0 -1 28 0 0 0
[pid=15345/tid=15372] ppid=15343 vsize=2062812 CPUtime=0.61 cores=0-31
/proc/15345/task/15372/stat : 15372 (claspmt) S 15343 15345 15319 0 -1 4202560 7102 0 0 0 14 47 0 0 18 0 32 0 10280564 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604497804869088 0 0 -1 18 0 0 0
[pid=15345/tid=15373] ppid=15343 vsize=2062812 CPUtime=0.75 cores=0-31
/proc/15345/task/15373/stat : 15373 (claspmt) S 15343 15345 15319 0 -1 4202560 8024 0 0 0 8 67 0 0 18 0 32 0 10280564 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499483606496 0 0 -1 0 0 0 0
[pid=15345/tid=15374] ppid=15343 vsize=2062812 CPUtime=0.63 cores=0-31
/proc/15345/task/15374/stat : 15374 (claspmt) S 15343 15345 15319 0 -1 4202560 5717 0 0 0 6 57 0 0 18 0 32 0 10280564 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499294846432 0 0 -1 9 0 0 0
[pid=15345/tid=15375] ppid=15343 vsize=2062812 CPUtime=2.63 cores=0-31
/proc/15345/task/15375/stat : 15375 (claspmt) R 15343 15345 15319 0 -1 4202560 7021 0 0 0 203 60 0 0 20 0 32 0 10280564 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=15345/tid=15376] ppid=15343 vsize=2062812 CPUtime=0.71 cores=0-31
/proc/15345/task/15376/stat : 15376 (claspmt) S 15343 15345 15319 0 -1 4202560 7776 0 0 0 11 60 0 0 18 0 32 0 10280564 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604497804860896 0 0 -1 25 0 0 0
[pid=15345/tid=15377] ppid=15343 vsize=2062812 CPUtime=0.72 cores=0-31
/proc/15345/task/15377/stat : 15377 (claspmt) S 15343 15345 15319 0 -1 4202560 6992 0 0 0 13 59 0 0 18 0 32 0 10280564 2112319488 195987 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499295174112 0 0 -1 21 0 0 0
Current children cumulated CPU time (s) 32.3
Current children cumulated vsize (KiB) 2062812

[startup+12.7014 s]
/proc/loadavg: 23.65 30.13 31.40 5/445 15377
/proc/meminfo: memFree=262571112/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=2062812 CPUtime=57.89 cores=0-31
/proc/15345/stat : 15345 (claspmt) S 15343 15345 15319 0 -1 4202496 254676 0 0 0 3884 1905 0 0 18 0 32 0 10280273 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/15345/statm: 515703 196176 435 189 0 510052 0
[pid=15345/tid=15347] ppid=15343 vsize=2062812 CPUtime=0.71 cores=0-31
/proc/15345/task/15347/stat : 15347 (claspmt) S 15343 15345 15319 0 -1 4202560 5253 0 0 0 6 65 0 0 18 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499571899872 0 0 -1 1 0 0 0
[pid=15345/tid=15348] ppid=15343 vsize=2062812 CPUtime=0.76 cores=0-31
/proc/15345/task/15348/stat : 15348 (claspmt) S 15343 15345 15319 0 -1 4202560 7814 0 0 0 11 65 0 0 18 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604505862987232 0 0 -1 14 0 0 0
[pid=15345/tid=15349] ppid=15343 vsize=2062812 CPUtime=0.7 cores=0-31
/proc/15345/task/15349/stat : 15349 (claspmt) S 15343 15345 15319 0 -1 4202560 7696 0 0 0 12 58 0 0 18 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499119152608 0 0 -1 19 0 0 0
[pid=15345/tid=15350] ppid=15343 vsize=2062812 CPUtime=0.62 cores=0-31
/proc/15345/task/15350/stat : 15350 (claspmt) S 15343 15345 15319 0 -1 4202560 5905 0 0 0 6 56 0 0 18 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498475113952 0 0 -1 11 0 0 0
[pid=15345/tid=15351] ppid=15343 vsize=2062812 CPUtime=9.07 cores=0-31
/proc/15345/task/15351/stat : 15351 (claspmt) R 15343 15345 15319 0 -1 4202560 6902 0 0 0 847 60 0 0 25 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=15345/tid=15352] ppid=15343 vsize=2062812 CPUtime=0.7 cores=0-31
/proc/15345/task/15352/stat : 15352 (claspmt) S 15343 15345 15319 0 -1 4202560 7818 0 0 0 9 61 0 0 18 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604500988517856 0 0 -1 7 0 0 0
[pid=15345/tid=15353] ppid=15343 vsize=2062812 CPUtime=0.76 cores=0-31
/proc/15345/task/15353/stat : 15353 (claspmt) S 15343 15345 15319 0 -1 4202560 7397 0 0 0 13 63 0 0 18 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604501659753952 0 0 -1 27 0 0 0
[pid=15345/tid=15354] ppid=15343 vsize=2062812 CPUtime=0.62 cores=0-31
/proc/15345/task/15354/stat : 15354 (claspmt) S 15343 15345 15319 0 -1 4202560 5888 0 0 0 3 59 0 0 18 0 32 0 10280562 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498979421664 0 0 -1 24 0 0 0
[pid=15345/tid=15355] ppid=15343 vsize=2062812 CPUtime=0.67 cores=0-31
/proc/15345/task/15355/stat : 15355 (claspmt) S 15343 15345 15319 0 -1 4202560 6178 0 0 0 4 63 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499295165920 0 0 -1 28 0 0 0
[pid=15345/tid=15356] ppid=15343 vsize=2062812 CPUtime=0.73 cores=0-31
/proc/15345/task/15356/stat : 15356 (claspmt) S 15343 15345 15319 0 -1 4202560 7744 0 0 0 10 63 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499571891680 0 0 -1 3 0 0 0
[pid=15345/tid=15357] ppid=15343 vsize=2062812 CPUtime=0.75 cores=0-31
/proc/15345/task/15357/stat : 15357 (claspmt) S 15343 15345 15319 0 -1 4202560 6389 0 0 0 10 65 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498546892256 0 0 -1 5 0 0 0
[pid=15345/tid=15358] ppid=15343 vsize=2062812 CPUtime=0.66 cores=0-31
/proc/15345/task/15358/stat : 15358 (claspmt) S 15343 15345 15319 0 -1 4202560 5481 0 0 0 7 59 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604573910941152 0 0 -1 6 0 0 0
[pid=15345/tid=15359] ppid=15343 vsize=2062812 CPUtime=8.96 cores=0-31
/proc/15345/task/15359/stat : 15359 (claspmt) R 15343 15345 15319 0 -1 4202560 7034 0 0 0 830 66 0 0 25 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 4399310 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=15345/tid=15360] ppid=15343 vsize=2062812 CPUtime=0.65 cores=0-31
/proc/15345/task/15360/stat : 15360 (claspmt) S 15343 15345 15319 0 -1 4202560 8450 0 0 0 11 54 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604502147718624 0 0 -1 10 0 0 0
[pid=15345/tid=15361] ppid=15343 vsize=2062812 CPUtime=0.77 cores=0-31
/proc/15345/task/15361/stat : 15361 (claspmt) S 15343 15345 15319 0 -1 4202560 7571 0 0 0 12 65 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499079011808 0 0 -1 4 0 0 0
[pid=15345/tid=15362] ppid=15343 vsize=2062812 CPUtime=0.67 cores=0-31
/proc/15345/task/15362/stat : 15362 (claspmt) S 15343 15345 15319 0 -1 4202560 6118 0 0 0 5 62 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604498383912416 0 0 -1 15 0 0 0
[pid=15345/tid=15363] ppid=15343 vsize=2062812 CPUtime=0.66 cores=0-31
/proc/15345/task/15363/stat : 15363 (claspmt) S 15343 15345 15319 0 -1 4202560 5993 0 0 0 4 62 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604504799870432 0 0 -1 20 0 0 0
[pid=15345/tid=15364] ppid=15343 vsize=2062812 CPUtime=0.71 cores=0-31
/proc/15345/task/15364/stat : 15364 (claspmt) S 15343 15345 15319 0 -1 4202560 8082 0 0 0 9 62 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604505456147936 0 0 -1 8 0 0 0
[pid=15345/tid=15365] ppid=15343 vsize=2062812 CPUtime=0.72 cores=0-31
/proc/15345/task/15365/stat : 15365 (claspmt) S 15343 15345 15319 0 -1 4202560 7649 0 0 0 10 62 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499295092192 0 0 -1 29 0 0 0
[pid=15345/tid=15366] ppid=15343 vsize=2062812 CPUtime=0.65 cores=0-31
/proc/15345/task/15366/stat : 15366 (claspmt) S 15343 15345 15319 0 -1 4202560 5874 0 0 0 5 60 0 0 18 0 32 0 10280563 2112319488 196176 270582939648 4194304 4964800 140736028740832 18446744073709551615 258662772457 0 0 4096 18946 18446604499483598304 0 0 -1 2 0 0 0
[pid=15345/tid=15367] ppid=15343 vsize=2062812 CPUtime=8.99 cores=0-31

################
# More data... #
################

[pid=15345/tid=15354] ppid=15343 vsize=15978244 CPUtime=5001.76 cores=0-31
/proc/15345/task/15354/stat : 15354 (claspmt) R 15343 15345 15319 0 -1 4202560 17197 0 0 0 500070 106 0 0 25 0 32 0 10280562 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=15345/tid=15355] ppid=15343 vsize=15978244 CPUtime=5001.75 cores=0-31
/proc/15345/task/15355/stat : 15355 (claspmt) R 15343 15345 15319 0 -1 4202560 12883 0 0 0 500080 95 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4678036 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=15345/tid=15356] ppid=15343 vsize=15978244 CPUtime=5001.08 cores=0-31
/proc/15345/task/15356/stat : 15356 (claspmt) R 15343 15345 15319 0 -1 4202560 121681 0 0 0 499766 342 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=15345/tid=15357] ppid=15343 vsize=15978244 CPUtime=4979.27 cores=0-31
/proc/15345/task/15357/stat : 15357 (claspmt) R 15343 15345 15319 0 -1 4202560 29280 0 0 0 497770 157 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=15345/tid=15358] ppid=15343 vsize=15978244 CPUtime=5001.06 cores=0-31
/proc/15345/task/15358/stat : 15358 (claspmt) R 15343 15345 15319 0 -1 4202560 139960 0 0 0 499719 387 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15345/tid=15359] ppid=15343 vsize=15978244 CPUtime=5017.41 cores=0-31
/proc/15345/task/15359/stat : 15359 (claspmt) R 15343 15345 15319 0 -1 4202560 45465 0 0 0 501604 137 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4660236 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=15345/tid=15360] ppid=15343 vsize=15978244 CPUtime=5001.72 cores=0-31
/proc/15345/task/15360/stat : 15360 (claspmt) R 15343 15345 15319 0 -1 4202560 34195 0 0 0 500019 153 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=15345/tid=15361] ppid=15343 vsize=15978244 CPUtime=4999.49 cores=0-31
/proc/15345/task/15361/stat : 15361 (claspmt) R 15343 15345 15319 0 -1 4202560 567597 0 0 0 498716 1233 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=15345/tid=15362] ppid=15343 vsize=15978244 CPUtime=5001.85 cores=0-31
/proc/15345/task/15362/stat : 15362 (claspmt) R 15343 15345 15319 0 -1 4202560 15895 0 0 0 500079 106 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668753 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=15345/tid=15363] ppid=15343 vsize=15978244 CPUtime=5001.66 cores=0-31
/proc/15345/task/15363/stat : 15363 (claspmt) R 15343 15345 15319 0 -1 4202560 14963 0 0 0 500064 102 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=15345/tid=15364] ppid=15343 vsize=15978244 CPUtime=5000.84 cores=0-31
/proc/15345/task/15364/stat : 15364 (claspmt) R 15343 15345 15319 0 -1 4202560 151945 0 0 0 499712 372 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=15345/tid=15365] ppid=15343 vsize=15978244 CPUtime=5001.76 cores=0-31
/proc/15345/task/15365/stat : 15365 (claspmt) R 15343 15345 15319 0 -1 4202560 35141 0 0 0 500024 152 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4653328 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=15345/tid=15366] ppid=15343 vsize=15978244 CPUtime=4996.77 cores=0-31
/proc/15345/task/15366/stat : 15366 (claspmt) R 15343 15345 15319 0 -1 4202560 79043 0 0 0 499404 273 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=15345/tid=15367] ppid=15343 vsize=15978244 CPUtime=5017.48 cores=0-31
/proc/15345/task/15367/stat : 15367 (claspmt) R 15343 15345 15319 0 -1 4202560 172092 0 0 0 501440 308 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=15345/tid=15368] ppid=15343 vsize=15978244 CPUtime=4994.95 cores=0-31
/proc/15345/task/15368/stat : 15368 (claspmt) R 15343 15345 15319 0 -1 4202560 32723 0 0 0 499338 157 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15345/tid=15369] ppid=15343 vsize=15978244 CPUtime=5001.56 cores=0-31
/proc/15345/task/15369/stat : 15369 (claspmt) R 15343 15345 15319 0 -1 4202560 436475 0 0 0 499056 1100 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=15345/tid=15370] ppid=15343 vsize=15978244 CPUtime=5001.78 cores=0-31
/proc/15345/task/15370/stat : 15370 (claspmt) R 15343 15345 15319 0 -1 4202560 17266 0 0 0 500075 103 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4370295 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=15345/tid=15371] ppid=15343 vsize=15978244 CPUtime=5000.58 cores=0-31
/proc/15345/task/15371/stat : 15371 (claspmt) R 15343 15345 15319 0 -1 4202560 13392 0 0 0 499965 93 0 0 25 0 32 0 10280563 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=15345/tid=15372] ppid=15343 vsize=15978244 CPUtime=5001.58 cores=0-31
/proc/15345/task/15372/stat : 15372 (claspmt) R 15343 15345 15319 0 -1 4202560 121300 0 0 0 499823 335 0 0 25 0 32 0 10280564 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=15345/tid=15373] ppid=15343 vsize=15978244 CPUtime=5001.66 cores=0-31
/proc/15345/task/15373/stat : 15373 (claspmt) R 15343 15345 15319 0 -1 4202560 22556 0 0 0 500036 130 0 0 25 0 32 0 10280564 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15345/tid=15374] ppid=15343 vsize=15978244 CPUtime=4979.75 cores=0-31
/proc/15345/task/15374/stat : 15374 (claspmt) R 15343 15345 15319 0 -1 4202560 73641 0 0 0 497732 243 0 0 25 0 32 0 10280564 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15345/tid=15375] ppid=15343 vsize=15978244 CPUtime=5002.83 cores=0-31
/proc/15345/task/15375/stat : 15375 (claspmt) R 15343 15345 15319 0 -1 4202560 64970 0 0 0 500083 200 0 0 25 0 32 0 10280564 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4627293 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15345/tid=15376] ppid=15343 vsize=15978244 CPUtime=5001.77 cores=0-31
/proc/15345/task/15376/stat : 15376 (claspmt) R 15343 15345 15319 0 -1 4202560 28358 0 0 0 500042 135 0 0 25 0 32 0 10280564 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=15345/tid=15377] ppid=15343 vsize=15978244 CPUtime=4993.85 cores=0-31
/proc/15345/task/15377/stat : 15377 (claspmt) R 15343 15345 15319 0 -1 4202560 822573 0 0 0 497763 1622 0 0 25 0 32 0 10280564 16361721856 3722575 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669413 0 0 4096 18946 0 0 0 -1 29 0 0 0
Current children cumulated CPU time (s) 159972
Current children cumulated vsize (KiB) 15978244



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5023.2 s]
/proc/loadavg: 32.13 32.09 32.07 33/445 15523
/proc/meminfo: memFree=248439696/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=15978244 CPUtime=160002 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 3818112 0 0 0 15989973 10210 0 0 25 0 32 0 10280273 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4653186 0 0 4096 18946 18446744073709551615 0 0 17 19 0 0 0
/proc/15345/statm: 3994561 3722779 463 189 0 3988910 0
[pid=15345/tid=15347] ppid=15343 vsize=15978244 CPUtime=5002.47 cores=0-31
/proc/15345/task/15347/stat : 15347 (claspmt) R 15343 15345 15319 0 -1 4202560 12343 0 0 0 500157 90 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670691 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=15345/tid=15348] ppid=15343 vsize=15978244 CPUtime=5002.67 cores=0-31
/proc/15345/task/15348/stat : 15348 (claspmt) R 15343 15345 15319 0 -1 4202560 151874 0 0 0 499876 391 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=15345/tid=15349] ppid=15343 vsize=15978244 CPUtime=4987.95 cores=0-31
/proc/15345/task/15349/stat : 15349 (claspmt) R 15343 15345 15319 0 -1 4202560 28903 0 0 0 498628 167 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=15345/tid=15350] ppid=15343 vsize=15978244 CPUtime=4986.44 cores=0-31
/proc/15345/task/15350/stat : 15350 (claspmt) R 15343 15345 15319 0 -1 4202560 58681 0 0 0 498425 219 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=15345/tid=15351] ppid=15343 vsize=15978244 CPUtime=5018.11 cores=0-31
/proc/15345/task/15351/stat : 15351 (claspmt) R 15343 15345 15319 0 -1 4202560 65477 0 0 0 501624 187 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=15345/tid=15352] ppid=15343 vsize=15978244 CPUtime=4983.42 cores=0-31
/proc/15345/task/15352/stat : 15352 (claspmt) R 15343 15345 15319 0 -1 4202560 32339 0 0 0 498189 153 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 11 0 0 0
[pid=15345/tid=15353] ppid=15343 vsize=15978244 CPUtime=5001.61 cores=0-31
/proc/15345/task/15353/stat : 15353 (claspmt) R 15343 15345 15319 0 -1 4202560 342088 0 0 0 499310 851 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=15345/tid=15354] ppid=15343 vsize=15978244 CPUtime=5002.68 cores=0-31
/proc/15345/task/15354/stat : 15354 (claspmt) R 15343 15345 15319 0 -1 4202560 17197 0 0 0 500162 106 0 0 25 0 32 0 10280562 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=15345/tid=15355] ppid=15343 vsize=15978244 CPUtime=5002.68 cores=0-31
/proc/15345/task/15355/stat : 15355 (claspmt) R 15343 15345 15319 0 -1 4202560 12883 0 0 0 500173 95 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=15345/tid=15356] ppid=15343 vsize=15978244 CPUtime=5002 cores=0-31
/proc/15345/task/15356/stat : 15356 (claspmt) R 15343 15345 15319 0 -1 4202560 121681 0 0 0 499858 342 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 16 0 0 0
[pid=15345/tid=15357] ppid=15343 vsize=15978244 CPUtime=4980.2 cores=0-31
/proc/15345/task/15357/stat : 15357 (claspmt) R 15343 15345 15319 0 -1 4202560 29280 0 0 0 497863 157 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=15345/tid=15358] ppid=15343 vsize=15978244 CPUtime=5001.98 cores=0-31
/proc/15345/task/15358/stat : 15358 (claspmt) R 15343 15345 15319 0 -1 4202560 139960 0 0 0 499811 387 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4338784 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=15345/tid=15359] ppid=15343 vsize=15978244 CPUtime=5018.33 cores=0-31
/proc/15345/task/15359/stat : 15359 (claspmt) R 15343 15345 15319 0 -1 4202560 45465 0 0 0 501696 137 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=15345/tid=15360] ppid=15343 vsize=15978244 CPUtime=5002.64 cores=0-31
/proc/15345/task/15360/stat : 15360 (claspmt) R 15343 15345 15319 0 -1 4202560 34195 0 0 0 500111 153 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4627652 0 0 4096 18946 0 0 0 -1 10 0 0 0
[pid=15345/tid=15361] ppid=15343 vsize=15978244 CPUtime=5000.37 cores=0-31
/proc/15345/task/15361/stat : 15361 (claspmt) R 15343 15345 15319 0 -1 4202560 567597 0 0 0 498804 1233 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 31 0 0 0
[pid=15345/tid=15362] ppid=15343 vsize=15978244 CPUtime=5002.78 cores=0-31
/proc/15345/task/15362/stat : 15362 (claspmt) R 15343 15345 15319 0 -1 4202560 15895 0 0 0 500172 106 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=15345/tid=15363] ppid=15343 vsize=15978244 CPUtime=5002.58 cores=0-31
/proc/15345/task/15363/stat : 15363 (claspmt) R 15343 15345 15319 0 -1 4202560 14963 0 0 0 500156 102 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669403 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=15345/tid=15364] ppid=15343 vsize=15978244 CPUtime=5001.76 cores=0-31
/proc/15345/task/15364/stat : 15364 (claspmt) R 15343 15345 15319 0 -1 4202560 151947 0 0 0 499804 372 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 28 0 0 0
[pid=15345/tid=15365] ppid=15343 vsize=15978244 CPUtime=5002.68 cores=0-31
/proc/15345/task/15365/stat : 15365 (claspmt) R 15343 15345 15319 0 -1 4202560 35141 0 0 0 500116 152 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 13 0 0 0
[pid=15345/tid=15366] ppid=15343 vsize=15978244 CPUtime=4997.69 cores=0-31
/proc/15345/task/15366/stat : 15366 (claspmt) R 15343 15345 15319 0 -1 4202560 79043 0 0 0 499496 273 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=15345/tid=15367] ppid=15343 vsize=15978244 CPUtime=5018.4 cores=0-31
/proc/15345/task/15367/stat : 15367 (claspmt) R 15343 15345 15319 0 -1 4202560 172092 0 0 0 501532 308 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669276 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=15345/tid=15368] ppid=15343 vsize=15978244 CPUtime=4995.88 cores=0-31
/proc/15345/task/15368/stat : 15368 (claspmt) R 15343 15345 15319 0 -1 4202560 32730 0 0 0 499431 157 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4653587 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=15345/tid=15369] ppid=15343 vsize=15978244 CPUtime=5002.48 cores=0-31
/proc/15345/task/15369/stat : 15369 (claspmt) R 15343 15345 15319 0 -1 4202560 436475 0 0 0 499148 1100 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669479 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=15345/tid=15370] ppid=15343 vsize=15978244 CPUtime=5002.7 cores=0-31
/proc/15345/task/15370/stat : 15370 (claspmt) R 15343 15345 15319 0 -1 4202560 17266 0 0 0 500167 103 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=15345/tid=15371] ppid=15343 vsize=15978244 CPUtime=5001.51 cores=0-31
/proc/15345/task/15371/stat : 15371 (claspmt) R 15343 15345 15319 0 -1 4202560 13414 0 0 0 500057 94 0 0 25 0 32 0 10280563 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=15345/tid=15372] ppid=15343 vsize=15978244 CPUtime=5002.5 cores=0-31
/proc/15345/task/15372/stat : 15372 (claspmt) R 15343 15345 15319 0 -1 4202560 121300 0 0 0 499915 335 0 0 25 0 32 0 10280564 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=15345/tid=15373] ppid=15343 vsize=15978244 CPUtime=5002.58 cores=0-31
/proc/15345/task/15373/stat : 15373 (claspmt) R 15343 15345 15319 0 -1 4202560 22556 0 0 0 500128 130 0 0 25 0 32 0 10280564 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=15345/tid=15374] ppid=15343 vsize=15978244 CPUtime=4980.67 cores=0-31
/proc/15345/task/15374/stat : 15374 (claspmt) R 15343 15345 15319 0 -1 4202560 73641 0 0 0 497824 243 0 0 25 0 32 0 10280564 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4668748 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=15345/tid=15375] ppid=15343 vsize=15978244 CPUtime=5003.75 cores=0-31
/proc/15345/task/15375/stat : 15375 (claspmt) R 15343 15345 15319 0 -1 4202560 64970 0 0 0 500175 200 0 0 25 0 32 0 10280564 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4669509 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=15345/tid=15376] ppid=15343 vsize=15978244 CPUtime=5002.69 cores=0-31
/proc/15345/task/15376/stat : 15376 (claspmt) R 15343 15345 15319 0 -1 4202560 28358 0 0 0 500134 135 0 0 25 0 32 0 10280564 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4332327 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=15345/tid=15377] ppid=15343 vsize=15978244 CPUtime=4994.77 cores=0-31
/proc/15345/task/15377/stat : 15377 (claspmt) R 15343 15345 15319 0 -1 4202560 822573 0 0 0 497855 1622 0 0 25 0 32 0 10280564 16361721856 3722779 270582939648 4194304 4964800 140736028740832 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 29 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 15978244

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -15345

Solver just ended. Dumping a history of the last processes samples

[startup+5023.9 s]
/proc/loadavg: 32.13 32.09 32.07 2/415 15524
/proc/meminfo: memFree=248464612/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=15853396 CPUtime=160003 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 3818125 0 0 0 15990073 10211 0 0 25 0 1 0 10280273 16233877504 3714070 270582939648 4194304 4964800 140736028740832 18446744073709551615 258650613570 0 0 4096 18946 0 0 0 17 19 0 0 0
/proc/15345/statm: 3963349 3714070 474 189 0 3957698 0
Current children cumulated CPU time (s) 160003
Current children cumulated vsize (KiB) 15853396

[startup+5024.7 s]
/proc/loadavg: 32.13 32.09 32.07 2/415 15524
/proc/meminfo: memFree=248464612/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=15815812 CPUtime=160004 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4202496 3818125 0 0 0 15990152 10211 0 0 25 0 1 0 10280273 16195391488 3704674 270582939648 4194304 4964800 140736028740832 18446744073709551615 258650604748 0 0 4096 18946 0 0 0 17 19 0 0 0
/proc/15345/statm: 3953953 3704674 474 189 0 3948302 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 15815812

[startup+5025.5 s]
/proc/loadavg: 32.13 32.09 32.07 2/415 15524
/proc/meminfo: memFree=248510492/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=0 CPUtime=160004 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4203524 3818125 0 0 0 15990205 10238 0 0 25 0 1 0 10280273 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 19 0 0 0
/proc/15345/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 0

[startup+5025.9 s]
/proc/loadavg: 32.13 32.09 32.07 2/414 15524
/proc/meminfo: memFree=262735532/264257624 swapFree=67111528/67111528
[pid=15345] ppid=15343 vsize=0 CPUtime=160005 cores=0-31
/proc/15345/stat : 15345 (claspmt) R 15343 15345 15319 0 -1 4203524 3818125 0 0 0 15990205 10280 0 0 25 0 1 0 10280273 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 19 0 0 0
/proc/15345/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 160005
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5025.93
CPU time (s): 160005
CPU user time (s): 159902
CPU system time (s): 102.819
CPU usage (%): 3183.59
Max. virtual memory (cumulated for all children) (KiB): 15978244

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 159902
system time used= 102.819
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3818125
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 278659
involuntary context switches= 402742

runsolver used 29.6345 second user time and 112.458 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-07 14:15:34
IDJOB=3472308
IDBENCH=70815
IDSOLVER=1961
FILE ID=node200/3472308-1307448933
RUNJOBID= node200-1307448933-15333
PBS_JOBID= 13509187
Free space on /tmp= 61732 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/APPLICATIONS/bitverif/maxxor/maxxor128.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472308-1307448933/watcher-3472308-1307448933 -o /tmp/evaluation-result-3472308-1307448933/solver-3472308-1307448933 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472308-1307448933.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= 065c126f0d55807310a398728b44423f
RANDOM SEED=2091739868

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      263355472 kB
Buffers:        125988 kB
Cached:         516524 kB
SwapCached:          0 kB
Active:         181096 kB
Inactive:       524328 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      263355472 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           11812 kB
Writeback:           0 kB
AnonPages:       64004 kB
Mapped:          14416 kB
Slab:           119112 kB
PageTables:       4488 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   166224 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61732 MiB
End job on node200 at 2011-06-07 15:39:20