Trace number 3472285

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 20430.1 5102.25

General information on the benchmark

Nameindustrial/vliw_sat_4.0/
9vliw_m_9stages_iq3_C1_bug5.cnf
MD5SUMc77a56af24208a6401c763e11a38de01
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark84.6771
Satisfiable
(Un)Satisfiability was proved
Number of variables520770
Number of clauses13380350
Sum of the clauses size39210884
Maximum clause length6689
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 28710587
Number of clauses of size 34215482
Number of clauses of size 425225
Number of clauses of size 515593
Number of clauses of size over 5413462

Solver Data 

0.00/0.01	c clasp version 2.0.0-RC2
0.00/0.01	c Reading from ...398834/instance-3472285-1307398834.cnf
6.08/6.18	c Solving...
20427.82/5100.02	
20427.82/5100.02	*** INTERRUPTED! ***
20427.82/5100.02	*** Shutting down threads...

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3472285-1307398834/watcher-3472285-1307398834 -o /tmp/evaluation-result-3472285-1307398834/solver-3472285-1307398834 -C 160000 -W 5100 -M 256000 HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472285-1307398834.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 262144000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 262195200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 23.16 29.69 28.43 1/413 11671
/proc/meminfo: memFree=263273676/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=87000 CPUtime=0.02 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 1874 0 0 0 2 0 0 0 25 0 1 0 5271427 89088000 1804 270582939648 4194304 4964800 140735297745008 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 21750 1808 373 189 0 16624 0

[startup+0.045237 s]
/proc/loadavg: 23.16 29.69 28.43 1/413 11671
/proc/meminfo: memFree=263273676/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=87000 CPUtime=0.03 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 2644 0 0 0 3 0 0 0 25 0 1 0 5271427 89088000 2575 270582939648 4194304 4964800 140735297745008 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 21750 2579 373 189 0 16624 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 87000

[startup+0.101339 s]
/proc/loadavg: 23.16 29.69 28.43 1/413 11671
/proc/meminfo: memFree=263273676/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=87000 CPUtime=0.09 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 6950 0 0 0 8 1 0 0 25 0 1 0 5271427 89088000 6880 270582939648 4194304 4964800 140735297745008 18446744073709551615 4700412 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 21750 6885 373 189 0 16624 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 87000

[startup+0.300353 s]
/proc/loadavg: 23.16 29.69 28.43 1/413 11671
/proc/meminfo: memFree=263273676/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=117520 CPUtime=0.28 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 27803 0 0 0 23 5 0 0 25 0 1 0 5271427 120340480 24042 270582939648 4194304 4964800 140735297745008 18446744073709551615 4679106 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 29380 24047 374 189 0 24254 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 117520

[startup+0.700357 s]
/proc/loadavg: 23.16 29.69 28.43 1/413 11671
/proc/meminfo: memFree=263273676/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=135428 CPUtime=0.69 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 32813 0 0 0 61 8 0 0 25 0 1 0 5271427 138678272 28592 270582939648 4194304 4964800 140735297745008 18446744073709551615 4581548 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 33857 28593 385 189 0 28731 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 135428

[startup+1.50039 s]
/proc/loadavg: 23.16 29.69 28.43 2/414 11672
/proc/meminfo: memFree=263150164/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=172144 CPUtime=1.49 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 42270 0 0 0 137 12 0 0 25 0 1 0 5271427 176275456 37695 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650955760 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 43036 37695 385 189 0 37910 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 172144

[startup+3.10045 s]
/proc/loadavg: 23.16 29.69 28.43 2/414 11673
/proc/meminfo: memFree=263093092/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=258336 CPUtime=3.08 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 64598 0 0 0 284 24 0 0 25 0 1 0 5271427 264536064 58832 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 64584 58832 385 189 0 59458 0
Current children cumulated CPU time (s) 3.08
Current children cumulated vsize (KiB) 258336

[startup+6.30047 s]
/proc/loadavg: 21.38 29.22 28.28 2/414 11673
/proc/meminfo: memFree=262905724/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=819184 CPUtime=9.06 cores=0-31
/proc/11671/stat : 11671 (claspmt) S 11669 11671 11645 0 -1 4202496 154688 0 0 0 579 327 0 0 18 0 32 0 5271427 838844416 137783 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 29 0 0 0
/proc/11671/statm: 204796 137854 424 189 0 199145 0
Current children cumulated CPU time (s) 9.06
Current children cumulated vsize (KiB) 819184

[startup+12.7027 s]
/proc/loadavg: 22.15 29.25 28.30 2/445 11704
/proc/meminfo: memFree=260911400/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=3768052 CPUtime=71.15 cores=0-31
/proc/11671/stat : 11671 (claspmt) S 11669 11671 11645 0 -1 4202496 777123 0 0 0 1283 5832 0 0 18 0 32 0 5271427 3858485248 634104 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 29 0 0 0
/proc/11671/statm: 942013 634112 428 189 0 936362 0
[pid=11671/tid=11674] ppid=11669 vsize=3768052 CPUtime=2.13 cores=0-31
/proc/11671/task/11674/stat : 11674 (claspmt) R 11669 11671 11645 0 -1 4202560 21411 0 0 0 28 185 0 0 16 0 32 0 5272046 3858485248 634119 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 26 0 0 0
[pid=11671/tid=11675] ppid=11669 vsize=3768052 CPUtime=2.1 cores=0-31
/proc/11671/task/11675/stat : 11675 (claspmt) S 11669 11671 11645 0 -1 4202560 19131 0 0 0 19 191 0 0 16 0 32 0 5272046 3858485248 634126 270582939648 4194304 4964800 140735297745008 18446744073709551615 258651060958 0 0 4096 18946 18446744071566926424 0 0 -1 3 0 0 0
[pid=11671/tid=11676] ppid=11669 vsize=3768052 CPUtime=2.1 cores=0-31
/proc/11671/task/11676/stat : 11676 (claspmt) R 11669 11671 11645 0 -1 4202560 19409 0 0 0 21 189 0 0 16 0 32 0 5272046 3858485248 634130 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=11671/tid=11677] ppid=11669 vsize=3768052 CPUtime=2.11 cores=0-31
/proc/11671/task/11677/stat : 11677 (claspmt) D 11669 11671 11645 0 -1 4202560 17937 0 0 0 21 190 0 0 16 0 32 0 5272046 3858485248 634134 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=11671/tid=11678] ppid=11669 vsize=3768052 CPUtime=2.05 cores=0-31
/proc/11671/task/11678/stat : 11678 (claspmt) D 11669 11671 11645 0 -1 4202560 21664 0 0 0 23 182 0 0 16 0 32 0 5272046 3858485248 634135 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=11671/tid=11679] ppid=11669 vsize=3768052 CPUtime=2.04 cores=0-31
/proc/11671/task/11679/stat : 11679 (claspmt) D 11669 11671 11645 0 -1 4202560 21539 0 0 0 21 183 0 0 16 0 32 0 5272046 3858485248 634138 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 20 0 0 0
[pid=11671/tid=11680] ppid=11669 vsize=3768052 CPUtime=2.07 cores=0-31
/proc/11671/task/11680/stat : 11680 (claspmt) D 11669 11671 11645 0 -1 4202560 22316 0 0 0 18 189 0 0 16 0 32 0 5272046 3858485248 634143 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650653371 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=11671/tid=11681] ppid=11669 vsize=3768052 CPUtime=2.04 cores=0-31
/proc/11671/task/11681/stat : 11681 (claspmt) S 11669 11671 11645 0 -1 4202560 21389 0 0 0 23 181 0 0 15 0 32 0 5272046 3858485248 634146 270582939648 4194304 4964800 140735297745008 18446744073709551615 258651060958 0 0 4096 18946 18446604575210470880 0 0 -1 19 0 0 0
[pid=11671/tid=11682] ppid=11669 vsize=3768052 CPUtime=2.07 cores=0-31
/proc/11671/task/11682/stat : 11682 (claspmt) R 11669 11671 11645 0 -1 4202560 18403 0 0 0 21 186 0 0 15 0 32 0 5272046 3858485248 634156 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=11671/tid=11683] ppid=11669 vsize=3768052 CPUtime=2.12 cores=0-31
/proc/11671/task/11683/stat : 11683 (claspmt) D 11669 11671 11645 0 -1 4202560 19759 0 0 0 21 191 0 0 16 0 32 0 5272046 3858485248 634161 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=11671/tid=11684] ppid=11669 vsize=3768052 CPUtime=2.07 cores=0-31
/proc/11671/task/11684/stat : 11684 (claspmt) D 11669 11671 11645 0 -1 4202560 23284 0 0 0 27 180 0 0 15 0 32 0 5272047 3858485248 634165 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 14 0 0 0
[pid=11671/tid=11685] ppid=11669 vsize=3768052 CPUtime=2.06 cores=0-31
/proc/11671/task/11685/stat : 11685 (claspmt) D 11669 11671 11645 0 -1 4202560 21767 0 0 0 27 179 0 0 16 0 32 0 5272047 3858485248 634165 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=11671/tid=11686] ppid=11669 vsize=3768052 CPUtime=2.13 cores=0-31
/proc/11671/task/11686/stat : 11686 (claspmt) D 11669 11671 11645 0 -1 4202560 19811 0 0 0 26 187 0 0 16 0 32 0 5272047 3858485248 634172 270582939648 4194304 4964800 140735297745008 18446744073709551615 4685574 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=11671/tid=11687] ppid=11669 vsize=3768052 CPUtime=2.09 cores=0-31
/proc/11671/task/11687/stat : 11687 (claspmt) S 11669 11671 11645 0 -1 4202560 22014 0 0 0 21 188 0 0 15 0 32 0 5272047 3858485248 634172 270582939648 4194304 4964800 140735297745008 18446744073709551615 258651060958 0 0 4096 18946 18446744071566925424 0 0 -1 10 0 0 0
[pid=11671/tid=11688] ppid=11669 vsize=3768052 CPUtime=2.08 cores=0-31
/proc/11671/task/11688/stat : 11688 (claspmt) D 11669 11671 11645 0 -1 4202560 22661 0 0 0 21 187 0 0 16 0 32 0 5272047 3858485248 634173 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616632 0 0 4096 18946 0 0 0 -1 18 0 0 0
[pid=11671/tid=11689] ppid=11669 vsize=3768052 CPUtime=2.13 cores=0-31
/proc/11671/task/11689/stat : 11689 (claspmt) D 11669 11671 11645 0 -1 4202560 20193 0 0 0 24 189 0 0 16 0 32 0 5272047 3858485248 634174 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650653361 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=11671/tid=11690] ppid=11669 vsize=3768052 CPUtime=2.09 cores=0-31
/proc/11671/task/11690/stat : 11690 (claspmt) R 11669 11671 11645 0 -1 4202560 21852 0 0 0 28 181 0 0 16 0 32 0 5272047 3858485248 634191 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 8 0 0 0
[pid=11671/tid=11691] ppid=11669 vsize=3768052 CPUtime=2.05 cores=0-31
/proc/11671/task/11691/stat : 11691 (claspmt) D 11669 11671 11645 0 -1 4202560 22781 0 0 0 22 183 0 0 16 0 32 0 5272047 3858485248 634193 270582939648 4194304 4964800 140735297745008 18446744073709551615 4697753 0 0 4096 18946 0 0 0 -1 30 0 0 0
[pid=11671/tid=11692] ppid=11669 vsize=3768052 CPUtime=2.09 cores=0-31
/proc/11671/task/11692/stat : 11692 (claspmt) D 11669 11671 11645 0 -1 4202560 22055 0 0 0 23 186 0 0 16 0 32 0 5272047 3858485248 634200 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=11671/tid=11693] ppid=11669 vsize=3768052 CPUtime=2.12 cores=0-31
/proc/11671/task/11693/stat : 11693 (claspmt) D 11669 11671 11645 0 -1 4202560 21176 0 0 0 24 188 0 0 16 0 32 0 5272047 3858485248 634204 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 9 0 0 0
[pid=11671/tid=11694] ppid=11669 vsize=3768052 CPUtime=2.12 cores=0-31
/proc/11671/task/11694/stat : 11694 (claspmt) S 11669 11671 11645 0 -1 4202560 21576 0 0 0 21 191 0 0 16 0 32 0 5272047 3858485248 634205 270582939648 4194304 4964800 140735297745008 18446744073709551615 258651060958 0 0 4096 18946 18446744071566931024 0 0 -1 11 0 0 0
[pid=11671/tid=11695] ppid=11669 vsize=3768052 CPUtime=2.06 cores=0-31
/proc/11671/task/11695/stat : 11695 (claspmt) D 11669 11671 11645 0 -1 4202560 21411 0 0 0 20 186 0 0 16 0 32 0 5272047 3858485248 634207 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650653299 0 0 4096 18946 0 0 0 -1 21 0 0 0
[pid=11671/tid=11696] ppid=11669 vsize=3768052 CPUtime=1.99 cores=0-31
/proc/11671/task/11696/stat : 11696 (claspmt) D 11669 11671 11645 0 -1 4202560 22500 0 0 0 21 178 0 0 16 0 32 0 5272047 3858485248 634211 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 23 0 0 0
[pid=11671/tid=11697] ppid=11669 vsize=3768052 CPUtime=2.13 cores=0-31
/proc/11671/task/11697/stat : 11697 (claspmt) D 11669 11671 11645 0 -1 4202560 21127 0 0 0 21 192 0 0 16 0 32 0 5272047 3858485248 634211 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 17 0 0 0
[pid=11671/tid=11698] ppid=11669 vsize=3768052 CPUtime=2.09 cores=0-31
/proc/11671/task/11698/stat : 11698 (claspmt) S 11669 11671 11645 0 -1 4202560 21017 0 0 0 24 185 0 0 15 0 32 0 5272047 3858485248 634219 270582939648 4194304 4964800 140735297745008 18446744073709551615 258651060958 0 0 4096 18946 18446744071566928184 0 0 -1 28 0 0 0
[pid=11671/tid=11699] ppid=11669 vsize=3768052 CPUtime=2.12 cores=0-31
/proc/11671/task/11699/stat : 11699 (claspmt) R 11669 11671 11645 0 -1 4202560 21022 0 0 0 19 193 0 0 16 0 32 0 5272047 3858485248 634233 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650653351 0 0 4096 18946 0 0 0 -1 29 0 0 0
[pid=11671/tid=11700] ppid=11669 vsize=3768052 CPUtime=2.12 cores=0-31
/proc/11671/task/11700/stat : 11700 (claspmt) D 11669 11671 11645 0 -1 4202560 21771 0 0 0 22 190 0 0 15 0 32 0 5272047 3858485248 634235 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 12 0 0 0
[pid=11671/tid=11701] ppid=11669 vsize=3768052 CPUtime=2.05 cores=0-31
/proc/11671/task/11701/stat : 11701 (claspmt) S 11669 11671 11645 0 -1 4202560 21593 0 0 0 22 183 0 0 16 0 32 0 5272047 3858485248 634235 270582939648 4194304 4964800 140735297745008 18446744073709551615 258651060958 0 0 4096 18946 18446744071566924824 0 0 -1 16 0 0 0
[pid=11671/tid=11702] ppid=11669 vsize=3768052 CPUtime=2.06 cores=0-31
/proc/11671/task/11702/stat : 11702 (claspmt) D 11669 11671 11645 0 -1 4202560 21298 0 0 0 25 181 0 0 16 0 32 0 5272047 3858485248 634238 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650616986 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=11671/tid=11703] ppid=11669 vsize=3768052 CPUtime=2.08 cores=0-31
/proc/11671/task/11703/stat : 11703 (claspmt) D 11669 11671 11645 0 -1 4202560 21787 0 0 0 24 184 0 0 15 0 32 0 5272047 3858485248 634239 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 15 0 0 0
[pid=11671/tid=11704] ppid=11669 vsize=3768052 CPUtime=2.09 cores=0-31
/proc/11671/task/11704/stat : 11704 (claspmt) D 11669 11671 11645 0 -1 4202560 22397 0 0 0 20 189 0 0 16 0 32 0 5272048 3858485248 634241 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650999207 0 0 4096 18946 0 0 0 -1 13 0 0 0
Current children cumulated CPU time (s) 71.15
Current children cumulated vsize (KiB) 3768052

[startup+25.5022 s]
/proc/loadavg: 19.78 28.39 28.03 5/445 11705
/proc/meminfo: memFree=259485160/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=5186196 CPUtime=133.75 cores=0-31
/proc/11671/stat : 11671 (claspmt) S 11669 11671 11645 0 -1 4202496 1091378 0 0 0 6591 6784 0 0 18 0 32 0 5271427 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 29 0 0 0
/proc/11671/statm: 1296549 947121 433 189 0 1290898 0
[pid=11671/tid=11674] ppid=11669 vsize=5186196 CPUtime=2.51 cores=0-31
/proc/11671/task/11674/stat : 11674 (claspmt) S 11669 11671 11645 0 -1 4202560 27624 0 0 0 41 210 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574963678688 0 0 -1 10 0 0 0
[pid=11671/tid=11675] ppid=11669 vsize=5186196 CPUtime=3.31 cores=0-31
/proc/11671/task/11675/stat : 11675 (claspmt) S 11669 11671 11645 0 -1 4202560 32537 0 0 0 106 225 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572921855456 0 0 -1 15 0 0 0
[pid=11671/tid=11676] ppid=11669 vsize=5186196 CPUtime=3.33 cores=0-31
/proc/11671/task/11676/stat : 11676 (claspmt) S 11669 11671 11645 0 -1 4202560 32983 0 0 0 107 226 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569194782176 0 0 -1 26 0 0 0
[pid=11671/tid=11677] ppid=11669 vsize=5186196 CPUtime=2.61 cores=0-31
/proc/11671/task/11677/stat : 11677 (claspmt) S 11669 11671 11645 0 -1 4202560 26785 0 0 0 42 219 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940295648 0 0 -1 0 0 0 0
[pid=11671/tid=11678] ppid=11669 vsize=5186196 CPUtime=11.51 cores=0-31
/proc/11671/task/11678/stat : 11678 (claspmt) R 11669 11671 11645 0 -1 4202560 32905 0 0 0 937 214 0 0 25 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=11671/tid=11679] ppid=11669 vsize=5186196 CPUtime=3.19 cores=0-31
/proc/11671/task/11679/stat : 11679 (claspmt) S 11669 11671 11645 0 -1 4202560 36027 0 0 0 104 215 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604568951389664 0 0 -1 20 0 0 0
[pid=11671/tid=11680] ppid=11669 vsize=5186196 CPUtime=3.6 cores=0-31
/proc/11671/task/11680/stat : 11680 (claspmt) S 11669 11671 11645 0 -1 4202560 31774 0 0 0 132 228 0 0 17 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572921830880 0 0 -1 22 0 0 0
[pid=11671/tid=11681] ppid=11669 vsize=5186196 CPUtime=2.41 cores=0-31
/proc/11671/task/11681/stat : 11681 (claspmt) S 11669 11671 11645 0 -1 4202560 27583 0 0 0 37 204 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574963793376 0 0 -1 19 0 0 0
[pid=11671/tid=11682] ppid=11669 vsize=5186196 CPUtime=2.58 cores=0-31
/proc/11671/task/11682/stat : 11682 (claspmt) S 11669 11671 11645 0 -1 4202560 26666 0 0 0 40 218 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572921839072 0 0 -1 25 0 0 0
[pid=11671/tid=11683] ppid=11669 vsize=5186196 CPUtime=3.38 cores=0-31
/proc/11671/task/11683/stat : 11683 (claspmt) S 11669 11671 11645 0 -1 4202560 33320 0 0 0 114 224 0 0 16 0 32 0 5272046 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940189152 0 0 -1 2 0 0 0
[pid=11671/tid=11684] ppid=11669 vsize=5186196 CPUtime=3.23 cores=0-31
/proc/11671/task/11684/stat : 11684 (claspmt) S 11669 11671 11645 0 -1 4202560 35971 0 0 0 114 209 0 0 16 0 32 0 5272047 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878494688 0 0 -1 14 0 0 0
[pid=11671/tid=11685] ppid=11669 vsize=5186196 CPUtime=2.43 cores=0-31
/proc/11671/task/11685/stat : 11685 (claspmt) S 11669 11671 11645 0 -1 4202560 27417 0 0 0 42 201 0 0 15 0 32 0 5272047 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878470112 0 0 -1 26 0 0 0
[pid=11671/tid=11686] ppid=11669 vsize=5186196 CPUtime=11.38 cores=0-31
/proc/11671/task/11686/stat : 11686 (claspmt) R 11669 11671 11645 0 -1 4202560 32563 0 0 0 915 223 0 0 25 0 32 0 5272047 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=11671/tid=11687] ppid=11669 vsize=5186196 CPUtime=3.23 cores=0-31
/proc/11671/task/11687/stat : 11687 (claspmt) S 11669 11671 11645 0 -1 4202560 35088 0 0 0 102 221 0 0 16 0 32 0 5272047 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569194773984 0 0 -1 18 0 0 0
[pid=11671/tid=11688] ppid=11669 vsize=5186196 CPUtime=3.62 cores=0-31
/proc/11671/task/11688/stat : 11688 (claspmt) S 11669 11671 11645 0 -1 4202560 31965 0 0 0 137 225 0 0 17 0 32 0 5272047 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574091648480 0 0 -1 6 0 0 0
[pid=11671/tid=11689] ppid=11669 vsize=5186196 CPUtime=2.57 cores=0-31
/proc/11671/task/11689/stat : 11689 (claspmt) S 11669 11671 11645 0 -1 4202560 27329 0 0 0 42 215 0 0 15 0 32 0 5272047 5310664704 947121 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569021767136 0 0 -1 29 0 0 0

################
# More data... #
################

[pid=11671/tid=11681] ppid=11669 vsize=5251732 CPUtime=2.41 cores=0-31
/proc/11671/task/11681/stat : 11681 (claspmt) S 11669 11671 11645 0 -1 4202560 27583 0 0 0 37 204 0 0 16 0 32 0 5272046 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574963793376 0 0 -1 19 0 0 0
[pid=11671/tid=11682] ppid=11669 vsize=5251732 CPUtime=2.58 cores=0-31
/proc/11671/task/11682/stat : 11682 (claspmt) S 11669 11671 11645 0 -1 4202560 26666 0 0 0 40 218 0 0 16 0 32 0 5272046 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572921839072 0 0 -1 25 0 0 0
[pid=11671/tid=11683] ppid=11669 vsize=5251732 CPUtime=3.38 cores=0-31
/proc/11671/task/11683/stat : 11683 (claspmt) S 11669 11671 11645 0 -1 4202560 33320 0 0 0 114 224 0 0 16 0 32 0 5272046 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940189152 0 0 -1 2 0 0 0
[pid=11671/tid=11684] ppid=11669 vsize=5251732 CPUtime=3.23 cores=0-31
/proc/11671/task/11684/stat : 11684 (claspmt) S 11669 11671 11645 0 -1 4202560 35971 0 0 0 114 209 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878494688 0 0 -1 14 0 0 0
[pid=11671/tid=11685] ppid=11669 vsize=5251732 CPUtime=2.43 cores=0-31
/proc/11671/task/11685/stat : 11685 (claspmt) S 11669 11671 11645 0 -1 4202560 27417 0 0 0 42 201 0 0 15 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878470112 0 0 -1 26 0 0 0
[pid=11671/tid=11686] ppid=11669 vsize=5251732 CPUtime=5067.22 cores=0-31
/proc/11671/task/11686/stat : 11686 (claspmt) R 11669 11671 11645 0 -1 4202560 35416 0 0 0 506492 230 0 0 25 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=11671/tid=11687] ppid=11669 vsize=5251732 CPUtime=3.23 cores=0-31
/proc/11671/task/11687/stat : 11687 (claspmt) S 11669 11671 11645 0 -1 4202560 35088 0 0 0 102 221 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569194773984 0 0 -1 18 0 0 0
[pid=11671/tid=11688] ppid=11669 vsize=5251732 CPUtime=3.62 cores=0-31
/proc/11671/task/11688/stat : 11688 (claspmt) S 11669 11671 11645 0 -1 4202560 31965 0 0 0 137 225 0 0 17 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574091648480 0 0 -1 6 0 0 0
[pid=11671/tid=11689] ppid=11669 vsize=5251732 CPUtime=2.57 cores=0-31
/proc/11671/task/11689/stat : 11689 (claspmt) S 11669 11671 11645 0 -1 4202560 27329 0 0 0 42 215 0 0 15 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569021767136 0 0 -1 29 0 0 0
[pid=11671/tid=11690] ppid=11669 vsize=5251732 CPUtime=2.44 cores=0-31
/proc/11671/task/11690/stat : 11690 (claspmt) S 11669 11671 11645 0 -1 4202560 27539 0 0 0 42 202 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604640110042592 0 0 -1 8 0 0 0
[pid=11671/tid=11691] ppid=11669 vsize=5251732 CPUtime=3.25 cores=0-31
/proc/11671/task/11691/stat : 11691 (claspmt) S 11669 11671 11645 0 -1 4202560 33975 0 0 0 113 212 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604575210470880 0 0 -1 30 0 0 0
[pid=11671/tid=11692] ppid=11669 vsize=5251732 CPUtime=3.26 cores=0-31
/proc/11671/task/11692/stat : 11692 (claspmt) S 11669 11671 11645 0 -1 4202560 34959 0 0 0 109 217 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878478304 0 0 -1 1 0 0 0
[pid=11671/tid=11693] ppid=11669 vsize=5251732 CPUtime=2.5 cores=0-31
/proc/11671/task/11693/stat : 11693 (claspmt) S 11669 11671 11645 0 -1 4202560 27549 0 0 0 39 211 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574091689440 0 0 -1 1 0 0 0
[pid=11671/tid=11694] ppid=11669 vsize=5251732 CPUtime=5067.35 cores=0-31
/proc/11671/task/11694/stat : 11694 (claspmt) R 11669 11671 11645 0 -1 4202560 36605 0 0 0 506501 234 0 0 25 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 4668592 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=11671/tid=11695] ppid=11669 vsize=5251732 CPUtime=3.3 cores=0-31
/proc/11671/task/11695/stat : 11695 (claspmt) S 11669 11671 11645 0 -1 4202560 33398 0 0 0 112 218 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878584800 0 0 -1 9 0 0 0
[pid=11671/tid=11696] ppid=11669 vsize=5251732 CPUtime=3.61 cores=0-31
/proc/11671/task/11696/stat : 11696 (claspmt) S 11669 11671 11645 0 -1 4202560 32384 0 0 0 145 216 0 0 17 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940197344 0 0 -1 22 0 0 0
[pid=11671/tid=11697] ppid=11669 vsize=5251732 CPUtime=2.54 cores=0-31
/proc/11671/task/11697/stat : 11697 (claspmt) S 11669 11671 11645 0 -1 4202560 27637 0 0 0 36 218 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604575210446304 0 0 -1 21 0 0 0
[pid=11671/tid=11698] ppid=11669 vsize=5251732 CPUtime=2.48 cores=0-31
/proc/11671/task/11698/stat : 11698 (claspmt) S 11669 11671 11645 0 -1 4202560 27440 0 0 0 39 209 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569194741216 0 0 -1 28 0 0 0
[pid=11671/tid=11699] ppid=11669 vsize=5251732 CPUtime=3.27 cores=0-31
/proc/11671/task/11699/stat : 11699 (claspmt) S 11669 11671 11645 0 -1 4202560 33976 0 0 0 101 226 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604568951266784 0 0 -1 17 0 0 0
[pid=11671/tid=11700] ppid=11669 vsize=5251732 CPUtime=3.3 cores=0-31
/proc/11671/task/11700/stat : 11700 (claspmt) S 11669 11671 11645 0 -1 4202560 33849 0 0 0 109 221 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574091705824 0 0 -1 12 0 0 0
[pid=11671/tid=11701] ppid=11669 vsize=5251732 CPUtime=2.43 cores=0-31
/proc/11671/task/11701/stat : 11701 (claspmt) S 11669 11671 11645 0 -1 4202560 27612 0 0 0 37 206 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604570291371488 0 0 -1 16 0 0 0
[pid=11671/tid=11702] ppid=11669 vsize=5251732 CPUtime=5067.13 cores=0-31
/proc/11671/task/11702/stat : 11702 (claspmt) R 11669 11671 11645 0 -1 4202560 36496 0 0 0 506468 245 0 0 25 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 4668721 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=11671/tid=11703] ppid=11669 vsize=5251732 CPUtime=3.37 cores=0-31
/proc/11671/task/11703/stat : 11703 (claspmt) S 11669 11671 11645 0 -1 4202560 36174 0 0 0 122 215 0 0 16 0 32 0 5272047 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940271072 0 0 -1 8 0 0 0
[pid=11671/tid=11704] ppid=11669 vsize=5251732 CPUtime=3.71 cores=0-31
/proc/11671/task/11704/stat : 11704 (claspmt) S 11669 11671 11645 0 -1 4202560 31490 0 0 0 148 223 0 0 17 0 32 0 5272048 5377773568 964278 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604571888860640 0 0 -1 13 0 0 0
Current children cumulated CPU time (s) 20356.9
Current children cumulated vsize (KiB) 5251732



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5100 s]
/proc/loadavg: 4.00 4.02 4.11 5/445 11854
/proc/meminfo: memFree=259411024/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=5251732 CPUtime=20427.8 cores=0-31
/proc/11671/stat : 11671 (claspmt) S 11669 11671 11645 0 -1 4202496 1108543 0 0 0 2035914 6868 0 0 18 0 32 0 5271427 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446744073709551615 0 0 17 29 0 0 0
/proc/11671/statm: 1312933 964280 433 189 0 1307282 0
[pid=11671/tid=11674] ppid=11669 vsize=5251732 CPUtime=2.51 cores=0-31
/proc/11671/task/11674/stat : 11674 (claspmt) S 11669 11671 11645 0 -1 4202560 27624 0 0 0 41 210 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574963678688 0 0 -1 10 0 0 0
[pid=11671/tid=11675] ppid=11669 vsize=5251732 CPUtime=3.31 cores=0-31
/proc/11671/task/11675/stat : 11675 (claspmt) S 11669 11671 11645 0 -1 4202560 32537 0 0 0 106 225 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572921855456 0 0 -1 15 0 0 0
[pid=11671/tid=11676] ppid=11669 vsize=5251732 CPUtime=3.33 cores=0-31
/proc/11671/task/11676/stat : 11676 (claspmt) S 11669 11671 11645 0 -1 4202560 32983 0 0 0 107 226 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569194782176 0 0 -1 26 0 0 0
[pid=11671/tid=11677] ppid=11669 vsize=5251732 CPUtime=2.61 cores=0-31
/proc/11671/task/11677/stat : 11677 (claspmt) S 11669 11671 11645 0 -1 4202560 26785 0 0 0 42 219 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940295648 0 0 -1 0 0 0 0
[pid=11671/tid=11678] ppid=11669 vsize=5251732 CPUtime=5084.99 cores=0-31
/proc/11671/task/11678/stat : 11678 (claspmt) R 11669 11671 11645 0 -1 4202560 39825 0 0 0 508240 259 0 0 25 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 24 0 0 0
[pid=11671/tid=11679] ppid=11669 vsize=5251732 CPUtime=3.19 cores=0-31
/proc/11671/task/11679/stat : 11679 (claspmt) S 11669 11671 11645 0 -1 4202560 36027 0 0 0 104 215 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604568951389664 0 0 -1 20 0 0 0
[pid=11671/tid=11680] ppid=11669 vsize=5251732 CPUtime=3.6 cores=0-31
/proc/11671/task/11680/stat : 11680 (claspmt) S 11669 11671 11645 0 -1 4202560 31774 0 0 0 132 228 0 0 17 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572921830880 0 0 -1 22 0 0 0
[pid=11671/tid=11681] ppid=11669 vsize=5251732 CPUtime=2.41 cores=0-31
/proc/11671/task/11681/stat : 11681 (claspmt) S 11669 11671 11645 0 -1 4202560 27583 0 0 0 37 204 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574963793376 0 0 -1 19 0 0 0
[pid=11671/tid=11682] ppid=11669 vsize=5251732 CPUtime=2.58 cores=0-31
/proc/11671/task/11682/stat : 11682 (claspmt) S 11669 11671 11645 0 -1 4202560 26666 0 0 0 40 218 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572921839072 0 0 -1 25 0 0 0
[pid=11671/tid=11683] ppid=11669 vsize=5251732 CPUtime=3.38 cores=0-31
/proc/11671/task/11683/stat : 11683 (claspmt) S 11669 11671 11645 0 -1 4202560 33320 0 0 0 114 224 0 0 16 0 32 0 5272046 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940189152 0 0 -1 2 0 0 0
[pid=11671/tid=11684] ppid=11669 vsize=5251732 CPUtime=3.23 cores=0-31
/proc/11671/task/11684/stat : 11684 (claspmt) S 11669 11671 11645 0 -1 4202560 35971 0 0 0 114 209 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878494688 0 0 -1 14 0 0 0
[pid=11671/tid=11685] ppid=11669 vsize=5251732 CPUtime=2.43 cores=0-31
/proc/11671/task/11685/stat : 11685 (claspmt) S 11669 11671 11645 0 -1 4202560 27417 0 0 0 42 201 0 0 15 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878470112 0 0 -1 26 0 0 0
[pid=11671/tid=11686] ppid=11669 vsize=5251732 CPUtime=5084.94 cores=0-31
/proc/11671/task/11686/stat : 11686 (claspmt) R 11669 11671 11645 0 -1 4202560 35417 0 0 0 508264 230 0 0 25 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 22 0 0 0
[pid=11671/tid=11687] ppid=11669 vsize=5251732 CPUtime=3.23 cores=0-31
/proc/11671/task/11687/stat : 11687 (claspmt) S 11669 11671 11645 0 -1 4202560 35088 0 0 0 102 221 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569194773984 0 0 -1 18 0 0 0
[pid=11671/tid=11688] ppid=11669 vsize=5251732 CPUtime=3.62 cores=0-31
/proc/11671/task/11688/stat : 11688 (claspmt) S 11669 11671 11645 0 -1 4202560 31965 0 0 0 137 225 0 0 17 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574091648480 0 0 -1 6 0 0 0
[pid=11671/tid=11689] ppid=11669 vsize=5251732 CPUtime=2.57 cores=0-31
/proc/11671/task/11689/stat : 11689 (claspmt) S 11669 11671 11645 0 -1 4202560 27329 0 0 0 42 215 0 0 15 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569021767136 0 0 -1 29 0 0 0
[pid=11671/tid=11690] ppid=11669 vsize=5251732 CPUtime=2.44 cores=0-31
/proc/11671/task/11690/stat : 11690 (claspmt) S 11669 11671 11645 0 -1 4202560 27539 0 0 0 42 202 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604640110042592 0 0 -1 8 0 0 0
[pid=11671/tid=11691] ppid=11669 vsize=5251732 CPUtime=3.25 cores=0-31
/proc/11671/task/11691/stat : 11691 (claspmt) S 11669 11671 11645 0 -1 4202560 33975 0 0 0 113 212 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604575210470880 0 0 -1 30 0 0 0
[pid=11671/tid=11692] ppid=11669 vsize=5251732 CPUtime=3.26 cores=0-31
/proc/11671/task/11692/stat : 11692 (claspmt) S 11669 11671 11645 0 -1 4202560 34959 0 0 0 109 217 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878478304 0 0 -1 1 0 0 0
[pid=11671/tid=11693] ppid=11669 vsize=5251732 CPUtime=2.5 cores=0-31
/proc/11671/task/11693/stat : 11693 (claspmt) S 11669 11671 11645 0 -1 4202560 27549 0 0 0 39 211 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574091689440 0 0 -1 1 0 0 0
[pid=11671/tid=11694] ppid=11669 vsize=5251732 CPUtime=5085.07 cores=0-31
/proc/11671/task/11694/stat : 11694 (claspmt) R 11669 11671 11645 0 -1 4202560 36606 0 0 0 508273 234 0 0 25 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 4678045 0 0 4096 18946 0 0 0 -1 25 0 0 0
[pid=11671/tid=11695] ppid=11669 vsize=5251732 CPUtime=3.3 cores=0-31
/proc/11671/task/11695/stat : 11695 (claspmt) S 11669 11671 11645 0 -1 4202560 33398 0 0 0 112 218 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569878584800 0 0 -1 9 0 0 0
[pid=11671/tid=11696] ppid=11669 vsize=5251732 CPUtime=3.61 cores=0-31
/proc/11671/task/11696/stat : 11696 (claspmt) S 11669 11671 11645 0 -1 4202560 32384 0 0 0 145 216 0 0 17 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940197344 0 0 -1 22 0 0 0
[pid=11671/tid=11697] ppid=11669 vsize=5251732 CPUtime=2.54 cores=0-31
/proc/11671/task/11697/stat : 11697 (claspmt) S 11669 11671 11645 0 -1 4202560 27637 0 0 0 36 218 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604575210446304 0 0 -1 21 0 0 0
[pid=11671/tid=11698] ppid=11669 vsize=5251732 CPUtime=2.48 cores=0-31
/proc/11671/task/11698/stat : 11698 (claspmt) S 11669 11671 11645 0 -1 4202560 27440 0 0 0 39 209 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604569194741216 0 0 -1 28 0 0 0
[pid=11671/tid=11699] ppid=11669 vsize=5251732 CPUtime=3.27 cores=0-31
/proc/11671/task/11699/stat : 11699 (claspmt) S 11669 11671 11645 0 -1 4202560 33976 0 0 0 101 226 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604568951266784 0 0 -1 17 0 0 0
[pid=11671/tid=11700] ppid=11669 vsize=5251732 CPUtime=3.3 cores=0-31
/proc/11671/task/11700/stat : 11700 (claspmt) S 11669 11671 11645 0 -1 4202560 33849 0 0 0 109 221 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604574091705824 0 0 -1 12 0 0 0
[pid=11671/tid=11701] ppid=11669 vsize=5251732 CPUtime=2.43 cores=0-31
/proc/11671/task/11701/stat : 11701 (claspmt) S 11669 11671 11645 0 -1 4202560 27612 0 0 0 37 206 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604570291371488 0 0 -1 16 0 0 0
[pid=11671/tid=11702] ppid=11669 vsize=5251732 CPUtime=5084.85 cores=0-31
/proc/11671/task/11702/stat : 11702 (claspmt) R 11669 11671 11645 0 -1 4202560 36496 0 0 0 508240 245 0 0 25 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 27 0 0 0
[pid=11671/tid=11703] ppid=11669 vsize=5251732 CPUtime=3.37 cores=0-31
/proc/11671/task/11703/stat : 11703 (claspmt) S 11669 11671 11645 0 -1 4202560 36174 0 0 0 122 215 0 0 16 0 32 0 5272047 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604572940271072 0 0 -1 8 0 0 0
[pid=11671/tid=11704] ppid=11669 vsize=5251732 CPUtime=3.71 cores=0-31
/proc/11671/task/11704/stat : 11704 (claspmt) S 11669 11671 11645 0 -1 4202560 31490 0 0 0 148 223 0 0 17 0 32 0 5272048 5377773568 964280 270582939648 4194304 4964800 140735297745008 18446744073709551615 258662772457 0 0 4096 18946 18446604571888860640 0 0 -1 13 0 0 0
Current children cumulated CPU time (s) 20427.8
Current children cumulated vsize (KiB) 5251732

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Sending SIGKILL to process tree (bottom up)

Sending SIGKILL to -11671

Solver just ended. Dumping a history of the last processes samples

[startup+5100.6 s]
/proc/loadavg: 4.00 4.02 4.11 2/415 11855
/proc/meminfo: memFree=259524972/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=5047096 CPUtime=20428.5 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 1108554 0 0 0 2035978 6869 0 0 18 0 1 0 5271427 5168226304 932688 270582939648 4194304 4964800 140735297745008 18446744073709551615 4628834 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 1261774 932688 444 189 0 1256123 0
Current children cumulated CPU time (s) 20428.5
Current children cumulated vsize (KiB) 5047096

[startup+5101.4 s]
/proc/loadavg: 4.00 4.02 4.11 2/415 11855
/proc/meminfo: memFree=259524972/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=4859500 CPUtime=20429.2 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 1108554 0 0 0 2036053 6870 0 0 19 0 1 0 5271427 4976128000 890471 270582939648 4194304 4964800 140735297745008 18446744073709551615 4628834 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 1214875 890471 444 189 0 1209224 0
Current children cumulated CPU time (s) 20429.2
Current children cumulated vsize (KiB) 4859500

[startup+5101.83 s]
/proc/loadavg: 4.00 4.02 4.11 2/415 11855
/proc/meminfo: memFree=259753876/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=4786264 CPUtime=20429.7 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4202496 1108554 0 0 0 2036096 6870 0 0 20 0 1 0 5271427 4901134336 872162 270582939648 4194304 4964800 140735297745008 18446744073709551615 258650604632 0 0 4096 18946 0 0 0 17 29 0 0 0
/proc/11671/statm: 1196566 872162 444 189 0 1190915 0
Current children cumulated CPU time (s) 20429.7
Current children cumulated vsize (KiB) 4786264

[startup+5102.2 s]
/proc/loadavg: 4.00 4.02 4.11 2/415 11855
/proc/meminfo: memFree=259753876/264257624 swapFree=67111528/67111528
[pid=11671] ppid=11669 vsize=0 CPUtime=20430 cores=0-31
/proc/11671/stat : 11671 (claspmt) R 11669 11671 11645 0 -1 4203524 1108554 0 0 0 2036116 6888 0 0 21 0 1 0 5271427 0 0 270582939648 0 0 0 0 0 0 0 4096 18946 18446744073709551615 0 0 17 29 0 0 0
/proc/11671/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 20430
Current children cumulated vsize (KiB) 0

Child ended because it received signal 9 (SIGKILL)
Real time (s): 5102.25
CPU time (s): 20430.1
CPU user time (s): 20361.2
CPU system time (s): 68.9275
CPU usage (%): 400.414
Max. virtual memory (cumulated for all children) (KiB): 5251732

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 20361.2
system time used= 68.9275
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1108554
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1101595
involuntary context switches= 30668

runsolver used 27.1809 second user time and 110.648 second system time

The end

Launcher Data 

Begin job on node200 at 2011-06-07 00:20:45
IDJOB=3472285
IDBENCH=20582
IDSOLVER=1961
FILE ID=node200/3472285-1307398834
RUNJOBID= node200-1307398834-11659
PBS_JOBID= 13509172
Free space on /tmp= 61444 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT07/industrial/vliw_sat_4.0/9vliw_m_9stages_iq3_C1_bug5.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3472285-1307398834/watcher-3472285-1307398834 -o /tmp/evaluation-result-3472285-1307398834/solver-3472285-1307398834 -C 160000 -W 5100 -M 256000  HOME/claspmt -t32 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3472285-1307398834.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 256000 MiB
NBCORE= 32

MD5SUM BENCH= c77a56af24208a6401c763e11a38de01
RANDOM SEED=1083955259

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.045
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      263274204 kB
Buffers:         57208 kB
Cached:         710752 kB
SwapCached:          0 kB
Active:         448136 kB
Inactive:       382992 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      263274204 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:          308676 kB
Writeback:           0 kB
AnonPages:       62996 kB
Mapped:          14416 kB
Slab:            75860 kB
PageTables:       4324 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   161852 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 61440 MiB
End job on node200 at 2011-06-07 01:45:48