Trace number 3347465

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 160001 5012.04

General information on the benchmark

Nameapplication/jarvisalo/HWMCC10-combination-timeframe-expansion/
hwmcc10-timeframe-expansion-k45-pdtvisns3p02-tseitin.cnf
MD5SUMa0ec1baa890da62d0c6df7f1e62c6af3
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark31478.4
Satisfiable
(Un)Satisfiability was proved
Number of variables163622
Number of clauses488120
Sum of the clauses size1138944
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2325412
Number of clauses of size 3162706
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347465-1305193425/watcher-3347465-1305193425 -o /tmp/evaluation-result-3347465-1305193425/solver-3347465-1305193425 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347465-1305193425.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 27.41 31.13 31.69 1/413 17027
/proc/meminfo: memFree=255736956/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 23 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0

[startup+0.0616411 s]
/proc/loadavg: 27.41 31.13 31.69 1/413 17027
/proc/meminfo: memFree=255736956/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 23 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100611 s]
/proc/loadavg: 27.41 31.13 31.69 1/413 17027
/proc/meminfo: memFree=255736956/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 23 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.30062 s]
/proc/loadavg: 27.41 31.13 31.69 1/413 17027
/proc/meminfo: memFree=255736956/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 21 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700656 s]
/proc/loadavg: 27.41 31.13 31.69 1/413 17027
/proc/meminfo: memFree=255736956/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 21 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50068 s]
/proc/loadavg: 27.41 31.13 31.69 6/419 17038
/proc/meminfo: memFree=255419824/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 21 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
[pid=17028] ppid=17027 vsize=83696 CPUtime=1.48 cores=0
/proc/17028/stat : 17028 (cryptominisat) R 17027 17027 17001 0 -1 4202496 14772 0 0 0 146 2 0 0 20 0 1 0 630829002 85704704 14273 268435456000 4194304 4587354 140734731610016 18446744073709551615 4248764 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/17028/statm: 20924 14273 304 96 0 17486 0
[pid=17029] ppid=17027 vsize=105184 CPUtime=1.48 cores=1
/proc/17029/stat : 17029 (clasp) R 17027 17027 17001 0 -1 4202496 26950 0 0 0 141 7 0 0 20 0 1 0 630829002 107708416 22486 268435456000 4194304 4785740 140737149889056 18446744073709551615 258650616632 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/17029/statm: 26296 22486 317 145 0 23341 0
[pid=17030] ppid=17027 vsize=486056 CPUtime=1.48 cores=2
/proc/17030/stat : 17030 (TNM) R 17027 17027 17001 0 -1 4202496 17001 0 0 0 144 4 0 0 20 0 1 0 630829002 497721344 16899 268435456000 134512640 134534564 4291563200 18446744073709551615 134529953 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17030/statm: 121514 16899 95 6 0 120179 0
[pid=17031] ppid=17027 vsize=86792 CPUtime=1.49 cores=3
/proc/17031/stat : 17031 (march_hi) R 17027 17027 17001 0 -1 4194304 35365 0 0 0 138 11 0 0 20 0 1 0 630829002 88875008 14377 268435456000 4194304 4331988 140736413374560 18446744073709551615 4251963 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/17031/statm: 21698 14377 126 34 0 20140 0
[pid=17032] ppid=17027 vsize=175328 CPUtime=1.48 cores=4-31
/proc/17032/stat : 17032 (plingeling) R 17027 17027 17001 0 -1 4202496 42592 1240 0 0 139 9 0 0 21 0 1 0 630829002 179535872 42461 268435456000 4194304 4356188 140737428388944 18446744073709551615 258650649821 0 0 4096 17442 0 0 0 17 12 0 0 0
/proc/17032/statm: 43832 42473 129 40 0 42376 0
Current children cumulated CPU time (s) 7.41
Current children cumulated vsize (KiB) 955380

[startup+3.10074 s]
/proc/loadavg: 27.41 31.13 31.69 6/419 17038
/proc/meminfo: memFree=255176920/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 15 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
[pid=17028] ppid=17027 vsize=181124 CPUtime=3.08 cores=0
/proc/17028/stat : 17028 (cryptominisat) R 17027 17027 17001 0 -1 4202496 33917 0 0 0 302 6 0 0 25 0 1 0 630829002 185470976 30036 268435456000 4194304 4587354 140734731610016 18446744073709551615 4375056 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/17028/statm: 45281 30036 315 96 0 41843 0
[pid=17029] ppid=17027 vsize=104524 CPUtime=3.08 cores=1
/proc/17029/stat : 17029 (clasp) R 17027 17027 17001 0 -1 4202496 29810 0 0 0 300 8 0 0 25 0 1 0 630829002 107032576 22794 268435456000 4194304 4785740 140737149889056 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/17029/statm: 26131 22794 338 145 0 23176 0
[pid=17030] ppid=17027 vsize=486056 CPUtime=3.08 cores=2
/proc/17030/stat : 17030 (TNM) R 17027 17027 17001 0 -1 4202496 17001 0 0 0 304 4 0 0 25 0 1 0 630829002 497721344 16899 268435456000 134512640 134534564 4291563200 18446744073709551615 134529366 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17030/statm: 121514 16899 95 6 0 120179 0
[pid=17031] ppid=17027 vsize=100188 CPUtime=3.09 cores=3
/proc/17031/stat : 17031 (march_hi) R 17027 17027 17001 0 -1 4202496 101410 0 0 0 284 25 0 0 25 0 1 0 630829002 102592512 18284 268435456000 4194304 4331988 140736413374560 18446744073709551615 4271412 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/17031/statm: 25047 18284 141 34 0 23489 0
[pid=17032] ppid=17027 vsize=342416 CPUtime=3.08 cores=4-31
/proc/17032/stat : 17032 (plingeling) R 17027 17027 17001 0 -1 4202496 84525 1240 0 0 290 18 0 0 25 0 1 0 630829002 350633984 84263 268435456000 4194304 4356188 140737428388944 18446744073709551615 4273853 0 0 4096 17442 0 0 0 17 12 0 0 0
/proc/17032/statm: 85604 84263 129 40 0 84148 0
Current children cumulated CPU time (s) 15.41
Current children cumulated vsize (KiB) 1232632

[startup+6.30084 s]
/proc/loadavg: 25.61 30.70 31.55 6/419 17038
/proc/meminfo: memFree=254803504/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 15 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
[pid=17028] ppid=17027 vsize=210688 CPUtime=6.29 cores=0
/proc/17028/stat : 17028 (cryptominisat) R 17027 17027 17001 0 -1 4202496 43218 0 0 0 620 9 0 0 25 0 1 0 630829002 215744512 39053 268435456000 4194304 4587354 140734731610016 18446744073709551615 4293982 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/17028/statm: 52672 39053 333 96 0 49234 0
[pid=17029] ppid=17027 vsize=104524 CPUtime=6.28 cores=1
/proc/17029/stat : 17029 (clasp) R 17027 17027 17001 0 -1 4202496 29810 0 0 0 620 8 0 0 25 0 1 0 630829002 107032576 22794 268435456000 4194304 4785740 140737149889056 18446744073709551615 4553148 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/17029/statm: 26131 22794 338 145 0 23176 0
[pid=17030] ppid=17027 vsize=486056 CPUtime=6.28 cores=2
/proc/17030/stat : 17030 (TNM) R 17027 17027 17001 0 -1 4202496 17001 0 0 0 624 4 0 0 25 0 1 0 630829002 497721344 16899 268435456000 134512640 134534564 4291563200 18446744073709551615 134528661 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17030/statm: 121514 16899 95 6 0 120179 0
[pid=17031] ppid=17027 vsize=328496 CPUtime=6.29 cores=3
/proc/17031/stat : 17031 (march_hi) R 17027 17027 17001 0 -1 4202496 157961 0 0 0 593 36 0 0 25 0 1 0 630829002 336379904 48585 268435456000 4194304 4331988 140736413374560 18446744073709551615 4229471 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/17031/statm: 82124 48585 172 34 0 80566 0
[pid=17032] ppid=17027 vsize=676412 CPUtime=6.27 cores=4-31
/proc/17032/stat : 17032 (plingeling) R 17027 17027 17001 0 -1 4202496 168154 1240 0 0 591 36 0 0 25 0 1 0 630829002 692645888 167762 268435456000 4194304 4356188 140737428388944 18446744073709551615 4278256 0 0 4096 17442 0 0 0 17 12 0 0 0
/proc/17032/statm: 169103 167762 129 40 0 167647 0
Current children cumulated CPU time (s) 31.41
Current children cumulated vsize (KiB) 1824500

[startup+12.702 s]
/proc/loadavg: 26.12 30.72 31.55 33/447 17066
/proc/meminfo: memFree=254223672/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 15 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
[pid=17028] ppid=17027 vsize=210688 CPUtime=12.69 cores=0
/proc/17028/stat : 17028 (cryptominisat) R 17027 17027 17001 0 -1 4202496 43425 0 0 0 1260 9 0 0 25 0 1 0 630829002 215744512 39238 268435456000 4194304 4587354 140734731610016 18446744073709551615 4238728 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/17028/statm: 52672 39238 333 96 0 49234 0
[pid=17029] ppid=17027 vsize=104524 CPUtime=12.68 cores=1
/proc/17029/stat : 17029 (clasp) R 17027 17027 17001 0 -1 4202496 29810 0 0 0 1260 8 0 0 25 0 1 0 630829002 107032576 22794 268435456000 4194304 4785740 140737149889056 18446744073709551615 4553068 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/17029/statm: 26131 22794 338 145 0 23176 0
[pid=17030] ppid=17027 vsize=486056 CPUtime=12.68 cores=2
/proc/17030/stat : 17030 (TNM) R 17027 17027 17001 0 -1 4202496 17001 0 0 0 1264 4 0 0 25 0 1 0 630829002 497721344 16899 268435456000 134512640 134534564 4291563200 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17030/statm: 121514 16899 95 6 0 120179 0
[pid=17031] ppid=17027 vsize=340128 CPUtime=12.69 cores=3
/proc/17031/stat : 17031 (march_hi) R 17027 17027 17001 0 -1 4202496 184726 0 0 0 1227 42 0 0 25 0 1 0 630829002 348291072 75350 268435456000 4194304 4331988 140736413374560 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/17031/statm: 85032 75350 172 34 0 83474 0
[pid=17032] ppid=17027 vsize=1514364 CPUtime=154.84 cores=4-31
/proc/17032/stat : 17032 (plingeling) S 17027 17027 17001 0 -1 4202496 439191 1240 0 0 15177 307 0 0 18 0 29 0 630829002 1550708736 222389 268435456000 4194304 4356188 140737428388944 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 16 0 0 0
/proc/17032/statm: 378591 222393 159 40 0 377135 0
[pid=17032/tid=17039] ppid=17027 vsize=1514364 CPUtime=5.27 cores=4-31
/proc/17032/task/17039/stat : 17039 (plingeling) R 17027 17027 17001 0 -1 4202560 8049 1240 0 0 517 10 0 0 25 0 29 0 630829732 1550708736 222398 268435456000 4194304 4356188 140737428388944 18446744073709551615 4321963 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17032/tid=17040] ppid=17027 vsize=1514364 CPUtime=5.29 cores=4-31
/proc/17032/task/17040/stat : 17040 (plingeling) R 17027 17027 17001 0 -1 4202560 9356 1240 0 0 521 8 0 0 25 0 29 0 630829732 1550708736 222405 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17032/tid=17041] ppid=17027 vsize=1514364 CPUtime=5.25 cores=4-31
/proc/17032/task/17041/stat : 17041 (plingeling) R 17027 17027 17001 0 -1 4202560 7550 1240 0 0 517 8 0 0 25 0 29 0 630829732 1550708736 222412 268435456000 4194304 4356188 140737428388944 18446744073709551615 4290570 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=17032/tid=17042] ppid=17027 vsize=1514364 CPUtime=5.26 cores=4-31
/proc/17032/task/17042/stat : 17042 (plingeling) R 17027 17027 17001 0 -1 4202560 12014 1240 0 0 513 13 0 0 25 0 29 0 630829732 1550708736 222419 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=17032/tid=17043] ppid=17027 vsize=1514364 CPUtime=5.27 cores=4-31
/proc/17032/task/17043/stat : 17043 (plingeling) R 17027 17027 17001 0 -1 4202560 13218 1240 0 0 516 11 0 0 25 0 29 0 630829732 1550708736 222425 268435456000 4194304 4356188 140737428388944 18446744073709551615 4232192 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17032/tid=17044] ppid=17027 vsize=1514364 CPUtime=5.28 cores=4-31
/proc/17032/task/17044/stat : 17044 (plingeling) R 17027 17027 17001 0 -1 4202560 11901 1240 0 0 518 10 0 0 25 0 29 0 630829732 1550708736 222432 268435456000 4194304 4356188 140737428388944 18446744073709551615 4286706 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=17032/tid=17045] ppid=17027 vsize=1514364 CPUtime=5.28 cores=4-31
/proc/17032/task/17045/stat : 17045 (plingeling) R 17027 17027 17001 0 -1 4202560 6274 1240 0 0 520 8 0 0 25 0 29 0 630829732 1550708736 222439 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=17032/tid=17046] ppid=17027 vsize=1514364 CPUtime=5.28 cores=4-31
/proc/17032/task/17046/stat : 17046 (plingeling) R 17027 17027 17001 0 -1 4202560 11368 1240 0 0 518 10 0 0 25 0 29 0 630829732 1550708736 222446 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322106 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=17032/tid=17047] ppid=17027 vsize=1514364 CPUtime=5.25 cores=4-31
/proc/17032/task/17047/stat : 17047 (plingeling) R 17027 17027 17001 0 -1 4202560 7464 1240 0 0 517 8 0 0 25 0 29 0 630829732 1550708736 222452 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322106 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=17032/tid=17048] ppid=17027 vsize=1514364 CPUtime=5.27 cores=4-31
/proc/17032/task/17048/stat : 17048 (plingeling) R 17027 17027 17001 0 -1 4202560 9228 1240 0 0 518 9 0 0 25 0 29 0 630829733 1550708736 222459 268435456000 4194304 4356188 140737428388944 18446744073709551615 4289359 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=17032/tid=17049] ppid=17027 vsize=1514364 CPUtime=5.27 cores=4-31
/proc/17032/task/17049/stat : 17049 (plingeling) R 17027 17027 17001 0 -1 4202560 9491 1240 0 0 518 9 0 0 25 0 29 0 630829733 1550708736 222466 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=17032/tid=17050] ppid=17027 vsize=1514364 CPUtime=5.27 cores=4-31
/proc/17032/task/17050/stat : 17050 (plingeling) R 17027 17027 17001 0 -1 4202560 5958 1240 0 0 520 7 0 0 25 0 29 0 630829733 1550708736 222473 268435456000 4194304 4356188 140737428388944 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=17032/tid=17051] ppid=17027 vsize=1514364 CPUtime=5.25 cores=4-31
/proc/17032/task/17051/stat : 17051 (plingeling) R 17027 17027 17001 0 -1 4202560 10938 1240 0 0 515 10 0 0 25 0 29 0 630829733 1550708736 222480 268435456000 4194304 4356188 140737428388944 18446744073709551615 4268476 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=17032/tid=17052] ppid=17027 vsize=1514364 CPUtime=5.28 cores=4-31
/proc/17032/task/17052/stat : 17052 (plingeling) R 17027 17027 17001 0 -1 4202560 11376 1240 0 0 517 11 0 0 25 0 29 0 630829733 1550708736 222486 268435456000 4194304 4356188 140737428388944 18446744073709551615 4210596 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=17032/tid=17053] ppid=17027 vsize=1514364 CPUtime=5.25 cores=4-31
/proc/17032/task/17053/stat : 17053 (plingeling) R 17027 17027 17001 0 -1 4202560 9475 1240 0 0 515 10 0 0 25 0 29 0 630829733 1550708736 222493 268435456000 4194304 4356188 140737428388944 18446744073709551615 4232872 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=17032/tid=17054] ppid=17027 vsize=1514364 CPUtime=5.26 cores=4-31
/proc/17032/task/17054/stat : 17054 (plingeling) R 17027 17027 17001 0 -1 4202560 10779 1240 0 0 516 10 0 0 25 0 29 0 630829733 1550708736 222503 268435456000 4194304 4356188 140737428388944 18446744073709551615 4233312 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=17032/tid=17055] ppid=17027 vsize=1514364 CPUtime=5.27 cores=4-31
/proc/17032/task/17055/stat : 17055 (plingeling) R 17027 17027 17001 0 -1 4202560 9747 1240 0 0 517 10 0 0 25 0 29 0 630829734 1550708736 222512 268435456000 4194304 4356188 140737428388944 18446744073709551615 4290570 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17032/tid=17056] ppid=17027 vsize=1514364 CPUtime=5.27 cores=4-31
/proc/17032/task/17056/stat : 17056 (plingeling) R 17027 17027 17001 0 -1 4202560 6798 1240 0 0 519 8 0 0 25 0 29 0 630829734 1550708736 222521 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=17032/tid=17057] ppid=17027 vsize=1514364 CPUtime=5.12 cores=4-31
/proc/17032/task/17057/stat : 17057 (plingeling) R 17027 17027 17001 0 -1 4202560 5426 1240 0 0 504 8 0 0 25 0 29 0 630829734 1550708736 222530 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=17032/tid=17058] ppid=17027 vsize=1514364 CPUtime=5.28 cores=4-31
/proc/17032/task/17058/stat : 17058 (plingeling) R 17027 17027 17001 0 -1 4202560 8654 1240 0 0 519 9 0 0 25 0 29 0 630829734 1550708736 222540 268435456000 4194304 4356188 140737428388944 18446744073709551615 4210765 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=17032/tid=17059] ppid=17027 vsize=1514364 CPUtime=5.21 cores=4-31
/proc/17032/task/17059/stat : 17059 (plingeling) R 17027 17027 17001 0 -1 4202560 6303 1240 0 0 519 2 0 0 25 0 29 0 630829734 1550708736 222548 268435456000 4194304 4356188 140737428388944 18446744073709551615 4322106 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=17032/tid=17060] ppid=17027 vsize=1514364 CPUtime=5.26 cores=4-31

################
# More data... #
################

[pid=17030] ppid=17027 vsize=486060 CPUtime=4961.04 cores=2
/proc/17030/stat : 17030 (TNM) R 17027 17027 17001 0 -1 4202496 17010 0 0 0 496100 4 0 0 25 0 1 0 630829002 497725440 16908 268435456000 134512640 134534564 4291563200 18446744073709551615 134528275 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17030/statm: 121515 16908 103 6 0 120180 0
[pid=17031] ppid=17027 vsize=347100 CPUtime=4961.35 cores=3
/proc/17031/stat : 17031 (march_hi) R 17027 17027 17001 0 -1 4202496 186140 0 0 0 496091 44 0 0 25 0 1 0 630829002 355430400 76764 268435456000 4194304 4331988 140736413374560 18446744073709551615 4211509 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/17031/statm: 86775 76764 172 34 0 85217 0
[pid=17032] ppid=17027 vsize=3441096 CPUtime=138566 cores=4-31
/proc/17032/stat : 17032 (plingeling) S 17027 17027 17001 0 -1 4202496 2734039 1240 0 0 13854816 1813 0 0 18 0 29 0 630829002 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 16 0 0 0
/proc/17032/statm: 860274 698314 159 40 0 858818 0
[pid=17032/tid=17039] ppid=17027 vsize=3441096 CPUtime=4953.4 cores=4-31
/proc/17032/task/17039/stat : 17039 (plingeling) R 17027 17027 17001 0 -1 4202560 84804 1240 0 0 495265 75 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331007 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17032/tid=17040] ppid=17027 vsize=3441096 CPUtime=4940.43 cores=4-31
/proc/17032/task/17040/stat : 17040 (plingeling) R 17027 17027 17001 0 -1 4202560 68821 1240 0 0 494021 22 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=17032/tid=17041] ppid=17027 vsize=3441096 CPUtime=4953.46 cores=4-31
/proc/17032/task/17041/stat : 17041 (plingeling) R 17027 17027 17001 0 -1 4202560 61095 1240 0 0 495309 37 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17032/tid=17042] ppid=17027 vsize=3441096 CPUtime=4953.85 cores=4-31
/proc/17032/task/17042/stat : 17042 (plingeling) R 17027 17027 17001 0 -1 4202560 32885 1240 0 0 495358 27 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330896 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=17032/tid=17043] ppid=17027 vsize=3441096 CPUtime=4952.38 cores=4-31
/proc/17032/task/17043/stat : 17043 (plingeling) R 17027 17027 17001 0 -1 4202560 93784 1240 0 0 495182 56 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17032/tid=17044] ppid=17027 vsize=3441096 CPUtime=4953.57 cores=4-31
/proc/17032/task/17044/stat : 17044 (plingeling) R 17027 17027 17001 0 -1 4202560 85632 1240 0 0 495306 51 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4281498 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=17032/tid=17045] ppid=17027 vsize=3441096 CPUtime=4895.35 cores=4-31
/proc/17032/task/17045/stat : 17045 (plingeling) R 17027 17027 17001 0 -1 4202560 72627 1240 0 0 489474 61 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17032/tid=17046] ppid=17027 vsize=3441096 CPUtime=4953.84 cores=4-31
/proc/17032/task/17046/stat : 17046 (plingeling) R 17027 17027 17001 0 -1 4202560 212212 1240 0 0 495243 141 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=17032/tid=17047] ppid=17027 vsize=3441096 CPUtime=4953.87 cores=4-31
/proc/17032/task/17047/stat : 17047 (plingeling) R 17027 17027 17001 0 -1 4202560 72498 1240 0 0 495332 55 0 0 25 0 29 0 630829732 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=17032/tid=17048] ppid=17027 vsize=3441096 CPUtime=4953.68 cores=4-31
/proc/17032/task/17048/stat : 17048 (plingeling) R 17027 17027 17001 0 -1 4202560 115059 1240 0 0 495307 61 0 0 25 0 29 0 630829733 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=17032/tid=17049] ppid=17027 vsize=3441096 CPUtime=4953.56 cores=4-31
/proc/17032/task/17049/stat : 17049 (plingeling) R 17027 17027 17001 0 -1 4202560 94364 1240 0 0 495279 77 0 0 25 0 29 0 630829733 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=17032/tid=17050] ppid=17027 vsize=3441096 CPUtime=4953.62 cores=4-31
/proc/17032/task/17050/stat : 17050 (plingeling) R 17027 17027 17001 0 -1 4202560 22283 1240 0 0 495351 11 0 0 25 0 29 0 630829733 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4210591 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=17032/tid=17051] ppid=17027 vsize=3441096 CPUtime=4953.77 cores=4-31
/proc/17032/task/17051/stat : 17051 (plingeling) R 17027 17027 17001 0 -1 4202560 98156 1240 0 0 495322 55 0 0 25 0 29 0 630829733 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=17032/tid=17052] ppid=17027 vsize=3441096 CPUtime=4953.78 cores=4-31
/proc/17032/task/17052/stat : 17052 (plingeling) R 17027 17027 17001 0 -1 4202560 86799 1240 0 0 495312 66 0 0 25 0 29 0 630829733 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=17032/tid=17053] ppid=17027 vsize=3441096 CPUtime=4951.03 cores=4-31
/proc/17032/task/17053/stat : 17053 (plingeling) R 17027 17027 17001 0 -1 4202560 92198 1240 0 0 494988 115 0 0 25 0 29 0 630829733 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=17032/tid=17054] ppid=17027 vsize=3441096 CPUtime=4953.56 cores=4-31
/proc/17032/task/17054/stat : 17054 (plingeling) R 17027 17027 17001 0 -1 4202560 217810 1240 0 0 495198 158 0 0 25 0 29 0 630829733 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=17032/tid=17055] ppid=17027 vsize=3441096 CPUtime=4953.5 cores=4-31
/proc/17032/task/17055/stat : 17055 (plingeling) R 17027 17027 17001 0 -1 4202560 72330 1240 0 0 495300 50 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330976 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=17032/tid=17056] ppid=17027 vsize=3441096 CPUtime=4953.62 cores=4-31
/proc/17032/task/17056/stat : 17056 (plingeling) R 17027 17027 17001 0 -1 4202560 79034 1240 0 0 495310 52 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4332041 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=17032/tid=17057] ppid=17027 vsize=3441096 CPUtime=4953.64 cores=4-31
/proc/17032/task/17057/stat : 17057 (plingeling) R 17027 17027 17001 0 -1 4202560 73174 1240 0 0 495304 60 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=17032/tid=17058] ppid=17027 vsize=3441096 CPUtime=4953.53 cores=4-31
/proc/17032/task/17058/stat : 17058 (plingeling) R 17027 17027 17001 0 -1 4202560 27256 1240 0 0 495331 22 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=17032/tid=17059] ppid=17027 vsize=3441096 CPUtime=4953.31 cores=4-31
/proc/17032/task/17059/stat : 17059 (plingeling) R 17027 17027 17001 0 -1 4202560 98085 1240 0 0 495305 26 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=17032/tid=17060] ppid=17027 vsize=3441096 CPUtime=4953.9 cores=4-31
/proc/17032/task/17060/stat : 17060 (plingeling) R 17027 17027 17001 0 -1 4202560 100506 1240 0 0 495314 76 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=17032/tid=17061] ppid=17027 vsize=3441096 CPUtime=4953.48 cores=4-31
/proc/17032/task/17061/stat : 17061 (plingeling) R 17027 17027 17001 0 -1 4202560 87964 1240 0 0 495301 47 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=17032/tid=17062] ppid=17027 vsize=3441096 CPUtime=4885.55 cores=4-31
/proc/17032/task/17062/stat : 17062 (plingeling) R 17027 17027 17001 0 -1 4202560 180601 1240 0 0 488426 129 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=17032/tid=17063] ppid=17027 vsize=3441096 CPUtime=4953.85 cores=4-31
/proc/17032/task/17063/stat : 17063 (plingeling) R 17027 17027 17001 0 -1 4202560 105597 1240 0 0 495279 106 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4284062 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=17032/tid=17064] ppid=17027 vsize=3441096 CPUtime=4953.76 cores=4-31
/proc/17032/task/17064/stat : 17064 (plingeling) R 17027 17027 17001 0 -1 4202560 95770 1240 0 0 495347 29 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=17032/tid=17065] ppid=17027 vsize=3441096 CPUtime=4953.74 cores=4-31
/proc/17032/task/17065/stat : 17065 (plingeling) R 17027 17027 17001 0 -1 4202560 104723 1240 0 0 495302 72 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=17032/tid=17066] ppid=17027 vsize=3441096 CPUtime=4953.68 cores=4-31
/proc/17032/task/17066/stat : 17066 (plingeling) R 17027 17027 17001 0 -1 4202560 29782 1240 0 0 495345 23 0 0 25 0 29 0 630829734 3523682304 698314 268435456000 4194304 4356188 140737428388944 18446744073709551615 4236874 0 0 4096 17442 0 0 0 -1 17 0 0 0
Current children cumulated CPU time (s) 158411
Current children cumulated vsize (KiB) 6825172



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5012 s]
/proc/loadavg: 32.02 32.03 32.02 33/447 17221
/proc/meminfo: memFree=251910616/264257624 swapFree=67111528/67111528
[pid=17027] ppid=17025 vsize=18324 CPUtime=0 cores=0-31
/proc/17027/stat : 17027 (ppfolio) S 17025 17027 17001 0 -1 4202496 510 0 0 0 0 0 0 0 15 0 1 0 630829001 18763776 385 268435456000 4194304 4287625 140734642604640 18446744073709551615 258650975999 0 0 4096 65536 9179084125099786239 0 0 17 8 0 0 0
/proc/17027/statm: 4581 385 337 23 0 91 0
[pid=17028] ppid=17027 vsize=2088148 CPUtime=5011.04 cores=0
/proc/17028/stat : 17028 (cryptominisat) R 17027 17027 17001 0 -1 4202496 633345 0 0 0 500800 304 0 0 25 0 1 0 630829002 2138263552 66284 268435456000 4194304 4587354 140734731610016 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/17028/statm: 522037 66284 338 96 0 518599 0
[pid=17029] ppid=17027 vsize=446132 CPUtime=5011.03 cores=1
/proc/17029/stat : 17029 (clasp) R 17027 17027 17001 0 -1 4202496 113962 0 0 0 501030 73 0 0 25 0 1 0 630829002 456839168 106946 268435456000 4194304 4785740 140737149889056 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/17029/statm: 111533 106946 338 145 0 108578 0
[pid=17030] ppid=17027 vsize=486060 CPUtime=5010.76 cores=2
/proc/17030/stat : 17030 (TNM) R 17027 17027 17001 0 -1 4202496 17010 0 0 0 501072 4 0 0 25 0 1 0 630829002 497725440 16908 268435456000 134512640 134534564 4291563200 18446744073709551615 134529651 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/17030/statm: 121515 16908 103 6 0 120180 0
[pid=17031] ppid=17027 vsize=347100 CPUtime=5011.06 cores=3
/proc/17031/stat : 17031 (march_hi) R 17027 17027 17001 0 -1 4202496 186149 0 0 0 501062 44 0 0 25 0 1 0 630829002 355430400 76773 268435456000 4194304 4331988 140736413374560 18446744073709551615 4226457 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/17031/statm: 86775 76773 172 34 0 85217 0
[pid=17032] ppid=17027 vsize=3404232 CPUtime=139957 cores=4-31
/proc/17032/stat : 17032 (plingeling) S 17027 17027 17001 0 -1 4202496 2741207 1240 0 0 13993853 1826 0 0 18 0 29 0 630829002 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 16 0 0 0
/proc/17032/statm: 851058 688586 159 40 0 849602 0
[pid=17032/tid=17039] ppid=17027 vsize=3404232 CPUtime=5003.11 cores=4-31
/proc/17032/task/17039/stat : 17039 (plingeling) R 17027 17027 17001 0 -1 4202560 84804 1240 0 0 500236 75 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=17032/tid=17040] ppid=17027 vsize=3404232 CPUtime=4990.14 cores=4-31
/proc/17032/task/17040/stat : 17040 (plingeling) R 17027 17027 17001 0 -1 4202560 68821 1240 0 0 498992 22 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=17032/tid=17041] ppid=17027 vsize=3404232 CPUtime=5003.17 cores=4-31
/proc/17032/task/17041/stat : 17041 (plingeling) R 17027 17027 17001 0 -1 4202560 61095 1240 0 0 500280 37 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=17032/tid=17042] ppid=17027 vsize=3404232 CPUtime=5003.56 cores=4-31
/proc/17032/task/17042/stat : 17042 (plingeling) R 17027 17027 17001 0 -1 4202560 32885 1240 0 0 500329 27 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=17032/tid=17043] ppid=17027 vsize=3404232 CPUtime=5002.08 cores=4-31
/proc/17032/task/17043/stat : 17043 (plingeling) R 17027 17027 17001 0 -1 4202560 93784 1240 0 0 500152 56 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=17032/tid=17044] ppid=17027 vsize=3404232 CPUtime=5003.29 cores=4-31
/proc/17032/task/17044/stat : 17044 (plingeling) R 17027 17027 17001 0 -1 4202560 85632 1240 0 0 500278 51 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=17032/tid=17045] ppid=17027 vsize=3404232 CPUtime=4945.07 cores=4-31
/proc/17032/task/17045/stat : 17045 (plingeling) R 17027 17027 17001 0 -1 4202560 72627 1240 0 0 494446 61 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=17032/tid=17046] ppid=17027 vsize=3404232 CPUtime=5003.56 cores=4-31
/proc/17032/task/17046/stat : 17046 (plingeling) R 17027 17027 17001 0 -1 4202560 212212 1240 0 0 500215 141 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=17032/tid=17047] ppid=17027 vsize=3404232 CPUtime=5003.59 cores=4-31
/proc/17032/task/17047/stat : 17047 (plingeling) R 17027 17027 17001 0 -1 4202560 76594 1240 0 0 500294 65 0 0 25 0 29 0 630829732 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331401 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=17032/tid=17048] ppid=17027 vsize=3404232 CPUtime=5003.4 cores=4-31
/proc/17032/task/17048/stat : 17048 (plingeling) R 17027 17027 17001 0 -1 4202560 116083 1240 0 0 500277 63 0 0 25 0 29 0 630829733 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=17032/tid=17049] ppid=17027 vsize=3404232 CPUtime=5003.27 cores=4-31
/proc/17032/task/17049/stat : 17049 (plingeling) R 17027 17027 17001 0 -1 4202560 94364 1240 0 0 500250 77 0 0 25 0 29 0 630829733 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331028 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=17032/tid=17050] ppid=17027 vsize=3404232 CPUtime=5003.33 cores=4-31
/proc/17032/task/17050/stat : 17050 (plingeling) R 17027 17027 17001 0 -1 4202560 22283 1240 0 0 500322 11 0 0 25 0 29 0 630829733 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=17032/tid=17051] ppid=17027 vsize=3404232 CPUtime=5003.48 cores=4-31
/proc/17032/task/17051/stat : 17051 (plingeling) R 17027 17027 17001 0 -1 4202560 98156 1240 0 0 500293 55 0 0 25 0 29 0 630829733 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=17032/tid=17052] ppid=17027 vsize=3404232 CPUtime=5003.49 cores=4-31
/proc/17032/task/17052/stat : 17052 (plingeling) R 17027 17027 17001 0 -1 4202560 86799 1240 0 0 500283 66 0 0 25 0 29 0 630829733 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=17032/tid=17053] ppid=17027 vsize=3404232 CPUtime=5000.72 cores=4-31
/proc/17032/task/17053/stat : 17053 (plingeling) R 17027 17027 17001 0 -1 4202560 92198 1240 0 0 499956 116 0 0 25 0 29 0 630829733 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=17032/tid=17054] ppid=17027 vsize=3404232 CPUtime=5003.27 cores=4-31
/proc/17032/task/17054/stat : 17054 (plingeling) R 17027 17027 17001 0 -1 4202560 217810 1240 0 0 500169 158 0 0 25 0 29 0 630829733 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4332272 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=17032/tid=17055] ppid=17027 vsize=3404232 CPUtime=5003.22 cores=4-31
/proc/17032/task/17055/stat : 17055 (plingeling) R 17027 17027 17001 0 -1 4202560 72330 1240 0 0 500272 50 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=17032/tid=17056] ppid=17027 vsize=3404232 CPUtime=5003.34 cores=4-31
/proc/17032/task/17056/stat : 17056 (plingeling) R 17027 17027 17001 0 -1 4202560 79034 1240 0 0 500282 52 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331111 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=17032/tid=17057] ppid=17027 vsize=3404232 CPUtime=5003.35 cores=4-31
/proc/17032/task/17057/stat : 17057 (plingeling) R 17027 17027 17001 0 -1 4202560 73174 1240 0 0 500275 60 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=17032/tid=17058] ppid=17027 vsize=3404232 CPUtime=5003.24 cores=4-31
/proc/17032/task/17058/stat : 17058 (plingeling) R 17027 17027 17001 0 -1 4202560 27256 1240 0 0 500302 22 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4236939 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=17032/tid=17059] ppid=17027 vsize=3404232 CPUtime=5003.02 cores=4-31
/proc/17032/task/17059/stat : 17059 (plingeling) R 17027 17027 17001 0 -1 4202560 99109 1240 0 0 500276 26 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=17032/tid=17060] ppid=17027 vsize=3404232 CPUtime=5003.62 cores=4-31
/proc/17032/task/17060/stat : 17060 (plingeling) R 17027 17027 17001 0 -1 4202560 100506 1240 0 0 500285 77 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4332007 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=17032/tid=17061] ppid=17027 vsize=3404232 CPUtime=5003.2 cores=4-31
/proc/17032/task/17061/stat : 17061 (plingeling) R 17027 17027 17001 0 -1 4202560 87964 1240 0 0 500273 47 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=17032/tid=17062] ppid=17027 vsize=3404232 CPUtime=4933.85 cores=4-31
/proc/17032/task/17062/stat : 17062 (plingeling) R 17027 17027 17001 0 -1 4202560 181625 1240 0 0 493256 129 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4281672 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=17032/tid=17063] ppid=17027 vsize=3404232 CPUtime=5003.57 cores=4-31
/proc/17032/task/17063/stat : 17063 (plingeling) R 17027 17027 17001 0 -1 4202560 105597 1240 0 0 500251 106 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=17032/tid=17064] ppid=17027 vsize=3404232 CPUtime=5003.47 cores=4-31
/proc/17032/task/17064/stat : 17064 (plingeling) R 17027 17027 17001 0 -1 4202560 95770 1240 0 0 500318 29 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=17032/tid=17065] ppid=17027 vsize=3404232 CPUtime=5003.45 cores=4-31
/proc/17032/task/17065/stat : 17065 (plingeling) R 17027 17027 17001 0 -1 4202560 104723 1240 0 0 500273 72 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4331007 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=17032/tid=17066] ppid=17027 vsize=3404232 CPUtime=5003.4 cores=4-31
/proc/17032/task/17066/stat : 17066 (plingeling) R 17027 17027 17001 0 -1 4202560 29782 1240 0 0 500317 23 0 0 25 0 29 0 630829734 3485933568 688586 268435456000 4194304 4356188 140737428388944 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 17 0 0 0
Current children cumulated CPU time (s) 160001
Current children cumulated vsize (KiB) 6789996

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 17027 and gives
#  childrusage.ru_utime.tv_sec=5010
#  childrusage.ru_utime.tv_usec=726253
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=61990
# CPU time returned by wait4() is 5010.79
# while last known CPU time is 160001
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5012.04
CPU time (s): 160001
CPU user time (s): 159978
CPU system time (s): 22.51
CPU usage (%): 3192.33
Max. virtual memory (cumulated for all children) (KiB): 8705836

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5010.73
system time used= 0.06199
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 17522
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 463
involuntary context switches= 12632

runsolver used 28.0767 second user time and 113.266 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-12 11:43:46
IDJOB=3347465
IDBENCH=82468
IDSOLVER=1808
FILE ID=node200/3347465-1305193425
RUNJOBID= node200-1305193425-17015
PBS_JOBID= 13282081
Free space on /tmp= 62044 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/application/jarvisalo/HWMCC10-combination-timeframe-expansion/hwmcc10-timeframe-expansion-k45-pdtvisns3p02-tseitin.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347465-1305193425/watcher-3347465-1305193425 -o /tmp/evaluation-result-3347465-1305193425/solver-3347465-1305193425 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347465-1305193425.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= a0ec1baa890da62d0c6df7f1e62c6af3
RANDOM SEED=1425278561

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      255737368 kB
Buffers:        426600 kB
Cached:        7695808 kB
SwapCached:          0 kB
Active:        7267072 kB
Inactive:       924204 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      255737368 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           18500 kB
Writeback:           0 kB
AnonPages:       68788 kB
Mapped:          14480 kB
Slab:           249240 kB
PageTables:       4440 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   178844 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62032 MiB
End job on node200 at 2011-05-12 13:07:20