Trace number 3347187

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 160004 5009.84

General information on the benchmark

Nameindustrial/IBM_FV_2002_13_rule_1/
SAT_dat.k45.cnf
MD5SUM8a76c8add64173ae39149aae572a02d9
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark6444.1
Satisfiable
(Un)Satisfiability was proved
Number of variables37432
Number of clauses147345
Sum of the clauses size401667
Maximum clause length37
Minimum clause length1
Number of clauses of size 1603
Number of clauses of size 269621
Number of clauses of size 371450
Number of clauses of size 4266
Number of clauses of size 5505
Number of clauses of size over 54900

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347187-1304817913/watcher-3347187-1304817913 -o /tmp/evaluation-result-3347187-1304817913/solver-3347187-1304817913 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347187-1304817913.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 27.21 31.01 31.73 1/413 9453
/proc/meminfo: memFree=256774600/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 18 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0

[startup+0.109624 s]
/proc/loadavg: 27.21 31.01 31.73 1/413 9453
/proc/meminfo: memFree=256774600/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 18 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.200624 s]
/proc/loadavg: 27.21 31.01 31.73 1/413 9453
/proc/meminfo: memFree=256774600/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 16 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300673 s]
/proc/loadavg: 27.21 31.01 31.73 1/413 9453
/proc/meminfo: memFree=256774600/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 16 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700679 s]
/proc/loadavg: 27.21 31.01 31.73 1/413 9453
/proc/meminfo: memFree=256774600/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 16 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50065 s]
/proc/loadavg: 27.21 31.01 31.73 6/419 9464
/proc/meminfo: memFree=256569556/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 16 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
[pid=9454] ppid=9453 vsize=38920 CPUtime=1.48 cores=0
/proc/9454/stat : 9454 (cryptominisat) R 9453 9453 9427 0 -1 4202496 4702 0 0 0 147 1 0 0 25 0 1 0 593277737 39854080 4481 268435456000 4194304 4587354 140735773368368 18446744073709551615 4237627 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9454/statm: 9730 4481 308 96 0 6292 0
[pid=9455] ppid=9453 vsize=37152 CPUtime=1.48 cores=1
/proc/9455/stat : 9455 (clasp) R 9453 9453 9427 0 -1 4202496 8447 0 0 0 145 3 0 0 20 0 1 0 593277737 38043648 6473 268435456000 4194304 4785740 140733578986096 18446744073709551615 4552550 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9455/statm: 9288 6473 338 145 0 6333 0
[pid=9456] ppid=9453 vsize=462956 CPUtime=1.48 cores=2
/proc/9456/stat : 9456 (TNM) R 9453 9453 9427 0 -1 4202496 5281 0 0 0 146 2 0 0 20 0 1 0 593277737 474066944 5179 268435456000 134512640 134534564 4287769776 18446744073709551615 134530614 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9456/statm: 115739 5179 95 6 0 114404 0
[pid=9457] ppid=9453 vsize=33948 CPUtime=1.48 cores=3
/proc/9457/stat : 9457 (march_hi) R 9453 9453 9427 0 -1 4202496 24171 0 0 0 142 6 0 0 20 0 1 0 593277737 34762752 5721 268435456000 4194304 4331988 140735863302960 18446744073709551615 4252160 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9457/statm: 8487 5721 147 34 0 6929 0
[pid=9458] ppid=9453 vsize=129536 CPUtime=1.47 cores=4-31
/proc/9458/stat : 9458 (plingeling) R 9453 9453 9427 0 -1 4202496 31154 1239 0 0 140 7 0 0 21 0 1 0 593277737 132644864 31039 268435456000 4194304 4356188 140734679173232 18446744073709551615 258650649933 0 0 4096 17442 0 0 0 17 10 0 0 0
/proc/9458/statm: 32384 31043 129 40 0 30928 0
Current children cumulated CPU time (s) 7.39
Current children cumulated vsize (KiB) 720836

[startup+3.10267 s]
/proc/loadavg: 27.21 31.01 31.73 27/446 9491
/proc/meminfo: memFree=256474064/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 15 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
[pid=9454] ppid=9453 vsize=62812 CPUtime=3.08 cores=0
/proc/9454/stat : 9454 (cryptominisat) R 9453 9453 9427 0 -1 4202496 13181 0 0 0 306 2 0 0 25 0 1 0 593277737 64319488 10838 268435456000 4194304 4587354 140735773368368 18446744073709551615 4233824 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9454/statm: 15703 10838 328 96 0 12265 0
[pid=9455] ppid=9453 vsize=37548 CPUtime=3.08 cores=1
/proc/9455/stat : 9455 (clasp) R 9453 9453 9427 0 -1 4202496 8594 0 0 0 305 3 0 0 25 0 1 0 593277737 38449152 6620 268435456000 4194304 4785740 140733578986096 18446744073709551615 258650616632 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9455/statm: 9387 6620 338 145 0 6432 0
[pid=9456] ppid=9453 vsize=462956 CPUtime=3.08 cores=2
/proc/9456/stat : 9456 (TNM) R 9453 9453 9427 0 -1 4202496 5281 0 0 0 306 2 0 0 25 0 1 0 593277737 474066944 5179 268435456000 134512640 134534564 4287769776 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9456/statm: 115739 5179 95 6 0 114404 0
[pid=9457] ppid=9453 vsize=33948 CPUtime=3.09 cores=3
/proc/9457/stat : 9457 (march_hi) R 9453 9453 9427 0 -1 4202496 24182 0 0 0 303 6 0 0 25 0 1 0 593277737 34762752 5732 268435456000 4194304 4331988 140735863302960 18446744073709551615 4222704 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9457/statm: 8487 5732 148 34 0 6929 0
[pid=9458] ppid=9453 vsize=657612 CPUtime=27.79 cores=4-31
/proc/9458/stat : 9458 (plingeling) S 9453 9453 9427 0 -1 4202496 109289 1239 0 0 2751 28 0 0 18 0 29 0 593277737 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 27 0 0 0
/proc/9458/statm: 164403 53843 153 40 0 162947 0
[pid=9458/tid=9465] ppid=9453 vsize=657612 CPUtime=0.93 cores=4-31
/proc/9458/task/9465/stat : 9465 (plingeling) R 9453 9453 9427 0 -1 4202560 1207 1239 0 0 93 0 0 0 19 0 29 0 593277952 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244402 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9458/tid=9466] ppid=9453 vsize=657612 CPUtime=0.93 cores=4-31
/proc/9458/task/9466/stat : 9466 (plingeling) R 9453 9453 9427 0 -1 4202560 1074 1239 0 0 93 0 0 0 25 0 29 0 593277952 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4321963 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=9458/tid=9467] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9467/stat : 9467 (plingeling) R 9453 9453 9427 0 -1 4202560 1683 1239 0 0 92 0 0 0 19 0 29 0 593277952 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4314913 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9458/tid=9468] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9468/stat : 9468 (plingeling) R 9453 9453 9427 0 -1 4202560 3525 1239 0 0 92 0 0 0 19 0 29 0 593277952 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330896 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=9458/tid=9469] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9469/stat : 9469 (plingeling) R 9453 9453 9427 0 -1 4202560 2186 1239 0 0 92 0 0 0 19 0 29 0 593277952 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4322024 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=9458/tid=9470] ppid=9453 vsize=657612 CPUtime=0.91 cores=4-31
/proc/9458/task/9470/stat : 9470 (plingeling) R 9453 9453 9427 0 -1 4202560 1427 1239 0 0 91 0 0 0 25 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244384 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=9458/tid=9471] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9471/stat : 9471 (plingeling) R 9453 9453 9427 0 -1 4202560 1202 1239 0 0 92 0 0 0 21 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4321938 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=9458/tid=9472] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9472/stat : 9472 (plingeling) R 9453 9453 9427 0 -1 4202560 987 1239 0 0 92 0 0 0 19 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4314496 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=9458/tid=9473] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9473/stat : 9473 (plingeling) R 9453 9453 9427 0 -1 4202560 1173 1239 0 0 92 0 0 0 21 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4322041 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=9458/tid=9474] ppid=9453 vsize=657612 CPUtime=0.91 cores=4-31
/proc/9458/task/9474/stat : 9474 (plingeling) R 9453 9453 9427 0 -1 4202560 2146 1239 0 0 91 0 0 0 18 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=9458/tid=9475] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9475/stat : 9475 (plingeling) R 9453 9453 9427 0 -1 4202560 2025 1239 0 0 91 1 0 0 19 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4315013 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=9458/tid=9476] ppid=9453 vsize=657612 CPUtime=0.92 cores=4-31
/proc/9458/task/9476/stat : 9476 (plingeling) R 9453 9453 9427 0 -1 4202560 1976 1239 0 0 92 0 0 0 19 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4210544 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=9458/tid=9477] ppid=9453 vsize=657612 CPUtime=0.91 cores=4-31
/proc/9458/task/9477/stat : 9477 (plingeling) R 9453 9453 9427 0 -1 4202560 3450 1239 0 0 91 0 0 0 21 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4322112 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=9458/tid=9478] ppid=9453 vsize=657612 CPUtime=0.9 cores=4-31
/proc/9458/task/9478/stat : 9478 (plingeling) R 9453 9453 9427 0 -1 4202560 2419 1239 0 0 90 0 0 0 21 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4245442 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=9458/tid=9479] ppid=9453 vsize=657612 CPUtime=0.91 cores=4-31
/proc/9458/task/9479/stat : 9479 (plingeling) R 9453 9453 9427 0 -1 4202560 2606 1239 0 0 91 0 0 0 21 0 29 0 593277953 673394688 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4322000 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9458/tid=9480] ppid=9453 vsize=658068 CPUtime=0.9 cores=4-31
/proc/9458/task/9480/stat : 9480 (plingeling) R 9453 9453 9427 0 -1 4202560 2657 1239 0 0 90 0 0 0 21 0 29 0 593277953 673861632 53843 268435456000 4194304 4356188 140734679173232 18446744073709551615 4322000 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=9458/tid=9481] ppid=9453 vsize=658068 CPUtime=0.89 cores=4-31
/proc/9458/task/9481/stat : 9481 (plingeling) R 9453 9453 9427 0 -1 4202560 2688 1239 0 0 89 0 0 0 18 0 29 0 593277953 673861632 53852 268435456000 4194304 4356188 140734679173232 18446744073709551615 4210575 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=9458/tid=9482] ppid=9453 vsize=658068 CPUtime=0.92 cores=4-31
/proc/9458/task/9482/stat : 9482 (plingeling) R 9453 9453 9427 0 -1 4202560 1881 1239 0 0 92 0 0 0 19 0 29 0 593277953 673861632 53861 268435456000 4194304 4356188 140734679173232 18446744073709551615 4317354 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=9458/tid=9483] ppid=9453 vsize=658068 CPUtime=0.92 cores=4-31
/proc/9458/task/9483/stat : 9483 (plingeling) R 9453 9453 9427 0 -1 4202560 1570 1239 0 0 92 0 0 0 23 0 29 0 593277953 673861632 53869 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=9458/tid=9484] ppid=9453 vsize=658068 CPUtime=0.87 cores=4-31
/proc/9458/task/9484/stat : 9484 (plingeling) R 9453 9453 9427 0 -1 4202560 1987 1239 0 0 87 0 0 0 21 0 29 0 593277953 673861632 53878 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=9458/tid=9485] ppid=9453 vsize=658068 CPUtime=0.91 cores=4-31
/proc/9458/task/9485/stat : 9485 (plingeling) R 9453 9453 9427 0 -1 4202560 1768 1239 0 0 91 0 0 0 19 0 29 0 593277954 673861632 53888 268435456000 4194304 4356188 140734679173232 18446744073709551615 4207569 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=9458/tid=9486] ppid=9453 vsize=658068 CPUtime=0.9 cores=4-31
/proc/9458/task/9486/stat : 9486 (plingeling) R 9453 9453 9427 0 -1 4202560 915 1239 0 0 90 0 0 0 19 0 29 0 593277954 673861632 53896 268435456000 4194304 4356188 140734679173232 18446744073709551615 4315078 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=9458/tid=9487] ppid=9453 vsize=658068 CPUtime=0.9 cores=4-31
/proc/9458/task/9487/stat : 9487 (plingeling) R 9453 9453 9427 0 -1 4202560 1603 1239 0 0 90 0 0 0 19 0 29 0 593277954 673861632 53906 268435456000 4194304 4356188 140734679173232 18446744073709551615 4317324 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=9458/tid=9488] ppid=9453 vsize=658068 CPUtime=0.89 cores=4-31
/proc/9458/task/9488/stat : 9488 (plingeling) R 9453 9453 9427 0 -1 4202560 915 1239 0 0 89 0 0 0 21 0 29 0 593277955 673861632 53915 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244344 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=9458/tid=9489] ppid=9453 vsize=658068 CPUtime=0.88 cores=4-31
/proc/9458/task/9489/stat : 9489 (plingeling) R 9453 9453 9427 0 -1 4202560 1218 1239 0 0 88 0 0 0 19 0 29 0 593277955 673861632 53925 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244731 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=9458/tid=9490] ppid=9453 vsize=658068 CPUtime=0.88 cores=4-31
/proc/9458/task/9490/stat : 9490 (plingeling) R 9453 9453 9427 0 -1 4202560 943 1239 0 0 88 0 0 0 21 0 29 0 593277956 673861632 53932 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244494 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=9458/tid=9491] ppid=9453 vsize=658068 CPUtime=0.88 cores=4-31
/proc/9458/task/9491/stat : 9491 (plingeling) R 9453 9453 9427 0 -1 4202560 1603 1239 0 0 88 0 0 0 21 0 29 0 593277956 673861632 53942 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=9458/tid=9492] ppid=9453 vsize=658068 CPUtime=0.88 cores=4-31
/proc/9458/task/9492/stat : 9492 (plingeling) R 9453 9453 9427 0 -1 4202560 1813 1239 0 0 88 0 0 0 21 0 29 0 593277956 673861632 53951 268435456000 4194304 4356188 140734679173232 18446744073709551615 4244356 0 0 4096 17442 0 0 0 -1 17 0 0 0
Current children cumulated CPU time (s) 40.12
Current children cumulated vsize (KiB) 1273200

[startup+6.30178 s]
/proc/loadavg: 27.60 31.03 31.73 33/447 9492
/proc/meminfo: memFree=256250384/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 15 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
[pid=9454] ppid=9453 vsize=100316 CPUtime=6.29 cores=0
/proc/9454/stat : 9454 (cryptominisat) R 9453 9453 9427 0 -1 4202496 15044 0 0 0 626 3 0 0 25 0 1 0 593277737 102723584 12683 268435456000 4194304 4587354 140735773368368 18446744073709551615 4230306 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9454/statm: 25079 12683 331 96 0 21641 0
[pid=9455] ppid=9453 vsize=40704 CPUtime=6.29 cores=1
/proc/9455/stat : 9455 (clasp) R 9453 9453 9427 0 -1 4202496 9363 0 0 0 625 4 0 0 25 0 1 0 593277737 41680896 7389 268435456000 4194304 4785740 140733578986096 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9455/statm: 10176 7389 338 145 0 7221 0
[pid=9456] ppid=9453 vsize=462956 CPUtime=6.28 cores=2
/proc/9456/stat : 9456 (TNM) R 9453 9453 9427 0 -1 4202496 5281 0 0 0 626 2 0 0 25 0 1 0 593277737 474066944 5179 268435456000 134512640 134534564 4287769776 18446744073709551615 134530664 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9456/statm: 115739 5179 95 6 0 114404 0
[pid=9457] ppid=9453 vsize=62496 CPUtime=6.28 cores=3
/proc/9457/stat : 9457 (march_hi) R 9453 9453 9427 0 -1 4202496 40713 0 0 0 619 9 0 0 25 0 1 0 593277737 63995904 12885 268435456000 4194304 4331988 140735863302960 18446744073709551615 4261508 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9457/statm: 15624 12885 150 34 0 14066 0
[pid=9458] ppid=9453 vsize=844268 CPUtime=114.57 cores=4-31
/proc/9458/stat : 9458 (plingeling) S 9453 9453 9427 0 -1 4202496 156551 1239 0 0 11370 87 0 0 18 0 29 0 593277737 864530432 97161 268435456000 4194304 4356188 140734679173232 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 27 0 0 0
/proc/9458/statm: 211067 97161 159 40 0 209611 0
[pid=9458/tid=9465] ppid=9453 vsize=844268 CPUtime=4.07 cores=4-31
/proc/9458/task/9465/stat : 9465 (plingeling) R 9453 9453 9427 0 -1 4202560 3518 1239 0 0 406 1 0 0 25 0 29 0 593277952 864530432 97161 268435456000 4194304 4356188 140734679173232 18446744073709551615 4312507 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9458/tid=9466] ppid=9453 vsize=844268 CPUtime=3.96 cores=4-31
/proc/9458/task/9466/stat : 9466 (plingeling) R 9453 9453 9427 0 -1 4202560 2580 1239 0 0 393 3 0 0 25 0 29 0 593277952 864530432 97161 268435456000 4194304 4356188 140734679173232 18446744073709551615 4270780 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=9458/tid=9467] ppid=9453 vsize=844268 CPUtime=3.99 cores=4-31
/proc/9458/task/9467/stat : 9467 (plingeling) R 9453 9453 9427 0 -1 4202560 3888 1239 0 0 396 3 0 0 20 0 29 0 593277952 864530432 97161 268435456000 4194304 4356188 140734679173232 18446744073709551615 4270759 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9458/tid=9468] ppid=9453 vsize=844268 CPUtime=4.06 cores=4-31
/proc/9458/task/9468/stat : 9468 (plingeling) R 9453 9453 9427 0 -1 4202560 4773 1239 0 0 405 1 0 0 25 0 29 0 593277952 864530432 97161 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=9458/tid=9469] ppid=9453 vsize=844268 CPUtime=4.08 cores=4-31
/proc/9458/task/9469/stat : 9469 (plingeling) R 9453 9453 9427 0 -1 4202560 3964 1239 0 0 405 3 0 0 21 0 29 0 593277952 864530432 97161 268435456000 4194304 4356188 140734679173232 18446744073709551615 4270892 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=9458/tid=9470] ppid=9453 vsize=844268 CPUtime=4.01 cores=4-31

################
# More data... #
################

/proc/9454/statm: 448943 47825 338 96 0 445505 0
[pid=9455] ppid=9453 vsize=499644 CPUtime=4998.02 cores=1
/proc/9455/stat : 9455 (clasp) R 9453 9453 9427 0 -1 4202496 123219 0 0 0 499713 89 0 0 25 0 1 0 593277737 511635456 121245 268435456000 4194304 4785740 140733578986096 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9455/statm: 124911 121245 338 145 0 121956 0
[pid=9456] ppid=9453 vsize=462960 CPUtime=4996.04 cores=2
/proc/9456/stat : 9456 (TNM) R 9453 9453 9427 0 -1 4202496 5291 0 0 0 499598 6 0 0 25 0 1 0 593277737 474071040 5189 268435456000 134512640 134534564 4287769776 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9456/statm: 115740 5189 103 6 0 114405 0
[pid=9457] ppid=9453 vsize=176600 CPUtime=4989.72 cores=3
/proc/9457/stat : 9457 (march_hi) R 9453 9453 9427 0 -1 4202496 83357 0 0 0 498948 24 0 0 25 0 1 0 593277737 180838400 28882 268435456000 4194304 4331988 140735863302960 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9457/statm: 44150 28882 175 34 0 42592 0
[pid=9458] ppid=9453 vsize=1623032 CPUtime=140026 cores=4-31
/proc/9458/stat : 9458 (plingeling) S 9453 9453 9427 0 -1 4202496 1775283 1239 0 0 14001118 1439 0 0 18 0 29 0 593277737 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 27 0 0 0
/proc/9458/statm: 405758 309919 159 40 0 404302 0
[pid=9458/tid=9465] ppid=9453 vsize=1623032 CPUtime=5006.5 cores=4-31
/proc/9458/task/9465/stat : 9465 (plingeling) R 9453 9453 9427 0 -1 4202560 70087 1239 0 0 500590 60 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9458/tid=9466] ppid=9453 vsize=1623032 CPUtime=5005.94 cores=4-31
/proc/9458/task/9466/stat : 9466 (plingeling) R 9453 9453 9427 0 -1 4202560 54994 1239 0 0 500545 49 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=9458/tid=9467] ppid=9453 vsize=1623032 CPUtime=5006.3 cores=4-31
/proc/9458/task/9467/stat : 9467 (plingeling) R 9453 9453 9427 0 -1 4202560 46007 1239 0 0 500601 29 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9458/tid=9468] ppid=9453 vsize=1623032 CPUtime=5005.7 cores=4-31
/proc/9458/task/9468/stat : 9468 (plingeling) R 9453 9453 9427 0 -1 4202560 104286 1239 0 0 500515 55 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=9458/tid=9469] ppid=9453 vsize=1623032 CPUtime=5006.02 cores=4-31
/proc/9458/task/9469/stat : 9469 (plingeling) R 9453 9453 9427 0 -1 4202560 55926 1239 0 0 500546 56 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=9458/tid=9470] ppid=9453 vsize=1623032 CPUtime=5006.5 cores=4-31
/proc/9458/task/9470/stat : 9470 (plingeling) R 9453 9453 9427 0 -1 4202560 59475 1239 0 0 500588 62 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=9458/tid=9471] ppid=9453 vsize=1623032 CPUtime=5005.82 cores=4-31
/proc/9458/task/9471/stat : 9471 (plingeling) R 9453 9453 9427 0 -1 4202560 65177 1239 0 0 500521 61 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4280034 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=9458/tid=9472] ppid=9453 vsize=1623032 CPUtime=5005.91 cores=4-31
/proc/9458/task/9472/stat : 9472 (plingeling) R 9453 9453 9427 0 -1 4202560 81332 1239 0 0 500514 77 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=9458/tid=9473] ppid=9453 vsize=1623032 CPUtime=5006.44 cores=4-31
/proc/9458/task/9473/stat : 9473 (plingeling) R 9453 9453 9427 0 -1 4202560 55360 1239 0 0 500601 43 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4212130 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=9458/tid=9474] ppid=9453 vsize=1623032 CPUtime=5004.47 cores=4-31
/proc/9458/task/9474/stat : 9474 (plingeling) R 9453 9453 9427 0 -1 4202560 53826 1239 0 0 500365 82 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=9458/tid=9475] ppid=9453 vsize=1623032 CPUtime=5006.16 cores=4-31
/proc/9458/task/9475/stat : 9475 (plingeling) R 9453 9453 9427 0 -1 4202560 53192 1239 0 0 500572 44 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=9458/tid=9476] ppid=9453 vsize=1623032 CPUtime=5005.71 cores=4-31
/proc/9458/task/9476/stat : 9476 (plingeling) R 9453 9453 9427 0 -1 4202560 83130 1239 0 0 500522 49 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4234297 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=9458/tid=9477] ppid=9453 vsize=1623032 CPUtime=5005.5 cores=4-31
/proc/9458/task/9477/stat : 9477 (plingeling) R 9453 9453 9427 0 -1 4202560 61429 1239 0 0 500512 38 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=9458/tid=9478] ppid=9453 vsize=1623032 CPUtime=5006.12 cores=4-31
/proc/9458/task/9478/stat : 9478 (plingeling) R 9453 9453 9427 0 -1 4202560 56036 1239 0 0 500570 42 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=9458/tid=9479] ppid=9453 vsize=1623032 CPUtime=5005.89 cores=4-31
/proc/9458/task/9479/stat : 9479 (plingeling) R 9453 9453 9427 0 -1 4202560 64036 1239 0 0 500538 51 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331360 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9458/tid=9480] ppid=9453 vsize=1623032 CPUtime=5006.07 cores=4-31
/proc/9458/task/9480/stat : 9480 (plingeling) R 9453 9453 9427 0 -1 4202560 56614 1239 0 0 500558 49 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=9458/tid=9481] ppid=9453 vsize=1623032 CPUtime=4977.06 cores=4-31
/proc/9458/task/9481/stat : 9481 (plingeling) R 9453 9453 9427 0 -1 4202560 65643 1239 0 0 497652 54 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4211026 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=9458/tid=9482] ppid=9453 vsize=1623032 CPUtime=5006.5 cores=4-31
/proc/9458/task/9482/stat : 9482 (plingeling) R 9453 9453 9427 0 -1 4202560 42991 1239 0 0 500600 50 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=9458/tid=9483] ppid=9453 vsize=1623032 CPUtime=4997.3 cores=4-31
/proc/9458/task/9483/stat : 9483 (plingeling) R 9453 9453 9427 0 -1 4202560 61636 1239 0 0 499679 51 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331792 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=9458/tid=9484] ppid=9453 vsize=1623032 CPUtime=5005.91 cores=4-31
/proc/9458/task/9484/stat : 9484 (plingeling) R 9453 9453 9427 0 -1 4202560 61850 1239 0 0 500550 41 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=9458/tid=9485] ppid=9453 vsize=1623032 CPUtime=5006.2 cores=4-31
/proc/9458/task/9485/stat : 9485 (plingeling) R 9453 9453 9427 0 -1 4202560 51517 1239 0 0 500567 53 0 0 25 0 29 0 593277954 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=9458/tid=9486] ppid=9453 vsize=1623032 CPUtime=5003.93 cores=4-31
/proc/9458/task/9486/stat : 9486 (plingeling) R 9453 9453 9427 0 -1 4202560 54431 1239 0 0 500360 33 0 0 25 0 29 0 593277954 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331199 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=9458/tid=9487] ppid=9453 vsize=1623032 CPUtime=5006.38 cores=4-31
/proc/9458/task/9487/stat : 9487 (plingeling) R 9453 9453 9427 0 -1 4202560 64993 1239 0 0 500571 67 0 0 25 0 29 0 593277954 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=9458/tid=9488] ppid=9453 vsize=1623032 CPUtime=4900.78 cores=4-31
/proc/9458/task/9488/stat : 9488 (plingeling) R 9453 9453 9427 0 -1 4202560 74403 1239 0 0 490014 64 0 0 25 0 29 0 593277955 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=9458/tid=9489] ppid=9453 vsize=1623032 CPUtime=5006.4 cores=4-31
/proc/9458/task/9489/stat : 9489 (plingeling) R 9453 9453 9427 0 -1 4202560 69561 1239 0 0 500590 50 0 0 25 0 29 0 593277955 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331483 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=9458/tid=9490] ppid=9453 vsize=1623032 CPUtime=5006.26 cores=4-31
/proc/9458/task/9490/stat : 9490 (plingeling) R 9453 9453 9427 0 -1 4202560 55997 1239 0 0 500588 38 0 0 25 0 29 0 593277956 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4281686 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=9458/tid=9491] ppid=9453 vsize=1623032 CPUtime=5005.14 cores=4-31
/proc/9458/task/9491/stat : 9491 (plingeling) R 9453 9453 9427 0 -1 4202560 51520 1239 0 0 500476 38 0 0 25 0 29 0 593277956 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=9458/tid=9492] ppid=9453 vsize=1623032 CPUtime=5006.24 cores=4-31
/proc/9458/task/9492/stat : 9492 (plingeling) R 9453 9453 9427 0 -1 4202560 41192 1239 0 0 500601 23 0 0 25 0 29 0 593277956 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 17 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 4576332

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5009.8 s]
/proc/loadavg: 33.32 33.18 32.95 34/455 15706
/proc/meminfo: memFree=254709732/264257624 swapFree=67111528/67111528
[pid=9453] ppid=9451 vsize=18324 CPUtime=0 cores=0-31
/proc/9453/stat : 9453 (ppfolio) S 9451 9453 9427 0 -1 4202496 513 0 0 0 0 0 0 0 15 0 1 0 593277736 18763776 386 268435456000 4194304 4287625 140734057575136 18446744073709551615 258650975999 0 0 4096 65536 13273577466007388159 0 0 17 20 0 0 0
/proc/9453/statm: 4581 386 337 23 0 91 0
[pid=9454] ppid=9453 vsize=1795772 CPUtime=4994.48 cores=0
/proc/9454/stat : 9454 (cryptominisat) R 9453 9453 9427 0 -1 4202496 2304712 0 0 0 498635 813 0 0 25 0 1 0 593277737 1838870528 47825 268435456000 4194304 4587354 140735773368368 18446744073709551615 258650652598 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9454/statm: 448943 47825 338 96 0 445505 0
[pid=9455] ppid=9453 vsize=499644 CPUtime=4998.02 cores=1
/proc/9455/stat : 9455 (clasp) R 9453 9453 9427 0 -1 4202496 123219 0 0 0 499713 89 0 0 25 0 1 0 593277737 511635456 121245 268435456000 4194304 4785740 140733578986096 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9455/statm: 124911 121245 338 145 0 121956 0
[pid=9456] ppid=9453 vsize=462960 CPUtime=4996.04 cores=2
/proc/9456/stat : 9456 (TNM) R 9453 9453 9427 0 -1 4202496 5291 0 0 0 499598 6 0 0 25 0 1 0 593277737 474071040 5189 268435456000 134512640 134534564 4287769776 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9456/statm: 115740 5189 103 6 0 114405 0
[pid=9457] ppid=9453 vsize=176600 CPUtime=4989.72 cores=3
/proc/9457/stat : 9457 (march_hi) R 9453 9453 9427 0 -1 4202496 83357 0 0 0 498948 24 0 0 25 0 1 0 593277737 180838400 28882 268435456000 4194304 4331988 140735863302960 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9457/statm: 44150 28882 175 34 0 42592 0
[pid=9458] ppid=9453 vsize=1623032 CPUtime=140026 cores=4-31
/proc/9458/stat : 9458 (plingeling) S 9453 9453 9427 0 -1 4202496 1775283 1239 0 0 14001118 1439 0 0 18 0 29 0 593277737 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 27 0 0 0
/proc/9458/statm: 405758 309919 159 40 0 404302 0
[pid=9458/tid=9465] ppid=9453 vsize=1623032 CPUtime=5006.5 cores=4-31
/proc/9458/task/9465/stat : 9465 (plingeling) R 9453 9453 9427 0 -1 4202560 70087 1239 0 0 500590 60 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9458/tid=9466] ppid=9453 vsize=1623032 CPUtime=5005.94 cores=4-31
/proc/9458/task/9466/stat : 9466 (plingeling) R 9453 9453 9427 0 -1 4202560 54994 1239 0 0 500545 49 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=9458/tid=9467] ppid=9453 vsize=1623032 CPUtime=5006.3 cores=4-31
/proc/9458/task/9467/stat : 9467 (plingeling) R 9453 9453 9427 0 -1 4202560 46007 1239 0 0 500601 29 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9458/tid=9468] ppid=9453 vsize=1623032 CPUtime=5005.7 cores=4-31
/proc/9458/task/9468/stat : 9468 (plingeling) R 9453 9453 9427 0 -1 4202560 104286 1239 0 0 500515 55 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=9458/tid=9469] ppid=9453 vsize=1623032 CPUtime=5006.02 cores=4-31
/proc/9458/task/9469/stat : 9469 (plingeling) R 9453 9453 9427 0 -1 4202560 55926 1239 0 0 500546 56 0 0 25 0 29 0 593277952 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=9458/tid=9470] ppid=9453 vsize=1623032 CPUtime=5006.5 cores=4-31
/proc/9458/task/9470/stat : 9470 (plingeling) R 9453 9453 9427 0 -1 4202560 59475 1239 0 0 500588 62 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=9458/tid=9471] ppid=9453 vsize=1623032 CPUtime=5005.82 cores=4-31
/proc/9458/task/9471/stat : 9471 (plingeling) R 9453 9453 9427 0 -1 4202560 65177 1239 0 0 500521 61 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4280034 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=9458/tid=9472] ppid=9453 vsize=1623032 CPUtime=5005.91 cores=4-31
/proc/9458/task/9472/stat : 9472 (plingeling) R 9453 9453 9427 0 -1 4202560 81332 1239 0 0 500514 77 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=9458/tid=9473] ppid=9453 vsize=1623032 CPUtime=5006.44 cores=4-31
/proc/9458/task/9473/stat : 9473 (plingeling) R 9453 9453 9427 0 -1 4202560 55360 1239 0 0 500601 43 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4212130 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=9458/tid=9474] ppid=9453 vsize=1623032 CPUtime=5004.47 cores=4-31
/proc/9458/task/9474/stat : 9474 (plingeling) R 9453 9453 9427 0 -1 4202560 53826 1239 0 0 500365 82 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=9458/tid=9475] ppid=9453 vsize=1623032 CPUtime=5006.16 cores=4-31
/proc/9458/task/9475/stat : 9475 (plingeling) R 9453 9453 9427 0 -1 4202560 53192 1239 0 0 500572 44 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=9458/tid=9476] ppid=9453 vsize=1623032 CPUtime=5005.71 cores=4-31
/proc/9458/task/9476/stat : 9476 (plingeling) R 9453 9453 9427 0 -1 4202560 83130 1239 0 0 500522 49 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4234297 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=9458/tid=9477] ppid=9453 vsize=1623032 CPUtime=5005.5 cores=4-31
/proc/9458/task/9477/stat : 9477 (plingeling) R 9453 9453 9427 0 -1 4202560 61429 1239 0 0 500512 38 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=9458/tid=9478] ppid=9453 vsize=1623032 CPUtime=5006.12 cores=4-31
/proc/9458/task/9478/stat : 9478 (plingeling) R 9453 9453 9427 0 -1 4202560 56036 1239 0 0 500570 42 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=9458/tid=9479] ppid=9453 vsize=1623032 CPUtime=5005.89 cores=4-31
/proc/9458/task/9479/stat : 9479 (plingeling) R 9453 9453 9427 0 -1 4202560 64036 1239 0 0 500538 51 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331360 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9458/tid=9480] ppid=9453 vsize=1623032 CPUtime=5006.07 cores=4-31
/proc/9458/task/9480/stat : 9480 (plingeling) R 9453 9453 9427 0 -1 4202560 56614 1239 0 0 500558 49 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=9458/tid=9481] ppid=9453 vsize=1623032 CPUtime=4977.06 cores=4-31
/proc/9458/task/9481/stat : 9481 (plingeling) R 9453 9453 9427 0 -1 4202560 65643 1239 0 0 497652 54 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4211026 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=9458/tid=9482] ppid=9453 vsize=1623032 CPUtime=5006.5 cores=4-31
/proc/9458/task/9482/stat : 9482 (plingeling) R 9453 9453 9427 0 -1 4202560 42991 1239 0 0 500600 50 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=9458/tid=9483] ppid=9453 vsize=1623032 CPUtime=4997.3 cores=4-31
/proc/9458/task/9483/stat : 9483 (plingeling) R 9453 9453 9427 0 -1 4202560 61636 1239 0 0 499679 51 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331792 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=9458/tid=9484] ppid=9453 vsize=1623032 CPUtime=5005.91 cores=4-31
/proc/9458/task/9484/stat : 9484 (plingeling) R 9453 9453 9427 0 -1 4202560 61850 1239 0 0 500550 41 0 0 25 0 29 0 593277953 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=9458/tid=9485] ppid=9453 vsize=1623032 CPUtime=5006.2 cores=4-31
/proc/9458/task/9485/stat : 9485 (plingeling) R 9453 9453 9427 0 -1 4202560 51517 1239 0 0 500567 53 0 0 25 0 29 0 593277954 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=9458/tid=9486] ppid=9453 vsize=1623032 CPUtime=5003.93 cores=4-31
/proc/9458/task/9486/stat : 9486 (plingeling) R 9453 9453 9427 0 -1 4202560 54431 1239 0 0 500360 33 0 0 25 0 29 0 593277954 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331199 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=9458/tid=9487] ppid=9453 vsize=1623032 CPUtime=5006.38 cores=4-31
/proc/9458/task/9487/stat : 9487 (plingeling) R 9453 9453 9427 0 -1 4202560 64993 1239 0 0 500571 67 0 0 25 0 29 0 593277954 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=9458/tid=9488] ppid=9453 vsize=1623032 CPUtime=4900.78 cores=4-31
/proc/9458/task/9488/stat : 9488 (plingeling) R 9453 9453 9427 0 -1 4202560 74403 1239 0 0 490014 64 0 0 25 0 29 0 593277955 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=9458/tid=9489] ppid=9453 vsize=1623032 CPUtime=5006.4 cores=4-31
/proc/9458/task/9489/stat : 9489 (plingeling) R 9453 9453 9427 0 -1 4202560 69561 1239 0 0 500590 50 0 0 25 0 29 0 593277955 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331483 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=9458/tid=9490] ppid=9453 vsize=1623032 CPUtime=5006.26 cores=4-31
/proc/9458/task/9490/stat : 9490 (plingeling) R 9453 9453 9427 0 -1 4202560 55997 1239 0 0 500588 38 0 0 25 0 29 0 593277956 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4281686 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=9458/tid=9491] ppid=9453 vsize=1623032 CPUtime=5005.14 cores=4-31
/proc/9458/task/9491/stat : 9491 (plingeling) R 9453 9453 9427 0 -1 4202560 51520 1239 0 0 500476 38 0 0 25 0 29 0 593277956 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=9458/tid=9492] ppid=9453 vsize=1623032 CPUtime=5006.24 cores=4-31
/proc/9458/task/9492/stat : 9492 (plingeling) R 9453 9453 9427 0 -1 4202560 41192 1239 0 0 500601 23 0 0 25 0 29 0 593277956 1661984768 309919 268435456000 4194304 4356188 140734679173232 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 17 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 4576332

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 9453 and gives
#  childrusage.ru_utime.tv_sec=4995
#  childrusage.ru_utime.tv_usec=991493
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=73988
# CPU time returned by wait4() is 4996.07
# while last known CPU time is 160004
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5009.84
CPU time (s): 160004
CPU user time (s): 159980
CPU system time (s): 23.71
CPU usage (%): 3193.79
Max. virtual memory (cumulated for all children) (KiB): 6268500

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4995.99
system time used= 0.073988
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 5805
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 418
involuntary context switches= 27939

runsolver used 28.5727 second user time and 113.722 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-08 03:25:13
IDJOB=3347187
IDBENCH=20523
IDSOLVER=1808
FILE ID=node200/3347187-1304817913
RUNJOBID= node200-1304817913-9441
PBS_JOBID= 13281969
Free space on /tmp= 62048 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT07/industrial/IBM_FV_2002_13_rule_1/SAT_dat.k45.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347187-1304817913/watcher-3347187-1304817913 -o /tmp/evaluation-result-3347187-1304817913/solver-3347187-1304817913 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347187-1304817913.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 8a76c8add64173ae39149aae572a02d9
RANDOM SEED=596932790

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      256775136 kB
Buffers:        352188 kB
Cached:        6790472 kB
SwapCached:          0 kB
Active:        6323496 kB
Inactive:       887772 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      256775136 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           13844 kB
Writeback:           0 kB
AnonPages:       70492 kB
Mapped:          14208 kB
Slab:           191400 kB
PageTables:       4192 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   185548 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62036 MiB
End job on node200 at 2011-05-08 04:48:45