Trace number 3347237

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 160004 5009.45

General information on the benchmark

NameAPPLICATIONS/bitverif/
maxxororand/maxxororand064.cnf
MD5SUM49dbd2fb16b3d5ce554d30c48729a56d
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark83265.2
Satisfiable
(Un)Satisfiability was proved
Number of variables92464
Number of clauses276623
Sum of the clauses size645451
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2184414
Number of clauses of size 392207
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347237-1304908802/watcher-3347237-1304908802 -o /tmp/evaluation-result-3347237-1304908802/solver-3347237-1304908802 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347237-1304908802.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 27.16 31.00 31.67 1/413 23634
/proc/meminfo: memFree=256752996/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 21 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0

[startup+0.085348 s]
/proc/loadavg: 27.16 31.00 31.67 1/413 23634
/proc/meminfo: memFree=256752996/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 21 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100349 s]
/proc/loadavg: 27.16 31.00 31.67 1/413 23634
/proc/meminfo: memFree=256752996/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 21 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300347 s]
/proc/loadavg: 27.16 31.00 31.67 1/413 23634
/proc/meminfo: memFree=256752996/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 19 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700373 s]
/proc/loadavg: 27.16 31.00 31.67 1/413 23634
/proc/meminfo: memFree=256752996/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 19 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.5014 s]
/proc/loadavg: 27.16 31.00 31.67 6/419 23645
/proc/meminfo: memFree=256490428/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 17 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
[pid=23635] ppid=23634 vsize=69396 CPUtime=1.49 cores=0
/proc/23635/stat : 23635 (cryptominisat) R 23634 23634 23608 0 -1 4202496 11413 0 0 0 146 3 0 0 25 0 1 0 602366654 71061504 11144 268435456000 4194304 4587354 140733641813952 18446744073709551615 258650616632 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/23635/statm: 17349 11146 308 96 0 13911 0
[pid=23636] ppid=23634 vsize=69824 CPUtime=1.48 cores=1
/proc/23636/stat : 23636 (clasp) R 23634 23634 23608 0 -1 4202496 17896 0 0 0 143 5 0 0 20 0 1 0 602366654 71499776 14357 268435456000 4194304 4785740 140734330010944 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/23636/statm: 17456 14357 321 145 0 14501 0
[pid=23637] ppid=23634 vsize=471140 CPUtime=1.48 cores=2
/proc/23637/stat : 23637 (TNM) R 23634 23634 23608 0 -1 4202496 9723 0 0 0 146 2 0 0 20 0 1 0 602366654 482447360 9620 268435456000 134512640 134534564 4291805680 18446744073709551615 134529352 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/23637/statm: 117785 9620 95 6 0 116450 0
[pid=23638] ppid=23634 vsize=64000 CPUtime=1.48 cores=3
/proc/23638/stat : 23638 (march_hi) R 23634 23634 23608 0 -1 4202496 51334 0 0 0 136 12 0 0 20 0 1 0 602366654 65536000 11870 268435456000 4194304 4331988 140735209632224 18446744073709551615 4252200 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/23638/statm: 16000 11870 140 34 0 14442 0
[pid=23639] ppid=23634 vsize=120012 CPUtime=1.49 cores=4-31
/proc/23639/stat : 23639 (plingeling) R 23634 23634 23608 0 -1 4202496 28794 1240 0 0 140 8 0 1 21 0 1 0 602366654 122892288 28662 268435456000 4194304 4356188 140733321386368 18446744073709551615 4225315 0 0 4096 17442 0 0 0 17 8 0 0 0
/proc/23639/statm: 30003 28662 129 40 0 28547 0
Current children cumulated CPU time (s) 7.42
Current children cumulated vsize (KiB) 812696

[startup+3.10146 s]
/proc/loadavg: 27.16 31.00 31.67 6/419 23645
/proc/meminfo: memFree=256292440/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
[pid=23635] ppid=23634 vsize=73976 CPUtime=3.08 cores=0
/proc/23635/stat : 23635 (cryptominisat) R 23634 23634 23608 0 -1 4202496 12770 0 0 0 305 3 0 0 25 0 1 0 602366654 75751424 11778 268435456000 4194304 4587354 140733641813952 18446744073709551615 4356723 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/23635/statm: 18494 11778 342 96 0 15056 0
[pid=23636] ppid=23634 vsize=69824 CPUtime=3.08 cores=1
/proc/23636/stat : 23636 (clasp) R 23634 23634 23608 0 -1 4202496 17924 0 0 0 303 5 0 0 25 0 1 0 602366654 71499776 14385 268435456000 4194304 4785740 140734330010944 18446744073709551615 4552974 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/23636/statm: 17456 14385 338 145 0 14501 0
[pid=23637] ppid=23634 vsize=471140 CPUtime=3.08 cores=2
/proc/23637/stat : 23637 (TNM) R 23634 23634 23608 0 -1 4202496 9723 0 0 0 306 2 0 0 25 0 1 0 602366654 482447360 9620 268435456000 134512640 134534564 4291805680 18446744073709551615 134530367 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/23637/statm: 117785 9620 95 6 0 116450 0
[pid=23638] ppid=23634 vsize=182204 CPUtime=3.08 cores=3
/proc/23638/stat : 23638 (march_hi) R 23634 23634 23608 0 -1 4202496 88997 0 0 0 288 20 0 0 25 0 1 0 602366654 186576896 29370 268435456000 4194304 4331988 140735209632224 18446744073709551615 258650616986 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/23638/statm: 45551 29370 172 34 0 43993 0
[pid=23639] ppid=23634 vsize=343184 CPUtime=3.09 cores=4-31
/proc/23639/stat : 23639 (plingeling) R 23634 23634 23608 0 -1 4202496 84644 1240 0 0 290 18 0 1 25 0 1 0 602366654 351420416 84455 268435456000 4194304 4356188 140733321386368 18446744073709551615 4277662 0 0 4096 17442 0 0 0 17 8 0 0 0
/proc/23639/statm: 85796 84455 129 40 0 84340 0
Current children cumulated CPU time (s) 15.41
Current children cumulated vsize (KiB) 1158652

[startup+6.30251 s]
/proc/loadavg: 25.38 30.56 31.53 33/447 23673
/proc/meminfo: memFree=255882360/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
[pid=23635] ppid=23634 vsize=155156 CPUtime=6.29 cores=0
/proc/23635/stat : 23635 (cryptominisat) R 23634 23634 23608 0 -1 4202496 31126 0 0 0 622 7 0 0 25 0 1 0 602366654 158879744 25673 268435456000 4194304 4587354 140733641813952 18446744073709551615 4230266 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/23635/statm: 38789 25673 328 96 0 35351 0
[pid=23636] ppid=23634 vsize=69824 CPUtime=6.28 cores=1
/proc/23636/stat : 23636 (clasp) R 23634 23634 23608 0 -1 4202496 17924 0 0 0 623 5 0 0 25 0 1 0 602366654 71499776 14385 268435456000 4194304 4785740 140734330010944 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/23636/statm: 17456 14385 338 145 0 14501 0
[pid=23637] ppid=23634 vsize=471140 CPUtime=6.28 cores=2
/proc/23637/stat : 23637 (TNM) R 23634 23634 23608 0 -1 4202496 9723 0 0 0 626 2 0 0 25 0 1 0 602366654 482447360 9620 268435456000 134512640 134534564 4291805680 18446744073709551615 134530268 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/23637/statm: 117785 9620 95 6 0 116450 0
[pid=23638] ppid=23634 vsize=194360 CPUtime=6.28 cores=3
/proc/23638/stat : 23638 (march_hi) R 23634 23634 23608 0 -1 4202496 101596 0 0 0 606 22 0 0 25 0 1 0 602366654 199024640 41969 268435456000 4194304 4331988 140735209632224 18446744073709551615 4263258 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/23638/statm: 48590 41969 172 34 0 47032 0
[pid=23639] ppid=23634 vsize=879116 CPUtime=57.17 cores=4-31
/proc/23639/stat : 23639 (plingeling) S 23634 23634 23608 0 -1 4202496 239224 1240 0 0 5648 68 0 1 19 0 29 0 602366654 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 12 0 0 0
/proc/23639/statm: 219779 126079 155 40 0 218323 0
[pid=23639/tid=23646] ppid=23634 vsize=879116 CPUtime=1.88 cores=4-31
/proc/23639/task/23646/stat : 23646 (plingeling) R 23634 23634 23608 0 -1 4202560 3253 1240 0 0 186 1 0 1 25 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=23639/tid=23647] ppid=23634 vsize=879116 CPUtime=1.92 cores=4-31
/proc/23639/task/23647/stat : 23647 (plingeling) R 23634 23634 23608 0 -1 4202560 2654 1240 0 0 191 0 0 1 20 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23639/tid=23648] ppid=23634 vsize=879116 CPUtime=1.91 cores=4-31
/proc/23639/task/23648/stat : 23648 (plingeling) R 23634 23634 23608 0 -1 4202560 4618 1240 0 0 189 1 0 1 18 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4321560 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23639/tid=23649] ppid=23634 vsize=879116 CPUtime=1.92 cores=4-31
/proc/23639/task/23649/stat : 23649 (plingeling) R 23634 23634 23608 0 -1 4202560 5482 1240 0 0 189 2 0 1 20 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23639/tid=23650] ppid=23634 vsize=879116 CPUtime=1.9 cores=4-31
/proc/23639/task/23650/stat : 23650 (plingeling) R 23634 23634 23608 0 -1 4202560 5635 1240 0 0 187 2 0 1 20 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4321963 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23639/tid=23651] ppid=23634 vsize=879116 CPUtime=1.89 cores=4-31
/proc/23639/task/23651/stat : 23651 (plingeling) R 23634 23634 23608 0 -1 4202560 7122 1240 0 0 186 2 0 1 20 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4210605 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23639/tid=23652] ppid=23634 vsize=879116 CPUtime=1.9 cores=4-31
/proc/23639/task/23652/stat : 23652 (plingeling) R 23634 23634 23608 0 -1 4202560 7257 1240 0 0 187 2 0 1 18 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322106 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23639/tid=23653] ppid=23634 vsize=879116 CPUtime=1.85 cores=4-31
/proc/23639/task/23653/stat : 23653 (plingeling) R 23634 23634 23608 0 -1 4202560 1992 1240 0 0 184 0 0 1 25 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23639/tid=23654] ppid=23634 vsize=879116 CPUtime=1.91 cores=4-31
/proc/23639/task/23654/stat : 23654 (plingeling) R 23634 23634 23608 0 -1 4202560 2768 1240 0 0 189 1 0 1 25 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322000 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23639/tid=23655] ppid=23634 vsize=879116 CPUtime=1.9 cores=4-31
/proc/23639/task/23655/stat : 23655 (plingeling) R 23634 23634 23608 0 -1 4202560 4775 1240 0 0 188 1 0 1 20 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23639/tid=23656] ppid=23634 vsize=879116 CPUtime=1.89 cores=4-31
/proc/23639/task/23656/stat : 23656 (plingeling) R 23634 23634 23608 0 -1 4202560 6195 1240 0 0 187 1 0 1 19 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322106 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23639/tid=23657] ppid=23634 vsize=879116 CPUtime=1.9 cores=4-31
/proc/23639/task/23657/stat : 23657 (plingeling) R 23634 23634 23608 0 -1 4202560 5890 1240 0 0 188 1 0 1 20 0 29 0 602367091 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4279701 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23639/tid=23658] ppid=23634 vsize=879116 CPUtime=1.9 cores=4-31
/proc/23639/task/23658/stat : 23658 (plingeling) R 23634 23634 23608 0 -1 4202560 3067 1240 0 0 188 1 0 1 20 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23639/tid=23659] ppid=23634 vsize=879116 CPUtime=1.9 cores=4-31
/proc/23639/task/23659/stat : 23659 (plingeling) R 23634 23634 23608 0 -1 4202560 3110 1240 0 0 188 1 0 1 20 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4321566 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23639/tid=23660] ppid=23634 vsize=879116 CPUtime=1.87 cores=4-31
/proc/23639/task/23660/stat : 23660 (plingeling) R 23634 23634 23608 0 -1 4202560 3845 1240 0 0 185 1 0 1 18 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4317350 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23639/tid=23661] ppid=23634 vsize=879116 CPUtime=1.81 cores=4-31
/proc/23639/task/23661/stat : 23661 (plingeling) R 23634 23634 23608 0 -1 4202560 2892 1240 0 0 180 0 0 1 23 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4207739 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23639/tid=23662] ppid=23634 vsize=879116 CPUtime=1.88 cores=4-31
/proc/23639/task/23662/stat : 23662 (plingeling) R 23634 23634 23608 0 -1 4202560 3966 1240 0 0 186 1 0 1 18 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23639/tid=23663] ppid=23634 vsize=879116 CPUtime=1.89 cores=4-31
/proc/23639/task/23663/stat : 23663 (plingeling) R 23634 23634 23608 0 -1 4202560 6435 1240 0 0 186 2 0 1 20 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4321585 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23639/tid=23664] ppid=23634 vsize=879116 CPUtime=1.9 cores=4-31
/proc/23639/task/23664/stat : 23664 (plingeling) R 23634 23634 23608 0 -1 4202560 4495 1240 0 0 188 1 0 1 18 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23639/tid=23665] ppid=23634 vsize=879116 CPUtime=1.88 cores=4-31
/proc/23639/task/23665/stat : 23665 (plingeling) R 23634 23634 23608 0 -1 4202560 4238 1240 0 0 187 0 0 1 20 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4334582 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23639/tid=23666] ppid=23634 vsize=879116 CPUtime=1.87 cores=4-31
/proc/23639/task/23666/stat : 23666 (plingeling) R 23634 23634 23608 0 -1 4202560 6052 1240 0 0 185 1 0 1 25 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4317354 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=23639/tid=23667] ppid=23634 vsize=879116 CPUtime=1.89 cores=4-31
/proc/23639/task/23667/stat : 23667 (plingeling) R 23634 23634 23608 0 -1 4202560 4104 1240 0 0 187 1 0 1 25 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=23639/tid=23668] ppid=23634 vsize=879116 CPUtime=1.86 cores=4-31
/proc/23639/task/23668/stat : 23668 (plingeling) R 23634 23634 23608 0 -1 4202560 3976 1240 0 0 184 1 0 1 25 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4320759 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=23639/tid=23669] ppid=23634 vsize=879116 CPUtime=1.87 cores=4-31
/proc/23639/task/23669/stat : 23669 (plingeling) R 23634 23634 23608 0 -1 4202560 2745 1240 0 0 186 0 0 1 25 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322106 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=23639/tid=23670] ppid=23634 vsize=879116 CPUtime=1.89 cores=4-31
/proc/23639/task/23670/stat : 23670 (plingeling) R 23634 23634 23608 0 -1 4202560 6780 1240 0 0 186 2 0 1 18 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322678 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=23639/tid=23671] ppid=23634 vsize=879116 CPUtime=1.88 cores=4-31
/proc/23639/task/23671/stat : 23671 (plingeling) R 23634 23634 23608 0 -1 4202560 2856 1240 0 0 187 0 0 1 18 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322106 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23639/tid=23672] ppid=23634 vsize=879116 CPUtime=1.86 cores=4-31
/proc/23639/task/23672/stat : 23672 (plingeling) R 23634 23634 23608 0 -1 4202560 4432 1240 0 0 185 0 0 1 18 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4322016 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23639/tid=23673] ppid=23634 vsize=879116 CPUtime=1.88 cores=4-31
/proc/23639/task/23673/stat : 23673 (plingeling) R 23634 23634 23608 0 -1 4202560 5236 1240 0 0 186 1 0 1 25 0 29 0 602367092 900214784 126079 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 11 0 0 0
Current children cumulated CPU time (s) 82.3
Current children cumulated vsize (KiB) 1787920

[startup+12.7017 s]
/proc/loadavg: 25.91 30.59 31.53 33/447 23673
/proc/meminfo: memFree=255513108/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
[pid=23635] ppid=23634 vsize=155156 CPUtime=12.68 cores=0
/proc/23635/stat : 23635 (cryptominisat) R 23634 23634 23608 0 -1 4202496 31796 0 0 0 1261 7 0 0 25 0 1 0 602366654 158879744 26326 268435456000 4194304 4587354 140733641813952 18446744073709551615 4253222 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=23637] ppid=23634 vsize=471144 CPUtime=4961.32 cores=2
/proc/23637/stat : 23637 (TNM) R 23634 23634 23608 0 -1 4202496 9733 0 0 0 496130 2 0 0 25 0 1 0 602366654 482451456 9630 268435456000 134512640 134534564 4291805680 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/23637/statm: 117786 9630 103 6 0 116451 0
[pid=23638] ppid=23634 vsize=256404 CPUtime=4961.24 cores=3
/proc/23638/stat : 23638 (march_hi) R 23634 23634 23608 0 -1 4202496 116410 0 0 0 496085 39 0 0 25 0 1 0 602366654 262557696 56783 268435456000 4194304 4331988 140735209632224 18446744073709551615 4226434 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/23638/statm: 64101 56783 172 34 0 62543 0
[pid=23639] ppid=23634 vsize=2322320 CPUtime=138652 cores=4-31
/proc/23639/stat : 23639 (plingeling) S 23634 23634 23608 0 -1 4202496 1744499 1240 0 0 13864304 859 0 1 19 0 29 0 602366654 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 12 0 0 0
/proc/23639/statm: 580580 490170 159 40 0 579124 0
[pid=23639/tid=23646] ppid=23634 vsize=2322320 CPUtime=4956.83 cores=4-31
/proc/23639/task/23646/stat : 23646 (plingeling) R 23634 23634 23608 0 -1 4202560 62318 1240 0 0 495654 28 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331002 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=23639/tid=23647] ppid=23634 vsize=2322320 CPUtime=4956.87 cores=4-31
/proc/23639/task/23647/stat : 23647 (plingeling) R 23634 23634 23608 0 -1 4202560 62407 1240 0 0 495662 24 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23639/tid=23648] ppid=23634 vsize=2322320 CPUtime=4956.9 cores=4-31
/proc/23639/task/23648/stat : 23648 (plingeling) R 23634 23634 23608 0 -1 4202560 63537 1240 0 0 495664 25 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4332165 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23639/tid=23649] ppid=23634 vsize=2322320 CPUtime=4956.94 cores=4-31
/proc/23639/task/23649/stat : 23649 (plingeling) R 23634 23634 23608 0 -1 4202560 35664 1240 0 0 495666 27 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23639/tid=23650] ppid=23634 vsize=2322320 CPUtime=4953.84 cores=4-31
/proc/23639/task/23650/stat : 23650 (plingeling) R 23634 23634 23608 0 -1 4202560 73977 1240 0 0 495310 73 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4281624 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23639/tid=23651] ppid=23634 vsize=2322320 CPUtime=4956.84 cores=4-31
/proc/23639/task/23651/stat : 23651 (plingeling) R 23634 23634 23608 0 -1 4202560 76134 1240 0 0 495653 30 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23639/tid=23652] ppid=23634 vsize=2322320 CPUtime=4956.3 cores=4-31
/proc/23639/task/23652/stat : 23652 (plingeling) R 23634 23634 23608 0 -1 4202560 62689 1240 0 0 495600 29 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331792 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23639/tid=23653] ppid=23634 vsize=2322320 CPUtime=4817.19 cores=4-31
/proc/23639/task/23653/stat : 23653 (plingeling) R 23634 23634 23608 0 -1 4202560 67773 1240 0 0 481658 60 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=23639/tid=23654] ppid=23634 vsize=2322320 CPUtime=4956.82 cores=4-31
/proc/23639/task/23654/stat : 23654 (plingeling) R 23634 23634 23608 0 -1 4202560 34783 1240 0 0 495662 19 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330995 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23639/tid=23655] ppid=23634 vsize=2322320 CPUtime=4956.94 cores=4-31
/proc/23639/task/23655/stat : 23655 (plingeling) R 23634 23634 23608 0 -1 4202560 60264 1240 0 0 495666 27 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331704 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23639/tid=23656] ppid=23634 vsize=2322320 CPUtime=4956.88 cores=4-31
/proc/23639/task/23656/stat : 23656 (plingeling) R 23634 23634 23608 0 -1 4202560 53438 1240 0 0 495670 17 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23639/tid=23657] ppid=23634 vsize=2322320 CPUtime=4956.88 cores=4-31
/proc/23639/task/23657/stat : 23657 (plingeling) R 23634 23634 23608 0 -1 4202560 44076 1240 0 0 495659 28 0 1 25 0 29 0 602367091 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23639/tid=23658] ppid=23634 vsize=2322320 CPUtime=4956.89 cores=4-31
/proc/23639/task/23658/stat : 23658 (plingeling) R 23634 23634 23608 0 -1 4202560 78423 1240 0 0 495664 24 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23639/tid=23659] ppid=23634 vsize=2322320 CPUtime=4955.77 cores=4-31
/proc/23639/task/23659/stat : 23659 (plingeling) R 23634 23634 23608 0 -1 4202560 56897 1240 0 0 495546 30 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23639/tid=23660] ppid=23634 vsize=2322320 CPUtime=4956.82 cores=4-31
/proc/23639/task/23660/stat : 23660 (plingeling) R 23634 23634 23608 0 -1 4202560 68946 1240 0 0 495662 19 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23639/tid=23661] ppid=23634 vsize=2322320 CPUtime=4956.83 cores=4-31
/proc/23639/task/23661/stat : 23661 (plingeling) R 23634 23634 23608 0 -1 4202560 79603 1240 0 0 495619 63 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4279478 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23639/tid=23662] ppid=23634 vsize=2322320 CPUtime=4956.83 cores=4-31
/proc/23639/task/23662/stat : 23662 (plingeling) R 23634 23634 23608 0 -1 4202560 55802 1240 0 0 495655 27 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23639/tid=23663] ppid=23634 vsize=2322320 CPUtime=4956.75 cores=4-31
/proc/23639/task/23663/stat : 23663 (plingeling) R 23634 23634 23608 0 -1 4202560 80704 1240 0 0 495636 38 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4210544 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23639/tid=23664] ppid=23634 vsize=2322320 CPUtime=4956.57 cores=4-31
/proc/23639/task/23664/stat : 23664 (plingeling) R 23634 23634 23608 0 -1 4202560 35612 1240 0 0 495641 15 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23639/tid=23665] ppid=23634 vsize=2322320 CPUtime=4956.82 cores=4-31
/proc/23639/task/23665/stat : 23665 (plingeling) R 23634 23634 23608 0 -1 4202560 32550 1240 0 0 495674 7 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4236939 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23639/tid=23666] ppid=23634 vsize=2322320 CPUtime=4956.78 cores=4-31
/proc/23639/task/23666/stat : 23666 (plingeling) R 23634 23634 23608 0 -1 4202560 28108 1240 0 0 495666 11 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331456 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=23639/tid=23667] ppid=23634 vsize=2322320 CPUtime=4956.76 cores=4-31
/proc/23639/task/23667/stat : 23667 (plingeling) R 23634 23634 23608 0 -1 4202560 47927 1240 0 0 495663 12 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23639/tid=23668] ppid=23634 vsize=2322320 CPUtime=4956.81 cores=4-31
/proc/23639/task/23668/stat : 23668 (plingeling) R 23634 23634 23608 0 -1 4202560 56489 1240 0 0 495665 15 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4274275 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23639/tid=23669] ppid=23634 vsize=2322320 CPUtime=4956.86 cores=4-31
/proc/23639/task/23669/stat : 23669 (plingeling) R 23634 23634 23608 0 -1 4202560 80040 1240 0 0 495607 78 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=23639/tid=23670] ppid=23634 vsize=2322320 CPUtime=4956.87 cores=4-31
/proc/23639/task/23670/stat : 23670 (plingeling) R 23634 23634 23608 0 -1 4202560 58209 1240 0 0 495651 35 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4210549 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=23639/tid=23671] ppid=23634 vsize=2322320 CPUtime=4956.89 cores=4-31
/proc/23639/task/23671/stat : 23671 (plingeling) R 23634 23634 23608 0 -1 4202560 66381 1240 0 0 495669 19 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23639/tid=23672] ppid=23634 vsize=2322320 CPUtime=4956.92 cores=4-31
/proc/23639/task/23672/stat : 23672 (plingeling) R 23634 23634 23608 0 -1 4202560 68428 1240 0 0 495662 29 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23639/tid=23673] ppid=23634 vsize=2322320 CPUtime=4956.86 cores=4-31
/proc/23639/task/23673/stat : 23673 (plingeling) R 23634 23634 23608 0 -1 4202560 39966 1240 0 0 495677 8 0 1 25 0 29 0 602367092 2378055680 490170 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 158497
Current children cumulated vsize (KiB) 4976708



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009.42 s]
/proc/loadavg: 32.08 32.09 32.05 33/447 23828
/proc/meminfo: memFree=253451904/264257624 swapFree=67111528/67111528
[pid=23634] ppid=23632 vsize=18324 CPUtime=0 cores=0-31
/proc/23634/stat : 23634 (ppfolio) S 23632 23634 23608 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 602366654 18763776 386 268435456000 4194304 4287625 140734012946128 18446744073709551615 258650975999 0 0 4096 65536 9170076650967138303 0 0 17 4 0 0 0
/proc/23634/statm: 4581 386 337 23 0 91 0
[pid=23635] ppid=23634 vsize=1313760 CPUtime=5008.48 cores=0
/proc/23635/stat : 23635 (cryptominisat) R 23634 23634 23608 0 -1 4202496 1216195 0 0 0 500210 638 0 0 25 0 1 0 602366654 1345290240 124824 268435456000 4194304 4587354 140733641813952 18446744073709551615 4230266 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/23635/statm: 328440 124824 334 96 0 325002 0
[pid=23636] ppid=23634 vsize=585036 CPUtime=5008.48 cores=1
/proc/23636/stat : 23636 (clasp) R 23634 23634 23608 0 -1 4202496 146020 0 0 0 500780 68 0 0 25 0 1 0 602366654 599076864 142481 268435456000 4194304 4785740 140734330010944 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/23636/statm: 146259 142481 338 145 0 143304 0
[pid=23637] ppid=23634 vsize=471144 CPUtime=5008.46 cores=2
/proc/23637/stat : 23637 (TNM) R 23634 23634 23608 0 -1 4202496 9733 0 0 0 500844 2 0 0 25 0 1 0 602366654 482451456 9630 268435456000 134512640 134534564 4291805680 18446744073709551615 134530712 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/23637/statm: 117786 9630 103 6 0 116451 0
[pid=23638] ppid=23634 vsize=256404 CPUtime=5008.37 cores=3
/proc/23638/stat : 23638 (march_hi) R 23634 23634 23608 0 -1 4202496 116537 0 0 0 500798 39 0 0 25 0 1 0 602366654 262557696 56910 268435456000 4194304 4331988 140735209632224 18446744073709551615 4211435 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/23638/statm: 64101 56910 172 34 0 62543 0
[pid=23639] ppid=23634 vsize=2319764 CPUtime=139970 cores=4-31
/proc/23639/stat : 23639 (plingeling) S 23634 23634 23608 0 -1 4202496 1754103 1240 0 0 13996129 864 0 1 19 0 29 0 602366654 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 12 0 0 0
/proc/23639/statm: 579941 489531 159 40 0 578485 0
[pid=23639/tid=23646] ppid=23634 vsize=2319764 CPUtime=5003.96 cores=4-31
/proc/23639/task/23646/stat : 23646 (plingeling) R 23634 23634 23608 0 -1 4202560 63342 1240 0 0 500367 28 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=23639/tid=23647] ppid=23634 vsize=2319764 CPUtime=5004.01 cores=4-31
/proc/23639/task/23647/stat : 23647 (plingeling) R 23634 23634 23608 0 -1 4202560 62407 1240 0 0 500376 24 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=23639/tid=23648] ppid=23634 vsize=2319764 CPUtime=5004.03 cores=4-31
/proc/23639/task/23648/stat : 23648 (plingeling) R 23634 23634 23608 0 -1 4202560 63537 1240 0 0 500377 25 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=23639/tid=23649] ppid=23634 vsize=2319764 CPUtime=5004.08 cores=4-31
/proc/23639/task/23649/stat : 23649 (plingeling) R 23634 23634 23608 0 -1 4202560 35664 1240 0 0 500380 27 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=23639/tid=23650] ppid=23634 vsize=2319764 CPUtime=5000.95 cores=4-31
/proc/23639/task/23650/stat : 23650 (plingeling) R 23634 23634 23608 0 -1 4202560 75001 1240 0 0 500020 74 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331360 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=23639/tid=23651] ppid=23634 vsize=2319764 CPUtime=5003.98 cores=4-31
/proc/23639/task/23651/stat : 23651 (plingeling) R 23634 23634 23608 0 -1 4202560 76522 1240 0 0 500367 30 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=23639/tid=23652] ppid=23634 vsize=2319764 CPUtime=5003.43 cores=4-31
/proc/23639/task/23652/stat : 23652 (plingeling) R 23634 23634 23608 0 -1 4202560 63713 1240 0 0 500312 30 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=23639/tid=23653] ppid=23634 vsize=2319764 CPUtime=4862.97 cores=4-31
/proc/23639/task/23653/stat : 23653 (plingeling) R 23634 23634 23608 0 -1 4202560 67773 1240 0 0 486236 60 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=23639/tid=23654] ppid=23634 vsize=2319764 CPUtime=5003.96 cores=4-31
/proc/23639/task/23654/stat : 23654 (plingeling) R 23634 23634 23608 0 -1 4202560 34783 1240 0 0 500376 19 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=23639/tid=23655] ppid=23634 vsize=2319764 CPUtime=5004.07 cores=4-31
/proc/23639/task/23655/stat : 23655 (plingeling) R 23634 23634 23608 0 -1 4202560 60264 1240 0 0 500379 27 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=23639/tid=23656] ppid=23634 vsize=2319764 CPUtime=5004.01 cores=4-31
/proc/23639/task/23656/stat : 23656 (plingeling) R 23634 23634 23608 0 -1 4202560 53438 1240 0 0 500383 17 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=23639/tid=23657] ppid=23634 vsize=2319764 CPUtime=5004.02 cores=4-31
/proc/23639/task/23657/stat : 23657 (plingeling) R 23634 23634 23608 0 -1 4202560 44076 1240 0 0 500373 28 0 1 25 0 29 0 602367091 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=23639/tid=23658] ppid=23634 vsize=2319764 CPUtime=5004.02 cores=4-31
/proc/23639/task/23658/stat : 23658 (plingeling) R 23634 23634 23608 0 -1 4202560 78423 1240 0 0 500377 24 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=23639/tid=23659] ppid=23634 vsize=2319764 CPUtime=5002.9 cores=4-31
/proc/23639/task/23659/stat : 23659 (plingeling) R 23634 23634 23608 0 -1 4202560 56897 1240 0 0 500259 30 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4244438 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=23639/tid=23660] ppid=23634 vsize=2319764 CPUtime=5003.95 cores=4-31
/proc/23639/task/23660/stat : 23660 (plingeling) R 23634 23634 23608 0 -1 4202560 68946 1240 0 0 500375 19 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4334538 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=23639/tid=23661] ppid=23634 vsize=2319764 CPUtime=5003.96 cores=4-31
/proc/23639/task/23661/stat : 23661 (plingeling) R 23634 23634 23608 0 -1 4202560 83699 1240 0 0 500331 64 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330976 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=23639/tid=23662] ppid=23634 vsize=2319764 CPUtime=5003.96 cores=4-31
/proc/23639/task/23662/stat : 23662 (plingeling) R 23634 23634 23608 0 -1 4202560 55802 1240 0 0 500368 27 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331964 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=23639/tid=23663] ppid=23634 vsize=2319764 CPUtime=5003.89 cores=4-31
/proc/23639/task/23663/stat : 23663 (plingeling) R 23634 23634 23608 0 -1 4202560 80704 1240 0 0 500350 38 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331960 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=23639/tid=23664] ppid=23634 vsize=2319764 CPUtime=5003.7 cores=4-31
/proc/23639/task/23664/stat : 23664 (plingeling) R 23634 23634 23608 0 -1 4202560 35612 1240 0 0 500354 15 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4332293 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=23639/tid=23665] ppid=23634 vsize=2319764 CPUtime=5003.95 cores=4-31
/proc/23639/task/23665/stat : 23665 (plingeling) R 23634 23634 23608 0 -1 4202560 32550 1240 0 0 500387 7 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=23639/tid=23666] ppid=23634 vsize=2319764 CPUtime=5003.91 cores=4-31
/proc/23639/task/23666/stat : 23666 (plingeling) R 23634 23634 23608 0 -1 4202560 28108 1240 0 0 500379 11 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=23639/tid=23667] ppid=23634 vsize=2319764 CPUtime=5003.9 cores=4-31
/proc/23639/task/23667/stat : 23667 (plingeling) R 23634 23634 23608 0 -1 4202560 47927 1240 0 0 500377 12 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4283908 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=23639/tid=23668] ppid=23634 vsize=2319764 CPUtime=5003.93 cores=4-31
/proc/23639/task/23668/stat : 23668 (plingeling) R 23634 23634 23608 0 -1 4202560 57513 1240 0 0 500375 17 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 27 0 0 0
[pid=23639/tid=23669] ppid=23634 vsize=2319764 CPUtime=5003.99 cores=4-31
/proc/23639/task/23669/stat : 23669 (plingeling) R 23634 23634 23608 0 -1 4202560 80040 1240 0 0 500320 78 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=23639/tid=23670] ppid=23634 vsize=2319764 CPUtime=5004 cores=4-31
/proc/23639/task/23670/stat : 23670 (plingeling) R 23634 23634 23608 0 -1 4202560 58209 1240 0 0 500364 35 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4210557 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=23639/tid=23671] ppid=23634 vsize=2319764 CPUtime=5004.03 cores=4-31
/proc/23639/task/23671/stat : 23671 (plingeling) R 23634 23634 23608 0 -1 4202560 67405 1240 0 0 500382 20 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4279448 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=23639/tid=23672] ppid=23634 vsize=2319764 CPUtime=5004.05 cores=4-31
/proc/23639/task/23672/stat : 23672 (plingeling) R 23634 23634 23608 0 -1 4202560 68428 1240 0 0 500375 29 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=23639/tid=23673] ppid=23634 vsize=2319764 CPUtime=5003.99 cores=4-31
/proc/23639/task/23673/stat : 23673 (plingeling) R 23634 23634 23608 0 -1 4202560 39966 1240 0 0 500390 8 0 1 25 0 29 0 602367092 2375438336 489531 268435456000 4194304 4356188 140733321386368 18446744073709551615 4279753 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 160004
Current children cumulated vsize (KiB) 4964432

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 23634 and gives
#  childrusage.ru_utime.tv_sec=5008
#  childrusage.ru_utime.tv_usec=449599
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=29995
# CPU time returned by wait4() is 5008.48
# while last known CPU time is 160004
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5009.45
CPU time (s): 160004
CPU user time (s): 159988
CPU system time (s): 16.12
CPU usage (%): 3194.04
Max. virtual memory (cumulated for all children) (KiB): 10606468

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5008.45
system time used= 0.029995
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 10249
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 423
involuntary context switches= 8678

runsolver used 29.3655 second user time and 111.981 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-09 04:40:02
IDJOB=3347237
IDBENCH=70797
IDSOLVER=1808
FILE ID=node200/3347237-1304908802
RUNJOBID= node200-1304908802-23622
PBS_JOBID= 13281997
Free space on /tmp= 62048 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT09/APPLICATIONS/bitverif/maxxororand/maxxororand064.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347237-1304908802/watcher-3347237-1304908802 -o /tmp/evaluation-result-3347237-1304908802/solver-3347237-1304908802 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347237-1304908802.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 49dbd2fb16b3d5ce554d30c48729a56d
RANDOM SEED=117599408

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      256753532 kB
Buffers:        362836 kB
Cached:        6800552 kB
SwapCached:          0 kB
Active:        6340312 kB
Inactive:       891952 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      256753532 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           14608 kB
Writeback:           0 kB
AnonPages:       70080 kB
Mapped:          13484 kB
Slab:           191792 kB
PageTables:       3980 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   188480 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62036 MiB
End job on node200 at 2011-05-09 06:03:34