Trace number 3347245

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 160002 5014.14

General information on the benchmark

NameAPPLICATIONS/
bitverif/maxxor/maxxor128.cnf
MD5SUM065c126f0d55807310a398728b44423f
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark148113
Satisfiable
(Un)Satisfiability was proved
Number of variables200440
Number of clauses599015
Sum of the clauses size1397699
Maximum clause length3
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2399342
Number of clauses of size 3199671
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 32 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3347245-1304722848/watcher-3347245-1304722848 -o /tmp/evaluation-result-3347245-1304722848/solver-3347245-1304722848 -C 160000 -W 5100 -M 266338304 ppfolio HOME/instance-3347245-1304722848.cnf 

running on 32 cores: 0,4,8,12,16,20,24,28,1,5,9,13,17,21,25,29,2,6,10,14,18,22,26,30,3,7,11,15,19,23,27,31

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 160000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 160030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 272730423296 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 272730474496 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 1/413 2390
/proc/meminfo: memFree=256783896/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 20 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0

[startup+0.105352 s]
/proc/loadavg: 0.00 0.00 0.00 1/413 2390
/proc/meminfo: memFree=256783896/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 20 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.200363 s]
/proc/loadavg: 0.00 0.00 0.00 1/413 2390
/proc/meminfo: memFree=256783896/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 20 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300378 s]
/proc/loadavg: 0.00 0.00 0.00 1/413 2390
/proc/meminfo: memFree=256783896/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 20 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700354 s]
/proc/loadavg: 0.00 0.00 0.00 1/413 2390
/proc/meminfo: memFree=256783896/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 16 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50143 s]
/proc/loadavg: 0.40 0.08 0.03 6/419 2401
/proc/meminfo: memFree=256387224/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 16 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
[pid=2391] ppid=2390 vsize=89324 CPUtime=1.48 cores=0
/proc/2391/stat : 2391 (cryptominisat) R 2390 2390 2364 0 -1 4202496 17787 0 0 0 144 4 0 0 25 0 1 0 583771276 91467776 17251 268435456000 4194304 4587354 140734940142480 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/2391/statm: 22331 17251 304 96 0 18893 0
[pid=2392] ppid=2390 vsize=126096 CPUtime=1.48 cores=1
/proc/2392/stat : 2392 (clasp) R 2390 2390 2364 0 -1 4202496 33141 0 0 0 140 8 0 0 20 0 1 0 583771276 129122304 28037 268435456000 4194304 4785740 140737035064496 18446744073709551615 258650616632 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/2392/statm: 31524 28037 317 145 0 28569 0
[pid=2393] ppid=2390 vsize=487800 CPUtime=1.48 cores=2
/proc/2393/stat : 2393 (TNM) R 2390 2390 2364 0 -1 4202496 13128 0 0 0 144 4 0 0 20 0 1 0 583771276 499507200 13025 268435456000 134512640 134534564 4294523376 18446744073709551615 4155047 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2393/statm: 121950 13025 92 6 0 120615 0
[pid=2394] ppid=2390 vsize=115912 CPUtime=1.48 cores=3
/proc/2394/stat : 2394 (march_hi) R 2390 2390 2364 0 -1 4194304 42226 0 0 0 138 10 0 0 20 0 1 0 583771276 118693888 21684 268435456000 4194304 4331988 140734698601152 18446744073709551615 258650604480 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2394/statm: 28978 21684 126 34 0 27420 0
[pid=2395] ppid=2390 vsize=125168 CPUtime=1.49 cores=4-31
/proc/2395/stat : 2395 (plingeling) R 2390 2390 2364 0 -1 4202496 30067 1239 0 0 141 7 0 1 21 0 1 0 583771276 128172032 29951 268435456000 4194304 4356188 140735591860832 18446744073709551615 258650955760 0 0 4096 17442 0 0 0 17 28 0 0 0
/proc/2395/statm: 31292 29951 129 40 0 29836 0
Current children cumulated CPU time (s) 7.41
Current children cumulated vsize (KiB) 962624

[startup+3.10145 s]
/proc/loadavg: 0.40 0.08 0.03 6/419 2401
/proc/meminfo: memFree=256147056/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
[pid=2391] ppid=2390 vsize=123520 CPUtime=3.08 cores=0
/proc/2391/stat : 2391 (cryptominisat) R 2390 2390 2364 0 -1 4202496 26357 0 0 0 302 6 0 0 25 0 1 0 583771276 126484480 25821 268435456000 4194304 4587354 140734940142480 18446744073709551615 4421117 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/2391/statm: 30880 25821 310 96 0 27442 0
[pid=2392] ppid=2390 vsize=141520 CPUtime=3.08 cores=1
/proc/2392/stat : 2392 (clasp) R 2390 2390 2364 0 -1 4202496 37017 0 0 0 299 9 0 0 25 0 1 0 583771276 144916480 31913 268435456000 4194304 4785740 140737035064496 18446744073709551615 4500020 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/2392/statm: 35380 31913 317 145 0 32425 0
[pid=2393] ppid=2390 vsize=493608 CPUtime=3.08 cores=2
/proc/2393/stat : 2393 (TNM) R 2390 2390 2364 0 -1 4202496 20739 0 0 0 302 6 0 0 25 0 1 0 583771276 505454592 20636 268435456000 134512640 134534564 4294523376 18446744073709551615 134529374 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2393/statm: 123402 20636 95 6 0 122067 0
[pid=2394] ppid=2390 vsize=98668 CPUtime=3.08 cores=3
/proc/2394/stat : 2394 (march_hi) R 2390 2390 2364 0 -1 4202496 101952 0 0 0 285 23 0 0 25 0 1 0 583771276 101036032 17005 268435456000 4194304 4331988 140734698601152 18446744073709551615 4257032 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2394/statm: 24667 17005 140 34 0 23109 0
[pid=2395] ppid=2390 vsize=343216 CPUtime=3.09 cores=4-31
/proc/2395/stat : 2395 (plingeling) R 2390 2390 2364 0 -1 4202496 84626 1239 0 0 292 16 0 1 25 0 1 0 583771276 351453184 84463 268435456000 4194304 4356188 140735591860832 18446744073709551615 4278468 0 0 4096 17442 0 0 0 17 28 0 0 0
/proc/2395/statm: 85804 84463 129 40 0 84348 0
Current children cumulated CPU time (s) 15.41
Current children cumulated vsize (KiB) 1218856

[startup+6.30055 s]
/proc/loadavg: 0.77 0.16 0.05 6/419 2401
/proc/meminfo: memFree=255840320/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
[pid=2391] ppid=2390 vsize=123520 CPUtime=6.28 cores=0
/proc/2391/stat : 2391 (cryptominisat) R 2390 2390 2364 0 -1 4202496 26357 0 0 0 622 6 0 0 25 0 1 0 583771276 126484480 25821 268435456000 4194304 4587354 140734940142480 18446744073709551615 4367567 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/2391/statm: 30880 25821 310 96 0 27442 0
[pid=2392] ppid=2390 vsize=134536 CPUtime=6.29 cores=1
/proc/2392/stat : 2392 (clasp) R 2390 2390 2364 0 -1 4202496 38652 0 0 0 619 10 0 0 25 0 1 0 583771276 137764864 30420 268435456000 4194304 4785740 140737035064496 18446744073709551615 4319144 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/2392/statm: 33634 30420 338 145 0 30679 0
[pid=2393] ppid=2390 vsize=493608 CPUtime=6.28 cores=2
/proc/2393/stat : 2393 (TNM) R 2390 2390 2364 0 -1 4202496 20739 0 0 0 622 6 0 0 25 0 1 0 583771276 505454592 20636 268435456000 134512640 134534564 4294523376 18446744073709551615 134529944 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2393/statm: 123402 20636 95 6 0 122067 0
[pid=2394] ppid=2390 vsize=384960 CPUtime=6.29 cores=3
/proc/2394/stat : 2394 (march_hi) R 2390 2390 2364 0 -1 4202496 175560 0 0 0 591 38 0 0 25 0 1 0 583771276 394199040 49706 268435456000 4194304 4331988 140734698601152 18446744073709551615 4226440 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2394/statm: 96240 49706 172 34 0 94682 0
[pid=2395] ppid=2390 vsize=677348 CPUtime=6.29 cores=4-31
/proc/2395/stat : 2395 (plingeling) R 2390 2390 2364 0 -1 4202496 168159 1239 0 0 595 33 0 1 25 0 1 0 583771276 693604352 167996 268435456000 4194304 4356188 140735591860832 18446744073709551615 4206023 0 0 4096 17442 0 0 0 17 28 0 0 0
/proc/2395/statm: 169337 167996 129 40 0 167881 0
Current children cumulated CPU time (s) 31.43
Current children cumulated vsize (KiB) 1832296

[startup+12.7027 s]
/proc/loadavg: 3.27 0.69 0.22 33/447 2429
/proc/meminfo: memFree=254929160/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
[pid=2391] ppid=2390 vsize=261844 CPUtime=12.68 cores=0
/proc/2391/stat : 2391 (cryptominisat) R 2390 2390 2364 0 -1 4202496 65593 0 0 0 1249 19 0 0 25 0 1 0 583771276 268128256 53685 268435456000 4194304 4587354 140734940142480 18446744073709551615 4237686 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/2391/statm: 65461 53685 326 96 0 62023 0
[pid=2392] ppid=2390 vsize=134536 CPUtime=12.69 cores=1
/proc/2392/stat : 2392 (clasp) R 2390 2390 2364 0 -1 4202496 38653 0 0 0 1259 10 0 0 25 0 1 0 583771276 137764864 30421 268435456000 4194304 4785740 140737035064496 18446744073709551615 4552628 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/2392/statm: 33634 30421 338 145 0 30679 0
[pid=2393] ppid=2390 vsize=493608 CPUtime=12.68 cores=2
/proc/2393/stat : 2393 (TNM) R 2390 2390 2364 0 -1 4202496 20739 0 0 0 1262 6 0 0 25 0 1 0 583771276 505454592 20636 268435456000 134512640 134534564 4294523376 18446744073709551615 134525938 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2393/statm: 123402 20636 95 6 0 122067 0
[pid=2394] ppid=2390 vsize=406768 CPUtime=12.69 cores=3
/proc/2394/stat : 2394 (march_hi) R 2390 2390 2364 0 -1 4202496 215546 0 0 0 1223 46 0 0 25 0 1 0 583771276 416530432 89692 268435456000 4194304 4331988 140734698601152 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2394/statm: 101692 89692 172 34 0 100134 0
[pid=2395] ppid=2390 vsize=1606312 CPUtime=89.96 cores=4-31
/proc/2395/stat : 2395 (plingeling) S 2390 2390 2364 0 -1 4202496 477995 1239 0 0 8854 141 0 1 18 0 29 0 583771276 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 28 0 0 0
/proc/2395/statm: 401578 287174 149 40 0 400122 0
[pid=2395/tid=2402] ppid=2390 vsize=1606312 CPUtime=2.86 cores=4-31
/proc/2395/task/2402/stat : 2402 (plingeling) R 2390 2390 2364 0 -1 4202560 7521 1239 0 0 282 3 0 1 23 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4267675 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=2395/tid=2403] ppid=2390 vsize=1606312 CPUtime=2.9 cores=4-31
/proc/2395/task/2403/stat : 2403 (plingeling) R 2390 2390 2364 0 -1 4202560 5301 1239 0 0 288 1 0 1 25 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=2395/tid=2404] ppid=2390 vsize=1606312 CPUtime=2.9 cores=4-31
/proc/2395/task/2404/stat : 2404 (plingeling) R 2390 2390 2364 0 -1 4202560 10740 1239 0 0 286 3 0 1 23 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4314760 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=2395/tid=2405] ppid=2390 vsize=1606312 CPUtime=2.92 cores=4-31
/proc/2395/task/2405/stat : 2405 (plingeling) R 2390 2390 2364 0 -1 4202560 15272 1239 0 0 286 5 0 1 25 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4299429 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=2395/tid=2406] ppid=2390 vsize=1606312 CPUtime=2.93 cores=4-31
/proc/2395/task/2406/stat : 2406 (plingeling) R 2390 2390 2364 0 -1 4202560 4359 1239 0 0 290 2 0 1 23 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2395/tid=2407] ppid=2390 vsize=1606312 CPUtime=2.88 cores=4-31
/proc/2395/task/2407/stat : 2407 (plingeling) R 2390 2390 2364 0 -1 4202560 9781 1239 0 0 285 2 0 1 23 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2395/tid=2408] ppid=2390 vsize=1606312 CPUtime=2.87 cores=4-31
/proc/2395/task/2408/stat : 2408 (plingeling) R 2390 2390 2364 0 -1 4202560 10347 1239 0 0 283 3 0 1 18 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4267675 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2395/tid=2409] ppid=2390 vsize=1606312 CPUtime=2.89 cores=4-31
/proc/2395/task/2409/stat : 2409 (plingeling) R 2390 2390 2364 0 -1 4202560 14175 1239 0 0 284 4 0 1 23 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=2395/tid=2410] ppid=2390 vsize=1606312 CPUtime=2.87 cores=4-31
/proc/2395/task/2410/stat : 2410 (plingeling) R 2390 2390 2364 0 -1 4202560 3953 1239 0 0 285 1 0 1 25 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4267726 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=2395/tid=2411] ppid=2390 vsize=1606312 CPUtime=2.91 cores=4-31
/proc/2395/task/2411/stat : 2411 (plingeling) R 2390 2390 2364 0 -1 4202560 9375 1239 0 0 286 4 0 1 24 0 29 0 583772250 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=2395/tid=2412] ppid=2390 vsize=1606312 CPUtime=2.85 cores=4-31
/proc/2395/task/2412/stat : 2412 (plingeling) R 2390 2390 2364 0 -1 4202560 10461 1239 0 0 281 3 0 1 23 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244723 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=2395/tid=2413] ppid=2390 vsize=1606312 CPUtime=2.92 cores=4-31
/proc/2395/task/2413/stat : 2413 (plingeling) R 2390 2390 2364 0 -1 4202560 4972 1239 0 0 290 1 0 1 25 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4259314 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=2395/tid=2414] ppid=2390 vsize=1606312 CPUtime=2.86 cores=4-31
/proc/2395/task/2414/stat : 2414 (plingeling) R 2390 2390 2364 0 -1 4202560 12971 1239 0 0 280 5 0 1 19 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244483 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=2395/tid=2415] ppid=2390 vsize=1606312 CPUtime=2.81 cores=4-31
/proc/2395/task/2415/stat : 2415 (plingeling) R 2390 2390 2364 0 -1 4202560 7759 1239 0 0 278 2 0 1 25 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4260245 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=2395/tid=2416] ppid=2390 vsize=1606312 CPUtime=2.9 cores=4-31
/proc/2395/task/2416/stat : 2416 (plingeling) R 2390 2390 2364 0 -1 4202560 4900 1239 0 0 287 2 0 1 25 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4245442 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=2395/tid=2417] ppid=2390 vsize=1606312 CPUtime=2.89 cores=4-31
/proc/2395/task/2417/stat : 2417 (plingeling) R 2390 2390 2364 0 -1 4202560 6536 1239 0 0 286 2 0 1 25 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4267675 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=2395/tid=2418] ppid=2390 vsize=1606312 CPUtime=2.89 cores=4-31
/proc/2395/task/2418/stat : 2418 (plingeling) R 2390 2390 2364 0 -1 4202560 13994 1239 0 0 283 5 0 1 19 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4259263 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2395/tid=2419] ppid=2390 vsize=1606312 CPUtime=2.85 cores=4-31
/proc/2395/task/2419/stat : 2419 (plingeling) R 2390 2390 2364 0 -1 4202560 7060 1239 0 0 282 2 0 1 22 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4245552 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=2395/tid=2420] ppid=2390 vsize=1606312 CPUtime=2.91 cores=4-31
/proc/2395/task/2420/stat : 2420 (plingeling) R 2390 2390 2364 0 -1 4202560 2939 1239 0 0 289 1 0 1 18 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4261931 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=2395/tid=2421] ppid=2390 vsize=1606312 CPUtime=2.87 cores=4-31
/proc/2395/task/2421/stat : 2421 (plingeling) R 2390 2390 2364 0 -1 4202560 10390 1239 0 0 283 3 0 1 19 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4244400 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=2395/tid=2422] ppid=2390 vsize=1606312 CPUtime=2.9 cores=4-31
/proc/2395/task/2422/stat : 2422 (plingeling) R 2390 2390 2364 0 -1 4202560 5512 1239 0 0 287 2 0 1 23 0 29 0 583772251 1644863488 287174 268435456000 4194304 4356188 140735591860832 18446744073709551615 4267675 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=2395/tid=2423] ppid=2390 vsize=1606312 CPUtime=2.58 cores=4-31

################
# More data... #
################

[pid=2393] ppid=2390 vsize=493612 CPUtime=4961.34 cores=2
/proc/2393/stat : 2393 (TNM) R 2390 2390 2364 0 -1 4202496 20748 0 0 0 496128 6 0 0 25 0 1 0 583771276 505458688 20645 268435456000 134512640 134534564 4294523376 18446744073709551615 134530124 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2393/statm: 123403 20645 103 6 0 122068 0
[pid=2394] ppid=2390 vsize=424812 CPUtime=4961.36 cores=3
/proc/2394/stat : 2394 (march_hi) R 2390 2390 2364 0 -1 4202496 220118 0 0 0 496087 49 0 0 25 0 1 0 583771276 435007488 94264 268435456000 4194304 4331988 140734698601152 18446744073709551615 4229471 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2394/statm: 106203 94264 172 34 0 104645 0
[pid=2395] ppid=2390 vsize=3058916 CPUtime=138500 cores=4-31
/proc/2395/stat : 2395 (plingeling) S 2390 2390 2364 0 -1 4202496 1403346 1239 0 0 13849391 653 0 1 18 0 29 0 583771276 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 28 0 0 0
/proc/2395/statm: 764729 670439 159 40 0 763273 0
[pid=2395/tid=2402] ppid=2390 vsize=3058916 CPUtime=4951.46 cores=4-31
/proc/2395/task/2402/stat : 2402 (plingeling) R 2390 2390 2364 0 -1 4202560 34979 1239 0 0 495128 17 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330973 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=2395/tid=2403] ppid=2390 vsize=3058916 CPUtime=4866.01 cores=4-31
/proc/2395/task/2403/stat : 2403 (plingeling) R 2390 2390 2364 0 -1 4202560 48852 1239 0 0 486580 20 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331002 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=2395/tid=2404] ppid=2390 vsize=3058916 CPUtime=4951.52 cores=4-31
/proc/2395/task/2404/stat : 2404 (plingeling) R 2390 2390 2364 0 -1 4202560 40074 1239 0 0 495129 22 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=2395/tid=2405] ppid=2390 vsize=3058916 CPUtime=4947.71 cores=4-31
/proc/2395/task/2405/stat : 2405 (plingeling) R 2390 2390 2364 0 -1 4202560 46453 1239 0 0 494704 66 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=2395/tid=2406] ppid=2390 vsize=3058916 CPUtime=4951.05 cores=4-31
/proc/2395/task/2406/stat : 2406 (plingeling) R 2390 2390 2364 0 -1 4202560 48605 1239 0 0 495090 14 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2395/tid=2407] ppid=2390 vsize=3058916 CPUtime=4950.53 cores=4-31
/proc/2395/task/2407/stat : 2407 (plingeling) R 2390 2390 2364 0 -1 4202560 43116 1239 0 0 495031 21 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4332309 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2395/tid=2408] ppid=2390 vsize=3058916 CPUtime=4951.12 cores=4-31
/proc/2395/task/2408/stat : 2408 (plingeling) R 2390 2390 2364 0 -1 4202560 18001 1239 0 0 495097 14 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2395/tid=2409] ppid=2390 vsize=3058916 CPUtime=4951.04 cores=4-31
/proc/2395/task/2409/stat : 2409 (plingeling) R 2390 2390 2364 0 -1 4202560 59203 1239 0 0 495059 44 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=2395/tid=2410] ppid=2390 vsize=3058916 CPUtime=4951.19 cores=4-31
/proc/2395/task/2410/stat : 2410 (plingeling) R 2390 2390 2364 0 -1 4202560 39074 1239 0 0 495105 13 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331939 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=2395/tid=2411] ppid=2390 vsize=3058916 CPUtime=4899.06 cores=4-31
/proc/2395/task/2411/stat : 2411 (plingeling) R 2390 2390 2364 0 -1 4202560 51727 1239 0 0 489885 20 0 1 25 0 29 0 583772250 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=2395/tid=2412] ppid=2390 vsize=3058916 CPUtime=4951.22 cores=4-31
/proc/2395/task/2412/stat : 2412 (plingeling) R 2390 2390 2364 0 -1 4202560 37761 1239 0 0 495103 18 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=2395/tid=2413] ppid=2390 vsize=3058916 CPUtime=4951.6 cores=4-31
/proc/2395/task/2413/stat : 2413 (plingeling) R 2390 2390 2364 0 -1 4202560 26244 1239 0 0 495143 16 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=2395/tid=2414] ppid=2390 vsize=3058916 CPUtime=4951.21 cores=4-31
/proc/2395/task/2414/stat : 2414 (plingeling) R 2390 2390 2364 0 -1 4202560 47502 1239 0 0 495099 21 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330817 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=2395/tid=2415] ppid=2390 vsize=3058916 CPUtime=4951.25 cores=4-31
/proc/2395/task/2415/stat : 2415 (plingeling) R 2390 2390 2364 0 -1 4202560 45383 1239 0 0 495103 21 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=2395/tid=2416] ppid=2390 vsize=3058916 CPUtime=4951.19 cores=4-31
/proc/2395/task/2416/stat : 2416 (plingeling) R 2390 2390 2364 0 -1 4202560 52309 1239 0 0 495096 22 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330976 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=2395/tid=2417] ppid=2390 vsize=3058916 CPUtime=4951.32 cores=4-31
/proc/2395/task/2417/stat : 2417 (plingeling) R 2390 2390 2364 0 -1 4202560 48918 1239 0 0 495097 34 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=2395/tid=2418] ppid=2390 vsize=3058916 CPUtime=4951.35 cores=4-31
/proc/2395/task/2418/stat : 2418 (plingeling) R 2390 2390 2364 0 -1 4202560 48254 1239 0 0 495111 23 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4279478 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2395/tid=2419] ppid=2390 vsize=3058916 CPUtime=4951.36 cores=4-31
/proc/2395/task/2419/stat : 2419 (plingeling) R 2390 2390 2364 0 -1 4202560 25288 1239 0 0 495120 15 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=2395/tid=2420] ppid=2390 vsize=3058916 CPUtime=4951.32 cores=4-31
/proc/2395/task/2420/stat : 2420 (plingeling) R 2390 2390 2364 0 -1 4202560 25997 1239 0 0 495118 13 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4241608 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=2395/tid=2421] ppid=2390 vsize=3058916 CPUtime=4951.54 cores=4-31
/proc/2395/task/2421/stat : 2421 (plingeling) R 2390 2390 2364 0 -1 4202560 22890 1239 0 0 495146 7 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=2395/tid=2422] ppid=2390 vsize=3058916 CPUtime=4951.16 cores=4-31
/proc/2395/task/2422/stat : 2422 (plingeling) R 2390 2390 2364 0 -1 4202560 43206 1239 0 0 495099 16 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=2395/tid=2423] ppid=2390 vsize=3058916 CPUtime=4951.06 cores=4-31
/proc/2395/task/2423/stat : 2423 (plingeling) R 2390 2390 2364 0 -1 4202560 64832 1239 0 0 495090 15 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=2395/tid=2424] ppid=2390 vsize=3058916 CPUtime=4951.06 cores=4-31
/proc/2395/task/2424/stat : 2424 (plingeling) R 2390 2390 2364 0 -1 4202560 41118 1239 0 0 495083 22 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=2395/tid=2425] ppid=2390 vsize=3058916 CPUtime=4950.78 cores=4-31
/proc/2395/task/2425/stat : 2425 (plingeling) R 2390 2390 2364 0 -1 4202560 53593 1239 0 0 495044 33 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330973 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=2395/tid=2426] ppid=2390 vsize=3058916 CPUtime=4948.33 cores=4-31
/proc/2395/task/2426/stat : 2426 (plingeling) R 2390 2390 2364 0 -1 4202560 36503 1239 0 0 494809 23 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=2395/tid=2427] ppid=2390 vsize=3058916 CPUtime=4951.47 cores=4-31
/proc/2395/task/2427/stat : 2427 (plingeling) R 2390 2390 2364 0 -1 4202560 34510 1239 0 0 495127 19 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4237473 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=2395/tid=2428] ppid=2390 vsize=3058916 CPUtime=4951.43 cores=4-31
/proc/2395/task/2428/stat : 2428 (plingeling) R 2390 2390 2364 0 -1 4202560 47801 1239 0 0 495127 15 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=2395/tid=2429] ppid=2390 vsize=3058916 CPUtime=4951.38 cores=4-31
/proc/2395/task/2429/stat : 2429 (plingeling) R 2390 2390 2364 0 -1 4202560 45613 1239 0 0 495127 10 0 1 25 0 29 0 583772251 3132329984 670439 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 158345
Current children cumulated vsize (KiB) 6088148



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5014.1 s]
/proc/loadavg: 32.07 32.08 31.91 33/447 2620
/proc/meminfo: memFree=252661616/264257624 swapFree=67111528/67111528
[pid=2390] ppid=2388 vsize=18324 CPUtime=0 cores=0-31
/proc/2390/stat : 2390 (ppfolio) S 2388 2390 2364 0 -1 4202496 514 0 0 0 0 0 0 0 15 0 1 0 583771275 18763776 386 268435456000 4194304 4287625 140733580793584 18446744073709551615 258650975999 0 0 4096 65536 9201901190399393791 0 0 17 24 0 0 0
/proc/2390/statm: 4581 386 337 23 0 91 0
[pid=2391] ppid=2390 vsize=1473752 CPUtime=5012.81 cores=0
/proc/2391/stat : 2391 (cryptominisat) R 2390 2390 2364 0 -1 4202496 897303 0 0 0 500858 423 0 0 25 0 1 0 583771276 1509122048 85022 268435456000 4194304 4587354 140734940142480 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/2391/statm: 368438 85022 336 96 0 365000 0
[pid=2392] ppid=2390 vsize=650424 CPUtime=5013.12 cores=1
/proc/2392/stat : 2392 (clasp) R 2390 2390 2364 0 -1 4202496 165843 0 0 0 501188 124 0 0 25 0 1 0 583771276 666034176 157611 268435456000 4194304 4785740 140737035064496 18446744073709551615 4312643 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/2392/statm: 162606 157611 338 145 0 159651 0
[pid=2393] ppid=2390 vsize=493612 CPUtime=5013.14 cores=2
/proc/2393/stat : 2393 (TNM) R 2390 2390 2364 0 -1 4202496 20748 0 0 0 501308 6 0 0 25 0 1 0 583771276 505458688 20645 268435456000 134512640 134534564 4294523376 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2393/statm: 123403 20645 103 6 0 122068 0
[pid=2394] ppid=2390 vsize=424812 CPUtime=5013.16 cores=3
/proc/2394/stat : 2394 (march_hi) R 2390 2390 2364 0 -1 4202496 220118 0 0 0 501267 49 0 0 25 0 1 0 583771276 435007488 94264 268435456000 4194304 4331988 140734698601152 18446744073709551615 4229533 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/2394/statm: 106203 94264 172 34 0 104645 0
[pid=2395] ppid=2390 vsize=3068240 CPUtime=139949 cores=4-31
/proc/2395/stat : 2395 (plingeling) S 2390 2390 2364 0 -1 4202496 1405689 1239 0 0 13994278 654 0 1 18 0 29 0 583771276 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 258662759221 0 0 4096 17442 18446744073709551615 0 0 17 28 0 0 0
/proc/2395/statm: 767060 672781 159 40 0 765604 0
[pid=2395/tid=2402] ppid=2390 vsize=3068240 CPUtime=5003.27 cores=4-31
/proc/2395/task/2402/stat : 2402 (plingeling) R 2390 2390 2364 0 -1 4202560 34979 1239 0 0 500309 17 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 18 0 0 0
[pid=2395/tid=2403] ppid=2390 vsize=3068240 CPUtime=4917.82 cores=4-31
/proc/2395/task/2403/stat : 2403 (plingeling) R 2390 2390 2364 0 -1 4202560 48852 1239 0 0 491761 20 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4281627 0 0 4096 17442 0 0 0 -1 17 0 0 0
[pid=2395/tid=2404] ppid=2390 vsize=3068240 CPUtime=5003.33 cores=4-31
/proc/2395/task/2404/stat : 2404 (plingeling) R 2390 2390 2364 0 -1 4202560 40074 1239 0 0 500310 22 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 28 0 0 0
[pid=2395/tid=2405] ppid=2390 vsize=3068240 CPUtime=4999.52 cores=4-31
/proc/2395/task/2405/stat : 2405 (plingeling) R 2390 2390 2364 0 -1 4202560 46453 1239 0 0 499884 67 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 24 0 0 0
[pid=2395/tid=2406] ppid=2390 vsize=3068240 CPUtime=5002.87 cores=4-31
/proc/2395/task/2406/stat : 2406 (plingeling) R 2390 2390 2364 0 -1 4202560 48605 1239 0 0 500272 14 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4332165 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=2395/tid=2407] ppid=2390 vsize=3068240 CPUtime=5002.34 cores=4-31
/proc/2395/task/2407/stat : 2407 (plingeling) R 2390 2390 2364 0 -1 4202560 43116 1239 0 0 500212 21 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4284212 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=2395/tid=2408] ppid=2390 vsize=3068240 CPUtime=5002.93 cores=4-31
/proc/2395/task/2408/stat : 2408 (plingeling) R 2390 2390 2364 0 -1 4202560 18001 1239 0 0 500278 14 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=2395/tid=2409] ppid=2390 vsize=3068240 CPUtime=5002.85 cores=4-31
/proc/2395/task/2409/stat : 2409 (plingeling) R 2390 2390 2364 0 -1 4202560 59203 1239 0 0 500240 44 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 9 0 0 0
[pid=2395/tid=2410] ppid=2390 vsize=3068240 CPUtime=5003 cores=4-31
/proc/2395/task/2410/stat : 2410 (plingeling) R 2390 2390 2364 0 -1 4202560 39074 1239 0 0 500286 13 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331050 0 0 4096 17442 0 0 0 -1 10 0 0 0
[pid=2395/tid=2411] ppid=2390 vsize=3068240 CPUtime=4949.39 cores=4-31
/proc/2395/task/2411/stat : 2411 (plingeling) R 2390 2390 2364 0 -1 4202560 51727 1239 0 0 494918 20 0 1 25 0 29 0 583772250 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 31 0 0 0
[pid=2395/tid=2412] ppid=2390 vsize=3068240 CPUtime=5003.03 cores=4-31
/proc/2395/task/2412/stat : 2412 (plingeling) R 2390 2390 2364 0 -1 4202560 37761 1239 0 0 500284 18 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 13 0 0 0
[pid=2395/tid=2413] ppid=2390 vsize=3068240 CPUtime=5003.41 cores=4-31
/proc/2395/task/2413/stat : 2413 (plingeling) R 2390 2390 2364 0 -1 4202560 26244 1239 0 0 500324 16 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 11 0 0 0
[pid=2395/tid=2414] ppid=2390 vsize=3068240 CPUtime=5003.02 cores=4-31
/proc/2395/task/2414/stat : 2414 (plingeling) R 2390 2390 2364 0 -1 4202560 47502 1239 0 0 500280 21 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331019 0 0 4096 17442 0 0 0 -1 20 0 0 0
[pid=2395/tid=2415] ppid=2390 vsize=3068240 CPUtime=5003.06 cores=4-31
/proc/2395/task/2415/stat : 2415 (plingeling) R 2390 2390 2364 0 -1 4202560 45383 1239 0 0 500284 21 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 26 0 0 0
[pid=2395/tid=2416] ppid=2390 vsize=3068240 CPUtime=5003 cores=4-31
/proc/2395/task/2416/stat : 2416 (plingeling) R 2390 2390 2364 0 -1 4202560 52340 1239 0 0 500277 22 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331383 0 0 4096 17442 0 0 0 -1 21 0 0 0
[pid=2395/tid=2417] ppid=2390 vsize=3068240 CPUtime=5003.13 cores=4-31
/proc/2395/task/2417/stat : 2417 (plingeling) R 2390 2390 2364 0 -1 4202560 48918 1239 0 0 500278 34 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4283920 0 0 4096 17442 0 0 0 -1 22 0 0 0
[pid=2395/tid=2418] ppid=2390 vsize=3068240 CPUtime=5003.16 cores=4-31
/proc/2395/task/2418/stat : 2418 (plingeling) R 2390 2390 2364 0 -1 4202560 48254 1239 0 0 500292 23 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=2395/tid=2419] ppid=2390 vsize=3068240 CPUtime=5003.16 cores=4-31
/proc/2395/task/2419/stat : 2419 (plingeling) R 2390 2390 2364 0 -1 4202560 27589 1239 0 0 500300 15 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 19 0 0 0
[pid=2395/tid=2420] ppid=2390 vsize=3068240 CPUtime=5003.13 cores=4-31
/proc/2395/task/2420/stat : 2420 (plingeling) R 2390 2390 2364 0 -1 4202560 25997 1239 0 0 500299 13 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 29 0 0 0
[pid=2395/tid=2421] ppid=2390 vsize=3068240 CPUtime=5003.35 cores=4-31
/proc/2395/task/2421/stat : 2421 (plingeling) R 2390 2390 2364 0 -1 4202560 22890 1239 0 0 500327 7 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 25 0 0 0
[pid=2395/tid=2422] ppid=2390 vsize=3068240 CPUtime=5002.97 cores=4-31
/proc/2395/task/2422/stat : 2422 (plingeling) R 2390 2390 2364 0 -1 4202560 43206 1239 0 0 500280 16 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 14 0 0 0
[pid=2395/tid=2423] ppid=2390 vsize=3068240 CPUtime=5002.87 cores=4-31
/proc/2395/task/2423/stat : 2423 (plingeling) R 2390 2390 2364 0 -1 4202560 64843 1239 0 0 500271 15 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 23 0 0 0
[pid=2395/tid=2424] ppid=2390 vsize=3068240 CPUtime=5002.87 cores=4-31
/proc/2395/task/2424/stat : 2424 (plingeling) R 2390 2390 2364 0 -1 4202560 41118 1239 0 0 500264 22 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 15 0 0 0
[pid=2395/tid=2425] ppid=2390 vsize=3068240 CPUtime=5002.58 cores=4-31
/proc/2395/task/2425/stat : 2425 (plingeling) R 2390 2390 2364 0 -1 4202560 53593 1239 0 0 500224 33 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 8 0 0 0
[pid=2395/tid=2426] ppid=2390 vsize=3068240 CPUtime=5000.14 cores=4-31
/proc/2395/task/2426/stat : 2426 (plingeling) R 2390 2390 2364 0 -1 4202560 36503 1239 0 0 499990 23 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4273866 0 0 4096 17442 0 0 0 -1 30 0 0 0
[pid=2395/tid=2427] ppid=2390 vsize=3068240 CPUtime=5003.28 cores=4-31
/proc/2395/task/2427/stat : 2427 (plingeling) R 2390 2390 2364 0 -1 4202560 34510 1239 0 0 500308 19 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4236939 0 0 4096 17442 0 0 0 -1 12 0 0 0
[pid=2395/tid=2428] ppid=2390 vsize=3068240 CPUtime=5003.24 cores=4-31
/proc/2395/task/2428/stat : 2428 (plingeling) R 2390 2390 2364 0 -1 4202560 47801 1239 0 0 500308 15 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 16 0 0 0
[pid=2395/tid=2429] ppid=2390 vsize=3068240 CPUtime=5003.19 cores=4-31
/proc/2395/task/2429/stat : 2429 (plingeling) R 2390 2390 2364 0 -1 4202560 45613 1239 0 0 500308 10 0 1 25 0 29 0 583772251 3141877760 672781 268435456000 4194304 4356188 140735591860832 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 27 0 0 0
Current children cumulated CPU time (s) 160002
Current children cumulated vsize (KiB) 6129164

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2390 and gives
#  childrusage.ru_utime.tv_sec=5013
#  childrusage.ru_utime.tv_usec=103892
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=82987
# CPU time returned by wait4() is 5013.19
# while last known CPU time is 160002
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5014.14
CPU time (s): 160002
CPU user time (s): 159989
CPU system time (s): 12.57
CPU usage (%): 3191.01
Max. virtual memory (cumulated for all children) (KiB): 7743944

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5013.1
system time used= 0.082987
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 21264
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 417
involuntary context switches= 12549

runsolver used 26.057 second user time and 115.617 second system time

The end

Launcher Data 

Begin job on node200 at 2011-05-07 01:00:49
IDJOB=3347245
IDBENCH=70815
IDSOLVER=1808
FILE ID=node200/3347245-1304722848
RUNJOBID= node200-1304722848-2378
PBS_JOBID= 13281943
Free space on /tmp= 62040 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT09/APPLICATIONS/bitverif/maxxor/maxxor128.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3347245-1304722848/watcher-3347245-1304722848 -o /tmp/evaluation-result-3347245-1304722848/solver-3347245-1304722848 -C 160000 -W 5100 -M 266338304  ppfolio HOME/instance-3347245-1304722848.cnf

TIME LIMIT= 160000 seconds
MEMORY LIMIT= 266338304 MiB
NBCORE= 32

MD5SUM BENCH= 065c126f0d55807310a398728b44423f
RANDOM SEED=888161294

node200.alineos.net Linux 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 64
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 0
cpu cores	: 8
apicid		: 96
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 66
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 1
cpu cores	: 8
apicid		: 98
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 68
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 2
cpu cores	: 8
apicid		: 100
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.17
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 70
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3989.98
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 3
cpu cores	: 8
apicid		: 102
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 48
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 80
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 8
cpu cores	: 8
apicid		: 112
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 50
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 82
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 9
cpu cores	: 8
apicid		: 114
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 52
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 84
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 10
cpu cores	: 8
apicid		: 116
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 0
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 1
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 54
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 2
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 86
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 46
model name	: Intel(R) Xeon(R) CPU           X7550  @ 2.00GHz
stepping	: 6
cpu MHz		: 1995.049
cache size	: 18432 KB
physical id	: 3
siblings	: 8
core id		: 11
cpu cores	: 8
apicid		: 118
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 3990.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 44 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     264257624 kB
MemFree:      256785128 kB
Buffers:        343400 kB
Cached:        6789916 kB
SwapCached:          0 kB
Active:        6317340 kB
Inactive:       884980 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     264257624 kB
LowFree:      256785128 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           20280 kB
Writeback:           0 kB
AnonPages:       69364 kB
Mapped:          14208 kB
Slab:           190564 kB
PageTables:       3992 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  199240340 kB
Committed_AS:   188564 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296372 kB
VmallocChunk: 34359440603 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 62032 MiB
End job on node200 at 2011-05-07 02:24:25