Trace number 3191015

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.34 1201.71

General information on the benchmark

Namerandom/medium/
unif-k3-r4.26-v500-c2130-S1514125851-037.UNKNOWN.cnf
MD5SUM7ce6e28d94f25a5004daf47b6c05c693
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark935.616
Satisfiable
(Un)Satisfiability was proved
Number of variables500
Number of clauses2130
Sum of the clauses size6390
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 32130
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 4 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 1
0.00/0.00	Starting clasp 1.3.6 on core(s) 5
0.00/0.00	Starting TNM 2009 on core(s) 7
0.00/0.00	Starting lingeling 276-6264d55-100731 on core(s) 3
0.00/0.00	Starting march_hi 2009 on core(s) 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3191015-1304053154/watcher-3191015-1304053154 -o /tmp/evaluation-result-3191015-1304053154/solver-3191015-1304053154 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3191015-1304053154.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.16 10.13 9.55 6/182 30766
/proc/meminfo: memFree=24729972/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 24 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0

[startup+0.467926 s]
/proc/loadavg: 10.16 10.13 9.55 6/182 30766
/proc/meminfo: memFree=24729972/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 22 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.50092 s]
/proc/loadavg: 10.16 10.13 9.55 6/182 30766
/proc/meminfo: memFree=24729972/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 21 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.600901 s]
/proc/loadavg: 10.16 10.13 9.55 6/182 30766
/proc/meminfo: memFree=24729972/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 20 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700883 s]
/proc/loadavg: 10.16 10.13 9.55 6/182 30766
/proc/meminfo: memFree=24729972/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 19 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50673 s]
/proc/loadavg: 10.16 10.13 9.55 11/188 30772
/proc/meminfo: memFree=24723908/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=391156 CPUtime=1.49 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 3518 0 0 0 149 0 0 0 20 0 1 0 595545879 400543744 2379 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 97789 2379 337 96 0 94351 0
[pid=30768] ppid=30766 vsize=14028 CPUtime=1.5 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 954 0 0 0 150 0 0 0 25 0 1 0 595545879 14364672 842 33554432000 4194304 4785740 140733520338320 18446744073709551615 4550456 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 3507 842 340 145 0 552 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=0.74 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 74 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134528737 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=6624 CPUtime=1.5 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 1090 0 0 0 150 0 0 0 25 0 1 0 595545879 6782976 847 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328087 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 1656 847 127 39 0 736 0
[pid=30771] ppid=30766 vsize=7932 CPUtime=0.74 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 585 0 0 0 74 0 0 0 25 0 1 0 595545879 8122368 478 33554432000 4194304 4331988 140733237635072 18446744073709551615 4211493 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 1983 478 172 34 0 425 0
Current children cumulated CPU time (s) 5.97
Current children cumulated vsize (KiB) 893892

[startup+3.10044 s]
/proc/loadavg: 10.23 10.14 9.56 11/188 30772
/proc/meminfo: memFree=24721552/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=394996 CPUtime=3.08 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 5380 0 0 0 308 0 0 0 25 0 1 0 595545879 404475904 3416 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238728 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 98749 3416 337 96 0 95311 0
[pid=30768] ppid=30766 vsize=14696 CPUtime=3.09 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 1145 0 0 0 309 0 0 0 25 0 1 0 595545879 15048704 1033 33554432000 4194304 4785740 140733520338320 18446744073709551615 4553001 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 3674 1033 340 145 0 719 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=1.56 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 156 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 9813216 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=6496 CPUtime=3.09 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 1418 0 0 0 309 0 0 0 25 0 1 0 595545879 6651904 823 33554432000 4194304 4352236 140733271265600 18446744073709551615 4329043 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 1624 823 127 39 0 704 0
[pid=30771] ppid=30766 vsize=8100 CPUtime=1.51 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 624 0 0 0 151 0 0 0 25 0 1 0 595545879 8294400 517 33554432000 4194304 4331988 140733237635072 18446744073709551615 4263175 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 2025 517 173 34 0 467 0
Current children cumulated CPU time (s) 12.33
Current children cumulated vsize (KiB) 898440

[startup+6.30084 s]
/proc/loadavg: 10.23 10.14 9.56 11/188 30772
/proc/meminfo: memFree=24713892/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=395108 CPUtime=6.29 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 7472 0 0 0 628 1 0 0 25 0 1 0 595545879 404590592 3323 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238597 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 98777 3323 337 96 0 95339 0
[pid=30768] ppid=30766 vsize=15636 CPUtime=6.29 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 1383 0 0 0 629 0 0 0 25 0 1 0 595545879 16011264 1271 33554432000 4194304 4785740 140733520338320 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 3909 1271 340 145 0 954 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=3.15 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 315 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=8296 CPUtime=6.29 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 2468 0 0 0 629 0 0 0 25 0 1 0 595545879 8495104 1265 33554432000 4194304 4352236 140733271265600 18446744073709551615 4329060 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 2074 1265 127 39 0 1154 0
[pid=30771] ppid=30766 vsize=8252 CPUtime=3.11 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 690 0 0 0 311 0 0 0 25 0 1 0 595545879 8450048 583 33554432000 4194304 4331988 140733237635072 18446744073709551615 4288464 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 2063 583 173 34 0 505 0
Current children cumulated CPU time (s) 25.13
Current children cumulated vsize (KiB) 901444

[startup+12.7007 s]
/proc/loadavg: 10.27 10.15 9.57 11/188 30772
/proc/meminfo: memFree=24706840/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=402544 CPUtime=12.68 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 11092 0 0 0 1267 1 0 0 25 0 1 0 595545879 412205056 4164 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 100636 4164 337 96 0 97198 0
[pid=30768] ppid=30766 vsize=17028 CPUtime=12.69 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 1721 0 0 0 1269 0 0 0 25 0 1 0 595545879 17436672 1609 33554432000 4194304 4785740 140733520338320 18446744073709551615 4546074 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 4257 1609 340 145 0 1302 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=6.41 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 641 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134528341 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=8428 CPUtime=12.68 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 3565 0 0 0 1268 0 0 0 25 0 1 0 595545879 8630272 1306 33554432000 4194304 4352236 140733271265600 18446744073709551615 4329000 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 2107 1306 127 39 0 1187 0
[pid=30771] ppid=30766 vsize=9052 CPUtime=6.21 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 843 0 0 0 621 0 0 0 25 0 1 0 595545879 9269248 736 33554432000 4194304 4331988 140733237635072 18446744073709551615 4229592 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 2263 736 173 34 0 705 0
Current children cumulated CPU time (s) 50.67
Current children cumulated vsize (KiB) 911204

[startup+25.5003 s]
/proc/loadavg: 9.83 10.06 9.54 6/178 30789
/proc/meminfo: memFree=24852656/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=402740 CPUtime=25.49 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 17510 0 0 0 2546 3 0 0 25 0 1 0 595545879 412405760 5101 33554432000 4194304 4587354 140733987390656 18446744073709551615 4237681 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 100685 5101 337 96 0 97247 0
[pid=30768] ppid=30766 vsize=18816 CPUtime=25.49 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 2142 0 0 0 2549 0 0 0 25 0 1 0 595545879 19267584 2030 33554432000 4194304 4785740 140733520338320 18446744073709551615 4277703 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 4704 2030 340 145 0 1749 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=12.75 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 1275 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134528824 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=12016 CPUtime=25.49 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 7214 0 0 0 2548 1 0 0 25 0 1 0 595545879 12304384 2203 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328042 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 3004 2203 127 39 0 2084 0
[pid=30771] ppid=30766 vsize=10560 CPUtime=12.6 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 1083 0 0 0 1260 0 0 0 25 0 1 0 595545879 10813440 976 33554432000 4194304 4331988 140733237635072 18446744073709551615 4292589 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 2640 976 173 34 0 1082 0
Current children cumulated CPU time (s) 101.82
Current children cumulated vsize (KiB) 918284

[startup+51.1005 s]
/proc/loadavg: 8.24 9.67 9.43 6/177 30790
/proc/meminfo: memFree=24837112/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=415552 CPUtime=51.09 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 26004 0 0 0 5104 5 0 0 25 0 1 0 595545879 425525248 7907 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238739 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 103888 7907 337 96 0 100450 0
[pid=30768] ppid=30766 vsize=20804 CPUtime=51.09 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 2611 0 0 0 5109 0 0 0 25 0 1 0 595545879 21303296 2499 33554432000 4194304 4785740 140733520338320 18446744073709551615 4560579 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 5201 2499 340 145 0 2246 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=25.51 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 2551 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134530278 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=12532 CPUtime=51.08 cores=3

################
# More data... #
################

/proc/30771/statm: 20471 14668 173 34 0 18913 0
Current children cumulated CPU time (s) 3764.1
Current children cumulated vsize (KiB) 1056756

[startup+1002.3 s]
/proc/loadavg: 5.03 5.25 6.62 6/177 30824
/proc/meminfo: memFree=24712568/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0.01 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 1 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=431564 CPUtime=1002.32 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 132673 0 0 0 100179 53 0 0 25 0 1 0 595545879 441921536 14546 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238654 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 107891 14546 337 96 0 104453 0
[pid=30768] ppid=30766 vsize=38064 CPUtime=1002.32 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 6629 0 0 0 100231 1 0 0 25 0 1 0 595545879 38977536 6517 33554432000 4194304 4785740 140733520338320 18446744073709551615 4546132 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 9516 6517 340 145 0 6561 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=498.53 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 49853 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134530586 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=31224 CPUtime=1002.28 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 96081 0 0 0 100213 15 0 0 25 0 1 0 595545879 31973376 7005 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328194 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 7806 7005 127 39 0 6886 0
[pid=30771] ppid=30766 vsize=81884 CPUtime=498.32 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 15598 0 0 0 49822 10 0 0 25 0 1 0 595545879 83849216 15491 33554432000 4194304 4331988 140733237635072 18446744073709551615 4211503 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 20471 15491 173 34 0 18913 0
Current children cumulated CPU time (s) 4003.78
Current children cumulated vsize (KiB) 1056888

[startup+1062.3 s]
/proc/loadavg: 5.05 5.22 6.51 6/177 30826
/proc/meminfo: memFree=24683896/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0.01 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 1 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=463420 CPUtime=1062.32 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 140011 0 0 0 106177 55 0 0 25 0 1 0 595545879 474542080 21808 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238392 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 115855 21808 337 96 0 112417 0
[pid=30768] ppid=30766 vsize=38512 CPUtime=1062.32 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 6719 0 0 0 106231 1 0 0 25 0 1 0 595545879 39436288 6607 33554432000 4194304 4785740 140733520338320 18446744073709551615 4553366 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 9628 6607 340 145 0 6673 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=528.36 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 52836 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134529675 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=34552 CPUtime=1062.27 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 100529 0 0 0 106212 15 0 0 25 0 1 0 595545879 35381248 7837 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328109 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 8638 7837 127 39 0 7718 0
[pid=30771] ppid=30766 vsize=96772 CPUtime=528.17 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 16388 0 0 0 52806 11 0 0 25 0 1 0 595545879 99094528 16281 33554432000 4194304 4331988 140733237635072 18446744073709551615 4246409 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 24193 16281 173 34 0 22635 0
Current children cumulated CPU time (s) 4243.45
Current children cumulated vsize (KiB) 1107408

[startup+1122.3 s]
/proc/loadavg: 5.02 5.17 6.41 6/177 30828
/proc/meminfo: memFree=24693664/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0.01 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 1 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=449212 CPUtime=1122.32 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 140775 0 0 0 112177 55 0 0 25 0 1 0 595545879 459993088 18743 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238481 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 112303 18743 337 96 0 108865 0
[pid=30768] ppid=30766 vsize=38944 CPUtime=1122.32 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 6804 0 0 0 112231 1 0 0 25 0 1 0 595545879 39878656 6692 33554432000 4194304 4785740 140733520338320 18446744073709551615 4552983 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 9736 6692 340 145 0 6781 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=558.19 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 55819 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134530664 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=31992 CPUtime=1122.29 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 102897 0 0 0 112213 16 0 0 25 0 1 0 595545879 32759808 7197 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328194 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 7998 7197 127 39 0 7078 0
[pid=30771] ppid=30766 vsize=96772 CPUtime=558.02 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 17158 0 0 0 55790 12 0 0 25 0 1 0 595545879 99094528 17051 33554432000 4194304 4331988 140733237635072 18446744073709551615 4212625 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 24193 17051 173 34 0 22635 0
Current children cumulated CPU time (s) 4483.15
Current children cumulated vsize (KiB) 1091072

[startup+1182.3 s]
/proc/loadavg: 5.04 5.15 6.33 6/177 30829
/proc/meminfo: memFree=24685072/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0.01 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 1 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=448636 CPUtime=1182.33 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 141070 0 0 0 118177 56 0 0 25 0 1 0 595545879 459403264 19021 33554432000 4194304 4587354 140733987390656 18446744073709551615 4237677 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 112159 19021 337 96 0 108721 0
[pid=30768] ppid=30766 vsize=39120 CPUtime=1182.32 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 6848 0 0 0 118231 1 0 0 25 0 1 0 595545879 40058880 6736 33554432000 4194304 4785740 140733520338320 18446744073709551615 4552983 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 9780 6736 340 145 0 6825 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=588.05 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 58805 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 9906549 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=36728 CPUtime=1182.28 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 107697 0 0 0 118212 16 0 0 25 0 1 0 595545879 37609472 8381 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328194 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 9182 8381 127 39 0 8262 0
[pid=30771] ppid=30766 vsize=96772 CPUtime=587.82 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 17934 0 0 0 58770 12 0 0 25 0 1 0 595545879 99094528 17827 33554432000 4194304 4331988 140733237635072 18446744073709551615 4211529 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 24193 17827 173 34 0 22635 0
Current children cumulated CPU time (s) 4722.81
Current children cumulated vsize (KiB) 1095408



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 5.11 5.16 6.30 6/177 30830
/proc/meminfo: memFree=24688668/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0.01 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 1 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=448636 CPUtime=1201.74 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 141070 0 0 0 120118 56 0 0 25 0 1 0 595545879 459403264 19021 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 112159 19021 337 96 0 108721 0
[pid=30768] ppid=30766 vsize=39120 CPUtime=1201.73 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 6865 0 0 0 120172 1 0 0 25 0 1 0 595545879 40058880 6753 33554432000 4194304 4785740 140733520338320 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 9780 6753 340 145 0 6825 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=597.66 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 59766 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134528295 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=31224 CPUtime=1201.69 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 107697 0 0 0 120153 16 0 0 25 0 1 0 595545879 31973376 7005 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328039 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 7806 7005 127 39 0 6886 0
[pid=30771] ppid=30766 vsize=96772 CPUtime=597.51 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 18200 0 0 0 59739 12 0 0 25 0 1 0 595545879 99094528 18093 33554432000 4194304 4331988 140733237635072 18446744073709551615 4211493 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 24193 18093 173 34 0 22635 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 1089904

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 5.11 5.16 6.30 6/177 30830
/proc/meminfo: memFree=24688668/32951124 swapFree=67111348/67111528
[pid=30766] ppid=30764 vsize=18324 CPUtime=0.01 cores=1,3,5,7
/proc/30766/stat : 30766 (ppfolio) S 30764 30766 30062 0 -1 4202496 522 0 0 0 1 0 0 0 15 0 1 0 595545878 18763776 389 33554432000 4194304 4287625 140735037034896 18446744073709551615 252458085903 0 0 4096 65536 4605371527444037631 0 0 17 3 0 0 0
/proc/30766/statm: 4581 389 341 23 0 91 0
[pid=30767] ppid=30766 vsize=448636 CPUtime=1201.74 cores=1
/proc/30767/stat : 30767 (cryptominisat) R 30766 30766 30062 0 -1 4202496 141070 0 0 0 120118 56 0 0 25 0 1 0 595545879 459403264 19021 33554432000 4194304 4587354 140733987390656 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 1 0 0 0
/proc/30767/statm: 112159 19021 337 96 0 108721 0
[pid=30768] ppid=30766 vsize=39120 CPUtime=1201.73 cores=5
/proc/30768/stat : 30768 (clasp) R 30766 30766 30062 0 -1 4202496 6865 0 0 0 120172 1 0 0 25 0 1 0 595545879 40058880 6753 33554432000 4194304 4785740 140733520338320 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 5 0 0 0
/proc/30768/statm: 9780 6753 340 145 0 6825 0
[pid=30769] ppid=30766 vsize=455828 CPUtime=597.66 cores=7
/proc/30769/stat : 30769 (TNM) R 30766 30766 30062 0 -1 4202496 336 0 0 0 59766 0 0 0 25 0 1 0 595545879 466767872 233 33554432000 134512640 134534564 4291013344 18446744073709551615 134528295 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30769/statm: 113957 233 95 6 0 112623 0
[pid=30770] ppid=30766 vsize=31224 CPUtime=1201.69 cores=3
/proc/30770/stat : 30770 (lingeling) R 30766 30766 30062 0 -1 4202496 107697 0 0 0 120153 16 0 0 25 0 1 0 595545879 31973376 7005 33554432000 4194304 4352236 140733271265600 18446744073709551615 4328039 0 0 4096 17506 0 0 0 17 3 0 0 0
/proc/30770/statm: 7806 7005 127 39 0 6886 0
[pid=30771] ppid=30766 vsize=96772 CPUtime=597.51 cores=7
/proc/30771/stat : 30771 (march_hi) R 30766 30766 30062 0 -1 4202496 18200 0 0 0 59739 12 0 0 25 0 1 0 595545879 99094528 18093 33554432000 4194304 4331988 140733237635072 18446744073709551615 4211493 0 0 4096 0 0 0 0 17 7 0 0 0
/proc/30771/statm: 24193 18093 173 34 0 22635 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 1089904

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 30766 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=12998
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=3999
# CPU time returned by wait4() is 0.016997
# while last known CPU time is 4800.34
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.34
CPU user time (s): 4799.49
CPU system time (s): 0.85
CPU usage (%): 399.458
Max. virtual memory (cumulated for all children) (KiB): 1455436

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.012998
system time used= 0.003999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 522
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 597
involuntary context switches= 6

runsolver used 1.85672 second user time and 4.70129 second system time

The end

Launcher Data 

Begin job on node111 at 2011-04-29 06:59:14
IDJOB=3191015
IDBENCH=82746
IDSOLVER=1612
FILE ID=node111/3191015-1304053154
RUNJOBID= node111-1304051939-30579
PBS_JOBID= 13170403
Free space on /tmp= 70888 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k3-r4.26-v500-c2130-S1514125851-037.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3191015-1304053154/watcher-3191015-1304053154 -o /tmp/evaluation-result-3191015-1304053154/solver-3191015-1304053154 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3191015-1304053154.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 7ce6e28d94f25a5004daf47b6c05c693
RANDOM SEED=1101519959

node111.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5428.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24730376 kB
Buffers:       1269600 kB
Cached:        6248896 kB
SwapCached:        124 kB
Active:        3229092 kB
Inactive:      4503336 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24730376 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:           10128 kB
Writeback:           0 kB
AnonPages:      213880 kB
Mapped:          16712 kB
Slab:           424660 kB
PageTables:       5468 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   866132 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 70888 MiB
End job on node111 at 2011-04-29 07:19:18