Trace number 3263581

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 4800.24 1201.71

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_60_29_30_6-dir.shuffled-as.sat03-130.cnf
MD5SUMe826117d87949a5b285e72732bfa395b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables870
Number of clauses69964
Sum of the clauses size140740
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 269935
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 529

Solver Data 

0.00/0.81	This is pico-portfolio, a naive parallel portfolio
0.00/0.81	Copyright (C) 2011 Olivier Roussel
0.00/0.81	Using 4 processing units
0.00/0.81	This portfolio uses the following solvers:
0.00/0.81	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.81	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.81	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.81	[4] lingeling 276-6264d55-100731 (Armin Biere)
0.00/0.81	[5] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.81	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.81	Starting clasp 1.3.6 on core(s) 4
0.00/0.81	Starting TNM 2009 on core(s) 6
0.00/0.81	Starting lingeling 276-6264d55-100731 on core(s) 2
0.00/0.81	Starting march_hi 2009 on core(s) 6

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263581-1304091627/watcher-3263581-1304091627 -o /tmp/evaluation-result-3263581-1304091627/solver-3263581-1304091627 -C 4800 -W 1300 -M 15500 ppfolio HOME/instance-3263581-1304091627.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.99 10.12 9.35 7/182 24604
/proc/meminfo: memFree=12714636/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) R 24602 24604 24128 0 -1 4194304 507 0 0 0 0 0 0 0 25 0 1 0 599393390 18763776 388 33554432000 4194304 4287625 140734474400816 18446744073709551615 216734838195 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/24604/statm: 4581 388 339 23 0 91 0

[startup+0.0617841 s]
/proc/loadavg: 9.99 10.12 9.35 7/182 24604
/proc/meminfo: memFree=12714636/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) R 24602 24604 24128 0 -1 4194304 507 0 0 0 0 0 0 0 25 0 1 0 599393390 18763776 388 33554432000 4194304 4287625 140734474400816 18446744073709551615 216734838195 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/24604/statm: 4581 388 339 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100776 s]
/proc/loadavg: 9.99 10.12 9.35 7/182 24604
/proc/meminfo: memFree=12714636/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) R 24602 24604 24128 0 -1 4194304 507 0 0 0 0 0 0 0 25 0 1 0 599393390 18763776 388 33554432000 4194304 4287625 140734474400816 18446744073709551615 216734838195 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/24604/statm: 4581 388 339 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300738 s]
/proc/loadavg: 9.99 10.12 9.35 7/182 24604
/proc/meminfo: memFree=12714636/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) R 24602 24604 24128 0 -1 4194304 507 0 0 0 0 0 0 0 25 0 1 0 599393390 18763776 388 33554432000 4194304 4287625 140734474400816 18446744073709551615 216734838195 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/24604/statm: 4581 388 339 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700669 s]
/proc/loadavg: 9.99 10.12 9.35 7/182 24604
/proc/meminfo: memFree=12714636/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) R 24602 24604 24128 0 -1 4194304 507 0 0 0 0 0 0 0 25 0 1 0 599393390 18763776 388 33554432000 4194304 4287625 140734474400816 18446744073709551615 216734838195 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/24604/statm: 4581 388 339 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50053 s]
/proc/loadavg: 9.99 10.12 9.35 11/188 24610
/proc/meminfo: memFree=12670444/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 22 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=37320 CPUtime=1.48 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 5170 0 0 0 148 0 0 0 25 0 1 0 599393391 38215680 4632 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 9330 4632 326 96 0 5892 0
[pid=24606] ppid=24604 vsize=22508 CPUtime=1.48 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 3515 0 0 0 148 0 0 0 20 0 1 0 599393391 23048192 2969 33554432000 4194304 4785740 140733755385088 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 5627 2969 340 145 0 2672 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=1.18 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 118 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134528824 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=7788 CPUtime=1.49 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 1486 0 0 0 149 0 0 0 20 0 1 0 599393391 7974912 1124 33554432000 4194304 4352236 140733669826784 18446744073709551615 4328022 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 1947 1124 127 39 0 1027 0
[pid=24610] ppid=24604 vsize=16852 CPUtime=0.3 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 1851 0 0 0 30 0 0 0 25 0 1 0 599393472 17256448 1537 33554432000 4194304 4331988 140735695545584 18446744073709551615 4226474 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 4213 1537 170 34 0 2655 0
Current children cumulated CPU time (s) 5.93
Current children cumulated vsize (KiB) 558484

[startup+3.10125 s]
/proc/loadavg: 9.99 10.12 9.35 12/188 24610
/proc/meminfo: memFree=12652616/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1042432 CPUtime=3.09 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 6675 0 0 0 308 1 0 0 25 0 1 0 599393391 1067450368 4260 33554432000 4194304 4587354 140736107332464 18446744073709551615 216734875292 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 260608 4260 327 96 0 257170 0
[pid=24606] ppid=24604 vsize=28716 CPUtime=3.08 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 5043 0 0 0 307 1 0 0 25 0 1 0 599393391 29405184 4497 33554432000 4194304 4785740 140733755385088 18446744073709551615 4309561 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 7179 4497 340 145 0 4224 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=2.02 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 202 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=10336 CPUtime=3.09 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 2107 0 0 0 309 0 0 0 25 0 1 0 599393391 10584064 1745 33554432000 4194304 4352236 140733669826784 18446744073709551615 4328013 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 2584 1745 127 39 0 1664 0
[pid=24610] ppid=24604 vsize=17116 CPUtime=1.05 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 1935 0 0 0 105 0 0 0 25 0 1 0 599393472 17526784 1621 33554432000 4194304 4331988 140735695545584 18446744073709551615 4226440 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 4279 1621 174 34 0 2721 0
Current children cumulated CPU time (s) 12.33
Current children cumulated vsize (KiB) 1572616

[startup+6.30069 s]
/proc/loadavg: 9.99 10.12 9.35 11/188 24610
/proc/meminfo: memFree=12612692/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1073404 CPUtime=6.29 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 14953 0 0 0 626 3 0 0 25 0 1 0 599393391 1099165696 12528 33554432000 4194304 4587354 140736107332464 18446744073709551615 4357668 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 268351 12528 337 96 0 264913 0
[pid=24606] ppid=24604 vsize=33156 CPUtime=6.29 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 6125 0 0 0 627 2 0 0 25 0 1 0 599393391 33951744 5579 33554432000 4194304 4785740 140733755385088 18446744073709551615 4416823 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 8289 5579 340 145 0 5334 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=3.62 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 362 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134529651 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=11872 CPUtime=6.29 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 2491 0 0 0 629 0 0 0 25 0 1 0 599393391 12156928 2129 33554432000 4194304 4352236 140733669826784 18446744073709551615 4329086 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 2968 2129 127 39 0 2048 0
[pid=24610] ppid=24604 vsize=17328 CPUtime=2.63 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 1971 0 0 0 263 0 0 0 25 0 1 0 599393472 17743872 1657 33554432000 4194304 4331988 140735695545584 18446744073709551615 4226443 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 4332 1657 174 34 0 2774 0
Current children cumulated CPU time (s) 25.12
Current children cumulated vsize (KiB) 1609776

[startup+12.7006 s]
/proc/loadavg: 9.99 10.11 9.36 11/188 24610
/proc/meminfo: memFree=12565488/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1077636 CPUtime=12.69 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 16184 0 0 0 1266 3 0 0 25 0 1 0 599393391 1103499264 13747 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 269409 13747 337 96 0 265971 0
[pid=24606] ppid=24604 vsize=40504 CPUtime=12.69 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 7898 0 0 0 1267 2 0 0 25 0 1 0 599393391 41476096 7352 33554432000 4194304 4785740 140733755385088 18446744073709551615 4553440 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 10126 7352 340 145 0 7171 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=6.82 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 682 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=14048 CPUtime=12.69 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 3932 0 0 0 1269 0 0 0 25 0 1 0 599393391 14385152 2546 33554432000 4194304 4352236 140733669826784 18446744073709551615 4329077 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 3512 2546 127 39 0 2592 0
[pid=24610] ppid=24604 vsize=17800 CPUtime=5.8 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 2073 0 0 0 580 0 0 0 25 0 1 0 599393472 18227200 1759 33554432000 4194304 4331988 140735695545584 18446744073709551615 4210412 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 4450 1759 174 34 0 2892 0
Current children cumulated CPU time (s) 50.69
Current children cumulated vsize (KiB) 1624004

[startup+25.5003 s]
/proc/loadavg: 9.99 10.11 9.37 11/188 24610
/proc/meminfo: memFree=12526460/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1067764 CPUtime=25.48 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 17653 0 0 0 2545 3 0 0 25 0 1 0 599393391 1093390336 13107 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238606 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 266941 13107 337 96 0 263503 0
[pid=24606] ppid=24604 vsize=47672 CPUtime=25.48 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 9600 0 0 0 2546 2 0 0 25 0 1 0 599393391 48816128 9054 33554432000 4194304 4785740 140733755385088 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 11918 9054 340 145 0 8963 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=13.15 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 1315 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=16864 CPUtime=25.48 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 5724 0 0 0 2548 0 0 0 25 0 1 0 599393391 17268736 3314 33554432000 4194304 4352236 140733669826784 18446744073709551615 4207584 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 4216 3314 127 39 0 3296 0
[pid=24610] ppid=24604 vsize=19276 CPUtime=12.2 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 2383 0 0 0 1220 0 0 0 25 0 1 0 599393472 19738624 2069 33554432000 4194304 4331988 140735695545584 18446744073709551615 4228224 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 4819 2069 175 34 0 3261 0
Current children cumulated CPU time (s) 101.79
Current children cumulated vsize (KiB) 1625592

[startup+51.1009 s]
/proc/loadavg: 10.07 10.11 9.39 11/188 24611
/proc/meminfo: memFree=12458084/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1087200 CPUtime=51.09 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 22403 0 0 0 5105 4 0 0 25 0 1 0 599393391 1113292800 17853 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 271800 17853 337 96 0 268362 0
[pid=24606] ppid=24604 vsize=58856 CPUtime=51.09 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 12283 0 0 0 5106 3 0 0 25 0 1 0 599393391 60268544 11737 33554432000 4194304 4785740 140733755385088 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 14714 11737 340 145 0 11759 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=25.91 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 2591 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134530614 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=22244 CPUtime=51.08 cores=2

################
# More data... #
################

/proc/24610/statm: 23902 17028 175 34 0 22344 0
Current children cumulated CPU time (s) 3764
Current children cumulated vsize (KiB) 1892304

[startup+1002.3 s]
/proc/loadavg: 10.14 10.17 9.88 11/188 24669
/proc/meminfo: memFree=12159384/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1153348 CPUtime=1002.31 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 106791 0 0 0 100195 36 0 0 25 0 1 0 599393391 1181028352 35053 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238658 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 288337 35053 337 96 0 284899 0
[pid=24606] ppid=24604 vsize=130684 CPUtime=1002.3 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 29041 0 0 0 100222 8 0 0 25 0 1 0 599393391 133820416 28495 33554432000 4194304 4785740 140733755385088 18446744073709551615 4304328 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 32671 28495 340 145 0 29716 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=499.01 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 49901 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134528348 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=63904 CPUtime=1002.23 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 133765 0 0 0 100207 16 0 0 25 0 1 0 599393391 65437696 15169 33554432000 4194304 4352236 140733669826784 18446744073709551615 4328544 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 15976 15169 127 39 0 15056 0
[pid=24610] ppid=24604 vsize=95608 CPUtime=497.82 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 18358 0 0 0 49774 8 0 0 25 0 1 0 599393472 97902592 18044 33554432000 4194304 4331988 140735695545584 18446744073709551615 4226400 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 23902 18044 175 34 0 22344 0
Current children cumulated CPU time (s) 4003.67
Current children cumulated vsize (KiB) 1917560

[startup+1062.3 s]
/proc/loadavg: 10.31 10.21 9.91 11/188 24671
/proc/meminfo: memFree=12181704/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1148272 CPUtime=1062.31 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 110598 0 0 0 106193 38 0 0 25 0 1 0 599393391 1175830528 34527 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238571 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 287068 34527 337 96 0 283630 0
[pid=24606] ppid=24604 vsize=131824 CPUtime=1062.31 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 29314 0 0 0 106222 9 0 0 25 0 1 0 599393391 134987776 28768 33554432000 4194304 4785740 140733755385088 18446744073709551615 4553344 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 32956 28768 340 145 0 30001 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=528.83 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 52883 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=39328 CPUtime=1062.24 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 137477 0 0 0 106207 17 0 0 25 0 1 0 599393391 40271872 9025 33554432000 4194304 4352236 140733669826784 18446744073709551615 4328778 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 9832 9025 127 39 0 8912 0
[pid=24610] ppid=24604 vsize=95608 CPUtime=527.67 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 19471 0 0 0 52759 8 0 0 25 0 1 0 599393472 97902592 19157 33554432000 4194304 4331988 140735695545584 18446744073709551615 216734573770 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 23902 19157 175 34 0 22344 0
Current children cumulated CPU time (s) 4243.36
Current children cumulated vsize (KiB) 1889048

[startup+1122.3 s]
/proc/loadavg: 10.23 10.20 9.93 11/188 24673
/proc/meminfo: memFree=12180328/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1152016 CPUtime=1122.31 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 114756 0 0 0 112191 40 0 0 25 0 1 0 599393391 1179664384 36426 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238418 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 288004 36426 337 96 0 284566 0
[pid=24606] ppid=24604 vsize=133152 CPUtime=1122.31 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 29637 0 0 0 112222 9 0 0 25 0 1 0 599393391 136347648 29091 33554432000 4194304 4785740 140733755385088 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 33288 29091 340 145 0 30333 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=558.66 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 55866 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134525956 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=38816 CPUtime=1122.24 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 141381 0 0 0 112207 17 0 0 25 0 1 0 599393391 39747584 8897 33554432000 4194304 4352236 140733669826784 18446744073709551615 4329060 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 9704 8897 127 39 0 8784 0
[pid=24610] ppid=24604 vsize=95608 CPUtime=557.51 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 20581 0 0 0 55742 9 0 0 25 0 1 0 599393472 97902592 20267 33554432000 4194304 4331988 140735695545584 18446744073709551615 4299872 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 23902 20267 175 34 0 22344 0
Current children cumulated CPU time (s) 4483.03
Current children cumulated vsize (KiB) 1893608

[startup+1182.3 s]
/proc/loadavg: 10.20 10.20 9.94 11/188 24674
/proc/meminfo: memFree=12154544/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1158080 CPUtime=1182.32 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 117793 0 0 0 118190 42 0 0 25 0 1 0 599393391 1185873920 34528 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 289520 34528 337 96 0 286082 0
[pid=24606] ppid=24604 vsize=134668 CPUtime=1182.31 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 30019 0 0 0 118222 9 0 0 25 0 1 0 599393391 137900032 29473 33554432000 4194304 4785740 140733755385088 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 33667 29473 340 145 0 30712 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=588.53 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 58853 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=47520 CPUtime=1182.24 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 147717 0 0 0 118206 18 0 0 25 0 1 0 599393391 48660480 11073 33554432000 4194304 4352236 140733669826784 18446744073709551615 4328122 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 11880 11073 127 39 0 10960 0
[pid=24610] ppid=24604 vsize=125076 CPUtime=587.32 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 21550 0 0 0 58723 9 0 0 25 0 1 0 599393472 128077824 21236 33554432000 4194304 4331988 140735695545584 18446744073709551615 4237130 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 31269 21236 175 34 0 29711 0
Current children cumulated CPU time (s) 4722.72
Current children cumulated vsize (KiB) 1939360



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 10.22 10.21 9.95 11/188 24675
/proc/meminfo: memFree=12168432/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1158080 CPUtime=1201.72 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 119338 0 0 0 120130 42 0 0 25 0 1 0 599393391 1185873920 36053 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238608 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 289520 36053 337 96 0 286082 0
[pid=24606] ppid=24604 vsize=134668 CPUtime=1201.72 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 30019 0 0 0 120163 9 0 0 25 0 1 0 599393391 137900032 29473 33554432000 4194304 4785740 140733755385088 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 33667 29473 340 145 0 30712 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=598.14 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 59814 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=39328 CPUtime=1201.65 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 147717 0 0 0 120147 18 0 0 25 0 1 0 599393391 40271872 9025 33554432000 4194304 4352236 140733669826784 18446744073709551615 4328191 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 9832 9025 127 39 0 8912 0
[pid=24610] ppid=24604 vsize=143232 CPUtime=597.01 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 21861 0 0 0 59691 10 0 0 25 0 1 0 599393472 146669568 21547 33554432000 4194304 4331988 140735695545584 18446744073709551615 4228226 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 35808 21547 175 34 0 34250 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 1949324

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.7 s]
/proc/loadavg: 10.22 10.21 9.95 11/188 24675
/proc/meminfo: memFree=12168432/32951124 swapFree=67102140/67111528
[pid=24604] ppid=24602 vsize=18324 CPUtime=0 cores=0,2,4,6
/proc/24604/stat : 24604 (ppfolio) S 24602 24604 24128 0 -1 4202496 524 0 0 0 0 0 0 0 15 0 1 0 599393390 18763776 390 33554432000 4194304 4287625 140734474400816 18446744073709551615 216735198735 0 0 4096 65536 7374732629922283519 0 0 17 2 0 0 0
/proc/24604/statm: 4581 390 341 23 0 91 0
[pid=24605] ppid=24604 vsize=1158080 CPUtime=1201.72 cores=0
/proc/24605/stat : 24605 (cryptominisat) R 24604 24604 24128 0 -1 4202496 119338 0 0 0 120130 42 0 0 25 0 1 0 599393391 1185873920 36053 33554432000 4194304 4587354 140736107332464 18446744073709551615 4238608 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/24605/statm: 289520 36053 337 96 0 286082 0
[pid=24606] ppid=24604 vsize=134668 CPUtime=1201.72 cores=4
/proc/24606/stat : 24606 (clasp) R 24604 24604 24128 0 -1 4202496 30019 0 0 0 120163 9 0 0 25 0 1 0 599393391 137900032 29473 33554432000 4194304 4785740 140733755385088 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 4 0 0 0
/proc/24606/statm: 33667 29473 340 145 0 30712 0
[pid=24607] ppid=24604 vsize=455692 CPUtime=598.14 cores=6
/proc/24607/stat : 24607 (TNM) R 24604 24604 24128 0 -1 4202496 1875 0 0 0 59814 0 0 0 25 0 1 0 599393391 466628608 1771 33554432000 134512640 134534564 4292714400 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24607/statm: 113923 1771 95 6 0 112589 0
[pid=24608] ppid=24604 vsize=39328 CPUtime=1201.65 cores=2
/proc/24608/stat : 24608 (lingeling) R 24604 24604 24128 0 -1 4202496 147717 0 0 0 120147 18 0 0 25 0 1 0 599393391 40271872 9025 33554432000 4194304 4352236 140733669826784 18446744073709551615 4328191 0 0 4096 17506 0 0 0 17 2 0 0 0
/proc/24608/statm: 9832 9025 127 39 0 8912 0
[pid=24610] ppid=24604 vsize=143232 CPUtime=597.01 cores=6
/proc/24610/stat : 24610 (march_hi) R 24604 24604 24128 0 -1 4202496 21861 0 0 0 59691 10 0 0 25 0 1 0 599393472 146669568 21547 33554432000 4194304 4331988 140735695545584 18446744073709551615 4228226 0 0 4096 0 0 0 0 17 6 0 0 0
/proc/24610/statm: 35808 21547 175 34 0 34250 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 1949324

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 24604 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=4999
# CPU time returned by wait4() is 0.005998
# while last known CPU time is 4800.24
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1201.71
CPU time (s): 4800.24
CPU user time (s): 4799.45
CPU system time (s): 0.79
CPU usage (%): 399.449
Max. virtual memory (cumulated for all children) (KiB): 2893376

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.000999
system time used= 0.004999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 524
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 311
involuntary context switches= 27

runsolver used 1.9367 second user time and 4.6263 second system time

The end

Launcher Data 

Begin job on node119 at 2011-04-29 17:40:27
IDJOB=3263581
IDBENCH=83986
IDSOLVER=1612
FILE ID=node119/3263581-1304091627
RUNJOBID= node119-1304090423-24477
PBS_JOBID= 13170550
Free space on /tmp= 71804 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_60_29_30_6-dir.shuffled-as.sat03-130.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263581-1304091627/watcher-3263581-1304091627 -o /tmp/evaluation-result-3263581-1304091627/solver-3263581-1304091627 -C 4800 -W 1300 -M 15500  ppfolio HOME/instance-3263581-1304091627.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e826117d87949a5b285e72732bfa395b
RANDOM SEED=1791534488

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      12714668 kB
Buffers:       2234908 kB
Cached:       17324108 kB
SwapCached:       5504 kB
Active:        8019680 kB
Inactive:     11631492 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      12714668 kB
SwapTotal:    67111528 kB
SwapFree:     67102140 kB
Dirty:           20848 kB
Writeback:           0 kB
AnonPages:       89020 kB
Mapped:          16520 kB
Slab:           522192 kB
PageTables:       5216 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  1595140 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71796 MiB
End job on node119 at 2011-04-29 18:00:31