Trace number 314476

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds, and are wall clock time (not CPU time).

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
CMUSAT 2007-02-08UNSAT 107.41 107.44

General information on the benchmark

Nameindustrial/fuhs/medium/
AProVE07-22.cnf
MD5SUM3b3198fba6db1661781a1aa668754f11
Bench CategoryINDUST (industrial instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark81.0267
Satisfiable
(Un)Satisfiability was proved
Number of variables15589
Number of clauses54263
Sum of the clauses size140423
Maximum clause length96
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 226004
Number of clauses of size 325780
Number of clauses of size 42100
Number of clauses of size 5198
Number of clauses of size over 5180

Solver Data (download as text)

0.00/0.00	c Pre-processing provided 0 assignments
0.00/0.00	c |  Number of variables:  15589                                                |
0.03/0.06	c |  Number of clauses:    54263                                                |
107.37/107.43	c |  Parsing time:         0.05         sec                                       |
107.37/107.43	c #varS 15589 #clauseS 22190
107.37/107.43	c REMAP internally 1
107.37/107.43	c After remapping new number of variables: 4046
107.37/107.43	c DBreduced=0 GV=101[0]  GL=22291[ 22190,101] Prune=342 decisions=241 restart=0 est=6.33522e-06 DB Impl=9958[0]  
107.37/107.43	c DBreduced=0 GV=151[0]  GL=22341[ 22190,151] Prune=887 decisions=736 restart=1 est=3.05571e-05 DB Impl=18003[0]  
107.37/107.43	c DBreduced=0 GV=226[0]  GL=22416[ 22190,226] Prune=1117 decisions=891 restart=2 est=6.11171e-06 DB Impl=31984[0]  
107.37/107.43	c DBreduced=0 GV=338[0]  GL=22528[ 22190,338] Prune=1520 decisions=1182 restart=3 est=1.22976e-05 DB Impl=45941[0]  
107.37/107.43	c DBreduced=0 GV=507[0]  GL=22697[ 22190,507] Prune=2178 decisions=1671 restart=4 est=0.000714725 DB Impl=72975[0]  
107.37/107.43	c DBreduced=0 GV=761[0]  GL=22951[ 22190,761] Prune=3108 decisions=2347 restart=5 est=0.000152895 DB Impl=115604[0]  
107.37/107.43	c DBreduced=0 GV=1140[0]  GL=23330[ 22190,1140] Prune=4158 decisions=3018 restart=6 est=5.05213e-05 DB Impl=168148[0]  
107.37/107.43	c DBreduced=0 GV=1709[0]  GL=23899[ 22190,1709] Prune=5868 decisions=4159 restart=7 est=0.00095069 DB Impl=301122[0]  
107.37/107.43	c DBreduced=0 GV=2565[0]  GL=24755[ 22190,2565] Prune=8545 decisions=5980 restart=8 est=0.00128304 DB Impl=451038[0]  
107.37/107.43	c DBreduced=0 GV=3845[0]  GL=26035[ 22190,3845] Prune=11938 decisions=8093 restart=9 est=0.000195529 DB Impl=714101[0]  
107.37/107.43	c DBreduced=0 GV=5767[0]  GL=27957[ 22190,5767] Prune=16578 decisions=10811 restart=10 est=0.00101463 DB Impl=972899[0]  
107.37/107.43	c DBreduced=0 GV=8651[0]  GL=30840[ 22190,8650] Prune=23390 decisions=14739 restart=11 est=0.0495843 DB Impl=1331068[0]  
107.37/107.43	c DBreduced=2 GV=12977[0]  GL=30390[ 22190,8200] Prune=33508 decisions=20531 restart=12 est=0.0500913 DB Impl=1972111[0]  
107.37/107.43	c DBreduced=3 GV=19462[0]  GL=34292[ 22190,12102] Prune=48006 decisions=28544 restart=13 est=2.12571 DB Impl=2822710[0]  
107.37/107.43	c DBreduced=5 GV=29193[0]  GL=35294[ 22190,13104] Prune=69137 decisions=39944 restart=14 est=9.12027 DB Impl=3867066[0]  
107.37/107.43	c DBreduced=7 GV=43793[0]  GL=39575[ 22190,17385] Prune=102558 decisions=58765 restart=15 est=9.1697 DB Impl=5619925[0]  
107.37/107.43	c DBreduced=10 GV=65685[0]  GL=40807[ 22190,18617] Prune=150317 decisions=84632 restart=16 est=11.7649 DB Impl=8755497[0]  
107.37/107.43	c DBreduced=13 GV=98527[0]  GL=49865[ 22190,27675] Prune=220206 decisions=121679 restart=17 est=15.3984 DB Impl=13658971[0]  
107.37/107.43	c DBreduced=17 GV=147790[0]  GL=53532[ 22190,31342] Prune=324270 decisions=176480 restart=18 est=17.7069 DB Impl=23741753[0]  
107.37/107.43	c DBreduced=21 GV=221685[0]  GL=61348[ 22190,39158] Prune=479461 decisions=257776 restart=19 est=34.9295 DB Impl=33549677[0]  
107.37/107.43	c #Total time (in seconds): 107.322
107.37/107.43	c #Memory used (in MB): 42.5703
107.37/107.43	c 
107.37/107.43	s UNSATISFIABLE

Verifier Data (download as text)

No possible verification on an UNSAT instance

Watcher Data (download as text)

runsolver version 3.2.1 (c) roussel@cril.univ-artois.fr

command line: runsolver --timestamp -w ROOT/results/node3/watcher-314476-1172637639 -o ROOT/results/node3/solver-314476-1172637639 -C 1200 -W 2400 -M 1800 --output-limit 1,15 /tmp/evaluation/314476-1172637639/cmusat /tmp/evaluation/314476-1172637639/instance-314476-1172637639.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2400 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.88 0.93 0.97 3/77 7277
/proc/meminfo: memFree=1150624/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=2936 CPUtime=0
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 551 0 0 0 0 0 0 0 18 0 1 0 283718724 3006464 534 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134548958 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 734 535 95 243 0 488 0

[startup+0.045857 s]
/proc/loadavg: 0.88 0.93 0.97 3/77 7277
/proc/meminfo: memFree=1150624/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=6172 CPUtime=0.03
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 1297 0 0 0 3 0 0 0 18 0 1 0 283718724 6320128 1263 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 135092575 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 1543 1266 96 243 0 1297 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 6172

[startup+0.101862 s]
/proc/loadavg: 0.88 0.93 0.97 3/77 7277
/proc/meminfo: memFree=1150624/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=7912 CPUtime=0.09
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 1744 0 0 0 9 0 0 0 18 0 1 0 283718724 8101888 1710 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134539070 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 1978 1710 108 243 0 1732 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 7912

[startup+0.301878 s]
/proc/loadavg: 0.88 0.93 0.97 3/77 7277
/proc/meminfo: memFree=1150624/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=10624 CPUtime=0.29
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 2625 0 0 0 28 1 0 0 19 0 1 0 283718724 10878976 2381 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 135091182 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 2656 2381 108 243 0 2410 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 10624

[startup+0.70191 s]
/proc/loadavg: 0.88 0.93 0.97 3/77 7277
/proc/meminfo: memFree=1150624/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=5736 CPUtime=0.69
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 3173 0 0 0 68 1 0 0 23 0 1 0 283718724 5873664 1142 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 135092575 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 1434 1142 118 243 0 1188 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 5736

[startup+1.50198 s]
/proc/loadavg: 0.88 0.93 0.97 2/78 7278
/proc/meminfo: memFree=1145688/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=8224 CPUtime=1.49
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 3813 0 0 0 148 1 0 0 25 0 1 0 283718724 8421376 1782 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589593 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 2056 1782 118 243 0 1810 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 8224

[startup+3.10111 s]
/proc/loadavg: 0.88 0.93 0.97 2/78 7278
/proc/meminfo: memFree=1143768/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=10224 CPUtime=3.09
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 4448 0 0 0 307 2 0 0 25 0 1 0 283718724 10469376 2295 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589536 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 2556 2295 118 243 0 2310 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 10224

[startup+6.30137 s]
/proc/loadavg: 0.89 0.93 0.97 2/78 7278
/proc/meminfo: memFree=1140184/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=11436 CPUtime=6.29
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 4997 0 0 0 627 2 0 0 25 0 1 0 283718724 11710464 2640 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134581507 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 2859 2640 118 243 0 2613 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 11436

[startup+12.7029 s]
/proc/loadavg: 0.90 0.93 0.97 2/78 7278
/proc/meminfo: memFree=1136856/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=14668 CPUtime=12.69
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 6063 0 0 0 1266 3 0 0 25 0 1 0 283718724 15020032 3474 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589593 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 3667 3474 118 243 0 3421 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 14668

[startup+25.502 s]
/proc/loadavg: 0.92 0.93 0.97 2/78 7278
/proc/meminfo: memFree=1128792/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=23868 CPUtime=25.48
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 8627 0 0 0 2544 4 0 0 25 0 1 0 283718724 24440832 5500 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134590013 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 5967 5500 118 243 0 5721 0
Current children cumulated CPU time (s) 25.48
Current children cumulated vsize (KiB) 23868

[startup+51.1011 s]
/proc/loadavg: 0.95 0.94 0.97 2/78 7278
/proc/meminfo: memFree=1122392/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=29916 CPUtime=51.07
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 10668 0 0 0 5102 5 0 0 25 0 1 0 283718724 30633984 7098 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134590180 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 7479 7098 118 243 0 7233 0
Current children cumulated CPU time (s) 51.07
Current children cumulated vsize (KiB) 29916

[startup+102.305 s]
/proc/loadavg: 0.97 0.95 0.97 2/78 7278
/proc/meminfo: memFree=1116632/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=34972 CPUtime=102.27
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 12660 0 0 0 10220 7 0 0 25 0 1 0 283718724 35811328 8526 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589593 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 8743 8526 118 243 0 8497 0
Current children cumulated CPU time (s) 102.27
Current children cumulated vsize (KiB) 34972

Solver just ended. Dumping a history of the last processes samples

[startup+105.502 s]
/proc/loadavg: 0.98 0.95 0.97 2/78 7278
/proc/meminfo: memFree=1110744/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=43592 CPUtime=105.47
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 14975 0 0 0 10539 8 0 0 25 0 1 0 283718724 44638208 9981 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589593 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 10898 9981 118 243 0 10652 0
Current children cumulated CPU time (s) 105.47
Current children cumulated vsize (KiB) 43592

[startup+106.302 s]
/proc/loadavg: 0.98 0.95 0.97 2/78 7278
/proc/meminfo: memFree=1110744/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=43592 CPUtime=106.27
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 14975 0 0 0 10619 8 0 0 25 0 1 0 283718724 44638208 9981 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589663 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 10898 9981 118 243 0 10652 0
Current children cumulated CPU time (s) 106.27
Current children cumulated vsize (KiB) 43592

[startup+107.102 s]
/proc/loadavg: 0.98 0.95 0.97 2/78 7278
/proc/meminfo: memFree=1110744/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=43592 CPUtime=107.07
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 14975 0 0 0 10699 8 0 0 25 0 1 0 283718724 44638208 9981 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134590126 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 10898 9981 118 243 0 10652 0
Current children cumulated CPU time (s) 107.07
Current children cumulated vsize (KiB) 43592

[startup+107.302 s]
/proc/loadavg: 0.98 0.95 0.97 2/78 7278
/proc/meminfo: memFree=1110744/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=43592 CPUtime=107.27
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 14975 0 0 0 10719 8 0 0 25 0 1 0 283718724 44638208 9981 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134589524 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 10898 9981 118 243 0 10652 0
Current children cumulated CPU time (s) 107.27
Current children cumulated vsize (KiB) 43592

[startup+107.402 s]
/proc/loadavg: 0.98 0.95 0.97 2/78 7278
/proc/meminfo: memFree=1110744/2055920 swapFree=4192812/4192956
[pid=7277] ppid=7275 vsize=43592 CPUtime=107.37
/proc/7277/stat : 7277 (cmusat) R 7275 7277 7176 0 -1 4194304 14975 0 0 0 10729 8 0 0 25 0 1 0 283718724 44638208 9981 18446744073709551615 134512640 135508737 4294956624 18446744073709551615 134591331 0 0 4096 35 0 0 0 17 1 0 0
/proc/7277/statm: 10898 9981 118 243 0 10652 0
Current children cumulated CPU time (s) 107.37
Current children cumulated vsize (KiB) 43592

Child status: 20
Real time (s): 107.44
CPU time (s): 107.41
CPU user time (s): 107.322
CPU system time (s): 0.087986
CPU usage (%): 99.9715
Max. virtual memory (cumulated for all children) (KiB): 43592

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 107.322
system time used= 0.087986
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 14987
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 8
involuntary context switches= 2192

runsolver used 0.131979 second user time and 0.265959 second system time

The end

Launcher Data (download as text)

Begin job on node3 on Wed Feb 28 04:40:39 UTC 2007

IDJOB= 314476
IDBENCH= 20392
IDSOLVER= 121
FILE ID= node3/314476-1172637639

PBS_JOBID= 3980930

Free space on /tmp= 66522 MiB

SOLVER NAME= CMUSAT 2007-02-08
BENCH NAME= HOME/pub/bench/SAT07/industrial/fuhs/medium/AProVE07-22.cnf
COMMAND LINE= /tmp/evaluation/314476-1172637639/cmusat /tmp/evaluation/314476-1172637639/instance-314476-1172637639.cnf            
RUNSOLVER COMMAND LINE= runsolver  --timestamp  -w ROOT/results/node3/watcher-314476-1172637639 -o ROOT/results/node3/solver-314476-1172637639 -C 1200 -W 2400 -M 1800 --output-limit 1,15  /tmp/evaluation/314476-1172637639/cmusat /tmp/evaluation/314476-1172637639/instance-314476-1172637639.cnf            

META MD5SUM SOLVER= af990fb12f8926fdd756b397adb3b5cf
MD5SUM BENCH=  3b3198fba6db1661781a1aa668754f11

RANDOM SEED= 157741535

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB

Linux node3.alineos.net 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005 x86_64 x86_64 x86_64 GNU/Linux

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.241
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.241
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1151032 kB
Buffers:         39808 kB
Cached:         740084 kB
SwapCached:          0 kB
Active:         274036 kB
Inactive:       560292 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1151032 kB
SwapTotal:     4192956 kB
SwapFree:      4192812 kB
Dirty:            3340 kB
Writeback:           0 kB
Mapped:          73368 kB
Slab:            56236 kB
Committed_AS:  1247760 kB
PageTables:       1656 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66522 MiB

End job on node3 on Wed Feb 28 04:42:26 UTC 2007