Trace number 1519313

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
VARSAT-random 2009-03-22SAT 22.4546 22.462

General information on the benchmark

NameRANDOM/MEDIUM/3SAT/SATISFIABLE/360/
unif-k3-r4.25-v360-c1530-S144043535-002.cnf
MD5SUM7d1ad487c2fb43d49523e96ef25c43ea
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.012998
Satisfiable
(Un)Satisfiability was proved
Number of variables360
Number of clauses1530
Sum of the clauses size4590
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 31530
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data

0.00/0.00	c ============================[ Problem Statistics ]=============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Number of variables:  360                                                  |
0.00/0.00	c |  Number of clauses:    1530                                                 |
0.00/0.00	c |  Parsing time:         0.00         s                                       |
0.00/0.00	c |  Timeout:              43200 s
0.00/0.00	c |  Random Seed:          828754000.000000
0.00/0.00	c |  Heuristic:            EMSP-G, threshold 0.750                              |
0.00/0.00	c |  Branching Rule:       default (solution-guided)                            |
0.00/0.00	c |  Learned clauses in surveys?  Using NO learned clauses.                     |
0.00/0.00	c |                                                                             |
0.00/0.00	c ============================[ Search Statistics ]==============================
0.00/0.00	c | Conflicts |          ORIGINAL         |          LEARNT          | Progress |
0.00/0.00	c |           |    Vars  Clauses Literals |    Limit  Clauses Lit/Cl |          |
0.00/0.00	c ===============================================================================
0.00/0.00	c |         0 |     342     1485     4535 |      495        0    nan |  0.000 % |
0.00/0.01	c |       100 |     342     1485     4535 |      544      100     17 |  0.001 % |
0.00/0.02	c |       250 |     342     1485     4535 |      598      250     17 |  0.001 % |
0.00/0.03	c |       477 |     342     1485     4535 |      658      477     17 |  0.001 % |
0.04/0.04	c |       814 |     342     1485     4535 |      724      477     16 |  0.001 % |
0.04/0.06	c |      1320 |     342     1485     4535 |      797      598     15 |  0.001 % |
0.04/0.09	c |      2080 |     342     1485     4535 |      876      535     14 |  0.001 % |
0.09/0.13	c |      3219 |     342     1485     4535 |      964      755     16 |  0.001 % |
0.09/0.19	c |      4927 |     342     1485     4535 |     1061      968     16 |  0.001 % |
0.19/0.28	c |      7490 |     342     1485     4535 |     1167      789     15 |  0.001 % |
0.39/0.42	c |     11335 |     342     1485     4535 |     1283     1057     15 |  0.001 % |
0.59/0.63	c |     17101 |     342     1485     4535 |     1412      915     16 |  0.001 % |
0.89/0.95	c |     25750 |     342     1485     4535 |     1553      919     15 |  0.001 % |
1.39/1.45	c |     38724 |     342     1485     4535 |     1708     1251     15 |  0.001 % |
2.20/2.23	c |     58186 |     342     1485     4535 |     1879     1627     16 |  0.001 % |
3.39/3.43	c |     87378 |     342     1485     4535 |     2067     1217     15 |  0.001 % |
5.19/5.26	c |    131168 |     342     1485     4535 |     2274     1979     17 |  0.001 % |
8.09/8.12	c |    196853 |     342     1485     4535 |     2501     1937     17 |  0.001 % |
12.49/12.59	c |    295379 |     342     1485     4535 |     2752     1622     14 |  0.001 % |
19.49/19.50	c |    443170 |     342     1485     4535 |     3027     1845     16 |  0.001 % |
22.39/22.46	c ===============================================================================
22.39/22.46	c surveys--CPU time (sec)   : 0.0910         (0.41 % of total)
22.39/22.46	c surveys--attempted        : 40             (440 /sec)
22.39/22.46	c surveys--converged        : 40             (100.00 % successful)
22.39/22.46	c restarts              : 20
22.39/22.46	c conflicts             : 504606         (22567 /sec)
22.39/22.46	c decisions             : 595676         (1.82 % random) (26640 /sec)
22.39/22.46	c propagations          : 27670998       (1237489 /sec)
22.39/22.46	c conflict literals     : 9371532        (19.45 % deleted)
22.39/22.46	c Memory used           : 3.69 MB
22.39/22.46	c Total CPU time (sec)      : 22.4516     
22.39/22.46	c 
22.39/22.46	s SATISFIABLE
22.39/22.46	v 1 -2 3 4 -5 6 -7 8 9 10 11 12 -13 -14 15 -16 -17 18 -19 20 -21 22 -23 24 -25 26 -27 28 -29 -30 -31 -32 -33 -34 -35 -36 37 38 -39 -40 -41 42 -43 -44 45 46 -47 -48 -49 -50 51 52 53 54 55 -56 57 -58 -59 60 -61 62 63 64 -65 -66 -67 -68 -69 70 71 72 -73 74 75 -76 77 78 79 -80 81 82 -83 -84 -85 86 -87 -88 -89 90 91 -92 93 -94 -95 -96 -97 -98 99 100 -101 -102 -103 104 -105 -106 107 -108 -109 110 111 112 -113 -114 -115 -116 117 -118 -119 -120 121 -122 -123 124 -125 -126 -127 128 129 -130 131 132 133 134 135 136 137 138 139 140 141 -142 -143 144 -145 -146 -147 -148 149 150 -151 152 153 154 155 -156 -157 -158 -159 -160 -161 -162 -163 164 165 -166 -167 -168 169 -170 -171 -172 173 -174 175 -176 177 178 -179 -180 181 -182 -183 184 185 -186 -187 188 -189 -190 -191 192 -193 194 195 -196 197 198 199 -200 201 -202 -203 -204 205 206 -207 -208 -209 -210 211 -212 -213 214 215 216 217 -218 -219 -220 -221 222 223 224 -225 226 227 228 -229 230 -231 -232 -233 -234 -235 236 -237 238 239 240 -241 -242 243 -244 -245 -246 247 -248 -249 250 251 252 253 -254 255 256 -257 -258 259 260 -261 -262 263 264 -265 266 267 -268 -269 270 271 272 -273 274 -275 276 277 278 279 -280 -281 -282 283 -284 285 -286 -287 -288 289 -290 291 292 -293 294 -295 -296 -297 298 -299 -300 -301 -302 -303 304 305 306 307 -308 -309 310 311 312 -313 314 -315 316 317 318 -319 -320 -321 -322 323 -324 325 -326 -327 328 329 330 331 -332 333 -334 -335 -336 -337 -338 -339 -340 -341 -342 343 -344 345 -346 347 -348 349 350 -351 352 353 354 -355 -356 357 -358 -359 360 0

Verifier Data

OK

Watcher Data

runsolver version 3.2.9 (svn:489) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-1519313-1238309667/watcher-1519313-1238309667 -o /tmp/evaluation-result-1519313-1238309667/solver-1519313-1238309667 -C 1200 -W 1800 -M 1800 --output-limit 1,15 HOME/varsat-random_static -seed=828754000 HOME/instance-1519313-1238309667.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.03 1.02 1.00 3/64 17165
/proc/meminfo: memFree=1817696/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=2412 CPUtime=0
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 222 0 0 0 0 0 0 0 18 0 1 0 1103633963 2469888 207 1992294400 4194304 5102416 548682068800 18446744073709551615 4266266 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 603 207 98 221 0 379 0

[startup+0.0426491 s]
/proc/loadavg: 1.03 1.02 1.00 3/64 17165
/proc/meminfo: memFree=1817696/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=2412 CPUtime=0.04
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 236 0 0 0 4 0 0 0 18 0 1 0 1103633963 2469888 221 1992294400 4194304 5102416 548682068800 18446744073709551615 4249529 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 603 221 111 221 0 379 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 2412

[startup+0.101658 s]
/proc/loadavg: 1.03 1.02 1.00 3/64 17165
/proc/meminfo: memFree=1817696/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=2412 CPUtime=0.09
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 242 0 0 0 9 0 0 0 18 0 1 0 1103633963 2469888 227 1992294400 4194304 5102416 548682068800 18446744073709551615 4249638 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 603 227 111 221 0 379 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 2412

[startup+0.30169 s]
/proc/loadavg: 1.03 1.02 1.00 3/64 17165
/proc/meminfo: memFree=1817696/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=2544 CPUtime=0.29
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 268 0 0 0 29 0 0 0 19 0 1 0 1103633963 2605056 253 1992294400 4194304 5102416 548682068800 18446744073709551615 4249508 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 636 253 111 221 0 412 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 2544

[startup+0.701755 s]
/proc/loadavg: 1.03 1.02 1.00 3/64 17165
/proc/meminfo: memFree=1817696/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=2676 CPUtime=0.69
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 299 0 0 0 69 0 0 0 23 0 1 0 1103633963 2740224 284 1992294400 4194304 5102416 548682068800 18446744073709551615 4249520 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 669 284 111 221 0 445 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 2676

[startup+1.50188 s]
/proc/loadavg: 1.03 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1816856/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=2824 CPUtime=1.49
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 326 0 0 0 149 0 0 0 25 0 1 0 1103633963 2891776 311 1992294400 4194304 5102416 548682068800 18446744073709551615 4249508 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 706 311 111 221 0 482 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 2824

[startup+3.10114 s]
/proc/loadavg: 1.03 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1816600/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=2956 CPUtime=3.09
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 373 0 0 0 309 0 0 0 25 0 1 0 1103633963 3026944 358 1992294400 4194304 5102416 548682068800 18446744073709551615 4249652 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 739 358 111 221 0 515 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 2956

[startup+6.30166 s]
/proc/loadavg: 1.03 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1816344/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3220 CPUtime=6.29
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 428 0 0 0 629 0 0 0 25 0 1 0 1103633963 3297280 413 1992294400 4194304 5102416 548682068800 18446744073709551615 4249520 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 805 413 111 221 0 581 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 3220

[startup+12.7017 s]
/proc/loadavg: 1.02 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1815960/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3512 CPUtime=12.69
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 496 0 0 0 1269 0 0 0 25 0 1 0 1103633963 3596288 481 1992294400 4194304 5102416 548682068800 18446744073709551615 4249721 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 878 481 111 221 0 654 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 3512

Solver just ended. Dumping a history of the last processes samples

[startup+19.1017 s]
/proc/loadavg: 1.02 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1815768/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3644 CPUtime=19.09
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 547 0 0 0 1909 0 0 0 25 0 1 0 1103633963 3731456 532 1992294400 4194304 5102416 548682068800 18446744073709551615 4249619 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 911 532 111 221 0 687 0
Current children cumulated CPU time (s) 19.09
Current children cumulated vsize (KiB) 3644

[startup+20.702 s]
/proc/loadavg: 1.02 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1815704/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3776 CPUtime=20.69
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 562 0 0 0 2069 0 0 0 25 0 1 0 1103633963 3866624 547 1992294400 4194304 5102416 548682068800 18446744073709551615 4249655 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 944 547 111 221 0 720 0
Current children cumulated CPU time (s) 20.69
Current children cumulated vsize (KiB) 3776

[startup+21.5011 s]
/proc/loadavg: 1.02 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1815640/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3776 CPUtime=21.49
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 567 0 0 0 2149 0 0 0 25 0 1 0 1103633963 3866624 552 1992294400 4194304 5102416 548682068800 18446744073709551615 4250067 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 944 552 111 221 0 720 0
Current children cumulated CPU time (s) 21.49
Current children cumulated vsize (KiB) 3776

[startup+21.9012 s]
/proc/loadavg: 1.02 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1815640/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3776 CPUtime=21.89
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 569 0 0 0 2189 0 0 0 25 0 1 0 1103633963 3866624 554 1992294400 4194304 5102416 548682068800 18446744073709551615 4249619 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 944 554 111 221 0 720 0
Current children cumulated CPU time (s) 21.89
Current children cumulated vsize (KiB) 3776

[startup+22.3012 s]
/proc/loadavg: 1.02 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1815640/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3776 CPUtime=22.29
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 571 0 0 0 2229 0 0 0 25 0 1 0 1103633963 3866624 556 1992294400 4194304 5102416 548682068800 18446744073709551615 4255552 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 944 556 111 221 0 720 0
Current children cumulated CPU time (s) 22.29
Current children cumulated vsize (KiB) 3776

[startup+22.4013 s]
/proc/loadavg: 1.02 1.02 1.00 2/65 17166
/proc/meminfo: memFree=1815640/2055920 swapFree=4178496/4192956
[pid=17165] ppid=17163 vsize=3776 CPUtime=22.39
/proc/17165/stat : 17165 (varsat-random_s) R 17163 17165 16661 0 -1 4194304 571 0 0 0 2239 0 0 0 25 0 1 0 1103633963 3866624 556 1992294400 4194304 5102416 548682068800 18446744073709551615 4244075 0 0 4096 3 0 0 0 17 1 0 0
/proc/17165/statm: 944 556 111 221 0 720 0
Current children cumulated CPU time (s) 22.39
Current children cumulated vsize (KiB) 3776

Child status: 10
Real time (s): 22.462
CPU time (s): 22.4546
CPU user time (s): 22.4526
CPU system time (s): 0.001999
CPU usage (%): 99.9668
Max. virtual memory (cumulated for all children) (KiB): 3776

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 22.4526
system time used= 0.001999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 576
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 52

runsolver used 0.029995 second user time and 0.042993 second system time

The end

Launcher Data

Begin job on node3 at 2009-03-29 08:54:27
IDJOB=1519313
IDBENCH=70752
IDSOLVER=539
FILE ID=node3/1519313-1238309667
PBS_JOBID= 9061037
Free space on /tmp= 65916 MiB

SOLVER NAME= VARSAT-random 2009-03-22
BENCH NAME= SAT09/RANDOM/MEDIUM/3SAT/SATISFIABLE/360/unif-k3-r4.25-v360-c1530-S144043535-002.cnf
COMMAND LINE= HOME/varsat-random_static -seed=RANDOMSEED BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-1519313-1238309667/watcher-1519313-1238309667 -o /tmp/evaluation-result-1519313-1238309667/solver-1519313-1238309667 -C 1200 -W 1800 -M 1800 --output-limit 1,15  HOME/varsat-random_static -seed=828754000 HOME/instance-1519313-1238309667.cnf

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 7d1ad487c2fb43d49523e96ef25c43ea
RANDOM SEED=828754000

node3.alineos.net Linux 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.275
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5914.62
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.275
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1818112 kB
Buffers:         50544 kB
Cached:         114180 kB
SwapCached:       8268 kB
Active:         102048 kB
Inactive:        72540 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1818112 kB
SwapTotal:     4192956 kB
SwapFree:      4178496 kB
Dirty:            1276 kB
Writeback:           0 kB
Mapped:          15584 kB
Slab:            49204 kB
Committed_AS:  2686416 kB
PageTables:       1376 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 65916 MiB
End job on node3 at 2009-03-29 08:54:50