Trace number 1728897

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
tts tts-5-0? (MO) 378.675 378.886

General information on the benchmark

Namecrafted/Hard/contest02/
ezfact64_3.sat05-450.reshuffled-07.cnf
MD5SUMa4a7c528ede134442a2d3ba0a444cdbb
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark864.685
Satisfiable
(Un)Satisfiability was proved
Number of variables3073
Number of clauses19785
Sum of the clauses size74601
Maximum clause length4
Minimum clause length1
Number of clauses of size 165
Number of clauses of size 20
Number of clauses of size 34344
Number of clauses of size 415376
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.03/0.05	c Command: HOME/tts-5-0 HOME/instance-1728897-1240742643.cnf 
0.03/0.05	c Using SATTIMEOUT of 1200
0.03/0.05	c Using SATRAM of 1800
0.03/0.05	c Size of subset node = 16
0.03/0.05	c Variables = 3073
0.03/0.05	c Clauses = 19785
0.03/0.05	c Literals = 74601
0.03/0.05	c Reduced size:
0.03/0.05	c Variables = 3073
0.03/0.05	c Clauses = 19521
0.03/0.05	c Literals = 73602
0.03/0.05	c a2
0.09/0.15	c a1
0.49/0.52	c Independent sub-problems = 1
0.49/0.52	c Sub-problem with 3008 variables and 19521 clauses
0.49/0.52	c using permutation method 4
0.49/0.52	c count = 1
0.49/0.52	c Ordering finished 0.500 seconds
0.49/0.52	c mapped 0.510 seconds
0.49/0.52	c permuted 0.510 seconds
0.49/0.52	c about to sort
0.49/0.53	c sorted
0.59/0.64	c clauses now 19521
0.59/0.64	c init finished 0.540 seconds
0.59/0.64	c tree built 0.630 seconds
0.59/0.64	c 
0.59/0.64	c PA progress                      | CPU progress Partial assignments | improvements      CPU time        PA/sec        nodes
0.59/0.64	c ---------------------------------|----------------------------------|------------------------------------------------------
0.59/0.64	c ................................ | ..........                       |                        0.6           0.0             
0.59/0.64	c ................................ | ..........                     1 |                        0.6           1.6             
0.59/0.64	c ................................ | ..........                     2 |                        0.6           3.2             
0.59/0.64	c ................................ | ..........                     3 |                        0.6           4.8             
0.59/0.64	c ................................ | ..........                     4 |                        0.6           6.3             
0.59/0.64	c ................................ | ..........                     5 |                        0.6           7.9             
0.59/0.64	c ................................ | ..........                  1000 | 9                      0.6        1562.5          298
0.59/0.66	c ****............................ | ..........                  2450 | 58                     0.7        3769.2         1422
0.59/0.66	c ****............................ | ..........                  2451 | 58                     0.7        3770.8         1422
0.59/0.66	c ****............................ | ..........                  2452 | 58                     0.7        3772.3         1422
24.09/24.14	c ****............................ | ..........                284000 | 21350                 24.1       11769.6      9201782
47.98/48.07	c ****............................ | ..........                527000 | 40233                 48.1       10965.5     17466917
71.97/72.01	c ****............................ | ..........                803000 | 64031                 72.0       11152.8     26487544
95.95/96.02	c ****............................ | ..........               1048000 | 85049                 96.0       10916.7     34478395
120.04/120.13	c ****............................ | *.........               1308000 | 107066               120.1       10890.9     42501013
143.94/144.05	c ****............................ | *.........               1536000 | 124295               144.0       10665.9     49981270
168.02/168.10	c ****............................ | *.........               1743000 | 140039               168.1       10371.9     56829563
192.01/192.13	c ****............................ | *.........               1951000 | 157919               192.1       10157.2     63502318
215.90/216.06	c ****............................ | *.........               2205000 | 179985               216.0       10208.3     71628698
239.90/240.09	c ****............................ | **........               2440000 | 200109               240.0       10165.4     79248351
263.89/264.07	c ****............................ | **........               2606000 | 213376               264.0        9871.2     84517567
287.88/288.09	c ****............................ | **........               2816000 | 229472               288.0        9777.1     91193766
312.06/312.22	c ****............................ | **........               3013000 | 244506               312.1        9653.0     97706902
335.86/336.08	c ****............................ | **........               3146000 | 255947               336.0        9363.1    101930724
359.93/360.11	c ****............................ | ***.......               3396000 | 277421               360.0        9433.3    110048254

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-1728897-1240742643/watcher-1728897-1240742643 -o /tmp/evaluation-result-1728897-1240742643/solver-1728897-1240742643 -C 1200 -W 1800 -M 1800 --output-limit 1,15 HOME/tts-5-0 HOME/instance-1728897-1240742643.cnf 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 1843200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 1894400 KiB
Solver output will be limited to a maximum of 15728640 bytes. The first 1048576 bytes and the last 14680064 bytes will be preserved
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 20551
/proc/meminfo: memFree=1828912/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=1008 CPUtime=0
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 129 0 0 0 0 0 0 0 18 0 1 0 102297680 1032192 113 1992294400 134512640 135191068 4294956224 18446744073709551615 134694054 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 252 113 67 165 0 85 0

[startup+0.0409039 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 20551
/proc/meminfo: memFree=1828912/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=2272 CPUtime=0.03
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 436 0 0 0 3 0 0 0 18 0 1 0 102297680 2326528 420 1992294400 134512640 135191068 4294956224 18446744073709551615 134716206 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 568 420 68 165 0 401 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 2272

[startup+0.101908 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 20551
/proc/meminfo: memFree=1828912/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=4260 CPUtime=0.09
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 943 0 0 0 9 0 0 0 18 0 1 0 102297680 4362240 926 1992294400 134512640 135191068 4294956224 18446744073709551615 134710785 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 1065 926 69 165 0 898 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 4260

[startup+0.301925 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 20551
/proc/meminfo: memFree=1828912/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=4424 CPUtime=0.29
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 1001 0 0 0 29 0 0 0 19 0 1 0 102297680 4530176 984 1992294400 134512640 135191068 4294956224 18446744073709551615 134555288 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 1106 984 71 165 0 939 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 4424

[startup+0.70196 s]
/proc/loadavg: 0.92 0.98 0.99 3/64 20551
/proc/meminfo: memFree=1828912/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=10428 CPUtime=0.69
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 2493 0 0 0 68 1 0 0 23 0 1 0 102297680 10678272 2476 1992294400 134512640 135191068 4294956224 18446744073709551615 134546224 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 2607 2476 89 165 0 2440 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 10428

[startup+1.50103 s]
/proc/loadavg: 0.92 0.98 0.99 2/65 20552
/proc/meminfo: memFree=1813096/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=22108 CPUtime=1.49
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 5429 0 0 0 146 3 0 0 25 0 1 0 102297680 22638592 5412 1992294400 134512640 135191068 4294956224 18446744073709551615 134710791 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 5527 5412 89 165 0 5360 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 22108

[startup+3.10117 s]
/proc/loadavg: 0.92 0.98 0.99 2/65 20552
/proc/meminfo: memFree=1801320/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=33788 CPUtime=3.09
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 8331 0 0 0 305 4 0 0 25 0 1 0 102297680 34598912 8314 1992294400 134512640 135191068 4294956224 18446744073709551615 134562321 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 8447 8314 89 165 0 8280 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 33788

[startup+6.30145 s]
/proc/loadavg: 0.93 0.98 0.99 2/65 20552
/proc/meminfo: memFree=1777712/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=57788 CPUtime=6.29
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 14367 0 0 0 622 7 0 0 25 0 1 0 102297680 59174912 14350 1992294400 134512640 135191068 4294956224 18446744073709551615 134592120 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 14447 14350 89 165 0 14280 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 57788

[startup+12.703 s]
/proc/loadavg: 0.93 0.98 0.99 2/65 20552
/proc/meminfo: memFree=1729392/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=105148 CPUtime=12.69
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 26185 0 0 0 1257 12 0 0 25 0 1 0 102297680 107671552 26168 1992294400 134512640 135191068 4294956224 18446744073709551615 134642167 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 26287 26168 89 165 0 26120 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 105148

[startup+25.5011 s]
/proc/loadavg: 0.95 0.98 0.99 2/65 20552
/proc/meminfo: memFree=1669040/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=160988 CPUtime=25.49
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 40169 0 0 0 2531 18 0 0 25 0 1 0 102297680 164851712 40152 1992294400 134512640 135191068 4294956224 18446744073709551615 134562386 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 40247 40152 89 165 0 40080 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 160988

[startup+51.1014 s]
/proc/loadavg: 0.96 0.98 0.99 2/65 20552
/proc/meminfo: memFree=1530800/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=300324 CPUtime=51.08
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 74950 0 0 0 5076 32 0 0 25 0 1 0 102297680 307531776 74933 1992294400 134512640 135191068 4294956224 18446744073709551615 134562389 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 75081 74933 89 165 0 74914 0
Current children cumulated CPU time (s) 51.08
Current children cumulated vsize (KiB) 300324

[startup+102.306 s]
/proc/loadavg: 0.98 0.98 0.99 2/65 20552
/proc/meminfo: memFree=1228464/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=600324 CPUtime=102.26
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 149960 0 0 0 10163 63 0 0 25 0 1 0 102297680 614731776 149943 1992294400 134512640 135191068 4294956224 18446744073709551615 134562428 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 150081 149943 89 165 0 149914 0
Current children cumulated CPU time (s) 102.26
Current children cumulated vsize (KiB) 600324

[startup+162.301 s]
/proc/loadavg: 0.99 0.98 0.99 2/65 20552
/proc/meminfo: memFree=934576/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=895364 CPUtime=162.23
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 223725 0 0 0 16131 92 0 0 25 0 1 0 102297680 916852736 223708 1992294400 134512640 135191068 4294956224 18446744073709551615 134561558 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 223841 223708 89 165 0 223674 0
Current children cumulated CPU time (s) 162.23
Current children cumulated vsize (KiB) 895364

[startup+222.301 s]
/proc/loadavg: 0.99 0.98 0.99 2/65 20552
/proc/meminfo: memFree=642800/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=1184644 CPUtime=222.2
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 296060 0 0 0 22101 119 0 0 25 0 1 0 102297680 1213075456 296043 1992294400 134512640 135191068 4294956224 18446744073709551615 134562399 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 296161 296043 89 165 0 295994 0
Current children cumulated CPU time (s) 222.2
Current children cumulated vsize (KiB) 1184644

[startup+282.302 s]
/proc/loadavg: 0.99 0.98 0.99 2/65 20552
/proc/meminfo: memFree=377456/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=1451044 CPUtime=282.18
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 362632 0 0 0 28071 147 0 0 25 0 1 0 102297680 1485869056 362615 1992294400 134512640 135191068 4294956224 18446744073709551615 134562453 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 362761 362615 89 165 0 362594 0
Current children cumulated CPU time (s) 282.18
Current children cumulated vsize (KiB) 1451044

[startup+342.302 s]
/proc/loadavg: 0.99 0.98 0.99 2/65 20552
/proc/meminfo: memFree=146864/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=1680964 CPUtime=342.16
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 420129 0 0 0 34045 171 0 0 25 0 1 0 102297680 1721307136 420112 1992294400 134512640 135191068 4294956224 18446744073709551615 134641748 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 420241 420112 89 165 0 420074 0
Current children cumulated CPU time (s) 342.16
Current children cumulated vsize (KiB) 1680964



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+378.602 s]
/proc/loadavg: 0.99 0.98 0.99 2/65 20552
/proc/meminfo: memFree=15920/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=1843684 CPUtime=378.39
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4194304 460787 0 0 0 37651 188 0 0 25 0 1 0 102297680 1887932416 460770 1992294400 134512640 135191068 4294956224 18446744073709551615 134562392 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 460921 460770 89 165 0 460754 0
Current children cumulated CPU time (s) 378.39
Current children cumulated vsize (KiB) 1843684

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+378.802 s]
/proc/loadavg: 0.99 0.98 0.99 2/65 20552
/proc/meminfo: memFree=15920/2055920 swapFree=4191900/4192956
[pid=20551] ppid=20549 vsize=0 CPUtime=378.58
/proc/20551/stat : 20551 (tts-5-0) R 20549 20551 20196 0 -1 4195332 460787 0 0 0 37651 207 0 0 25 0 1 0 102297680 0 0 1992294400 0 0 0 0 0 0 0 4096 0 0 0 0 17 0 0 0
/proc/20551/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 378.58
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 378.886
CPU time (s): 378.675
CPU user time (s): 376.517
CPU system time (s): 2.15867
CPU usage (%): 99.9445
Max. virtual memory (cumulated for all children) (KiB): 1843684

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 376.517
system time used= 2.15867
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 460787
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 906

runsolver used 0.332949 second user time and 1.05784 second system time

The end

Launcher Data 

Begin job on node15 at 2009-04-26 12:44:03
IDJOB=1728897
IDBENCH=24688
IDSOLVER=505
FILE ID=node15/1728897-1240742643
PBS_JOBID= 9186974
Free space on /tmp= 66268 MiB

SOLVER NAME= tts tts-5-0
BENCH NAME= SAT07/crafted/Hard/contest02/ezfact64_3.sat05-450.reshuffled-07.cnf
COMMAND LINE= HOME/tts-5-0 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-1728897-1240742643/watcher-1728897-1240742643 -o /tmp/evaluation-result-1728897-1240742643/solver-1728897-1240742643 -C 1200 -W 1800 -M 1800 --output-limit 1,15  HOME/tts-5-0 HOME/instance-1728897-1240742643.cnf

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 1800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= a4a7c528ede134442a2d3ba0a444cdbb
RANDOM SEED=1582370386

node15.alineos.net Linux 2.6.9-22.EL.rootsmp #1 SMP Mon Oct 3 08:59:52 CEST 2005

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.220
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5931.00
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 15
model		: 4
model name	:                   Intel(R) Xeon(TM) CPU 3.00GHz
stepping	: 3
cpu MHz		: 3000.220
cache size	: 2048 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
bogomips	: 5586.94
clflush size	: 64
cache_alignment	: 128
address sizes	: 36 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:      2055920 kB
MemFree:       1829392 kB
Buffers:         17720 kB
Cached:         140676 kB
SwapCached:        384 kB
Active:          14900 kB
Inactive:       156868 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      2055920 kB
LowFree:       1829392 kB
SwapTotal:     4192956 kB
SwapFree:      4191900 kB
Dirty:            1176 kB
Writeback:           0 kB
Mapped:          21768 kB
Slab:            40736 kB
Committed_AS:   364920 kB
PageTables:       1460 kB
VmallocTotal: 536870911 kB
VmallocUsed:    264952 kB
VmallocChunk: 536605679 kB
HugePages_Total:     0
HugePages_Free:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66268 MiB
End job on node15 at 2009-04-26 12:50:23