Trace number 3500304

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
pwbo 1.1SAT (TO)11 1800.26 900.154

General information on the benchmark

Namenormalized-PB06/OPT-SMALLINT/web/www.ps.uni-sb.de/~walser/
benchmarks/radar/normalized-10:10:4.5:0.95:100.opb
MD5SUMb258fc1dc8cb1827e842a0b9558712c7
Bench CategoryOPT-SMALLINT (optimisation, small integers)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark0.06299
Has Objective FunctionYES
SatisfiableYES
(Un)Satisfiability was provedYES
Best value of the objective function 0
Optimality of the best value was proved YES
Number of variables435
Total number of constraints501
Number of constraints which are clauses403
Number of constraints which are cardinality constraints (but not clauses)98
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint16
Number of terms in the objective function 435
Biggest coefficient in the objective function 282
Number of bits for the biggest coefficient in the objective function 9
Sum of the numbers in the objective function 1168
Number of bits of the sum of numbers in the objective function 11
Biggest number in a constraint 282
Number of bits of the biggest number in a constraint 9
Biggest sum of numbers in a constraint 1168
Number of bits of the biggest sum of numbers11
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Quality of the solution as a function of time


Download the above graph as a PDF file / as an EPS file

Solver Data 

0.00/0.00	c pwbo 1.1 beta - 20110321
0.00/0.00	c Parsing opb file format. File HOME/instance-3500304-1307993206.opb.
0.00/0.00	c Instance file HOME/instance-3500304-1307993206.opb
0.00/0.00	c File size is 20180 bytes.
0.00/0.00	c Highest Coefficient sum: 19
0.00/0.00	c ============================[ Problem Statistics ]=============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Parsing time:          0.00         s                                      |
0.00/0.00	c |  Number Variables:      435                                                 |
0.00/0.00	c |  Number Clauses:        404                                                 |
0.00/0.00	c |  Number Cardinality:    97                                                  |
0.00/0.00	c |  Number PB Constraints: 0                                                   |
0.00/0.00	c init lb 0
0.00/0.00	c init ub 1169
0.00/0.00	c lb -281 (thread 0)
0.00/0.00	c lb -280 (thread 0)
0.00/0.00	c lb -279 (thread 0)
0.00/0.00	c lb -278 (thread 0)
0.00/0.00	c lb -277 (thread 0)
0.00/0.00	c lb -276 (thread 0)
0.00/0.00	c lb -275 (thread 0)
0.00/0.00	c lb -274 (thread 0)
0.00/0.00	c lb -273 (thread 0)
0.00/0.00	c lb -272 (thread 0)
0.00/0.00	c lb -271 (thread 0)
0.00/0.00	c lb -270 (thread 0)
0.00/0.00	c lb -268 (thread 0)
0.00/0.00	c lb -267 (thread 0)
0.00/0.01	c lb -266 (thread 0)
0.00/0.01	c lb -265 (thread 0)
0.00/0.01	c lb -263 (thread 0)
0.00/0.01	c lb -262 (thread 0)
0.00/0.01	c lb -261 (thread 0)
0.00/0.01	c lb -260 (thread 0)
0.00/0.01	c lb -259 (thread 0)
0.00/0.01	c lb -258 (thread 0)
0.00/0.01	c lb -257 (thread 0)
0.00/0.01	c lb -256 (thread 0)
0.00/0.01	c lb -255 (thread 0)
0.00/0.01	c lb -254 (thread 0)
0.00/0.01	c lb -253 (thread 0)
0.00/0.01	c lb -252 (thread 0)
0.00/0.01	c lb -251 (thread 0)
0.00/0.01	c lb -250 (thread 0)
0.00/0.01	c lb -249 (thread 0)
0.00/0.01	c lb -248 (thread 0)
0.00/0.01	c lb -247 (thread 0)
0.00/0.01	c lb -246 (thread 0)
0.00/0.02	c lb -245 (thread 0)
0.00/0.02	c lb -244 (thread 0)
0.00/0.02	c lb -243 (thread 0)
0.00/0.02	c lb -242 (thread 0)
0.00/0.02	c lb -241 (thread 0)
0.00/0.02	c lb -240 (thread 0)
0.00/0.02	c lb -239 (thread 0)
0.00/0.02	c lb -238 (thread 0)
0.00/0.02	c lb -237 (thread 0)
0.00/0.02	c lb -236 (thread 0)
0.00/0.02	c lb -235 (thread 0)
0.00/0.02	c lb -233 (thread 0)
0.00/0.02	c lb -232 (thread 0)
0.00/0.02	c lb -231 (thread 0)
0.00/0.02	c lb -230 (thread 0)
0.00/0.02	c lb -229 (thread 0)
0.00/0.02	c lb -228 (thread 0)
0.00/0.03	c lb -227 (thread 0)
0.00/0.03	c lb -226 (thread 0)
0.00/0.03	c lb -225 (thread 0)
0.00/0.03	c lb -224 (thread 0)
0.00/0.03	c lb -223 (thread 0)
0.00/0.03	c lb -222 (thread 0)
0.00/0.03	c lb -221 (thread 0)
0.00/0.03	c lb -220 (thread 0)
0.00/0.03	c lb -219 (thread 0)
0.00/0.03	c lb -218 (thread 0)
0.00/0.03	c lb -217 (thread 0)
0.00/0.03	c lb -216 (thread 0)
0.00/0.03	c lb -215 (thread 0)
0.00/0.03	c lb -214 (thread 0)
0.00/0.03	c lb -213 (thread 0)
0.00/0.04	c lb -212 (thread 0)
0.00/0.04	c lb -211 (thread 0)
0.00/0.04	c lb -210 (thread 0)
0.00/0.04	c lb -209 (thread 0)
0.00/0.04	c lb -208 (thread 0)
0.00/0.04	c lb -207 (thread 0)
0.00/0.04	c lb -206 (thread 0)
0.00/0.04	c lb -205 (thread 0)
0.00/0.04	c lb -204 (thread 0)
0.00/0.04	c lb -203 (thread 0)
0.00/0.05	c lb -202 (thread 0)
0.00/0.05	c lb -201 (thread 0)
0.00/0.05	c lb -200 (thread 0)
0.00/0.05	c lb -199 (thread 0)
0.00/0.05	c lb -198 (thread 0)
0.00/0.05	c lb -197 (thread 0)
0.00/0.05	c lb -196 (thread 0)
0.00/0.05	c lb -195 (thread 0)
0.00/0.05	c lb -194 (thread 0)
0.00/0.05	c lb -193 (thread 0)
0.00/0.05	c lb -192 (thread 0)
0.00/0.05	c lb -191 (thread 0)
0.00/0.05	c lb -190 (thread 0)
0.00/0.06	c lb -189 (thread 0)
0.00/0.06	c lb -188 (thread 0)
0.00/0.06	c lb -187 (thread 0)
0.00/0.06	c lb -186 (thread 0)
0.00/0.06	c lb -185 (thread 0)
0.00/0.06	c lb -184 (thread 0)
0.00/0.06	c lb -183 (thread 0)
0.00/0.06	c lb -182 (thread 0)
0.00/0.06	c lb -181 (thread 0)
0.00/0.06	c lb -180 (thread 0)
0.00/0.07	c lb -179 (thread 0)
0.00/0.07	c lb -178 (thread 0)
0.00/0.07	c lb -177 (thread 0)
0.00/0.07	c lb -176 (thread 0)
0.00/0.07	c lb -175 (thread 0)
0.00/0.07	c lb -174 (thread 0)
0.00/0.07	c lb -173 (thread 0)
0.00/0.07	c lb -172 (thread 0)
0.00/0.07	c lb -171 (thread 0)
0.00/0.07	c lb -170 (thread 0)
0.00/0.08	c lb -169 (thread 0)
0.00/0.08	c lb -168 (thread 0)
0.00/0.08	c lb -167 (thread 0)
0.00/0.08	c lb -166 (thread 0)
0.00/0.08	c lb -165 (thread 0)
0.00/0.08	c lb -164 (thread 0)
0.00/0.08	c lb -163 (thread 0)
0.00/0.08	c lb -162 (thread 0)
0.00/0.09	c lb -161 (thread 0)
0.00/0.09	c lb -160 (thread 0)
0.00/0.09	c lb -159 (thread 0)
0.00/0.09	c lb -158 (thread 0)
0.00/0.09	c lb -157 (thread 0)
0.16/0.09	c lb -156 (thread 0)
0.17/0.10	c lb -155 (thread 0)
0.17/0.10	c lb -154 (thread 0)
0.17/0.10	c lb -153 (thread 0)
0.17/0.11	c lb -152 (thread 0)
0.17/0.14	c lb -151 (thread 0)
0.17/0.14	c lb -150 (thread 0)
0.17/0.16	c lb -149 (thread 0)
0.17/0.17	c lb -148 (thread 0)
0.17/0.17	c lb -147 (thread 0)
0.17/0.18	c lb -146 (thread 0)
0.17/0.18	c lb -145 (thread 0)
0.17/0.19	c lb -144 (thread 0)
0.17/0.19	c lb -143 (thread 0)
0.17/0.19	c lb -142 (thread 0)
0.17/0.19	c lb -141 (thread 0)
0.17/0.20	c lb -140 (thread 0)
0.36/0.20	c lb -139 (thread 0)
0.36/0.20	c lb -138 (thread 0)
0.36/0.20	c lb -137 (thread 0)
0.36/0.20	c lb -136 (thread 0)
0.36/0.21	c lb -135 (thread 0)
0.36/0.21	c lb -134 (thread 0)
0.36/0.21	c lb -133 (thread 0)
0.36/0.22	c lb -132 (thread 0)
0.36/0.22	c lb -131 (thread 0)
0.36/0.22	c ub 45 (thread 1)
0.36/0.23	o 45
0.36/0.23	c lb -130 (thread 0)
0.36/0.24	c lb -129 (thread 0)
0.36/0.24	c lb -128 (thread 0)
0.36/0.24	c lb -127 (thread 0)
0.36/0.24	c lb -126 (thread 0)
0.36/0.25	c lb -125 (thread 0)
0.36/0.25	c lb -124 (thread 0)
0.36/0.26	c lb -123 (thread 0)
0.36/0.27	c lb -122 (thread 0)
0.36/0.27	c lb -121 (thread 0)
0.36/0.29	c lb -120 (thread 0)
1.16/0.63	c lb -119 (thread 0)
1.16/0.64	c lb -118 (thread 0)
1.16/0.64	c lb -117 (thread 0)
1.36/0.79	c lb -116 (thread 0)
1.57/0.81	c lb -115 (thread 0)
1.77/0.97	c ub 29 (thread 1)
3.36/1.73	o 29
3.36/1.73	c lb -114 (thread 0)
3.36/1.78	c lb -113 (thread 0)
7.16/3.68	c ub 14 (thread 1)
21.56/10.89	o 14
21.56/10.89	c ub 13 (thread 1)
24.37/12.22	o 13
24.37/12.22	c ub 12 (thread 1)
39.57/19.84	o 12
39.57/19.84	c lb -112 (thread 0)
44.56/22.30	c lb -111 (thread 0)
175.77/87.94	c lb -110 (thread 0)
209.96/105.02	c ub 11 (thread 1)
1432.57/716.34	o 11
1432.57/716.34	c lb -109 (thread 0)
1800.20/900.11	s SATISFIABLE
1800.20/900.11	v x359 x235 x63 -x6 x341 x236 x400 x363 x340 -x240 -x7 x399 x361 -x239 -x8 x401 x342 -x237 -x144 -x11 x434 x362 x344 -x238 -x143 -x9 -x417 x366 -x145 -x10 x421 x345 -x435 x358 -x234 -x183 x62 x3 -x337 x233 -x187 -x5 x364 -x336 -x321 -x244 x4 -x12 x367 x343 x402 x365 x346 x416 -x410 x420 x406 -x146 x405 -x387 -x154 -x27 -x150 -x31 x356 -x317 -x252 -x247 -x182 x64 x360 -x256 -x248 x186 x2 x357 -x320 -x243 -x20 x368 -x338 -x16 -x407 -x339 -x241 x161 -x68 -x15 -x409 -x165 x418 -x383 x350 -x151 -x129 x422 -x153 x403 x386 -x302 -x26 -x149 -x30 -x316 -x274 -x251 -x245 x184 x65 -x17 x355 -x278 -x255 x188 -x19 x376 -x322 -x84 -x69 -x412 -x408 x372 -x88 -x67 -x411 x371 x353 -x242 -x190 x160 -x125 -x48 -x13 x354 -x191 -x164 -x152 x419 -x382 x349 -x325 -x298 -x128 -x14 x423 -x424 x404 x388 x347 -x301 -x28 -x425 -x147 -x32 x373 -x318 -x273 -x253 -x246 x185 x66 -x18 x375 -x277 -x257 x189 -x70 -x352 -x323 x193 x83 -x44 -x351 x192 -x87 -x378 x369 -x326 -x259 x162 x124 -x47 -x413 -x324 -x260 -x166 -x22 -x414 -x384 x370 -x297 x130 -x21 x415 -x429 x389 x348 -x303 x168 -x29 x169 -x148 -x33 x374 -x314 -x275 -x254 -x181 x78 -x319 -x279 -x258 x180 -x156 -x74 -x315 -x262 x197 -x155 -x120 x85 -x73 -x43 -x327 -x261 -x89 -x293 -x281 -x221 x163 x126 -x112 -x49 -x377 -x282 -x225 x167 -x432 -x379 -x299 x171 x131 -x91 -x433 -x385 x170 -x92 -x23 -x428 x381 -x304 -x211 x132 -x52 -x24 x390 x133 -x25 -x276 -x250 -x200 x80 x77 x39 -x313 x280 -x249 -x201 -x335 -x284 -x266 x196 -x108 x86 -x71 -x45 -x331 -x283 -x157 x119 -x90 -x431 -x330 -x220 x194 x158 x121 -x111 -x94 -x72 -x50 -x430 -x292 x224 x159 x127 -x93 -x294 -x207 x175 x123 -x53 x380 -x300 x134 -x51 -x426 x398 -x296 -x210 -x36 x394 -x305 -x37 -x332 -x272 -x269 -x198 x75 -x334 x271 -x270 x79 x38 -x288 -x265 -x107 x81 x40 x82 -x46 -x328 -x263 -x222 -x195 x178 -x113 -x98 x42 x226 x179 x122 -x54 -x395 -x329 -x206 x174 x142 -x35 -x397 -x295 x138 -x34 -x427 -x312 x228 -x212 -x172 x137 x116 x393 -x308 x229 -x333 x289 -x267 -x199 x105 -x76 x291 -x216 -x287 x215 x177 -x109 -x101 x176 -x102 x41 -x285 -x264 -x223 -x204 x139 -x114 -x97 x61 -x396 x227 x141 -x58 -x309 x231 -x208 x117 -x95 -x57 -x311 x230 x115 -x213 -x173 x135 x391 -x306 x290 -x268 -x100 x104 -x99 x103 x60 x217 -x140 -x110 x59 -x286 x218 x106 -x310 x219 -x203 x118 x232 -x202 -x96 -x55 -x209 -x205 x136 -x56 x392 -x307 -x214 x1
1800.20/900.11	c Memory used           : 1116.37 MB
1800.20/900.11	c CPU time              : 1782.18 s

Verifier Data 

OK	11

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /var/tmp/evaluation-result-3500304-1307993206/watcher-3500304-1307993206 -o /var/tmp/evaluation-result-3500304-1307993206/solver-3500304-1307993206 -C 1800 -W 1900 -M 15500 pwbo1.1 -file-format=opb HOME/instance-3500304-1307993206.opb 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.00 3.99 3.69 4/179 31717
/proc/meminfo: memFree=4122684/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=135552 CPUtime=0 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 695 0 0 0 0 0 0 0 24 0 3 0 910921547 138805248 643 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 33888 648 179 372 0 33513 0
[pid=31717/tid=31718] ppid=31715 vsize=135552 CPUtime=0 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 116 0 0 0 0 0 0 0 24 0 3 0 910921547 138805248 658 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=135552 CPUtime=0 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 187 0 0 0 0 0 0 0 18 0 3 0 910921547 138805248 661 33554432000 4194304 5714118 140735748110000 18446744073709551615 5013131 0 0 4096 16387 0 0 0 -1 5 0 0 0

[startup+0.0973909 s]
/proc/loadavg: 4.00 3.99 3.69 4/179 31717
/proc/meminfo: memFree=4122684/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=192664 CPUtime=0.16 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 16012 0 0 0 13 3 0 0 24 0 3 0 910921547 197287936 15706 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 48166 15709 180 372 0 47791 0
[pid=31717/tid=31718] ppid=31715 vsize=192664 CPUtime=0.08 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 525 0 0 0 8 0 0 0 25 0 3 0 910921547 197287936 15712 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=192664 CPUtime=0.07 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 15084 0 0 0 5 2 0 0 18 0 3 0 910921547 197287936 15714 33554432000 4194304 5714118 140735748110000 18446744073709551615 4970308 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 0.16
Current children cumulated vsize (KiB) 192664

[startup+0.100388 s]
/proc/loadavg: 4.00 3.99 3.69 4/179 31717
/proc/meminfo: memFree=4122684/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=192796 CPUtime=0.17 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 16303 0 0 0 14 3 0 0 24 0 3 0 910921547 197423104 15997 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 48199 15998 180 372 0 47824 0
[pid=31717/tid=31718] ppid=31715 vsize=192796 CPUtime=0.08 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 526 0 0 0 8 0 0 0 25 0 3 0 910921547 197423104 16000 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=192796 CPUtime=0.07 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 15372 0 0 0 5 2 0 0 18 0 3 0 910921547 197423104 16003 33554432000 4194304 5714118 140735748110000 18446744073709551615 4970308 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 192796

[startup+0.300366 s]
/proc/loadavg: 4.00 3.99 3.69 4/179 31717
/proc/meminfo: memFree=4122684/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=201036 CPUtime=0.56 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 29577 0 0 0 50 6 0 0 24 0 3 0 910921547 205860864 29200 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 50259 29200 181 372 0 49884 0
[pid=31717/tid=31718] ppid=31715 vsize=201036 CPUtime=0.29 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 953 0 0 0 26 3 0 0 18 0 3 0 910921547 205860864 29200 33554432000 4194304 5714118 140735748110000 18446744073709551615 4964730 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=201036 CPUtime=0.26 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 28213 0 0 0 23 3 0 0 18 0 3 0 910921547 205860864 29200 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 0.56
Current children cumulated vsize (KiB) 201036

[startup+0.700283 s]
/proc/loadavg: 4.00 3.99 3.69 4/179 31717
/proc/meminfo: memFree=4122684/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=201644 CPUtime=1.36 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 30324 0 0 0 126 10 0 0 24 0 3 0 910921547 206483456 29947 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 50411 29947 181 372 0 50036 0
[pid=31717/tid=31718] ppid=31715 vsize=201644 CPUtime=0.68 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 1578 0 0 0 62 6 0 0 18 0 3 0 910921547 206483456 29947 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=201644 CPUtime=0.66 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 28335 0 0 0 63 3 0 0 18 0 3 0 910921547 206483456 29947 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319195 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1.36
Current children cumulated vsize (KiB) 201644

[startup+1.50114 s]
/proc/loadavg: 4.00 3.99 3.69 5/182 31720
/proc/meminfo: memFree=4000296/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=202208 CPUtime=2.97 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 31505 0 0 0 282 15 0 0 24 0 3 0 910921547 207060992 31128 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 50552 31128 181 372 0 50177 0
[pid=31717/tid=31718] ppid=31715 vsize=202208 CPUtime=1.48 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 2002 0 0 0 137 11 0 0 19 0 3 0 910921547 207060992 31128 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=202208 CPUtime=1.46 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 29092 0 0 0 143 3 0 0 20 0 3 0 910921547 207060992 31128 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2.97
Current children cumulated vsize (KiB) 202208

[startup+3.10088 s]
/proc/loadavg: 4.00 3.99 3.69 5/182 31720
/proc/meminfo: memFree=3996100/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=202064 CPUtime=6.16 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 32156 0 0 0 594 22 0 0 24 0 3 0 910921547 206913536 31753 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 50516 31753 181 372 0 50141 0
[pid=31717/tid=31718] ppid=31715 vsize=202064 CPUtime=3.08 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 2571 0 0 0 290 18 0 0 24 0 3 0 910921547 206913536 31753 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=202064 CPUtime=3.07 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 29174 0 0 0 303 4 0 0 25 0 3 0 910921547 206913536 31753 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319200 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 6.16
Current children cumulated vsize (KiB) 202064

[startup+6.30029 s]
/proc/loadavg: 4.00 3.99 3.69 5/182 31720
/proc/meminfo: memFree=3988608/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=267816 CPUtime=12.56 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 34625 0 0 0 1222 34 0 0 24 0 3 0 910921547 274243584 34222 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 66954 34222 182 372 0 66579 0
[pid=31717/tid=31718] ppid=31715 vsize=267816 CPUtime=6.28 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 4235 0 0 0 600 28 0 0 25 0 3 0 910921547 274243584 34222 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=267816 CPUtime=6.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 29979 0 0 0 621 6 0 0 25 0 3 0 910921547 274243584 34222 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319139 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.56
Current children cumulated vsize (KiB) 267816

[startup+12.7012 s]
/proc/loadavg: 4.00 3.99 3.69 5/182 31720
/proc/meminfo: memFree=3972376/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=267980 CPUtime=25.37 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 39203 0 0 0 2480 57 0 0 24 0 3 0 910921547 274411520 38800 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 66995 38800 182 372 0 66620 0
[pid=31717/tid=31718] ppid=31715 vsize=267980 CPUtime=12.69 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 6892 0 0 0 1225 44 0 0 25 0 3 0 910921547 274411520 38800 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=267980 CPUtime=12.66 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 31900 0 0 0 1254 12 0 0 25 0 3 0 910921547 274411520 38800 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319218 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.37
Current children cumulated vsize (KiB) 267980

[startup+25.5009 s]
/proc/loadavg: 4.00 3.99 3.70 5/182 31720
/proc/meminfo: memFree=3942364/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=268028 CPUtime=50.96 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 45297 0 0 0 4996 100 0 0 24 0 3 0 910921547 274460672 44800 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 67007 44800 182 372 0 66632 0
[pid=31717/tid=31718] ppid=31715 vsize=268028 CPUtime=25.48 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 11058 0 0 0 2465 83 0 0 25 0 3 0 910921547 274460672 44800 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=268028 CPUtime=25.47 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 33828 0 0 0 2530 17 0 0 25 0 3 0 910921547 274460672 44800 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319134 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.96
Current children cumulated vsize (KiB) 268028

[startup+51.1004 s]
/proc/loadavg: 4.00 3.99 3.70 5/182 31721
/proc/meminfo: memFree=3921524/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=268436 CPUtime=102.16 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 50974 0 0 0 10051 165 0 0 24 0 3 0 910921547 274878464 50477 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 67109 50477 182 372 0 66734 0
[pid=31717/tid=31718] ppid=31715 vsize=268436 CPUtime=51.08 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 14304 0 0 0 4972 136 0 0 25 0 3 0 910921547 274878464 50477 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=268436 CPUtime=51.06 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 36259 0 0 0 5078 28 0 0 25 0 3 0 910921547 274878464 50477 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319293 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 102.16
Current children cumulated vsize (KiB) 268436

[startup+102.306 s]
/proc/loadavg: 4.00 3.99 3.72 5/182 31723
/proc/meminfo: memFree=3841748/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=335688 CPUtime=204.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 66705 0 0 0 20160 297 0 0 24 0 3 0 910921547 343744512 66147 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 83922 66148 182 372 0 83547 0
[pid=31717/tid=31718] ppid=31715 vsize=335688 CPUtime=102.28 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 24478 0 0 0 9980 248 0 0 25 0 3 0 910921547 343744512 66148 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319720 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=335688 CPUtime=102.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 41817 0 0 0 10179 48 0 0 25 0 3 0 910921547 343744512 66148 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 204.57
Current children cumulated vsize (KiB) 335688

[startup+162.301 s]
/proc/loadavg: 4.00 3.99 3.73 5/182 31765
/proc/meminfo: memFree=4015088/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=337168 CPUtime=324.56 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 68761 0 0 0 32051 405 0 0 24 0 3 0 910921547 345260032 68058 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 84292 68058 182 372 0 83917 0
[pid=31717/tid=31718] ppid=31715 vsize=337168 CPUtime=162.28 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 25940 0 0 0 15880 348 0 0 25 0 3 0 910921547 345260032 68058 33554432000 4194304 5714118 140735748110000 18446744073709551615 4199408 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=337168 CPUtime=162.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 42410 0 0 0 16170 57 0 0 25 0 3 0 910921547 345260032 68058 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319200 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 324.56
Current children cumulated vsize (KiB) 337168

[startup+222.301 s]
/proc/loadavg: 4.00 3.99 3.75 5/182 31767
/proc/meminfo: memFree=3873804/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=469636 CPUtime=444.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 93991 0 0 0 43939 518 0 0 24 0 3 0 910921547 480907264 93288 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 117409 93288 182 372 0 117034 0
[pid=31717/tid=31718] ppid=31715 vsize=469636 CPUtime=222.29 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 41277 0 0 0 21786 443 0 0 25 0 3 0 910921547 480907264 93288 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=469636 CPUtime=222.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 52303 0 0 0 22152 75 0 0 25 0 3 0 910921547 480907264 93288 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319195 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 444.57
Current children cumulated vsize (KiB) 469636

################
# More data... #
################

/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 54888 0 0 0 27696 533 0 0 25 0 3 0 910921547 550617088 113240 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319830 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=537712 CPUtime=282.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 58644 0 0 0 28141 86 0 0 25 0 3 0 910921547 550617088 113240 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319200 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 564.57
Current children cumulated vsize (KiB) 537712

[startup+342.301 s]
/proc/loadavg: 4.08 4.02 3.78 5/182 31810
/proc/meminfo: memFree=3706528/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=671032 CPUtime=684.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 133210 0 0 0 67743 714 0 0 24 0 3 0 910921547 687136768 132507 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 167758 132507 182 372 0 167383 0
[pid=31717/tid=31718] ppid=31715 vsize=671032 CPUtime=342.29 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 66976 0 0 0 33616 613 0 0 25 0 3 0 910921547 687136768 132507 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=671032 CPUtime=342.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 65823 0 0 0 34126 101 0 0 25 0 3 0 910921547 687136768 132507 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 684.57
Current children cumulated vsize (KiB) 671032

[startup+402.3 s]
/proc/loadavg: 4.03 4.01 3.79 5/182 31812
/proc/meminfo: memFree=3607776/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=673540 CPUtime=804.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 154747 0 0 0 79636 821 0 0 24 0 3 0 910921547 689704960 154044 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 168385 154044 182 372 0 168010 0
[pid=31717/tid=31718] ppid=31715 vsize=673540 CPUtime=402.28 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 79885 0 0 0 39526 702 0 0 25 0 3 0 910921547 689704960 154044 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 1 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=673540 CPUtime=402.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 74451 0 0 0 40109 118 0 0 25 0 3 0 910921547 689704960 154044 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319200 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 804.57
Current children cumulated vsize (KiB) 673540

[startup+462.301 s]
/proc/loadavg: 4.01 4.00 3.80 5/182 31813
/proc/meminfo: memFree=3387260/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=740792 CPUtime=924.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 171379 0 0 0 91546 911 0 0 24 0 3 0 910921547 758571008 170676 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 185198 170676 182 372 0 184823 0
[pid=31717/tid=31718] ppid=31715 vsize=740792 CPUtime=462.28 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 91284 0 0 0 45448 780 0 0 25 0 3 0 910921547 758571008 170676 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=740792 CPUtime=462.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 79684 0 0 0 46097 130 0 0 25 0 3 0 910921547 758571008 170676 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319134 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 924.57
Current children cumulated vsize (KiB) 740792

[startup+522.301 s]
/proc/loadavg: 4.00 4.00 3.82 5/182 31815
/proc/meminfo: memFree=3086268/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=872940 CPUtime=1044.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 188095 0 0 0 103429 1028 0 0 24 0 3 0 910921547 893890560 187392 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 218235 187392 182 372 0 217860 0
[pid=31717/tid=31718] ppid=31715 vsize=872940 CPUtime=522.28 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 103116 0 0 0 51347 881 0 0 25 0 3 0 910921547 893890560 187392 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=872940 CPUtime=522.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 84568 0 0 0 52081 146 0 0 25 0 3 0 910921547 893890560 187392 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319195 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1044.57
Current children cumulated vsize (KiB) 872940

[startup+582.3 s]
/proc/loadavg: 4.00 4.00 3.82 5/182 31816
/proc/meminfo: memFree=2888332/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=939544 CPUtime=1164.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 206904 0 0 0 115318 1139 0 0 24 0 3 0 910921547 962093056 206201 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 234886 206201 182 372 0 234511 0
[pid=31717/tid=31718] ppid=31715 vsize=939544 CPUtime=582.29 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 116247 0 0 0 57256 973 0 0 25 0 3 0 910921547 962093056 206201 33554432000 4194304 5714118 140735748110000 18446744073709551615 4299735 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=939544 CPUtime=582.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 90246 0 0 0 58061 166 0 0 25 0 3 0 910921547 962093056 206201 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319293 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1164.57
Current children cumulated vsize (KiB) 939544

[startup+642.301 s]
/proc/loadavg: 4.00 4.00 3.83 5/182 31818
/proc/meminfo: memFree=2805352/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=941296 CPUtime=1284.59 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 221056 0 0 0 127210 1249 0 0 24 0 3 0 910921547 963887104 220353 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 235324 220353 182 372 0 234949 0
[pid=31717/tid=31718] ppid=31715 vsize=941296 CPUtime=642.3 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 125655 0 0 0 63167 1063 0 0 25 0 3 0 910921547 963887104 220353 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=941296 CPUtime=642.28 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 94990 0 0 0 64042 186 0 0 25 0 3 0 910921547 963887104 220353 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1284.59
Current children cumulated vsize (KiB) 941296

[startup+702.301 s]
/proc/loadavg: 4.00 4.00 3.83 5/182 31820
/proc/meminfo: memFree=2704880/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=1073940 CPUtime=1404.57 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 236863 0 0 0 139085 1372 0 0 24 0 3 0 910921547 1099714560 236160 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 268485 236160 182 372 0 268110 0
[pid=31717/tid=31718] ppid=31715 vsize=1073940 CPUtime=702.29 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 135329 0 0 0 69062 1167 0 0 25 0 3 0 910921547 1099714560 236160 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=1073940 CPUtime=702.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 101123 0 0 0 70022 205 0 0 25 0 3 0 910921547 1099714560 236160 33554432000 4194304 5714118 140735748110000 18446744073709551615 4322628 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1404.57
Current children cumulated vsize (KiB) 1073940

[startup+762.306 s]
/proc/loadavg: 4.00 4.00 3.84 5/182 31821
/proc/meminfo: memFree=2618564/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=1074480 CPUtime=1524.59 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 248598 0 0 0 150952 1507 0 0 24 0 3 0 910921547 1100267520 247798 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 268620 247798 182 372 0 268245 0
[pid=31717/tid=31718] ppid=31715 vsize=1074480 CPUtime=762.3 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 141576 0 0 0 74949 1281 0 0 25 0 3 0 910921547 1100267520 247798 33554432000 4194304 5714118 140735748110000 18446744073709551615 4199408 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=1074480 CPUtime=762.28 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 106611 0 0 0 76002 226 0 0 25 0 3 0 910921547 1100267520 247798 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1524.59
Current children cumulated vsize (KiB) 1074480

[startup+822.301 s]
/proc/loadavg: 4.00 4.00 3.85 5/182 31823
/proc/meminfo: memFree=2569444/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=1075040 CPUtime=1644.58 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 255510 0 0 0 162821 1637 0 0 24 0 3 0 910921547 1100840960 254710 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 268760 254710 182 372 0 268385 0
[pid=31717/tid=31718] ppid=31715 vsize=1075040 CPUtime=822.29 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 141698 0 0 0 80849 1380 0 0 25 0 3 0 910921547 1100840960 254710 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=1075040 CPUtime=822.27 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 113401 0 0 0 81971 256 0 0 25 0 3 0 910921547 1100840960 254710 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319195 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1644.58
Current children cumulated vsize (KiB) 1075040

[startup+882.3 s]
/proc/loadavg: 4.07 4.02 3.86 5/182 31824
/proc/meminfo: memFree=2502704/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=1142692 CPUtime=1764.58 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 263484 0 0 0 174691 1767 0 0 24 0 3 0 910921547 1170116608 262684 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 285673 262684 182 372 0 285298 0
[pid=31717/tid=31718] ppid=31715 vsize=1142692 CPUtime=882.3 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 141767 0 0 0 86750 1480 0 0 25 0 3 0 910921547 1170116608 262684 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=1142692 CPUtime=882.28 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 121306 0 0 0 87941 287 0 0 25 0 3 0 910921547 1170116608 262684 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319200 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1764.58
Current children cumulated vsize (KiB) 1142692



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+900.101 s]
/proc/loadavg: 4.05 4.01 3.86 5/182 31825
/proc/meminfo: memFree=2482736/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=1143156 CPUtime=1800.2 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 265766 0 0 0 178217 1803 0 0 24 0 3 0 910921547 1170591744 264966 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 285789 264966 182 372 0 285414 0
[pid=31717/tid=31718] ppid=31715 vsize=1143156 CPUtime=900.1 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 141816 0 0 0 88504 1506 0 0 25 0 3 0 910921547 1170591744 264966 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=1143156 CPUtime=900.08 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 123539 0 0 0 89712 296 0 0 25 0 3 0 910921547 1170591744 264966 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319134 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1800.2
Current children cumulated vsize (KiB) 1143156

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+900.101 s]
/proc/loadavg: 4.05 4.01 3.86 5/182 31825
/proc/meminfo: memFree=2482736/32951124 swapFree=67052292/67111528
[pid=31717] ppid=31715 vsize=1143156 CPUtime=1800.2 cores=1,3,5,7
/proc/31717/stat : 31717 (pwbo1.1) S 31715 31717 30934 0 -1 4202496 265766 0 0 0 178217 1803 0 0 24 0 3 0 910921547 1170591744 264966 33554432000 4194304 5714118 140735748110000 18446744073709551615 4419421 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/31717/statm: 285789 264966 182 372 0 285414 0
[pid=31717/tid=31718] ppid=31715 vsize=1143156 CPUtime=900.1 cores=1,3,5,7
/proc/31717/task/31718/stat : 31718 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 141816 0 0 0 88504 1506 0 0 25 0 3 0 910921547 1170591744 264966 33554432000 4194304 5714118 140735748110000 18446744073709551615 5070743 0 0 4096 16387 0 0 0 -1 3 0 0 0
[pid=31717/tid=31719] ppid=31715 vsize=1143156 CPUtime=900.08 cores=1,3,5,7
/proc/31717/task/31719/stat : 31719 (pwbo1.1) R 31715 31717 30934 0 -1 4202560 123539 0 0 0 89712 296 0 0 25 0 3 0 910921547 1170591744 264966 33554432000 4194304 5714118 140735748110000 18446744073709551615 4319134 0 0 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1800.2
Current children cumulated vsize (KiB) 1143156

Child status: 0
Real time (s): 900.154
CPU time (s): 1800.26
CPU user time (s): 1782.18
CPU system time (s): 18.0753
CPU usage (%): 199.995
Max. virtual memory (cumulated for all children) (KiB): 1143156

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1782.18
system time used= 18.0753
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 265773
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 608
involuntary context switches= 10809

runsolver used 1.40279 second user time and 3.23451 second system time

The end

Launcher Data 

Begin job on node108 at 2011-06-13 21:26:46
IDJOB=3500304
IDBENCH=2225
IDSOLVER=1954
FILE ID=node108/3500304-1307993206
RUNJOBID= node108-1307991378-31184
PBS_JOBID= 13559746
Free space on /var/tmp= 2884 MiB

SOLVER NAME= pwbo 1.1
BENCH NAME= PB06//final/normalized-PB06/OPT-SMALLINT/web/www.ps.uni-sb.de/~walser/benchmarks/radar/normalized-10:10:4.5:0.95:100.opb
COMMAND LINE= pwbo1.1 -file-format=opb BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /var/tmp/evaluation-result-3500304-1307993206/watcher-3500304-1307993206 -o /var/tmp/evaluation-result-3500304-1307993206/solver-3500304-1307993206 -C 1800 -W 1900 -M 15500  pwbo1.1 -file-format=opb HOME/instance-3500304-1307993206.opb

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= b258fc1dc8cb1827e842a0b9558712c7
RANDOM SEED=1129985596

node108.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:       4122988 kB
Buffers:        134108 kB
Cached:       27393304 kB
SwapCached:      23260 kB
Active:       12247816 kB
Inactive:     15685136 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:       4122988 kB
SwapTotal:    67111528 kB
SwapFree:     67052292 kB
Dirty:            4292 kB
Writeback:           0 kB
AnonPages:      403120 kB
Mapped:          10016 kB
Slab:           829884 kB
PageTables:       5060 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   587492 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /var/tmp at the end= 2876 MiB
End job on node108 at 2011-06-13 21:41:48