Trace number 3378415

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 3351.17 843.52

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v90-c8010-S1475406203-080.UNKNOWN.cnf
MD5SUM6cadcf1602218f53da431eaffbc93148
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables90
Number of clauses8010
Sum of the clauses size56070
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58010

Solver Data 

0.31/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.31/0.47	c INFO: solving HOME/instance-3378415-1305465326.cnf
0.31/0.47	c INFO: computing task features
0.39/0.52	c INFO: computed variable-clause graph statistics
0.39/0.52	c INFO: computed clause balance statistics
0.70/0.81	c INFO: computed variable balance statistics
0.79/0.93	c INFO: computed small-clause counts
0.79/0.93	c INFO: computed Horn-clause counts
0.79/0.94	c INFO: computed variable graph statistics
0.79/0.94	c INFO: collected features for HOME/instance-3378415-1305465326.cnf in 0.47 s
0.79/0.95	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.47)
0.79/0.96	c INFO: running gnovelty+2@0 for 97 with 1199 remaining (b = 0.11)
0.79/0.97	c INFO: running adaptg2wsat2009++@0 for 97 with 1199 remaining (b = 0.06)
0.79/0.98	c INFO: running adaptg2wsat2009++@0 for 97 with 1199 remaining (b = 0.05)
396.77/100.36	c INFO: running adaptg2wsat2009++@0 for 97 with 1100 remaining (b = 0.04)
400.44/101.26	c INFO: running gnovelty+2@0 for 97 with 1099 remaining (b = 0.03)
400.73/101.33	c INFO: running FH@0 for 97 with 1099 remaining (b = 0.03)
400.73/101.34	c INFO: running adaptg2wsat2009++@0 for 97 with 1099 remaining (b = 0.02)
792.48/199.70	c INFO: running adaptg2wsat2009++@0 for 97 with 1000 remaining (b = 0.02)
800.42/201.62	c INFO: running FH@0 for 97 with 998 remaining (b = 0.02)
800.42/201.67	c INFO: running hybridGM3@0 for 194 with 998 remaining (b = 0.03)
800.42/201.69	c INFO: running gnovelty+2@0 for 97 with 998 remaining (b = 0.01)
1191.64/300.04	c INFO: running hybridGM3@0 for 194 with 900 remaining (b = 0.02)
1195.61/301.08	c INFO: running adaptg2wsat2009++@99 for 97 with 899 remaining (b = 0.01)
1199.18/301.96	c INFO: running hybridGM3@0 for 194 with 898 remaining (b = 0.02)
1582.78/398.42	c INFO: running FH@0 for 97 with 802 remaining (b = 0.01)
1602.25/403.38	c INFO: running hybridGM3@99 for 97 with 797 remaining (b = 0.01)
1973.57/496.72	c INFO: running hybridGM3@98 for 97 with 703 remaining (b = 0.01)
1981.12/498.70	c INFO: running hybridGM3@296 for 97 with 701 remaining (b = 0.01)
1981.50/498.77	c INFO: running march_hi@0 for 388 with 701 remaining (b = 0.04)
2004.95/504.65	c INFO: running hybridGM3@97 for 97 with 695 remaining (b = 0.01)
2352.33/592.04	c INFO: running hybridGM3@192 for 97 with 608 remaining (b = 0.01)
2383.72/599.97	c INFO: running hybridGM3@192 for 97 with 600 remaining (b = 0.01)
2384.12/600.04	c INFO: running adaptg2wsat2009++@0 for 97 with 600 remaining (b = 0.01)
2754.97/693.39	c INFO: running adaptg2wsat2009++@98 for 97 with 507 remaining (b = 0.01)
2770.92/697.39	c INFO: running hybridGM3@396 for 97 with 503 remaining (b = 0.01)
2786.75/701.32	c INFO: running hybridGM3@293 for 97 with 499 remaining (b = 0.01)
3158.00/794.73	c INFO: running march_hi@0 for 405 with 405 remaining (b = 0.03)
3162.36/795.81	c INFO: running gnovelty+2@0 for 97 with 404 remaining (b = 0.01)
3165.52/796.66	c INFO: running hybridGM3@391 for 97 with 403 remaining (b = 0.01)
3351.17/843.51	c INFO: running adaptg2wsat2009++@99 for 97 with 356 remaining (b = 0.01)
3351.17/843.51	Traceback (most recent call last):
3351.17/843.51	  File "HOME/solve", line 33, in <module>
3351.17/843.51	    plac.call(main)
3351.17/843.51	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
3351.17/843.51	    cmd, result = parser_from(obj).consume(arglist)
3351.17/843.51	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
3351.17/843.51	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
3351.17/843.51	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
3351.17/843.51	    (_, answer, _) = solver(input_path, budget / cores, cores)
3351.17/843.51	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
3351.17/843.51	    solver.go(max_cost * borg.defaults.machine_speed)
3351.17/843.51	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
3351.17/843.51	    self._process.start()
3351.17/843.51	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
3351.17/843.51	    assert self._popen is None, 'cannot start a process twice'
3351.17/843.51	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378415-1305465326/watcher-3378415-1305465326 -o /tmp/evaluation-result-3378415-1305465326/solver-3378415-1305465326 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378415-1305465326.cnf -seed 991091442 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.48 0.10 0.58 8/423 25451
/proc/meminfo: memFree=13719432/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=16240 CPUtime=0 cores=0,2,4,6
/proc/25451/stat : 25451 (python) R 25449 25451 25222 0 -1 4202496 1036 0 0 0 0 0 0 0 25 0 1 0 736763276 16629760 909 33554432000 4194304 5683644 140736249270960 18446744073709551615 216734839493 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25451/statm: 4060 909 380 364 0 547 0

[startup+0.022828 s]
/proc/loadavg: 0.48 0.10 0.58 8/423 25451
/proc/meminfo: memFree=13719432/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=16396 CPUtime=0 cores=0,2,4,6
/proc/25451/stat : 25451 (python) R 25449 25451 25222 0 -1 4202496 1120 0 0 0 0 0 0 0 25 0 1 0 736763276 16789504 989 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735176704 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25451/statm: 4099 989 413 364 0 586 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16396

[startup+0.100855 s]
/proc/loadavg: 0.48 0.10 0.58 8/423 25451
/proc/meminfo: memFree=13719432/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=104220 CPUtime=0.06 cores=0,2,4,6
/proc/25451/stat : 25451 (python) S 25449 25451 25222 0 -1 4202496 3044 0 0 0 4 2 0 0 17 0 1 0 736763276 106721280 2732 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735176704 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/25451/statm: 26055 2732 847 364 0 1803 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 104220

[startup+0.300773 s]
/proc/loadavg: 0.48 0.10 0.58 8/423 25451
/proc/meminfo: memFree=13719432/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=170756 CPUtime=0.23 cores=0,2,4,6
/proc/25451/stat : 25451 (python) R 25449 25451 25222 0 -1 4202496 6733 934 1 0 17 6 0 0 17 0 1 0 736763276 174854144 4806 33554432000 4194304 5683644 140736249270960 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25451/statm: 42688 4805 1336 364 0 3453 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 170756

[startup+0.700768 s]
/proc/loadavg: 0.48 0.10 0.58 8/423 25451
/proc/meminfo: memFree=13719432/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=280236 CPUtime=0.6 cores=0,2,4,6
/proc/25451/stat : 25451 (python) R 25449 25451 25222 0 -1 4202496 9998 934 1 0 52 8 0 0 17 0 1 0 736763276 286961664 7444 33554432000 4194304 5683644 140736249270960 18446744073709551615 47654034453546 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/25451/statm: 70059 7444 1826 364 0 5983 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 280236

[startup+1.5006 s]
/proc/loadavg: 0.48 0.10 0.58 6/424 25496
/proc/meminfo: memFree=13710800/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=320748 CPUtime=0.87 cores=0,2,4,6
/proc/25451/stat : 25451 (python) S 25449 25451 25222 0 -1 4202496 12938 934 1 0 78 9 0 0 19 0 5 0 736763276 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/25451/statm: 80187 7386 1883 364 0 16096 0
[pid=25485] ppid=25451 vsize=279416 CPUtime=0 cores=0,2,4,6
/proc/25485/stat : 25485 (python) S 25451 25451 25222 0 -1 4202560 727 0 0 0 0 0 0 0 19 0 1 0 736763371 286121984 5793 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198735 0 0 16781312 514 2447514262113878015 0 0 17 4 0 0 0
/proc/25485/statm: 69854 5793 354 364 0 5772 0
[pid=25487] ppid=25485 vsize=1018088 CPUtime=0.52 cores=0,2,4,6
/proc/25487/stat : 25487 (gnovelty+2) R 25485 25487 25487 0 -1 4202496 564 0 0 0 52 0 0 0 22 0 1 0 736763371 1042522112 305 33554432000 134512640 135663704 4293623344 18446744073709551615 134524553 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/25487/statm: 254522 305 113 282 0 254238 0
[pid=25488] ppid=25451 vsize=289992 CPUtime=0 cores=0,2,4,6
/proc/25488/stat : 25488 (python) S 25451 25451 25222 0 -1 4202560 732 0 0 0 0 0 0 0 21 0 1 0 736763372 296951808 5839 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 16025663179228971007 0 0 17 6 0 0 0
/proc/25488/statm: 72498 5839 348 364 0 8413 0
[pid=25490] ppid=25488 vsize=1018088 CPUtime=0.53 cores=0,2,4,6
/proc/25490/stat : 25490 (gnovelty+2) R 25488 25490 25490 0 -1 4202496 569 0 0 0 53 0 0 0 25 0 1 0 736763372 1042522112 306 33554432000 134512640 135663704 4289781856 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25490/statm: 254522 306 113 282 0 254238 0
[pid=25491] ppid=25451 vsize=300248 CPUtime=0 cores=0,2,4,6
/proc/25491/stat : 25491 (python) S 25451 25451 25222 0 -1 4202560 719 0 0 0 0 0 0 0 23 0 1 0 736763373 307453952 5839 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 15144079154124685311 0 0 17 4 0 0 0
/proc/25491/statm: 75062 5839 344 364 0 10974 0
[pid=25493] ppid=25491 vsize=417048 CPUtime=0.51 cores=0,2,4,6
/proc/25493/stat : 25493 (adaptg2wsat2009) R 25491 25493 25493 0 -1 4202496 681 0 0 0 51 0 0 0 25 0 1 0 736763373 427057152 418 33554432000 134512640 135107411 4291574112 18446744073709551615 134523296 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/25493/statm: 104262 418 49 146 0 103138 0
[pid=25494] ppid=25451 vsize=310504 CPUtime=0 cores=0,2,4,6
/proc/25494/stat : 25494 (python) S 25451 25451 25222 0 -1 4202560 732 0 0 0 0 0 0 0 20 0 1 0 736763374 317956096 5843 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 16025655482647576575 0 0 17 6 0 0 0
/proc/25494/statm: 77626 5843 344 364 0 13535 0
[pid=25496] ppid=25494 vsize=417048 CPUtime=0.51 cores=0,2,4,6
/proc/25496/stat : 25496 (adaptg2wsat2009) R 25494 25496 25496 0 -1 4202496 682 0 0 0 51 0 0 0 23 0 1 0 736763374 427057152 418 33554432000 134512640 135107411 4293081744 18446744073709551615 134523296 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25496/statm: 104262 418 49 146 0 103138 0
[pid=25451/tid=25486] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25486/stat : 25486 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 5 0 736763371 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25489] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25489/stat : 25489 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 5 0 736763372 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25492] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25492/stat : 25492 (python) S 25449 25451 25222 0 -1 4202560 22 934 0 0 0 0 0 0 24 0 5 0 736763373 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25495] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25495/stat : 25495 (python) S 25449 25451 25222 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 5 0 736763374 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2.94
Current children cumulated vsize (KiB) 4371180

[startup+3.10131 s]
/proc/loadavg: 0.48 0.10 0.58 6/420 25535
/proc/meminfo: memFree=13720132/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=320748 CPUtime=0.87 cores=0,2,4,6
/proc/25451/stat : 25451 (python) S 25449 25451 25222 0 -1 4202496 12938 934 1 0 78 9 0 0 19 0 5 0 736763276 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/25451/statm: 80187 7386 1883 364 0 16096 0
[pid=25485] ppid=25451 vsize=279416 CPUtime=0 cores=0,2,4,6
/proc/25485/stat : 25485 (python) S 25451 25451 25222 0 -1 4202560 727 0 0 0 0 0 0 0 18 0 1 0 736763371 286121984 5793 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198735 0 0 16781312 514 2447514262113845508 0 0 17 4 0 0 0
/proc/25485/statm: 69854 5793 354 364 0 5772 0
[pid=25487] ppid=25485 vsize=1018088 CPUtime=2.11 cores=0,2,4,6
/proc/25487/stat : 25487 (gnovelty+2) R 25485 25487 25487 0 -1 4202496 564 0 0 0 211 0 0 0 25 0 1 0 736763371 1042522112 305 33554432000 134512640 135663704 4293623344 18446744073709551615 134525224 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/25487/statm: 254522 305 113 282 0 254238 0
[pid=25488] ppid=25451 vsize=289992 CPUtime=0 cores=0,2,4,6
/proc/25488/stat : 25488 (python) S 25451 25451 25222 0 -1 4202560 732 0 0 0 0 0 0 0 18 0 1 0 736763372 296951808 5839 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 16025663179228938498 0 0 17 6 0 0 0
/proc/25488/statm: 72498 5839 348 364 0 8413 0
[pid=25490] ppid=25488 vsize=1018088 CPUtime=2.13 cores=0,2,4,6
/proc/25490/stat : 25490 (gnovelty+2) R 25488 25490 25490 0 -1 4202496 569 0 0 0 213 0 0 0 25 0 1 0 736763372 1042522112 306 33554432000 134512640 135663704 4289781856 18446744073709551615 134525260 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25490/statm: 254522 306 113 282 0 254238 0
[pid=25491] ppid=25451 vsize=300248 CPUtime=0 cores=0,2,4,6
/proc/25491/stat : 25491 (python) S 25451 25451 25222 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 736763373 307453952 5843 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 15144079154124652803 0 0 17 4 0 0 0
/proc/25491/statm: 75062 5843 348 364 0 10974 0
[pid=25493] ppid=25491 vsize=417192 CPUtime=2.1 cores=0,2,4,6
/proc/25493/stat : 25493 (adaptg2wsat2009) R 25491 25493 25493 0 -1 4202496 757 0 0 0 209 1 0 0 25 0 1 0 736763373 427204608 494 33554432000 134512640 135107411 4291574112 18446744073709551615 134532318 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/25493/statm: 104298 494 59 146 0 103174 0
[pid=25494] ppid=25451 vsize=310504 CPUtime=0 cores=0,2,4,6
/proc/25494/stat : 25494 (python) S 25451 25451 25222 0 -1 4202560 751 0 0 0 0 0 0 0 18 0 1 0 736763374 317956096 5847 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 16025655482647544067 0 0 17 6 0 0 0
/proc/25494/statm: 77626 5847 348 364 0 13535 0
[pid=25496] ppid=25494 vsize=417192 CPUtime=2.11 cores=0,2,4,6
/proc/25496/stat : 25496 (adaptg2wsat2009) R 25494 25496 25496 0 -1 4202496 758 0 0 0 211 0 0 0 25 0 1 0 736763374 427204608 494 33554432000 134512640 135107411 4293081744 18446744073709551615 134531792 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25496/statm: 104298 494 59 146 0 103174 0
[pid=25451/tid=25486] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25486/stat : 25486 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 5 0 736763371 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25489] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25489/stat : 25489 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 5 0 736763372 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25492] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25492/stat : 25492 (python) S 25449 25451 25222 0 -1 4202560 22 934 0 0 0 0 0 0 24 0 5 0 736763373 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25495] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25495/stat : 25495 (python) S 25449 25451 25222 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 5 0 736763374 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 9.32
Current children cumulated vsize (KiB) 4371468

[startup+6.30072 s]
/proc/loadavg: 0.84 0.18 0.60 9/436 25600
/proc/meminfo: memFree=13672812/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=320748 CPUtime=0.87 cores=0,2,4,6
/proc/25451/stat : 25451 (python) S 25449 25451 25222 0 -1 4202496 12938 934 1 0 78 9 0 0 19 0 5 0 736763276 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/25451/statm: 80187 7386 1883 364 0 16096 0
[pid=25485] ppid=25451 vsize=279416 CPUtime=0 cores=0,2,4,6
/proc/25485/stat : 25485 (python) S 25451 25451 25222 0 -1 4202560 891 0 0 0 0 0 0 0 18 0 1 0 736763371 286121984 5816 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198735 0 0 16781312 514 2447514262113845508 0 0 17 4 0 0 0
/proc/25485/statm: 69854 5816 377 364 0 5772 0
[pid=25487] ppid=25485 vsize=1018088 CPUtime=5.23 cores=0,2,4,6
/proc/25487/stat : 25487 (gnovelty+2) R 25485 25487 25487 0 -1 4202496 564 0 0 0 523 0 0 0 25 0 1 0 736763371 1042522112 305 33554432000 134512640 135663704 4293623344 18446744073709551615 134524882 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/25487/statm: 254522 305 113 282 0 254238 0
[pid=25488] ppid=25451 vsize=289992 CPUtime=0.01 cores=0,2,4,6
/proc/25488/stat : 25488 (python) S 25451 25451 25222 0 -1 4202560 892 0 0 0 0 1 0 0 18 0 1 0 736763372 296951808 5862 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 16025663179228938498 0 0 17 6 0 0 0
/proc/25488/statm: 72498 5862 371 364 0 8413 0
[pid=25490] ppid=25488 vsize=1018088 CPUtime=5.33 cores=0,2,4,6
/proc/25490/stat : 25490 (gnovelty+2) R 25488 25490 25490 0 -1 4202496 569 0 0 0 533 0 0 0 25 0 1 0 736763372 1042522112 306 33554432000 134512640 135663704 4289781856 18446744073709551615 134525678 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25490/statm: 254522 306 113 282 0 254238 0
[pid=25491] ppid=25451 vsize=300248 CPUtime=0.01 cores=0,2,4,6
/proc/25491/stat : 25491 (python) S 25451 25451 25222 0 -1 4202560 900 0 0 0 1 0 0 0 18 0 1 0 736763373 307453952 5866 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 15144079154124652803 0 0 17 4 0 0 0
/proc/25491/statm: 75062 5866 371 364 0 10974 0
[pid=25493] ppid=25491 vsize=417192 CPUtime=5.26 cores=0,2,4,6
/proc/25493/stat : 25493 (adaptg2wsat2009) R 25491 25493 25493 0 -1 4202496 757 0 0 0 525 1 0 0 25 0 1 0 736763373 427204608 494 33554432000 134512640 135107411 4291574112 18446744073709551615 134531232 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/25493/statm: 104298 494 59 146 0 103174 0
[pid=25494] ppid=25451 vsize=310504 CPUtime=0.01 cores=0,2,4,6
/proc/25494/stat : 25494 (python) S 25451 25451 25222 0 -1 4202560 911 0 0 0 1 0 0 0 18 0 1 0 736763374 317956096 5870 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 16025655482647544067 0 0 17 6 0 0 0
/proc/25494/statm: 77626 5870 371 364 0 13535 0
[pid=25496] ppid=25494 vsize=417192 CPUtime=5.31 cores=0,2,4,6
/proc/25496/stat : 25496 (adaptg2wsat2009) R 25494 25496 25496 0 -1 4202496 758 0 0 0 531 0 0 0 25 0 1 0 736763374 427204608 494 33554432000 134512640 135107411 4293081744 18446744073709551615 134531428 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25496/statm: 104298 494 59 146 0 103174 0
[pid=25451/tid=25486] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25486/stat : 25486 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 5 0 736763371 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25489] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25489/stat : 25489 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 5 0 736763372 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25492] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25492/stat : 25492 (python) S 25449 25451 25222 0 -1 4202560 22 934 0 0 0 0 0 0 24 0 5 0 736763373 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25495] ppid=25449 vsize=320748 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25495/stat : 25495 (python) S 25449 25451 25222 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 5 0 736763374 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22.03
Current children cumulated vsize (KiB) 4371468

[startup+12.7006 s]
/proc/loadavg: 1.50 0.33 0.64 9/436 25600
/proc/meminfo: memFree=13670472/32951124 swapFree=65586204/67111528
[pid=25451] ppid=25449 vsize=320748 CPUtime=0.87 cores=0,2,4,6
/proc/25451/stat : 25451 (python) S 25449 25451 25222 0 -1 4202496 12938 934 1 0 78 9 0 0 19 0 5 0 736763276 328445952 7386 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/25451/statm: 80187 7386 1883 364 0 16096 0
[pid=25485] ppid=25451 vsize=279416 CPUtime=0.02 cores=0,2,4,6
/proc/25485/stat : 25485 (python) S 25451 25451 25222 0 -1 4202560 894 0 0 0 1 1 0 0 18 0 1 0 736763371 286121984 5816 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198735 0 0 16781312 514 6646785284417487108 0 0 17 4 0 0 0
/proc/25485/statm: 69854 5816 377 364 0 5772 0
[pid=25487] ppid=25485 vsize=1018088 CPUtime=11.5 cores=0,2,4,6
/proc/25487/stat : 25487 (gnovelty+2) R 25485 25487 25487 0 -1 4202496 564 0 0 0 1150 0 0 0 25 0 1 0 736763371 1042522112 305 33554432000 134512640 135663704 4293623344 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/25487/statm: 254522 305 113 282 0 254238 0
[pid=25488] ppid=25451 vsize=289992 CPUtime=0.02 cores=0,2,4,6

################
# More data... #
################

/proc/25494/stat : 25494 (python) S 25451 25451 25222 0 -1 4202560 1134 0 0 0 36 29 0 0 18 0 2 0 736763374 317956096 5900 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/25494/statm: 77626 5900 400 364 0 13535 0
[pid=25496] ppid=25494 vsize=417192 CPUtime=196.68 cores=0,2,4,6
/proc/25496/stat : 25496 (adaptg2wsat2009) T 25494 25496 25496 0 -1 4202496 758 0 0 0 19668 0 0 0 25 0 1 0 736763374 427204608 494 33554432000 134512640 135107411 4293081744 18446744073709551615 134531795 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/25496/statm: 104298 494 59 146 0 103174 0
[pid=25494/tid=25603] ppid=25451 vsize=310504 CPUtime=0 cores=0,2,4,6
/proc/25494/task/25603/stat : 25603 (python) S 25451 25451 25222 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 736773311 317956096 5900 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604452348989632 0 0 -1 6 0 0 0
[pid=25604] ppid=25451 vsize=320760 CPUtime=0.7 cores=0,2,4,6
/proc/25604/stat : 25604 (python) S 25451 25451 25222 0 -1 4202560 1136 0 0 0 36 34 0 0 18 0 2 0 736773312 328458240 5904 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/25604/statm: 80190 5904 400 364 0 16096 0
[pid=25606] ppid=25604 vsize=417192 CPUtime=198.47 cores=0,2,4,6
/proc/25606/stat : 25606 (adaptg2wsat2009) T 25604 25606 25606 0 -1 4202496 758 0 0 0 19847 0 0 0 25 0 1 0 736773313 427204608 495 33554432000 134512640 135107411 4291224848 18446744073709551615 134532306 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/25606/statm: 104298 495 59 146 0 103174 0
[pid=25604/tid=25638] ppid=25451 vsize=320760 CPUtime=0 cores=0,2,4,6
/proc/25604/task/25638/stat : 25638 (python) S 25451 25451 25222 0 -1 4202560 49 0 0 0 0 0 0 0 18 0 2 0 736783245 328458240 5904 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604444314932224 0 0 -1 6 0 0 0
[pid=25608] ppid=25451 vsize=331016 CPUtime=0.32 cores=0,2,4,6
/proc/25608/stat : 25608 (python) S 25451 25451 25222 0 -1 4202560 1102 0 0 0 16 16 0 0 19 0 2 0 736773403 338960384 5908 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/25608/statm: 82754 5908 400 364 0 18657 0
[pid=25610] ppid=25608 vsize=1018088 CPUtime=100.33 cores=0,2,4,6
/proc/25610/stat : 25610 (gnovelty+2) T 25608 25610 25610 0 -1 4202496 569 0 0 0 10033 0 0 0 25 0 1 0 736773403 1042522112 305 33554432000 134512640 135663704 4291027184 18446744073709551615 134525255 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/25610/statm: 254522 305 113 282 0 254238 0
[pid=25608/tid=25642] ppid=25451 vsize=331016 CPUtime=0 cores=0,2,4,6
/proc/25608/task/25642/stat : 25642 (python) S 25451 25451 25222 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 736783437 338960384 5908 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604444314934016 0 0 -1 6 0 0 0
[pid=25613] ppid=25451 vsize=341272 CPUtime=0.32 cores=0,2,4,6
/proc/25613/stat : 25613 (python) S 25451 25451 25222 0 -1 4202560 1099 0 0 0 16 16 0 0 18 0 2 0 736773409 349462528 5912 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25613/statm: 85318 5912 400 364 0 21218 0
[pid=25615] ppid=25613 vsize=247344 CPUtime=98.29 cores=0,2,4,6
/proc/25615/stat : 25615 (FH) T 25613 25615 25615 0 -1 4202496 705 0 0 0 9829 0 0 0 25 0 1 0 736773409 253280256 443 33554432000 134512640 134936988 4292677952 18446744073709551615 134554229 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/25615/statm: 61836 443 41 104 0 61340 0
[pid=25613/tid=25646] ppid=25451 vsize=341272 CPUtime=0 cores=0,2,4,6
/proc/25613/task/25646/stat : 25646 (python) S 25451 25451 25222 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 736783442 349462528 5912 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604446290776512 0 0 -1 2 0 0 0
[pid=25616] ppid=25451 vsize=351528 CPUtime=0.32 cores=0,2,4,6
/proc/25616/stat : 25616 (python) S 25451 25451 25222 0 -1 4202560 1142 0 0 0 18 14 0 0 19 0 2 0 736773410 359964672 5922 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/25616/statm: 87882 5922 406 364 0 23779 0
[pid=25618] ppid=25616 vsize=417192 CPUtime=99.64 cores=0,2,4,6
/proc/25618/stat : 25618 (adaptg2wsat2009) T 25616 25618 25618 0 -1 4202496 763 0 0 0 9964 0 0 0 25 0 1 0 736773411 427204608 495 33554432000 134512640 135107411 4291861792 18446744073709551615 134531720 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/25618/statm: 104298 495 59 146 0 103174 0
[pid=25616/tid=25650] ppid=25451 vsize=351528 CPUtime=0 cores=0,2,4,6
/proc/25616/task/25650/stat : 25650 (python) S 25451 25451 25222 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 736783444 359964672 5922 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604452348990528 0 0 -1 4 0 0 0
[pid=25639] ppid=25451 vsize=361784 CPUtime=0.33 cores=0,2,4,6
/proc/25639/stat : 25639 (python) S 25451 25451 25222 0 -1 4202560 1149 0 0 0 16 17 0 0 19 0 2 0 736783246 370466816 5926 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/25639/statm: 90446 5926 406 364 0 26340 0
[pid=25641] ppid=25639 vsize=417192 CPUtime=100.28 cores=0,2,4,6
/proc/25641/stat : 25641 (adaptg2wsat2009) T 25639 25641 25641 0 -1 4202496 764 0 0 0 10028 0 0 0 25 0 1 0 736783246 427204608 496 33554432000 134512640 135107411 4294862864 18446744073709551615 134531781 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/25641/statm: 104298 496 59 146 0 103174 0
[pid=25639/tid=25672] ppid=25451 vsize=361784 CPUtime=0 cores=0,2,4,6
/proc/25639/task/25672/stat : 25672 (python) S 25451 25451 25222 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 736793279 370466816 5926 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604443947234432 0 0 -1 4 0 0 0
[pid=25643] ppid=25451 vsize=372040 CPUtime=0.33 cores=0,2,4,6
/proc/25643/stat : 25643 (python) S 25451 25451 25222 0 -1 4202560 1111 0 0 0 15 18 0 0 19 0 2 0 736783438 380968960 5950 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/25643/statm: 93010 5950 400 364 0 28901 0
[pid=25645] ppid=25643 vsize=247348 CPUtime=99.63 cores=0,2,4,6
/proc/25645/stat : 25645 (FH) T 25643 25645 25645 0 -1 4202496 710 0 0 0 9963 0 0 0 25 0 1 0 736783439 253284352 444 33554432000 134512640 134936988 4287594384 18446744073709551615 134553524 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/25645/statm: 61837 444 41 104 0 61341 0
[pid=25643/tid=25677] ppid=25451 vsize=372040 CPUtime=0 cores=0,2,4,6
/proc/25643/task/25677/stat : 25677 (python) S 25451 25451 25222 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 736793471 380968960 5950 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604443947235328 0 0 -1 4 0 0 0
[pid=25647] ppid=25451 vsize=382296 CPUtime=1.69 cores=0,2,4,6
/proc/25647/stat : 25647 (python) S 25451 25451 25222 0 -1 4202560 1174 0 0 0 89 80 0 0 18 0 2 0 736783443 391471104 5954 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25647/statm: 95574 5954 400 364 0 31462 0
[pid=25649] ppid=25647 vsize=268420 CPUtime=491.08 cores=0,2,4,6
/proc/25649/stat : 25649 (hybridGM3) T 25647 25649 25649 0 -1 4202496 4938 0 0 0 49108 0 0 0 25 0 1 0 736783443 274862080 4671 33554432000 134512640 135204385 4286811264 18446744073709551615 134537338 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/25649/statm: 67105 4671 64 169 0 66935 0
[pid=25647/tid=25695] ppid=25451 vsize=382296 CPUtime=0 cores=0,2,4,6
/proc/25647/task/25695/stat : 25695 (python) S 25451 25451 25222 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 736803613 391471104 5954 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604442396425216 0 0 -1 2 0 0 0
[pid=25651] ppid=25451 vsize=392552 CPUtime=0.32 cores=0,2,4,6
/proc/25651/stat : 25651 (python) S 25451 25451 25222 0 -1 4202560 1138 0 0 0 17 15 0 0 19 0 2 0 736783445 401973248 5962 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25651/statm: 98138 5962 400 364 0 34023 0
[pid=25653] ppid=25651 vsize=1018088 CPUtime=98.74 cores=0,2,4,6
/proc/25653/stat : 25653 (gnovelty+2) T 25651 25653 25653 0 -1 4202496 571 0 0 0 9874 0 0 0 25 0 1 0 736783445 1042522112 305 33554432000 134512640 135663704 4287422400 18446744073709551615 134525255 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/25653/statm: 254522 305 113 282 0 254238 0
[pid=25651/tid=25676] ppid=25451 vsize=392552 CPUtime=0 cores=0,2,4,6
/proc/25651/task/25676/stat : 25676 (python) S 25451 25451 25222 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 736793383 401973248 5962 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604442396423424 0 0 -1 2 0 0 0
[pid=25673] ppid=25451 vsize=402808 CPUtime=1.34 cores=0,2,4,6
/proc/25673/stat : 25673 (python) S 25451 25451 25222 0 -1 4202560 1169 0 0 0 68 66 0 0 18 0 2 0 736793280 412475392 5966 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25673/statm: 100702 5966 400 364 0 36584 0
[pid=25675] ppid=25673 vsize=268424 CPUtime=392.49 cores=0,2,4,6
/proc/25675/stat : 25675 (hybridGM3) T 25673 25675 25675 0 -1 4202496 4939 0 0 0 39249 0 0 0 25 0 1 0 736793280 274866176 4673 33554432000 134512640 135204385 4292458608 18446744073709551615 134537290 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/25675/statm: 67106 4673 64 169 0 66936 0
[pid=25673/tid=25704] ppid=25451 vsize=402808 CPUtime=0 cores=0,2,4,6
/proc/25673/task/25704/stat : 25704 (python) S 25451 25451 25222 0 -1 4202560 45 0 0 0 0 0 0 0 18 0 2 0 736812947 412475392 5966 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604453629039808 0 0 -1 2 0 0 0
[pid=25678] ppid=25451 vsize=413332 CPUtime=1.84 cores=0,2,4,6
/proc/25678/stat : 25678 (python) S 25451 25451 25222 0 -1 4202560 1176 0 0 0 94 90 0 0 18 0 2 0 736793472 423251968 6007 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25678/statm: 103333 6007 400 364 0 39212 0
[pid=25680] ppid=25678 vsize=268420 CPUtime=534.13 cores=0,2,4,6
/proc/25680/stat : 25680 (hybridGM3) R 25678 25680 25680 0 -1 4202496 4939 0 0 0 53412 1 0 0 25 0 1 0 736793473 274862080 4671 33554432000 134512640 135204385 4293612224 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/25680/statm: 67105 4671 64 169 0 66935 0
[pid=25678/tid=25712] ppid=25451 vsize=413332 CPUtime=0 cores=0,2,4,6
/proc/25678/task/25712/stat : 25712 (python) S 25451 25451 25222 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 736813740 423251968 6007 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604440928343296 0 0 -1 2 0 0 0
[pid=25692] ppid=25451 vsize=423588 CPUtime=0.34 cores=0,2,4,6
/proc/25692/stat : 25692 (python) S 25451 25451 25222 0 -1 4202560 1199 0 0 0 19 15 0 0 19 0 2 0 736803118 433754112 6018 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747005387 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/25692/statm: 105897 6018 407 364 0 41773 0
[pid=25694] ppid=25692 vsize=247344 CPUtime=100.23 cores=0,2,4,6
/proc/25694/stat : 25694 (FH) T 25692 25694 25694 0 -1 4202496 720 0 0 0 10023 0 0 0 25 0 1 0 736803119 253280256 443 33554432000 134512640 134936988 4294496896 18446744073709551615 134554064 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/25694/statm: 61836 443 41 104 0 61340 0
[pid=25692/tid=25705] ppid=25451 vsize=423588 CPUtime=0 cores=0,2,4,6
/proc/25692/task/25705/stat : 25705 (python) S 25451 25451 25222 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 736813152 433754112 6018 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 514 18446604444473771968 0 0 -1 0 0 0 0
[pid=25706] ppid=25451 vsize=434040 CPUtime=1.22 cores=0,2,4,6
/proc/25706/stat : 25706 (python) S 25451 25451 25222 0 -1 4202560 1055 0 0 0 69 53 0 0 15 0 1 0 736813153 444456960 6041 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 6646785284417487105 0 0 17 6 0 0 0
/proc/25706/statm: 108510 6041 378 364 0 44383 0
[pid=25708] ppid=25706 vsize=63264 CPUtime=341.49 cores=0,2,4,6
/proc/25708/stat : 25708 (march_hi) R 25706 25708 25708 0 -1 4202496 12085 0 0 0 34147 2 0 0 25 0 1 0 736813153 64782336 10530 33554432000 4194304 4955518 140737201117872 18446744073709551615 4297477 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/25708/statm: 15816 10530 106 186 0 15629 0
[pid=25726] ppid=25451 vsize=444484 CPUtime=0.16 cores=0,2,4,6
/proc/25726/stat : 25726 (python) S 25451 25451 25222 0 -1 4202560 1037 0 0 0 8 8 0 0 15 0 1 0 736842749 455151616 6093 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 11604179753145172225 0 0 17 0 0 0 0
/proc/25726/statm: 111121 6093 379 364 0 46991 0
[pid=25728] ppid=25726 vsize=10968 CPUtime=46.87 cores=0,2,4,6
/proc/25728/stat : 25728 (march_hi) R 25726 25728 25728 0 -1 4202496 3534 0 0 0 4687 0 0 0 25 0 1 0 736842750 11231232 1973 33554432000 4194304 4955518 140736203998016 18446744073709551615 4224468 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/25728/statm: 2742 1973 106 186 0 2555 0
[pid=25729] ppid=25451 vsize=454740 CPUtime=0.16 cores=0,2,4,6
/proc/25729/stat : 25729 (python) S 25451 25451 25222 0 -1 4202560 1000 0 0 0 8 8 0 0 18 0 1 0 736842857 465653760 6090 33554432000 4194304 5683644 140736249270960 18446744073709551615 216735198790 0 0 16781312 514 6646785284417487107 0 0 17 0 0 0 0
/proc/25729/statm: 113685 6090 372 364 0 49552 0
[pid=25731] ppid=25729 vsize=1018088 CPUtime=47.43 cores=0,2,4,6
/proc/25731/stat : 25731 (gnovelty+2) R 25729 25731 25731 0 -1 4202496 581 0 0 0 4743 0 0 0 25 0 1 0 736842858 1042522112 306 33554432000 134512640 135663704 4291404864 18446744073709551615 134524843 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/25731/statm: 254522 306 113 282 0 254238 0
[pid=25451/tid=25486] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25486/stat : 25486 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 19 0 736763371 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25489] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25489/stat : 25489 (python) S 25449 25451 25222 0 -1 4202560 25 934 0 0 0 0 0 0 22 0 19 0 736763372 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25492] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25492/stat : 25492 (python) S 25449 25451 25222 0 -1 4202560 26 934 0 0 0 0 0 0 18 0 19 0 736763373 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25495] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25495/stat : 25495 (python) S 25449 25451 25222 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 19 0 736763374 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25605] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25605/stat : 25605 (python) S 25449 25451 25222 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 19 0 736773312 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25609] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25609/stat : 25609 (python) S 25449 25451 25222 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 19 0 736773403 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25614] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25614/stat : 25614 (python) S 25449 25451 25222 0 -1 4202560 24 934 0 0 0 0 0 0 21 0 19 0 736773409 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 6 0 0 0
[pid=25451/tid=25617] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25617/stat : 25617 (python) S 25449 25451 25222 0 -1 4202560 30 934 0 0 0 0 0 0 23 0 19 0 736773411 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25640] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25640/stat : 25640 (python) S 25449 25451 25222 0 -1 4202560 22 934 0 0 0 0 0 0 20 0 19 0 736783246 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25644] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25644/stat : 25644 (python) S 25449 25451 25222 0 -1 4202560 26 934 0 0 0 0 0 0 20 0 19 0 736783438 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 4 0 0 0
[pid=25451/tid=25648] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25648/stat : 25648 (python) S 25449 25451 25222 0 -1 4202560 32 934 0 0 0 0 0 0 18 0 19 0 736783443 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 2 0 0 0
[pid=25451/tid=25652] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25652/stat : 25652 (python) S 25449 25451 25222 0 -1 4202560 26 934 0 0 0 0 0 0 22 0 19 0 736783445 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 2 0 0 0
[pid=25451/tid=25674] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25674/stat : 25674 (python) S 25449 25451 25222 0 -1 4202560 36 934 0 0 0 0 0 0 18 0 19 0 736793280 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 2 0 0 0
[pid=25451/tid=25679] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25679/stat : 25679 (python) S 25449 25451 25222 0 -1 4202560 38 934 0 0 0 0 0 0 18 0 19 0 736793472 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 2 0 0 0
[pid=25451/tid=25693] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25693/stat : 25693 (python) S 25449 25451 25222 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 736803119 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 2 0 0 0
[pid=25451/tid=25707] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25707/stat : 25707 (python) S 25449 25451 25222 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 736813153 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 2 0 0 0
[pid=25451/tid=25727] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25727/stat : 25727 (python) S 25449 25451 25222 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 19 0 736842749 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 0 0 0 0
[pid=25451/tid=25730] ppid=25449 vsize=464984 CPUtime=0 cores=0,2,4,6
/proc/25451/task/25730/stat : 25730 (python) S 25449 25451 25222 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 19 0 736842858 476143616 7638 33554432000 4194304 5683644 140736249270960 18446744073709551615 216747002289 0 0 16781312 2 18446604444314933120 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3351.17
Current children cumulated vsize (KiB) 15879560

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 25451 and gives
#  childrusage.ru_utime.tv_sec=396
#  childrusage.ru_utime.tv_usec=192769
#  childrusage.ru_stime.tv_sec=5
#  childrusage.ru_stime.tv_usec=629144
# CPU time returned by wait4() is 401.822
# while last known CPU time is 3351.17
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 843.52
CPU time (s): 3351.17
CPU user time (s): 3345.65
CPU system time (s): 5.52
CPU usage (%): 397.284
Max. virtual memory (cumulated for all children) (KiB): 15879560

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 396.193
system time used= 5.62914
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 61442
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6210
involuntary context switches= 2944

runsolver used 6.10207 second user time and 13.04 second system time

The end

Launcher Data 

Begin job on node119 at 2011-05-15 15:15:26
IDJOB=3378415
IDBENCH=82824
IDSOLVER=1859
FILE ID=node119/3378415-1305465326
RUNJOBID= node119-1305465319-25240
PBS_JOBID= 13325055
Free space on /tmp= 73280 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/medium/unif-k7-r89-v90-c8010-S1475406203-080.UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378415-1305465326/watcher-3378415-1305465326 -o /tmp/evaluation-result-3378415-1305465326/solver-3378415-1305465326 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378415-1305465326.cnf -seed 991091442 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 6cadcf1602218f53da431eaffbc93148
RANDOM SEED=991091442

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      13719712 kB
Buffers:        303724 kB
Cached:        4914120 kB
SwapCached:       4756 kB
Active:       15688284 kB
Inactive:      3195932 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      13719712 kB
SwapTotal:    67111528 kB
SwapFree:     65586204 kB
Dirty:           64248 kB
Writeback:           0 kB
AnonPages:    13664628 kB
Mapped:          38356 kB
Slab:           216764 kB
PageTables:      64088 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 115451656 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73248 MiB
End job on node119 at 2011-05-15 15:29:32