Trace number 3378113

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-par-random (fixed)? (MO) 2002.15 506.036

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S1143439968-082-UNKNOWN.cnf
MD5SUM4baebda329e22fa39a89c4e2a8acff70
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark468.292
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.41/0.79	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.41/0.79	c INFO: solving HOME/instance-3378113-1305462758.cnf
0.41/0.79	c INFO: computing task features
0.57/0.99	c INFO: computed variable-clause graph statistics
0.57/1.00	c INFO: computed clause balance statistics
1.79/2.12	c INFO: computed variable balance statistics
2.38/2.71	c INFO: computed small-clause counts
2.38/2.71	c INFO: computed Horn-clause counts
2.38/2.76	c INFO: computed variable graph statistics
2.38/2.76	c INFO: collected features for HOME/instance-3378113-1305462758.cnf in 1.96 s
2.38/2.77	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.38)
2.38/2.78	c INFO: running gnovelty+2@0 for 97 with 1198 remaining (b = 0.13)
2.38/2.79	c INFO: running adaptg2wsat2009++@0 for 97 with 1198 remaining (b = 0.08)
2.46/2.80	c INFO: running adaptg2wsat2009++@0 for 97 with 1197 remaining (b = 0.07)
398.62/102.29	c INFO: running adaptg2wsat2009++@0 for 97 with 1098 remaining (b = 0.05)
401.41/103.12	c INFO: running gnovelty+2@0 for 97 with 1097 remaining (b = 0.05)
401.64/103.28	c INFO: running FH@0 for 97 with 1097 remaining (b = 0.03)
401.64/103.29	c INFO: running adaptg2wsat2009++@0 for 97 with 1097 remaining (b = 0.03)
797.75/202.63	c INFO: running adaptg2wsat2009++@0 for 97 with 998 remaining (b = 0.02)
800.95/203.51	c INFO: running FH@0 for 97 with 997 remaining (b = 0.02)
801.62/203.62	c INFO: running hybridGM3@0 for 194 with 997 remaining (b = 0.04)
821.94/208.70	c INFO: running gnovelty+2@0 for 97 with 992 remaining (b = 0.02)
1197.19/303.13	c INFO: running hybridGM3@0 for 194 with 897 remaining (b = 0.03)
1221.08/309.13	c INFO: running adaptg2wsat2009++@0 for 97 with 891 remaining (b = 0.01)
1223.43/309.91	c INFO: running hybridGM3@0 for 194 with 890 remaining (b = 0.03)
1575.72/398.41	c INFO: running hybridGM3@102 for 97 with 802 remaining (b = 0.01)
1639.68/414.53	c INFO: running hybridGM3@0 for 194 with 786 remaining (b = 0.02)
1977.52/499.79	c INFO: running TNM@0 for 700 with 700 remaining (b = 0.08)
1986.62/502.02	c INFO: running hybridGM3@100 for 97 with 698 remaining (b = 0.01)
2000.88/505.69	c INFO: running FH@0 for 97 with 695 remaining (b = 0.01)
2002.15/506.01	c INFO: running adaptg2wsat2009++@0 for 97 with 694 remaining (b = 0.01)
2002.15/506.01	Traceback (most recent call last):
2002.15/506.01	  File "HOME/solve", line 33, in <module>
2002.15/506.01	    plac.call(main)
2002.15/506.01	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 311, in call
2002.15/506.01	    cmd, result = parser_from(obj).consume(arglist)
2002.15/506.01	  File "/home/evaluation/evaluation/tools64/lib/python2.7/site-packages/plac-0.8.1-py2.7.egg/plac_core.py", line 192, in consume
2002.15/506.01	    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
2002.15/506.01	  File "HOME/borg/build/lib/borg/tools/solve.py", line 84, in main
2002.15/506.01	    (_, answer, _) = solver(input_path, budget / cores, cores)
2002.15/506.01	  File "HOME/borg/build/lib/borg/bilevel.py", line 202, in __call__
2002.15/506.01	    solver.go(max_cost * borg.defaults.machine_speed)
2002.15/506.01	  File "HOME/borg/build/lib/borg/solvers.py", line 179, in go
2002.15/506.01	    self._process.start()
2002.15/506.01	  File "/home/evaluation/evaluation/tools64/lib/python2.7/multiprocessing/process.py", line 94, in start
2002.15/506.01	    assert self._popen is None, 'cannot start a process twice'
2002.15/506.01	AssertionError: cannot start a process twice

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3378113-1305462758/watcher-3378113-1305462758 -o /tmp/evaluation-result-3378113-1305462758/solver-3378113-1305462758 -C 4800 -W 1300 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378113-1305462758.cnf -seed 314927927 -budget 4800 -cores 4 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.41 3.89 3.87 3/415 25987
/proc/meminfo: memFree=16389316/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=16244 CPUtime=0 cores=0,2,4,6
/proc/25987/stat : 25987 (python) R 25985 25987 25950 0 -1 4202496 994 0 0 0 0 0 0 0 25 0 1 0 657873538 16633856 868 33554432000 4194304 5683644 140735660468272 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/25987/statm: 4061 868 368 364 0 548 0

[startup+0.0886429 s]
/proc/loadavg: 3.41 3.89 3.87 3/415 25987
/proc/meminfo: memFree=16389316/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=50016 CPUtime=0.05 cores=0,2,4,6
/proc/25987/stat : 25987 (python) S 25985 25987 25950 0 -1 4202496 1947 0 0 0 3 2 0 0 17 0 1 0 657873538 51216384 1713 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905729024 0 2147483391 16781312 2 0 0 0 17 6 0 0 0
/proc/25987/statm: 12504 1717 476 364 0 1204 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 50016

[startup+0.100702 s]
/proc/loadavg: 3.41 3.89 3.87 3/415 25987
/proc/meminfo: memFree=16389316/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=56784 CPUtime=0.06 cores=0,2,4,6
/proc/25987/stat : 25987 (python) D 25985 25987 25950 0 -1 4202496 2101 0 0 0 3 3 0 0 17 0 1 0 657873538 58146816 1857 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905729024 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/25987/statm: 14196 1857 499 364 0 1319 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 56784

[startup+0.300805 s]
/proc/loadavg: 3.41 3.89 3.87 3/415 25987
/proc/meminfo: memFree=16389316/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=135588 CPUtime=0.19 cores=0,2,4,6
/proc/25987/stat : 25987 (python) S 25985 25987 25950 0 -1 4202496 5885 932 1 0 12 7 0 0 17 0 1 0 657873538 138842112 4132 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905729024 0 2147483391 16781312 2 18446744071699953811 0 0 17 6 0 0 0
/proc/25987/statm: 33897 4132 1031 364 0 3083 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135588

[startup+0.700506 s]
/proc/loadavg: 3.41 3.89 3.87 3/415 25987
/proc/meminfo: memFree=16389316/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=246316 CPUtime=0.41 cores=0,2,4,6
/proc/25987/stat : 25987 (python) R 25985 25987 25950 0 -1 4202496 8141 932 1 0 26 15 0 0 16 0 1 0 657873538 252227584 6042 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905386441 0 0 16781312 2 0 0 0 17 6 0 0 0
/proc/25987/statm: 61579 6042 1621 364 0 4581 0
Current children cumulated CPU time (s) 0.41
Current children cumulated vsize (KiB) 246316

[startup+1.50039 s]
/proc/loadavg: 3.41 3.89 3.87 4/417 25997
/proc/meminfo: memFree=16323464/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=290804 CPUtime=1.18 cores=0,2,4,6
/proc/25987/stat : 25987 (python) R 25985 25987 25950 0 -1 4202496 14008 932 1 0 100 18 0 0 17 0 1 0 657873538 297783296 10085 33554432000 4194304 5683644 140735660468272 18446744073709551615 4602358 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/25987/statm: 72701 10085 1826 364 0 8625 0
Current children cumulated CPU time (s) 1.18
Current children cumulated vsize (KiB) 290804

[startup+3.10104 s]
/proc/loadavg: 3.41 3.89 3.87 4/417 25997
/proc/meminfo: memFree=16322596/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=325512 CPUtime=2.47 cores=0,2,4,6
/proc/25987/stat : 25987 (python) S 25985 25987 25950 0 -1 4202496 16818 932 1 0 229 18 0 0 25 0 5 0 657873538 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/25987/statm: 81378 8576 1883 364 0 17287 0
Current children cumulated CPU time (s) 2.47
Current children cumulated vsize (KiB) 325512
heavy processes:

[startup+6.30146 s]
/proc/loadavg: 3.54 3.90 3.87 9/441 26021
/proc/meminfo: memFree=16260420/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=325512 CPUtime=2.47 cores=0,2,4,6
/proc/25987/stat : 25987 (python) S 25985 25987 25950 0 -1 4202496 16818 932 1 0 229 18 0 0 25 0 5 0 657873538 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/25987/statm: 81378 8576 1883 364 0 17287 0
[pid=26010] ppid=25987 vsize=284984 CPUtime=0 cores=0,2,4,6
/proc/26010/stat : 26010 (python) S 25987 25987 25950 0 -1 4202560 720 0 0 0 0 0 0 0 18 0 1 0 657873815 291823616 6898 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751055 0 0 16781312 514 15484326875630633217 0 0 17 6 0 0 0
/proc/26010/statm: 71246 6898 355 364 0 7164 0
[pid=26012] ppid=26010 vsize=1020308 CPUtime=3.5 cores=0,2,4,6
/proc/26012/stat : 26012 (gnovelty+2) R 26010 26012 26012 0 -1 4202496 1156 0 0 0 349 1 0 0 25 0 1 0 657873816 1044795392 892 33554432000 134512640 135663704 4293875632 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/26012/statm: 255077 892 113 282 0 254793 0
[pid=26013] ppid=25987 vsize=294904 CPUtime=0 cores=0,2,4,6
/proc/26013/stat : 26013 (python) S 25987 25987 25950 0 -1 4202560 733 0 0 0 0 0 0 0 18 0 1 0 657873816 301981696 7031 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 3161025840284664066 0 0 17 6 0 0 0
/proc/26013/statm: 73726 7031 349 364 0 9641 0
[pid=26017] ppid=26013 vsize=1020308 CPUtime=3.5 cores=0,2,4,6
/proc/26017/stat : 26017 (gnovelty+2) R 26013 26017 26017 0 -1 4202496 1159 0 0 0 350 0 0 0 25 0 1 0 657873817 1044795392 892 33554432000 134512640 135663704 4288874384 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/26017/statm: 255077 892 113 282 0 254793 0
[pid=26015] ppid=25987 vsize=305160 CPUtime=0 cores=0,2,4,6
/proc/26015/stat : 26015 (python) S 25987 25987 25950 0 -1 4202560 737 0 0 0 0 0 0 0 18 0 1 0 657873817 312483840 7035 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 3161018143703269635 0 0 17 0 0 0 0
/proc/26015/statm: 76290 7035 349 364 0 12202 0
[pid=26018] ppid=26015 vsize=423400 CPUtime=3.44 cores=0,2,4,6
/proc/26018/stat : 26018 (adaptg2wsat2009) R 26015 26018 26018 0 -1 4202496 2622 0 0 0 344 0 0 0 25 0 1 0 657873818 433561600 2357 33554432000 134512640 135107411 4294129248 18446744073709551615 134532371 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/26018/statm: 105850 2357 59 146 0 104726 0
[pid=26019] ppid=25987 vsize=316020 CPUtime=0 cores=0,2,4,6
/proc/26019/stat : 26019 (python) S 25987 25987 25950 0 -1 4202560 740 0 0 0 0 0 0 0 18 0 1 0 657873819 323604480 7039 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 10860100652210553089 0 0 17 0 0 0 0
/proc/26019/statm: 79005 7039 349 364 0 14914 0
[pid=26021] ppid=26019 vsize=423400 CPUtime=3.48 cores=0,2,4,6
/proc/26021/stat : 26021 (adaptg2wsat2009) R 26019 26021 26021 0 -1 4202496 2620 0 0 0 348 0 0 0 25 0 1 0 657873819 433561600 2357 33554432000 134512640 135107411 4290123824 18446744073709551615 134532135 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/26021/statm: 105850 2357 59 146 0 104726 0
[pid=25987/tid=26011] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26011/stat : 26011 (python) S 25985 25987 25950 0 -1 4202560 23 932 0 0 0 0 0 0 21 0 5 0 657873815 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26014] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26014/stat : 26014 (python) S 25985 25987 25950 0 -1 4202560 40 932 0 0 0 0 0 0 23 0 5 0 657873816 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26016] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26016/stat : 26016 (python) S 25985 25987 25950 0 -1 4202560 16 932 0 0 0 0 0 0 25 0 5 0 657873817 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26020] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26020/stat : 26020 (python) S 25985 25987 25950 0 -1 4202560 29 932 0 0 0 0 0 0 25 0 5 0 657873819 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 16.39
Current children cumulated vsize (KiB) 4413996

[startup+12.7013 s]
/proc/loadavg: 3.89 3.97 3.89 9/441 26021
/proc/meminfo: memFree=16256868/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=325512 CPUtime=2.47 cores=0,2,4,6
/proc/25987/stat : 25987 (python) S 25985 25987 25950 0 -1 4202496 16818 932 1 0 229 18 0 0 25 0 5 0 657873538 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/25987/statm: 81378 8576 1883 364 0 17287 0
[pid=26010] ppid=25987 vsize=284984 CPUtime=0.02 cores=0,2,4,6
/proc/26010/stat : 26010 (python) S 25987 25987 25950 0 -1 4202560 876 0 0 0 1 1 0 0 18 0 1 0 657873815 291823616 6921 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751055 0 0 16781312 514 15484326875630633217 0 0 17 6 0 0 0
/proc/26010/statm: 71246 6921 378 364 0 7164 0
[pid=26012] ppid=26010 vsize=1020308 CPUtime=9.9 cores=0,2,4,6
/proc/26012/stat : 26012 (gnovelty+2) R 26010 26012 26012 0 -1 4202496 1156 0 0 0 989 1 0 0 25 0 1 0 657873816 1044795392 892 33554432000 134512640 135663704 4293875632 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/26012/statm: 255077 892 113 282 0 254793 0
[pid=26013] ppid=25987 vsize=294904 CPUtime=0.01 cores=0,2,4,6
/proc/26013/stat : 26013 (python) S 25987 25987 25950 0 -1 4202560 900 0 0 0 0 1 0 0 18 0 1 0 657873816 301981696 7054 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 3161025840284664066 0 0 17 6 0 0 0
/proc/26013/statm: 73726 7054 372 364 0 9641 0
[pid=26017] ppid=26013 vsize=1020308 CPUtime=9.89 cores=0,2,4,6
/proc/26017/stat : 26017 (gnovelty+2) R 26013 26017 26017 0 -1 4202496 1159 0 0 0 989 0 0 0 25 0 1 0 657873817 1044795392 892 33554432000 134512640 135663704 4288874384 18446744073709551615 134544196 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/26017/statm: 255077 892 113 282 0 254793 0
[pid=26015] ppid=25987 vsize=305160 CPUtime=0.01 cores=0,2,4,6
/proc/26015/stat : 26015 (python) S 25987 25987 25950 0 -1 4202560 900 0 0 0 0 1 0 0 18 0 1 0 657873817 312483840 7058 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 3161018143703269635 0 0 17 0 0 0 0
/proc/26015/statm: 76290 7058 372 364 0 12202 0
[pid=26018] ppid=26015 vsize=423400 CPUtime=9.71 cores=0,2,4,6
/proc/26018/stat : 26018 (adaptg2wsat2009) R 26015 26018 26018 0 -1 4202496 2622 0 0 0 971 0 0 0 25 0 1 0 657873818 433561600 2357 33554432000 134512640 135107411 4294129248 18446744073709551615 134532277 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/26018/statm: 105850 2357 59 146 0 104726 0
[pid=26019] ppid=25987 vsize=316020 CPUtime=0.01 cores=0,2,4,6
/proc/26019/stat : 26019 (python) S 25987 25987 25950 0 -1 4202560 907 0 0 0 1 0 0 0 18 0 1 0 657873819 323604480 7062 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 10860100652210553089 0 0 17 0 0 0 0
/proc/26019/statm: 79005 7062 372 364 0 14914 0
[pid=26021] ppid=26019 vsize=423400 CPUtime=9.84 cores=0,2,4,6
/proc/26021/stat : 26021 (adaptg2wsat2009) R 26019 26021 26021 0 -1 4202496 2620 0 0 0 984 0 0 0 25 0 1 0 657873819 433561600 2357 33554432000 134512640 135107411 4290123824 18446744073709551615 134531582 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/26021/statm: 105850 2357 59 146 0 104726 0
[pid=25987/tid=26011] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26011/stat : 26011 (python) S 25985 25987 25950 0 -1 4202560 23 932 0 0 0 0 0 0 21 0 5 0 657873815 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26014] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26014/stat : 26014 (python) S 25985 25987 25950 0 -1 4202560 40 932 0 0 0 0 0 0 23 0 5 0 657873816 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26016] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26016/stat : 26016 (python) S 25985 25987 25950 0 -1 4202560 16 932 0 0 0 0 0 0 25 0 5 0 657873817 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26020] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26020/stat : 26020 (python) S 25985 25987 25950 0 -1 4202560 29 932 0 0 0 0 0 0 25 0 5 0 657873819 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 41.86
Current children cumulated vsize (KiB) 4413996

[startup+25.501 s]
/proc/loadavg: 4.80 4.17 3.96 9/441 26022
/proc/meminfo: memFree=16259392/32951124 swapFree=65581060/67111528
[pid=25987] ppid=25985 vsize=325512 CPUtime=2.47 cores=0,2,4,6
/proc/25987/stat : 25987 (python) S 25985 25987 25950 0 -1 4202496 16818 932 1 0 229 18 0 0 25 0 5 0 657873538 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 2 18446744073709551615 0 0 17 0 0 0 0
/proc/25987/statm: 81378 8576 1883 364 0 17287 0
[pid=26010] ppid=25987 vsize=284984 CPUtime=0.07 cores=0,2,4,6
/proc/26010/stat : 26010 (python) S 25987 25987 25950 0 -1 4202560 885 0 0 0 4 3 0 0 18 0 1 0 657873815 291823616 6921 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751055 0 0 16781312 514 17473298051127804161 0 0 17 6 0 0 0
/proc/26010/statm: 71246 6921 378 364 0 7164 0
[pid=26012] ppid=26010 vsize=1020308 CPUtime=22.7 cores=0,2,4,6
/proc/26012/stat : 26012 (gnovelty+2) R 26010 26012 26012 0 -1 4202496 1156 0 0 0 2269 1 0 0 25 0 1 0 657873816 1044795392 892 33554432000 134512640 135663704 4293875632 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/26012/statm: 255077 892 113 282 0 254793 0
[pid=26013] ppid=25987 vsize=294904 CPUtime=0.07 cores=0,2,4,6
/proc/26013/stat : 26013 (python) S 25987 25987 25950 0 -1 4202560 909 0 0 0 3 4 0 0 18 0 1 0 657873816 301981696 7054 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 17473298051127804162 0 0 17 6 0 0 0
/proc/26013/statm: 73726 7054 372 364 0 9641 0
[pid=26017] ppid=26013 vsize=1020308 CPUtime=22.69 cores=0,2,4,6
/proc/26017/stat : 26017 (gnovelty+2) R 26013 26017 26017 0 -1 4202496 1159 0 0 0 2269 0 0 0 25 0 1 0 657873817 1044795392 892 33554432000 134512640 135663704 4288874384 18446744073709551615 134525222 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/26017/statm: 255077 892 113 282 0 254793 0
[pid=26015] ppid=25987 vsize=305160 CPUtime=0.06 cores=0,2,4,6
/proc/26015/stat : 26015 (python) S 25987 25987 25950 0 -1 4202560 909 0 0 0 3 3 0 0 18 0 1 0 657873817 312483840 7058 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 17473298051127804163 0 0 17 0 0 0 0
/proc/26015/statm: 76290 7058 372 364 0 12202 0
[pid=26018] ppid=26015 vsize=423400 CPUtime=22.22 cores=0,2,4,6
/proc/26018/stat : 26018 (adaptg2wsat2009) R 26015 26018 26018 0 -1 4202496 2622 0 0 0 2222 0 0 0 25 0 1 0 657873818 433561600 2357 33554432000 134512640 135107411 4294129248 18446744073709551615 134537867 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/26018/statm: 105850 2357 59 146 0 104726 0
[pid=26019] ppid=25987 vsize=316020 CPUtime=0.06 cores=0,2,4,6
/proc/26019/stat : 26019 (python) S 25987 25987 25950 0 -1 4202560 913 0 0 0 3 3 0 0 18 0 1 0 657873819 323604480 7062 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 17473298051127804161 0 0 17 0 0 0 0
/proc/26019/statm: 79005 7062 372 364 0 14914 0
[pid=26021] ppid=26019 vsize=423400 CPUtime=22.53 cores=0,2,4,6
/proc/26021/stat : 26021 (adaptg2wsat2009) R 26019 26021 26021 0 -1 4202496 2620 0 0 0 2253 0 0 0 25 0 1 0 657873819 433561600 2357 33554432000 134512640 135107411 4290123824 18446744073709551615 134531817 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/26021/statm: 105850 2357 59 146 0 104726 0
[pid=25987/tid=26011] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26011/stat : 26011 (python) S 25985 25987 25950 0 -1 4202560 23 932 0 0 0 0 0 0 21 0 5 0 657873815 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26014] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26014/stat : 26014 (python) S 25985 25987 25950 0 -1 4202560 40 932 0 0 0 0 0 0 23 0 5 0 657873816 333324288 8576 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26016] ppid=25985 vsize=325512 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

/proc/26019/stat : 26019 (python) S 25987 25987 25950 0 -1 4202560 1088 0 0 0 17 14 0 0 19 0 2 0 657873819 323604480 7091 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26019/statm: 79005 7091 400 364 0 14914 0
[pid=26021] ppid=26019 vsize=423400 CPUtime=99.85 cores=0,2,4,6
/proc/26021/stat : 26021 (adaptg2wsat2009) T 26019 26021 26021 0 -1 4202496 2620 0 0 0 9983 2 0 0 25 0 1 0 657873819 433561600 2357 33554432000 134512640 135107411 4290123824 18446744073709551615 134531600 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26021/statm: 105850 2357 59 146 0 104726 0
[pid=26019/tid=26049] ppid=25987 vsize=316020 CPUtime=0 cores=0,2,4,6
/proc/26019/task/26049/stat : 26049 (python) S 25987 25987 25950 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657883865 323604480 7091 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604442556355584 0 0 -1 4 0 0 0
[pid=26026] ppid=25987 vsize=325792 CPUtime=0.33 cores=0,2,4,6
/proc/26026/stat : 26026 (python) S 25987 25987 25950 0 -1 4202560 1090 0 0 0 17 16 0 0 19 0 2 0 657883768 333611008 7161 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26026/statm: 81448 7161 399 364 0 17354 0
[pid=26028] ppid=26026 vsize=423400 CPUtime=100.26 cores=0,2,4,6
/proc/26028/stat : 26028 (adaptg2wsat2009) T 26026 26028 26028 0 -1 4202496 2621 0 0 0 10026 0 0 0 25 0 1 0 657883768 433561600 2357 33554432000 134512640 135107411 4290544432 18446744073709551615 134531428 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26028/statm: 105850 2357 59 146 0 104726 0
[pid=26026/tid=26060] ppid=25987 vsize=325792 CPUtime=0 cores=0,2,4,6
/proc/26026/task/26060/stat : 26060 (python) S 25987 25987 25950 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657893801 333611008 7161 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604453906229248 0 0 -1 0 0 0 0
[pid=26046] ppid=25987 vsize=336236 CPUtime=0.33 cores=0,2,4,6
/proc/26046/stat : 26046 (python) S 25987 25987 25950 0 -1 4202560 1094 0 0 0 17 16 0 0 19 0 2 0 657883850 344305664 7166 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26046/statm: 84059 7166 400 364 0 19962 0
[pid=26048] ppid=26046 vsize=1020308 CPUtime=99.37 cores=0,2,4,6
/proc/26048/stat : 26048 (gnovelty+2) T 26046 26048 26048 0 -1 4202496 1155 0 0 0 9936 1 0 0 25 0 1 0 657883851 1044795392 892 33554432000 134512640 135663704 4288349360 18446744073709551615 134545786 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/26048/statm: 255077 892 113 282 0 254793 0
[pid=26046/tid=26080] ppid=25987 vsize=336236 CPUtime=0 cores=0,2,4,6
/proc/26046/task/26080/stat : 26080 (python) S 25987 25987 25950 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657893888 344305664 7166 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604449728840640 0 0 -1 0 0 0 0
[pid=26051] ppid=25987 vsize=346304 CPUtime=0.33 cores=0,2,4,6
/proc/26051/stat : 26051 (python) S 25987 25987 25950 0 -1 4202560 1100 0 0 0 20 13 0 0 19 0 2 0 657883866 354615296 7169 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/26051/statm: 86576 7169 399 364 0 22476 0
[pid=26053] ppid=26051 vsize=253032 CPUtime=100.32 cores=0,2,4,6
/proc/26053/stat : 26053 (FH) T 26051 26053 26053 0 -1 4202496 2498 0 0 0 10032 0 0 0 25 0 1 0 657883866 259104768 2234 33554432000 134512640 134936988 4287546240 18446744073709551615 134545294 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/26053/statm: 63258 2234 41 104 0 62762 0
[pid=26051/tid=26084] ppid=25987 vsize=346304 CPUtime=0 cores=0,2,4,6
/proc/26051/task/26084/stat : 26084 (python) S 25987 25987 25950 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657893899 354615296 7169 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604453879800256 0 0 -1 0 0 0 0
[pid=26054] ppid=25987 vsize=357184 CPUtime=0.35 cores=0,2,4,6
/proc/26054/stat : 26054 (python) S 25987 25987 25950 0 -1 4202560 1140 0 0 0 18 17 0 0 19 0 2 0 657883867 365756416 7180 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/26054/statm: 89296 7180 406 364 0 25193 0
[pid=26056] ppid=26054 vsize=423400 CPUtime=102.72 cores=0,2,4,6
/proc/26056/stat : 26056 (adaptg2wsat2009) T 26054 26056 26056 0 -1 4202496 2625 0 0 0 10272 0 0 0 25 0 1 0 657883868 433561600 2357 33554432000 134512640 135107411 4291532256 18446744073709551615 134532376 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26056/statm: 105850 2357 59 146 0 104726 0
[pid=26054/tid=26088] ppid=25987 vsize=357184 CPUtime=0 cores=0,2,4,6
/proc/26054/task/26088/stat : 26088 (python) S 25987 25987 25950 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657894407 365756416 7180 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604450469772288 0 0 -1 6 0 0 0
[pid=26061] ppid=25987 vsize=366956 CPUtime=0.34 cores=0,2,4,6
/proc/26061/stat : 26061 (python) S 25987 25987 25950 0 -1 4202560 1139 0 0 0 17 17 0 0 19 0 2 0 657893802 375762944 7218 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/26061/statm: 91739 7218 405 364 0 27633 0
[pid=26063] ppid=26061 vsize=423400 CPUtime=100.42 cores=0,2,4,6
/proc/26063/stat : 26063 (adaptg2wsat2009) T 26061 26063 26063 0 -1 4202496 2630 0 0 0 10042 0 0 0 25 0 1 0 657893802 433561600 2358 33554432000 134512640 135107411 4286796304 18446744073709551615 134531817 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/26063/statm: 105850 2358 59 146 0 104726 0
[pid=26061/tid=26094] ppid=25987 vsize=366956 CPUtime=0 cores=0,2,4,6
/proc/26061/task/26094/stat : 26094 (python) S 25987 25987 25950 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657903850 375762944 7218 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604453913051520 0 0 -1 2 0 0 0
[pid=26081] ppid=25987 vsize=377212 CPUtime=0.37 cores=0,2,4,6
/proc/26081/stat : 26081 (python) S 25987 25987 25950 0 -1 4202560 1128 0 0 0 19 18 0 0 19 0 2 0 657893889 386265088 7216 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/26081/statm: 94303 7216 399 364 0 30194 0
[pid=26083] ppid=26081 vsize=253032 CPUtime=102.73 cores=0,2,4,6
/proc/26083/stat : 26083 (FH) T 26081 26083 26083 0 -1 4202496 2500 0 0 0 10273 0 0 0 25 0 1 0 657893890 259104768 2234 33554432000 134512640 134936988 4290910032 18446744073709551615 134554591 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/26083/statm: 63258 2234 41 104 0 62762 0
[pid=26081/tid=26114] ppid=25987 vsize=377212 CPUtime=0 cores=0,2,4,6
/proc/26081/task/26114/stat : 26114 (python) S 25987 25987 25950 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657904528 386265088 7216 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604452883772608 0 0 -1 6 0 0 0
[pid=26085] ppid=25987 vsize=387644 CPUtime=1.04 cores=0,2,4,6
/proc/26085/stat : 26085 (python) S 25987 25987 25950 0 -1 4202560 1184 0 0 0 50 54 0 0 18 0 2 0 657893900 396947456 7221 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/26085/statm: 96911 7221 400 364 0 32799 0
[pid=26087] ppid=26085 vsize=268420 CPUtime=295.31 cores=0,2,4,6
/proc/26087/stat : 26087 (hybridGM3) T 26085 26087 26087 0 -1 4202496 6190 0 0 0 29530 1 0 0 25 0 1 0 657893900 274862080 5919 33554432000 134512640 135204385 4294104288 18446744073709551615 134537711 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26087/statm: 67105 5919 63 169 0 66935 0
[pid=26085/tid=26121] ppid=25987 vsize=387644 CPUtime=0 cores=0,2,4,6
/proc/26085/task/26121/stat : 26121 (python) S 25987 25987 25950 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 657913378 396947456 7221 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604451824357632 0 0 -1 6 0 0 0
[pid=26089] ppid=25987 vsize=397724 CPUtime=0.34 cores=0,2,4,6
/proc/26089/stat : 26089 (python) S 25987 25987 25950 0 -1 4202560 1133 0 0 0 17 17 0 0 19 0 2 0 657894408 407269376 7224 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/26089/statm: 99431 7224 399 364 0 35316 0
[pid=26091] ppid=26089 vsize=1020308 CPUtime=100.4 cores=0,2,4,6
/proc/26091/stat : 26091 (gnovelty+2) T 26089 26091 26091 0 -1 4202496 1162 0 0 0 10040 0 0 0 25 0 1 0 657894409 1044795392 891 33554432000 134512640 135663704 4289989312 18446744073709551615 134525678 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/26091/statm: 255077 891 113 282 0 254793 0
[pid=26089/tid=26110] ppid=25987 vsize=397724 CPUtime=0 cores=0,2,4,6
/proc/26089/task/26110/stat : 26110 (python) S 25987 25987 25950 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 657904449 407269376 7224 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604446611957696 0 0 -1 0 0 0 0
[pid=26095] ppid=25987 vsize=408644 CPUtime=0.72 cores=0,2,4,6
/proc/26095/stat : 26095 (python) S 25987 25987 25950 0 -1 4202560 1181 0 0 0 37 35 0 0 18 0 2 0 657903851 418451456 7229 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/26095/statm: 102161 7229 400 364 0 38043 0
[pid=26097] ppid=26095 vsize=268420 CPUtime=201.77 cores=0,2,4,6
/proc/26097/stat : 26097 (hybridGM3) R 26095 26097 26097 0 -1 4202496 6189 0 0 0 20176 1 0 0 25 0 1 0 657903851 274862080 5919 33554432000 134512640 135204385 4287833232 18446744073709551615 134538432 0 0 16781312 0 0 0 0 17 2 0 0 0
/proc/26097/statm: 67105 5919 63 169 0 66935 0
[pid=26095/tid=26147] ppid=25987 vsize=408644 CPUtime=0 cores=0,2,4,6
/proc/26095/task/26147/stat : 26147 (python) S 25987 25987 25950 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 657923738 418451456 7229 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604441753289088 0 0 -1 2 0 0 0
[pid=26111] ppid=25987 vsize=418416 CPUtime=0.37 cores=0,2,4,6
/proc/26111/stat : 26111 (python) S 25987 25987 25950 0 -1 4202560 1150 0 0 0 18 19 0 0 19 0 2 0 657904451 428457984 7276 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/26111/statm: 104604 7276 399 364 0 40483 0
[pid=26113] ppid=26111 vsize=423400 CPUtime=101.9 cores=0,2,4,6
/proc/26113/stat : 26113 (adaptg2wsat2009) T 26111 26113 26113 0 -1 4202496 2630 0 0 0 10190 0 0 0 25 0 1 0 657904451 433561600 2356 33554432000 134512640 135107411 4294392384 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 4 0 0 0
/proc/26113/statm: 105850 2356 59 146 0 104726 0
[pid=26111/tid=26127] ppid=25987 vsize=418416 CPUtime=0 cores=0,2,4,6
/proc/26111/task/26127/stat : 26127 (python) S 25987 25987 25950 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 657914990 428457984 7276 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604452181891904 0 0 -1 6 0 0 0
[pid=26115] ppid=25987 vsize=429256 CPUtime=0.69 cores=0,2,4,6
/proc/26115/stat : 26115 (python) S 25987 25987 25950 0 -1 4202560 1233 0 0 0 33 36 0 0 19 0 2 0 657904529 439558144 7290 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917557707 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/26115/statm: 107314 7290 408 364 0 43190 0
[pid=26117] ppid=26115 vsize=268424 CPUtime=195.76 cores=0,2,4,6
/proc/26117/stat : 26117 (hybridGM3) T 26115 26117 26117 0 -1 4202496 6198 0 0 0 19575 1 0 0 25 0 1 0 657904530 274866176 5920 33554432000 134512640 135204385 4289618928 18446744073709551615 134538375 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/26117/statm: 67106 5920 63 169 0 66936 0
[pid=26115/tid=26180] ppid=25987 vsize=429256 CPUtime=0 cores=0,2,4,6
/proc/26115/task/26180/stat : 26180 (python) S 25987 25987 25950 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 657924106 439558144 7290 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 514 18446604444804469824 0 0 -1 2 0 0 0
[pid=26128] ppid=25987 vsize=439152 CPUtime=0.31 cores=0,2,4,6
/proc/26128/stat : 26128 (python) S 25987 25987 25950 0 -1 4202560 1035 0 0 0 17 14 0 0 18 0 1 0 657914992 449691648 7288 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 17473298051127804162 0 0 17 6 0 0 0
/proc/26128/statm: 109788 7288 379 364 0 45661 0
[pid=26130] ppid=26128 vsize=268428 CPUtime=87.48 cores=0,2,4,6
/proc/26130/stat : 26130 (hybridGM3) R 26128 26130 26130 0 -1 4202496 6195 0 0 0 8746 2 0 0 25 0 1 0 657914992 274870272 5919 33554432000 134512640 135204385 4294416976 18446744073709551615 134538520 0 0 16781312 0 0 0 0 17 4 0 0 0
/proc/26130/statm: 67107 5919 63 169 0 66937 0
[pid=26133] ppid=25987 vsize=449828 CPUtime=0.02 cores=0,2,4,6
/proc/26133/stat : 26133 (python) S 25987 25987 25950 0 -1 4202560 1012 0 0 0 1 1 0 0 18 0 1 0 657923518 460623872 7292 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 1914259643857600769 0 0 17 6 0 0 0
/proc/26133/statm: 112457 7292 379 364 0 48327 0
[pid=26135] ppid=26133 vsize=461504 CPUtime=5.95 cores=0,2,4,6
/proc/26135/stat : 26135 (TNM) R 26133 26135 26135 0 -1 4202496 2471 0 0 0 594 1 0 0 25 0 1 0 657923519 472580096 2194 33554432000 134512640 135008748 4294049600 18446744073709551615 134527542 0 0 16781312 0 0 0 0 17 6 0 0 0
/proc/26135/statm: 115376 2194 46 122 0 114276 0
[pid=26181] ppid=25987 vsize=459836 CPUtime=0 cores=0,2,4,6
/proc/26181/stat : 26181 (python) S 25987 25987 25950 0 -1 4202560 822 0 0 0 0 0 0 0 19 0 1 0 657924107 470872064 7263 33554432000 4194304 5683644 140735660468272 18446744073709551615 218905751110 0 0 16781312 514 4271848419053535231 0 0 17 6 0 0 0
/proc/26181/statm: 114959 7263 346 364 0 50826 0
[pid=26183] ppid=26181 vsize=248616 CPUtime=0.23 cores=0,2,4,6
/proc/26183/stat : 26183 (FH) R 26181 26183 26183 0 -1 4202496 1487 0 0 0 23 0 0 0 21 0 1 0 657924107 254582784 1214 33554432000 134512640 134936988 4293226448 18446744073709551615 134523622 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/26183/statm: 62154 1214 37 104 0 61658 0
[pid=25987/tid=26011] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26011/stat : 26011 (python) S 25985 25987 25950 0 -1 4202560 23 932 0 0 0 0 0 0 21 0 19 0 657873815 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26014] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26014/stat : 26014 (python) S 25985 25987 25950 0 -1 4202560 40 932 0 0 0 0 0 0 23 0 19 0 657873816 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26016] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26016/stat : 26016 (python) S 25985 25987 25950 0 -1 4202560 16 932 0 0 0 0 0 0 25 0 19 0 657873817 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26020] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26020/stat : 26020 (python) S 25985 25987 25950 0 -1 4202560 29 932 0 0 0 0 0 0 25 0 19 0 657873819 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26027] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26027/stat : 26027 (python) S 25985 25987 25950 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 657883768 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26047] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26047/stat : 26047 (python) S 25985 25987 25950 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 657883851 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26052] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26052/stat : 26052 (python) S 25985 25987 25950 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 657883866 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 0 0 0 0
[pid=25987/tid=26055] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26055/stat : 26055 (python) S 25985 25987 25950 0 -1 4202560 26 932 0 0 0 0 0 0 22 0 19 0 657883868 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26062] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26062/stat : 26062 (python) S 25985 25987 25950 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 19 0 657893802 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26082] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26082/stat : 26082 (python) S 25985 25987 25950 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 657893890 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26086] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26086/stat : 26086 (python) S 25985 25987 25950 0 -1 4202560 27 932 0 0 0 0 0 0 18 0 19 0 657893900 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26090] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26090/stat : 26090 (python) S 25985 25987 25950 0 -1 4202560 24 932 0 0 0 0 0 0 20 0 19 0 657894409 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26096] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26096/stat : 26096 (python) S 25985 25987 25950 0 -1 4202560 29 932 0 0 0 0 0 0 18 0 19 0 657903851 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26112] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26112/stat : 26112 (python) S 25985 25987 25950 0 -1 4202560 30 932 0 0 0 0 0 0 20 0 19 0 657904451 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26116] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26116/stat : 26116 (python) S 25985 25987 25950 0 -1 4202560 25 932 0 0 0 0 0 0 20 0 19 0 657904530 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26129] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26129/stat : 26129 (python) S 25985 25987 25950 0 -1 4202560 28 932 0 0 0 0 0 0 20 0 19 0 657914992 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26134] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26134/stat : 26134 (python) S 25985 25987 25950 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 19 0 657923518 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
[pid=25987/tid=26182] ppid=25985 vsize=469684 CPUtime=0 cores=0,2,4,6
/proc/25987/task/26182/stat : 26182 (python) S 25985 25987 25950 0 -1 4202560 27 932 0 0 0 0 0 0 20 0 19 0 657924107 480956416 8832 33554432000 4194304 5683644 140735660468272 18446744073709551615 218917554609 0 0 16781312 2 18446604444804468928 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2002.15
Current children cumulated vsize (KiB) 16092688

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 25987 and gives
#  childrusage.ru_utime.tv_sec=301
#  childrusage.ru_utime.tv_usec=787121
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=737431
# CPU time returned by wait4() is 305.525
# while last known CPU time is 2002.15
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 506.036
CPU time (s): 2002.15
CPU user time (s): 1998.51
CPU system time (s): 3.64
CPU usage (%): 395.653
Max. virtual memory (cumulated for all children) (KiB): 16092688

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 301.787
system time used= 3.73743
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 70024
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17431
involuntary context switches= 2573

runsolver used 3.9704 second user time and 8.43372 second system time

The end

Launcher Data 

Begin job on node137 at 2011-05-15 14:32:38
IDJOB=3378113
IDBENCH=82522
IDSOLVER=1859
FILE ID=node137/3378113-1305462758
RUNJOBID= node137-1305462758-25968
PBS_JOBID= 13324972
Free space on /tmp= 73748 MiB

SOLVER NAME= borg-sat 11.03.02-par-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S1143439968-082-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT -cores NBCORE
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3378113-1305462758/watcher-3378113-1305462758 -o /tmp/evaluation-result-3378113-1305462758/solver-3378113-1305462758 -C 4800 -W 1300 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3378113-1305462758.cnf -seed 314927927 -budget 4800 -cores 4

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 4baebda329e22fa39a89c4e2a8acff70
RANDOM SEED=314927927

node137.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.844
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      16390092 kB
Buffers:        308800 kB
Cached:        4051216 kB
SwapCached:      36320 kB
Active:        6367724 kB
Inactive:      9936532 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      16390092 kB
SwapTotal:    67111528 kB
SwapFree:     65581060 kB
Dirty:           65344 kB
Writeback:           0 kB
AnonPages:    11918068 kB
Mapped:          26208 kB
Slab:           134860 kB
PageTables:      54432 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 106418660 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73676 MiB
End job on node137 at 2011-05-15 14:41:06