Trace number 3386610

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (MO) 3558.21 3547.54

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S922811046-076-UNKNOWN.cnf
MD5SUMe99bff721d695892a53f60adc73d403b
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark625.84
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.34/0.64	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.34/0.64	c INFO: solving HOME/instance-3386610-1305589590.cnf
0.34/0.64	c INFO: computing task features
0.52/0.85	c INFO: computed variable-clause graph statistics
0.52/0.86	c INFO: computed clause balance statistics
1.52/1.86	c INFO: computed variable balance statistics
2.12/2.42	c INFO: computed small-clause counts
2.12/2.42	c INFO: computed Horn-clause counts
2.12/2.47	c INFO: computed variable graph statistics
2.12/2.47	c INFO: collected features for HOME/instance-3386610-1305589590.cnf in 1.83 s
2.23/2.50	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.38)
102.73/102.89	c INFO: running gnovelty+2@0 for 97 with 4897 remaining (b = 0.13)
202.41/202.30	c INFO: running adaptg2wsat2009++@0 for 97 with 4798 remaining (b = 0.08)
303.04/302.64	c INFO: running adaptg2wsat2009++@0 for 97 with 4698 remaining (b = 0.07)
403.76/403.00	c INFO: running adaptg2wsat2009++@0 for 97 with 4597 remaining (b = 0.05)
503.34/502.37	c INFO: running gnovelty+2@0 for 97 with 4498 remaining (b = 0.05)
603.01/601.78	c INFO: running FH@0 for 97 with 4399 remaining (b = 0.03)
703.71/702.14	c INFO: running adaptg2wsat2009++@0 for 97 with 4298 remaining (b = 0.03)
804.43/802.51	c INFO: running adaptg2wsat2009++@0 for 97 with 4198 remaining (b = 0.02)
905.03/902.87	c INFO: running FH@0 for 97 with 4098 remaining (b = 0.02)
1005.72/1003.24	c INFO: running hybridGM3@0 for 194 with 3997 remaining (b = 0.04)
1202.81/1199.99	c INFO: running gnovelty+2@0 for 97 with 3801 remaining (b = 0.02)
1303.44/1300.39	c INFO: running hybridGM3@0 for 194 with 3701 remaining (b = 0.03)
1498.88/1495.20	c INFO: running adaptg2wsat2009++@0 for 97 with 3506 remaining (b = 0.01)
1598.52/1594.57	c INFO: running hybridGM3@0 for 194 with 3407 remaining (b = 0.03)
1798.99/1794.39	c INFO: running hybridGM3@0 for 194 with 3207 remaining (b = 0.02)
1995.39/1990.19	c INFO: running TNM@0 for 1552 with 3011 remaining (b = 0.19)
3557.36/3546.68	c INFO: running FH@0 for 97 with 1455 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386610-1305589590/watcher-3386610-1305589590 -o /tmp/evaluation-result-3386610-1305589590/solver-3386610-1305589590 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386610-1305589590.cnf -seed 785539133 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.07 4.07 4.05 5/473 4898
/proc/meminfo: memFree=19740708/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=15380 CPUtime=0 cores=1,3,5,7
/proc/4898/stat : 4898 (python) R 4896 4898 4581 0 -1 4202496 819 0 0 0 0 0 0 0 22 0 1 0 670557057 15749120 704 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122712064 0 2147483391 16781312 2 0 0 0 17 5 0 0 0
/proc/4898/statm: 3845 704 334 364 0 332 0

[startup+0.0780849 s]
/proc/loadavg: 4.07 4.07 4.05 5/473 4898
/proc/meminfo: memFree=19740708/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=50012 CPUtime=0.04 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 1948 0 0 0 3 1 0 0 17 0 1 0 670557057 51212288 1713 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122709861 0 2147483391 16781312 2 18446744071699207358 0 0 17 5 0 0 0
/proc/4898/statm: 12503 1713 476 364 0 1203 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 50012

[startup+0.101095 s]
/proc/loadavg: 4.07 4.07 4.05 5/473 4898
/proc/meminfo: memFree=19740708/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=93960 CPUtime=0.05 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 2692 0 0 0 4 1 0 0 17 0 1 0 670557057 96215040 2407 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122712064 0 2147483391 16781312 2 18446744071699207358 0 0 17 5 0 0 0
/proc/4898/statm: 23490 2407 764 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 93960

[startup+0.301012 s]
/proc/loadavg: 4.07 4.07 4.05 5/473 4898
/proc/meminfo: memFree=19740708/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=135748 CPUtime=0.17 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 5969 931 1 0 13 4 0 0 17 0 1 0 670557057 139005952 4149 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122710021 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/4898/statm: 33937 4149 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135748

[startup+0.706025 s]
/proc/loadavg: 4.07 4.07 4.05 5/473 4898
/proc/meminfo: memFree=19740708/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=282780 CPUtime=0.43 cores=1,3,5,7
/proc/4898/stat : 4898 (python) R 4896 4898 4581 0 -1 4202496 10299 931 1 0 34 9 0 0 15 0 1 0 670557057 289566720 7975 33554432000 4194304 5683644 140737361495184 18446744073709551615 4425442 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/4898/statm: 70695 7975 1761 364 0 6619 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 282780

[startup+1.50083 s]
/proc/loadavg: 4.07 4.07 4.05 5/474 4903
/proc/meminfo: memFree=19705360/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=290796 CPUtime=1.22 cores=1,3,5,7
/proc/4898/stat : 4898 (python) R 4896 4898 4581 0 -1 4202496 14015 931 1 0 113 9 0 0 16 0 1 0 670557057 297775104 10084 33554432000 4194304 5683644 140737361495184 18446744073709551615 4527938 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/4898/statm: 72699 10084 1826 364 0 8623 0
Current children cumulated CPU time (s) 1.22
Current children cumulated vsize (KiB) 290796

[startup+3.10051 s]
/proc/loadavg: 4.07 4.07 4.05 5/474 4903
/proc/meminfo: memFree=19703932/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=294132 CPUtime=2.23 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 14387 931 1 0 213 10 0 0 20 0 2 0 670557057 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/4898/statm: 73533 8423 1877 364 0 9451 0
Current children cumulated CPU time (s) 2.23
Current children cumulated vsize (KiB) 294132
heavy processes:

[startup+6.30097 s]
/proc/loadavg: 4.07 4.07 4.05 5/477 4906
/proc/meminfo: memFree=19704268/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=294132 CPUtime=2.23 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 14387 931 1 0 213 10 0 0 20 0 2 0 670557057 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/4898/statm: 73533 8423 1877 364 0 9451 0
[pid=4904] ppid=4898 vsize=284980 CPUtime=0 cores=1,3,5,7
/proc/4904/stat : 4904 (python) S 4898 4898 4581 0 -1 4202560 736 0 0 0 0 0 0 0 18 0 1 0 670557307 291819520 6898 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122734095 0 0 16781312 514 10290044280261017860 0 0 17 5 0 0 0
/proc/4904/statm: 71245 6898 355 364 0 7163 0
[pid=4906] ppid=4904 vsize=1020308 CPUtime=3.78 cores=1,3,5,7
/proc/4906/stat : 4906 (gnovelty+2) R 4904 4906 4906 0 -1 4202496 1157 0 0 0 378 0 0 0 25 0 1 0 670557307 1044795392 892 33554432000 134512640 135663704 4291396480 18446744073709551615 134525361 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4906/statm: 255077 892 113 282 0 254793 0
[pid=4898/tid=4905] ppid=4896 vsize=294132 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4905/stat : 4905 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 2 0 670557307 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 6.01
Current children cumulated vsize (KiB) 1599420

[startup+12.7008 s]
/proc/loadavg: 4.06 4.06 4.05 5/477 4906
/proc/meminfo: memFree=19703132/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=294132 CPUtime=2.23 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 14387 931 1 0 213 10 0 0 20 0 2 0 670557057 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/4898/statm: 73533 8423 1877 364 0 9451 0
[pid=4904] ppid=4898 vsize=284980 CPUtime=0.03 cores=1,3,5,7
/proc/4904/stat : 4904 (python) S 4898 4898 4581 0 -1 4202560 879 0 0 0 1 2 0 0 18 0 1 0 670557307 291819520 6921 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122734095 0 0 16781312 514 11448414135566434564 0 0 17 5 0 0 0
/proc/4904/statm: 71245 6921 378 364 0 7163 0
[pid=4906] ppid=4904 vsize=1020308 CPUtime=10.18 cores=1,3,5,7
/proc/4906/stat : 4906 (gnovelty+2) R 4904 4906 4906 0 -1 4202496 1157 0 0 0 1018 0 0 0 25 0 1 0 670557307 1044795392 892 33554432000 134512640 135663704 4291396480 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 7 0 0 0
/proc/4906/statm: 255077 892 113 282 0 254793 0
[pid=4898/tid=4905] ppid=4896 vsize=294132 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4905/stat : 4905 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 2 0 670557307 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.44
Current children cumulated vsize (KiB) 1599420

[startup+25.5005 s]
/proc/loadavg: 4.05 4.06 4.05 5/477 4906
/proc/meminfo: memFree=19702728/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=294132 CPUtime=2.23 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 14387 931 1 0 213 10 0 0 20 0 2 0 670557057 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/4898/statm: 73533 8423 1877 364 0 9451 0
[pid=4904] ppid=4898 vsize=284980 CPUtime=0.06 cores=1,3,5,7
/proc/4904/stat : 4904 (python) S 4898 4898 4581 0 -1 4202560 883 0 0 0 3 3 0 0 18 0 1 0 670557307 291819520 6921 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122734095 0 0 16781312 514 11448414135566434564 0 0 17 5 0 0 0
/proc/4904/statm: 71245 6921 378 364 0 7163 0
[pid=4906] ppid=4904 vsize=1020308 CPUtime=22.95 cores=1,3,5,7
/proc/4906/stat : 4906 (gnovelty+2) R 4904 4906 4906 0 -1 4202496 1157 0 0 0 2295 0 0 0 25 0 1 0 670557307 1044795392 892 33554432000 134512640 135663704 4291396480 18446744073709551615 134545646 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4906/statm: 255077 892 113 282 0 254793 0
[pid=4898/tid=4905] ppid=4896 vsize=294132 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4905/stat : 4905 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 2 0 670557307 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.24
Current children cumulated vsize (KiB) 1599420

[startup+51.1009 s]
/proc/loadavg: 4.03 4.05 4.05 5/477 4907
/proc/meminfo: memFree=19700116/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=294132 CPUtime=2.23 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 14387 931 1 0 213 10 0 0 20 0 2 0 670557057 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/4898/statm: 73533 8423 1877 364 0 9451 0
[pid=4904] ppid=4898 vsize=284980 CPUtime=0.14 cores=1,3,5,7
/proc/4904/stat : 4904 (python) S 4898 4898 4581 0 -1 4202560 895 0 0 0 7 7 0 0 18 0 1 0 670557307 291819520 6921 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122734095 0 0 16781312 514 11448414135566434564 0 0 17 5 0 0 0
/proc/4904/statm: 71245 6921 378 364 0 7163 0
[pid=4906] ppid=4904 vsize=1020308 CPUtime=48.54 cores=1,3,5,7
/proc/4906/stat : 4906 (gnovelty+2) R 4904 4906 4906 0 -1 4202496 1157 0 0 0 4854 0 0 0 25 0 1 0 670557307 1044795392 892 33554432000 134512640 135663704 4291396480 18446744073709551615 134544259 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4906/statm: 255077 892 113 282 0 254793 0
[pid=4898/tid=4905] ppid=4896 vsize=294132 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4905/stat : 4905 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 2 0 670557307 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.91
Current children cumulated vsize (KiB) 1599420

[startup+102.316 s]
/proc/loadavg: 4.01 4.04 4.04 5/478 4910
/proc/meminfo: memFree=19691680/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=294132 CPUtime=2.23 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 14387 931 1 0 213 10 0 0 20 0 2 0 670557057 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/4898/statm: 73533 8423 1877 364 0 9451 0
[pid=4904] ppid=4898 vsize=284980 CPUtime=0.3 cores=1,3,5,7
/proc/4904/stat : 4904 (python) S 4898 4898 4581 0 -1 4202560 912 0 0 0 15 15 0 0 18 0 1 0 670557307 291819520 6921 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122734095 0 0 16781312 514 11448414135566434564 0 0 17 5 0 0 0
/proc/4904/statm: 71245 6921 378 364 0 7163 0
[pid=4906] ppid=4904 vsize=1020308 CPUtime=99.71 cores=1,3,5,7
/proc/4906/stat : 4906 (gnovelty+2) R 4904 4906 4906 0 -1 4202496 1157 0 0 0 9971 0 0 0 25 0 1 0 670557307 1044795392 892 33554432000 134512640 135663704 4291396480 18446744073709551615 134524918 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/4906/statm: 255077 892 113 282 0 254793 0
[pid=4898/tid=4905] ppid=4896 vsize=294132 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4905/stat : 4905 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 2 0 670557307 301191168 8423 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 102.24
Current children cumulated vsize (KiB) 1599420

[startup+162.301 s]
/proc/loadavg: 4.07 4.06 4.05 5/482 4915
/proc/meminfo: memFree=19677344/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=304996 CPUtime=2.26 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 15269 931 1 0 216 10 0 0 19 0 3 0 670557057 312315904 8565 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/4898/statm: 76249 8565 1880 364 0 12164 0
[pid=4904] ppid=4898 vsize=295224 CPUtime=0.31 cores=1,3,5,7
/proc/4904/stat : 4904 (python) S 4898 4898 4581 0 -1 4202560 1076 0 0 0 16 15 0 0 19 0 2 0 670557307 302309376 6958 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/4904/statm: 73806 6958 410 364 0 9724 0
[pid=4906] ppid=4904 vsize=1020308 CPUtime=100.26 cores=1,3,5,7
/proc/4906/stat : 4906 (gnovelty+2) T 4904 4906 4906 0 -1 4202496 1157 0 0 0 10026 0 0 0 25 0 1 0 670557307 1044795392 892 33554432000 134512640 135663704 4291396480 18446744073709551615 134525544 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/4906/statm: 255077 892 113 282 0 254793 0
[pid=4904/tid=4911] ppid=4898 vsize=295224 CPUtime=0 cores=1,3,5,7
/proc/4904/task/4911/stat : 4911 (python) S 4898 4898 4581 0 -1 4202560 52 0 0 0 0 0 0 0 19 0 2 0 670567343 302309376 6958 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604455308530880 0 0 -1 7 0 0 0
[pid=4912] ppid=4898 vsize=294904 CPUtime=0.19 cores=1,3,5,7
/proc/4912/stat : 4912 (python) S 4898 4898 4581 0 -1 4202560 910 0 0 0 11 8 0 0 15 0 1 0 670567346 301981696 7054 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122734150 0 0 16781312 514 11448414135566434568 0 0 17 7 0 0 0
/proc/4912/statm: 73726 7054 372 364 0 9641 0
[pid=4914] ppid=4912 vsize=1020308 CPUtime=59.39 cores=1,3,5,7
/proc/4914/stat : 4914 (gnovelty+2) R 4912 4914 4914 0 -1 4202496 1156 0 0 0 5939 0 0 0 25 0 1 0 670567347 1044795392 891 33554432000 134512640 135663704 4294545520 18446744073709551615 134524498 0 0 16781312 0 0 0 0 17 1 0 0 0
/proc/4914/statm: 255077 891 113 282 0 254793 0
[pid=4898/tid=4905] ppid=4896 vsize=304996 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4905/stat : 4905 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 3 0 670557307 312315904 8565 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=4913] ppid=4896 vsize=304996 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4913/stat : 4913 (python) S 4896 4898 4581 0 -1 4202560 28 931 0 0 0 0 0 0 20 0 3 0 670567346 312315904 8565 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 162.41
Current children cumulated vsize (KiB) 2935740

[startup+222.301 s]
/proc/loadavg: 4.06 4.06 4.05 5/438 5065
/proc/meminfo: memFree=19735252/32951124 swapFree=49826708/67111528
[pid=4898] ppid=4896 vsize=315252 CPUtime=2.29 cores=1,3,5,7
/proc/4898/stat : 4898 (python) S 4896 4898 4581 0 -1 4202496 16103 931 1 0 219 10 0 0 20 0 4 0 670557057 322818048 8571 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0

################
# More data... #
################

/proc/5071/statm: 105850 2358 59 146 0 104726 0
[pid=5069/tid=5079] ppid=4898 vsize=315680 CPUtime=0 cores=1,3,5,7
/proc/5069/task/5079/stat : 5079 (python) S 4898 4898 4581 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 670597354 323256320 7160 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604458451059840 0 0 -1 5 0 0 0
[pid=5080] ppid=4898 vsize=326032 CPUtime=0.32 cores=1,3,5,7
/proc/5080/stat : 5080 (python) S 4898 4898 4581 0 -1 4202560 1108 0 0 0 18 14 0 0 19 0 2 0 670597358 333856768 7164 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5080/statm: 81508 7164 400 364 0 17414 0
[pid=5082] ppid=5080 vsize=423400 CPUtime=99.32 cores=1,3,5,7
/proc/5082/stat : 5082 (adaptg2wsat2009) T 5080 5082 5082 0 -1 4202496 2624 0 0 0 9932 0 0 0 25 0 1 0 670597358 433561600 2358 33554432000 134512640 135107411 4293166048 18446744073709551615 134532302 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5082/statm: 105850 2358 59 146 0 104726 0
[pid=5080/tid=5085] ppid=4898 vsize=326032 CPUtime=0 cores=1,3,5,7
/proc/5080/task/5085/stat : 5085 (python) S 4898 4898 4581 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670607291 333856768 7164 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604462499337280 0 0 -1 7 0 0 0
[pid=5086] ppid=4898 vsize=336236 CPUtime=0.32 cores=1,3,5,7
/proc/5086/stat : 5086 (python) S 4898 4898 4581 0 -1 4202560 1121 0 0 0 17 15 0 0 19 0 2 0 670607295 344305664 7168 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5086/statm: 84059 7168 400 364 0 19962 0
[pid=5088] ppid=5086 vsize=1020308 CPUtime=99.31 cores=1,3,5,7
/proc/5088/stat : 5088 (gnovelty+2) T 5086 5088 5088 0 -1 4202496 1159 0 0 0 9931 0 0 0 25 0 1 0 670607295 1044795392 891 33554432000 134512640 135663704 4290661616 18446744073709551615 134525181 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5088/statm: 255077 891 113 282 0 254793 0
[pid=5086/tid=5092] ppid=4898 vsize=336236 CPUtime=0 cores=1,3,5,7
/proc/5086/task/5092/stat : 5092 (python) S 4898 4898 4581 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670617232 344305664 7168 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604467647653888 0 0 -1 7 0 0 0
[pid=5093] ppid=4898 vsize=346604 CPUtime=0.31 cores=1,3,5,7
/proc/5093/stat : 5093 (python) S 4898 4898 4581 0 -1 4202560 1182 0 0 0 14 17 0 0 19 0 2 0 670617236 354922496 7178 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5093/statm: 86651 7178 406 364 0 22551 0
[pid=5095] ppid=5093 vsize=253032 CPUtime=100.31 cores=1,3,5,7
/proc/5095/stat : 5095 (FH) T 5093 5095 5095 0 -1 4202496 2502 0 0 0 10031 0 0 0 25 0 1 0 670617236 259104768 2233 33554432000 134512640 134936988 4292610800 18446744073709551615 134554610 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5095/statm: 63258 2233 41 104 0 62762 0
[pid=5093/tid=5103] ppid=4898 vsize=346604 CPUtime=0 cores=1,3,5,7
/proc/5093/task/5103/stat : 5103 (python) S 4898 4898 4581 0 -1 4202560 96 0 0 0 0 0 0 0 19 0 2 0 670627268 354922496 7178 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604459895310464 0 0 -1 5 0 0 0
[pid=5104] ppid=4898 vsize=356792 CPUtime=0.32 cores=1,3,5,7
/proc/5104/stat : 5104 (python) S 4898 4898 4581 0 -1 4202560 1161 0 0 0 15 17 0 0 19 0 2 0 670627272 365355008 7182 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5104/statm: 89198 7182 406 364 0 25095 0
[pid=5106] ppid=5104 vsize=423400 CPUtime=100.32 cores=1,3,5,7
/proc/5106/stat : 5106 (adaptg2wsat2009) T 5104 5106 5106 0 -1 4202496 2636 0 0 0 10032 0 0 0 25 0 1 0 670627272 433561600 2358 33554432000 134512640 135107411 4292083664 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5106/statm: 105850 2358 59 146 0 104726 0
[pid=5104/tid=5110] ppid=4898 vsize=356792 CPUtime=0 cores=1,3,5,7
/proc/5104/task/5110/stat : 5110 (python) S 4898 4898 4581 0 -1 4202560 52 0 0 0 0 0 0 0 19 0 2 0 670637304 365355008 7182 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604460347698176 0 0 -1 7 0 0 0
[pid=5111] ppid=4898 vsize=367176 CPUtime=0.33 cores=1,3,5,7
/proc/5111/stat : 5111 (python) S 4898 4898 4581 0 -1 4202560 1124 0 0 0 17 16 0 0 19 0 2 0 670637308 375988224 7180 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/5111/statm: 91794 7180 400 364 0 27688 0
[pid=5113] ppid=5111 vsize=423400 CPUtime=100.31 cores=1,3,5,7
/proc/5113/stat : 5113 (adaptg2wsat2009) T 5111 5113 5113 0 -1 4202496 2630 0 0 0 10031 0 0 0 25 0 1 0 670637309 433561600 2358 33554432000 134512640 135107411 4291287072 18446744073709551615 134531781 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5113/statm: 105850 2358 59 146 0 104726 0
[pid=5111/tid=5126] ppid=4898 vsize=367176 CPUtime=0 cores=1,3,5,7
/proc/5111/task/5126/stat : 5126 (python) S 4898 4898 4581 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670647341 375988224 7180 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604454413906944 0 0 -1 1 0 0 0
[pid=5127] ppid=4898 vsize=377408 CPUtime=0.33 cores=1,3,5,7
/proc/5127/stat : 5127 (python) S 4898 4898 4581 0 -1 4202560 1137 0 0 0 17 16 0 0 19 0 2 0 670647344 386465792 7184 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5127/statm: 94352 7184 400 364 0 30243 0
[pid=5129] ppid=5127 vsize=253032 CPUtime=100.26 cores=1,3,5,7
/proc/5129/stat : 5129 (FH) T 5127 5129 5129 0 -1 4202496 2506 0 0 0 10026 0 0 0 25 0 1 0 670647345 259104768 2234 33554432000 134512640 134936988 4287779648 18446744073709551615 134544735 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/5129/statm: 63258 2234 41 104 0 62762 0
[pid=5127/tid=5133] ppid=4898 vsize=377408 CPUtime=0 cores=1,3,5,7
/proc/5127/task/5133/stat : 5133 (python) S 4898 4898 4581 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670657377 386465792 7184 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604467459995840 0 0 -1 7 0 0 0
[pid=5134] ppid=4898 vsize=387804 CPUtime=0.65 cores=1,3,5,7
/proc/5134/stat : 5134 (python) S 4898 4898 4581 0 -1 4202560 1164 0 0 0 35 30 0 0 19 0 2 0 670657381 397111296 7188 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5134/statm: 96951 7188 400 364 0 32839 0
[pid=5136] ppid=5134 vsize=268428 CPUtime=196.5 cores=1,3,5,7
/proc/5136/stat : 5136 (hybridGM3) T 5134 5136 5136 0 -1 4202496 6191 0 0 0 19650 0 0 0 25 0 1 0 670657381 274870272 5919 33554432000 134512640 135204385 4294088496 18446744073709551615 134538013 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5136/statm: 67107 5919 63 169 0 66937 0
[pid=5134/tid=5145] ppid=4898 vsize=387804 CPUtime=0 cores=1,3,5,7
/proc/5134/task/5145/stat : 5145 (python) S 4898 4898 4581 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670677052 397111296 7188 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604470721050496 0 0 -1 7 0 0 0
[pid=5146] ppid=4898 vsize=398004 CPUtime=0.32 cores=1,3,5,7
/proc/5146/stat : 5146 (python) S 4898 4898 4581 0 -1 4202560 1152 0 0 0 17 15 0 0 19 0 2 0 670677056 407556096 7192 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/5146/statm: 99501 7192 400 364 0 35386 0
[pid=5148] ppid=5146 vsize=1020308 CPUtime=100.25 cores=1,3,5,7
/proc/5148/stat : 5148 (gnovelty+2) T 5146 5148 5148 0 -1 4202496 1163 0 0 0 10025 0 0 0 25 0 1 0 670677057 1044795392 891 33554432000 134512640 135663704 4288336400 18446744073709551615 134545190 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/5148/statm: 255077 891 113 282 0 254793 0
[pid=5146/tid=5151] ppid=4898 vsize=398004 CPUtime=0 cores=1,3,5,7
/proc/5146/task/5151/stat : 5151 (python) S 4898 4898 4581 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670687093 407556096 7192 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604470721048704 0 0 -1 7 0 0 0
[pid=5152] ppid=4898 vsize=408576 CPUtime=0.65 cores=1,3,5,7
/proc/5152/stat : 5152 (python) S 4898 4898 4581 0 -1 4202560 1165 0 0 0 33 32 0 0 19 0 2 0 670687097 418381824 7196 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5152/statm: 102144 7196 400 364 0 38026 0
[pid=5154] ppid=5152 vsize=268424 CPUtime=194.75 cores=1,3,5,7
/proc/5154/stat : 5154 (hybridGM3) T 5152 5154 5154 0 -1 4202496 6193 0 0 0 19475 0 0 0 25 0 1 0 670687097 274866176 5920 33554432000 134512640 135204385 4294853632 18446744073709551615 134538016 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5154/statm: 67106 5920 63 169 0 66936 0
[pid=5152/tid=5165] ppid=4898 vsize=408576 CPUtime=0 cores=1,3,5,7
/proc/5152/task/5165/stat : 5165 (python) S 4898 4898 4581 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670706573 418381824 7196 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604463894138944 0 0 -1 7 0 0 0
[pid=5166] ppid=4898 vsize=418544 CPUtime=0.33 cores=1,3,5,7
/proc/5166/stat : 5166 (python) S 4898 4898 4581 0 -1 4202560 1155 0 0 0 16 17 0 0 19 0 2 0 670706577 428589056 7262 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5166/statm: 104636 7262 400 364 0 40515 0
[pid=5168] ppid=5166 vsize=423400 CPUtime=99.27 cores=1,3,5,7
/proc/5168/stat : 5168 (adaptg2wsat2009) T 5166 5168 5168 0 -1 4202496 2631 0 0 0 9927 0 0 0 25 0 1 0 670706577 433561600 2357 33554432000 134512640 135107411 4294761328 18446744073709551615 134531942 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5168/statm: 105850 2357 59 146 0 104726 0
[pid=5166/tid=5171] ppid=4898 vsize=418544 CPUtime=0 cores=1,3,5,7
/proc/5166/task/5171/stat : 5171 (python) S 4898 4898 4581 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670716511 428589056 7262 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604454831793344 0 0 -1 7 0 0 0
[pid=5172] ppid=4898 vsize=428972 CPUtime=0.68 cores=1,3,5,7
/proc/5172/stat : 5172 (python) S 4898 4898 4581 0 -1 4202560 1195 0 0 0 37 31 0 0 19 0 2 0 670716514 439267328 7273 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5172/statm: 107243 7273 407 364 0 43119 0
[pid=5174] ppid=5172 vsize=268424 CPUtime=199.76 cores=1,3,5,7
/proc/5174/stat : 5174 (hybridGM3) T 5172 5174 5174 0 -1 4202496 6195 0 0 0 19976 0 0 0 25 0 1 0 670716515 274866176 5919 33554432000 134512640 135204385 4289941168 18446744073709551615 134537792 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5174/statm: 67106 5919 63 169 0 66936 0
[pid=5172/tid=5184] ppid=4898 vsize=428972 CPUtime=0 cores=1,3,5,7
/proc/5172/task/5184/stat : 5184 (python) S 4898 4898 4581 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 670736493 439267328 7273 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604469435192640 0 0 -1 7 0 0 0
[pid=5185] ppid=4898 vsize=439232 CPUtime=0.66 cores=1,3,5,7
/proc/5185/stat : 5185 (python) S 4898 4898 4581 0 -1 4202560 1217 0 0 0 31 35 0 0 19 0 2 0 670736496 449773568 7277 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5185/statm: 109808 7277 407 364 0 45681 0
[pid=5187] ppid=5185 vsize=268424 CPUtime=195.72 cores=1,3,5,7
/proc/5187/stat : 5187 (hybridGM3) T 5185 5187 5187 0 -1 4202496 6194 0 0 0 19572 0 0 0 25 0 1 0 670736497 274866176 5919 33554432000 134512640 135204385 4286650944 18446744073709551615 134537776 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/5187/statm: 67106 5919 63 169 0 66936 0
[pid=5185/tid=5194] ppid=4898 vsize=439232 CPUtime=0 cores=1,3,5,7
/proc/5185/task/5194/stat : 5194 (python) S 4898 4898 4581 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670756073 449773568 7277 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604469435193536 0 0 -1 7 0 0 0
[pid=5195] ppid=4898 vsize=449676 CPUtime=5.44 cores=1,3,5,7
/proc/5195/stat : 5195 (python) S 4898 4898 4581 0 -1 4202560 1201 0 0 0 287 257 0 0 19 0 2 0 670756077 460468224 7275 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/5195/statm: 112419 7275 401 364 0 48289 0
[pid=5197] ppid=5195 vsize=461508 CPUtime=1556.49 cores=1,3,5,7
/proc/5197/stat : 5197 (TNM) T 5195 5197 5197 0 -1 4202496 2471 0 0 0 155649 0 0 0 25 0 1 0 670756077 472584192 2194 33554432000 134512640 135008748 4290713456 18446744073709551615 134527680 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/5197/statm: 115377 2194 46 122 0 114277 0
[pid=5195/tid=5251] ppid=4898 vsize=449676 CPUtime=0 cores=1,3,5,7
/proc/5195/task/5251/stat : 5251 (python) S 4898 4898 4581 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670911723 460468224 7275 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 514 18446604464967798848 0 0 -1 7 0 0 0
[pid=5252] ppid=4898 vsize=460108 CPUtime=0 cores=1,3,5,7
/proc/5252/stat : 5252 (python) S 4898 4898 4581 0 -1 4202560 827 0 0 0 0 0 0 0 18 0 1 0 670911725 471150592 7242 33554432000 4194304 5683644 140737361495184 18446744073709551615 228122734150 0 0 16781312 514 10384355165011443711 0 0 17 1 0 0 0
/proc/5252/statm: 115027 7242 346 364 0 50894 0
[pid=5254] ppid=5252 vsize=248616 CPUtime=0.74 cores=1,3,5,7
/proc/5254/stat : 5254 (FH) R 5252 5254 5254 0 -1 4202496 1487 0 0 0 74 0 0 0 21 0 1 0 670911725 254582784 1214 33554432000 134512640 134936988 4294522896 18446744073709551615 134523650 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/5254/statm: 62154 1214 37 104 0 61658 0
[pid=4898/tid=4905] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4905/stat : 4905 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 19 0 670557307 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=4913] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/4913/stat : 4913 (python) S 4896 4898 4581 0 -1 4202560 28 931 0 0 0 0 0 0 20 0 19 0 670567346 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 7 0 0 0
[pid=4898/tid=5003] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5003/stat : 5003 (python) S 4896 4898 4581 0 -1 4202560 23 931 0 0 0 0 0 0 20 0 19 0 670577287 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5070] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5070/stat : 5070 (python) S 4896 4898 4581 0 -1 4202560 27 931 0 0 0 0 0 0 20 0 19 0 670587321 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 7 0 0 0
[pid=4898/tid=5081] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5081/stat : 5081 (python) S 4896 4898 4581 0 -1 4202560 30 931 0 0 0 0 0 0 20 0 19 0 670597358 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5087] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5087/stat : 5087 (python) S 4896 4898 4581 0 -1 4202560 30 931 0 0 0 0 0 0 20 0 19 0 670607295 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5094] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5094/stat : 5094 (python) S 4896 4898 4581 0 -1 4202560 33 931 0 0 0 0 0 0 20 0 19 0 670617236 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 1 0 0 0
[pid=4898/tid=5105] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5105/stat : 5105 (python) S 4896 4898 4581 0 -1 4202560 32 931 0 0 0 0 0 0 20 0 19 0 670627272 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 1 0 0 0
[pid=4898/tid=5112] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5112/stat : 5112 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 19 0 670637308 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5128] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5128/stat : 5128 (python) S 4896 4898 4581 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 19 0 670647344 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5135] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5135/stat : 5135 (python) S 4896 4898 4581 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 19 0 670657381 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5147] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5147/stat : 5147 (python) S 4896 4898 4581 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 19 0 670677056 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5153] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5153/stat : 5153 (python) S 4896 4898 4581 0 -1 4202560 28 931 0 0 0 0 0 0 20 0 19 0 670687097 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5167] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5167/stat : 5167 (python) S 4896 4898 4581 0 -1 4202560 27 931 0 0 0 0 0 0 20 0 19 0 670706577 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5173] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5173/stat : 5173 (python) S 4896 4898 4581 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 19 0 670716515 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 5 0 0 0
[pid=4898/tid=5186] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5186/stat : 5186 (python) S 4896 4898 4581 0 -1 4202560 24 931 0 0 0 0 0 0 20 0 19 0 670736496 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 1 0 0 0
[pid=4898/tid=5196] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5196/stat : 5196 (python) S 4896 4898 4581 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 19 0 670756077 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 1 0 0 0
[pid=4898/tid=5253] ppid=4896 vsize=469616 CPUtime=0 cores=1,3,5,7
/proc/4898/task/5253/stat : 5253 (python) S 4896 4898 4581 0 -1 4202560 28 931 0 0 0 0 0 0 20 0 19 0 670911725 480886784 8811 33554432000 4194304 5683644 140737361495184 18446744073709551615 228134537649 0 0 16781312 2 18446604455308532672 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3558.21
Current children cumulated vsize (KiB) 16093876

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 4898 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=711587
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=146977
# CPU time returned by wait4() is 2.85856
# while last known CPU time is 3558.21
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 3547.54
CPU time (s): 3558.21
CPU user time (s): 3552.38
CPU system time (s): 5.83
CPU usage (%): 100.301
Max. virtual memory (cumulated for all children) (KiB): 16093876

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.71159
system time used= 0.146977
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 31697
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2919
involuntary context switches= 75

runsolver used 29.3695 second user time and 65.901 second system time

The end

Launcher Data 

Begin job on node138 at 2011-05-17 01:46:31
IDJOB=3386610
IDBENCH=82675
IDSOLVER=1872
FILE ID=node138/3386610-1305589590
RUNJOBID= node138-1305586144-4600
PBS_JOBID= 13350340
Free space on /tmp= 73392 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S922811046-076-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386610-1305589590/watcher-3386610-1305589590 -o /tmp/evaluation-result-3386610-1305589590/solver-3386610-1305589590 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386610-1305589590.cnf -seed 785539133 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e99bff721d695892a53f60adc73d403b
RANDOM SEED=785539133

node138.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19740740 kB
Buffers:         12064 kB
Cached:         240152 kB
SwapCached:      70944 kB
Active:        2948264 kB
Inactive:     10012160 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19740740 kB
SwapTotal:    67111528 kB
SwapFree:     49826708 kB
Dirty:           32980 kB
Writeback:           0 kB
AnonPages:    12647584 kB
Mapped:          20928 kB
Slab:            78460 kB
PageTables:     102872 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 116698220 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73352 MiB
End job on node138 at 2011-05-17 02:45:40