Trace number 3386624

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (MO) 3459.48 3446.34

General information on the benchmark

Namerandom/large/
unif-k7-r85-v250-c21250-S58287803-001-UNKNOWN.cnf
MD5SUM0d5d355ade4a956f83626786ca6018b7
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables250
Number of clauses21250
Sum of the clauses size148750
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 521250

Solver Data 

0.35/0.65	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.35/0.65	c INFO: solving HOME/instance-3386624-1305589668.cnf
0.35/0.65	c INFO: computing task features
0.43/0.79	c INFO: computed variable-clause graph statistics
0.43/0.79	c INFO: computed clause balance statistics
1.33/1.60	c INFO: computed variable balance statistics
1.63/1.91	c INFO: computed small-clause counts
1.63/1.91	c INFO: computed Horn-clause counts
1.63/1.94	c INFO: computed variable graph statistics
1.63/1.94	c INFO: collected features for HOME/instance-3386624-1305589668.cnf in 1.29 s
1.63/1.97	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.36)
99.47/99.43	c INFO: running gnovelty+2@0 for 97 with 4901 remaining (b = 0.13)
199.16/198.86	c INFO: running adaptg2wsat2009++@0 for 97 with 4801 remaining (b = 0.08)
299.83/299.26	c INFO: running adaptg2wsat2009++@0 for 97 with 4701 remaining (b = 0.07)
402.71/401.73	c INFO: running adaptg2wsat2009++@0 for 97 with 4599 remaining (b = 0.05)
500.45/499.17	c INFO: running gnovelty+2@0 for 97 with 4501 remaining (b = 0.04)
601.34/599.64	c INFO: running FH@0 for 97 with 4401 remaining (b = 0.03)
700.11/698.07	c INFO: running adaptg2wsat2009++@0 for 97 with 4302 remaining (b = 0.03)
800.89/798.51	c INFO: running adaptg2wsat2009++@0 for 97 with 4202 remaining (b = 0.02)
903.90/901.00	c INFO: running FH@0 for 97 with 4100 remaining (b = 0.02)
1006.81/1003.50	c INFO: running hybridGM3@0 for 194 with 3997 remaining (b = 0.04)
1204.13/1200.33	c INFO: running gnovelty+2@0 for 97 with 3801 remaining (b = 0.02)
1302.99/1298.82	c INFO: running hybridGM3@0 for 194 with 3702 remaining (b = 0.03)
1499.66/1494.72	c INFO: running hybridGM3@0 for 194 with 3506 remaining (b = 0.03)
1698.35/1692.68	c INFO: running TNM@0 for 1552 with 3308 remaining (b = 0.20)
3259.86/3247.44	c INFO: running adaptg2wsat2009++@0 for 97 with 1753 remaining (b = 0.01)
3360.65/3347.88	c INFO: running FH@0 for 97 with 1653 remaining (b = 0.01)
3458.56/3445.34	c INFO: running hybridGM3@0 for 194 with 1556 remaining (b = 0.02)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386624-1305589668/watcher-3386624-1305589668 -o /tmp/evaluation-result-3386624-1305589668/solver-3386624-1305589668 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386624-1305589668.cnf -seed 1865723472 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.01 4.07 4.10 4/501 8905
/proc/meminfo: memFree=20176360/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=15384 CPUtime=0 cores=1,3,5,7
/proc/8905/stat : 8905 (python) R 8903 8905 8511 0 -1 4202496 809 0 0 0 0 0 0 0 22 0 1 0 670564764 15753216 696 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244344677 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/8905/statm: 3846 696 334 364 0 333 0

[startup+0.028895 s]
/proc/loadavg: 4.01 4.07 4.10 4/501 8905
/proc/meminfo: memFree=20176360/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=16400 CPUtime=0.01 cores=1,3,5,7
/proc/8905/stat : 8905 (python) R 8903 8905 8511 0 -1 4202496 1120 0 0 0 1 0 0 0 17 0 1 0 670564764 16793600 989 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244346880 0 2147483391 16781312 2 0 0 0 17 1 0 0 0
/proc/8905/statm: 4100 989 413 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+0.100881 s]
/proc/loadavg: 4.01 4.07 4.10 4/501 8905
/proc/meminfo: memFree=20176360/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=89772 CPUtime=0.06 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 2671 0 0 0 4 2 0 0 17 0 1 0 670564764 91926528 2388 33554432000 4194304 5683644 140733503542496 18446744073709551615 228239425159 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/8905/statm: 22443 2388 750 364 0 1623 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 89772

[startup+0.300848 s]
/proc/loadavg: 4.01 4.07 4.10 4/501 8905
/proc/meminfo: memFree=20176360/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=135752 CPUtime=0.17 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 5973 933 1 0 12 5 0 0 17 0 1 0 670564764 139010048 4149 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244344837 0 2147483391 16781312 2 18446744071699199166 0 0 17 1 0 0 0
/proc/8905/statm: 33938 4149 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135752

[startup+0.70078 s]
/proc/loadavg: 4.01 4.07 4.10 4/501 8905
/proc/meminfo: memFree=20176360/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=280288 CPUtime=0.43 cores=1,3,5,7
/proc/8905/stat : 8905 (python) R 8903 8905 8511 0 -1 4202496 9745 933 1 0 33 10 0 0 15 0 1 0 670564764 287014912 7416 33554432000 4194304 5683644 140733503542496 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/8905/statm: 70072 7416 1761 364 0 5996 0
Current children cumulated CPU time (s) 0.43
Current children cumulated vsize (KiB) 280288

[startup+1.50064 s]
/proc/loadavg: 4.01 4.07 4.10 5/502 8910
/proc/meminfo: memFree=20147584/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=283976 CPUtime=1.23 cores=1,3,5,7
/proc/8905/stat : 8905 (python) R 8903 8905 8511 0 -1 4202496 11753 933 1 0 113 10 0 0 16 0 1 0 670564764 290791424 8378 33554432000 4194304 5683644 140733503542496 18446744073709551615 4602441 0 0 16781312 2 0 0 0 17 1 0 0 0
/proc/8905/statm: 70994 8378 1826 364 0 6918 0
Current children cumulated CPU time (s) 1.23
Current children cumulated vsize (KiB) 283976

[startup+3.10037 s]
/proc/loadavg: 4.01 4.07 4.10 5/505 8913
/proc/meminfo: memFree=20146940/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=289868 CPUtime=1.69 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 12110 933 1 0 159 10 0 0 19 0 2 0 670564764 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/8905/statm: 72467 7339 1877 364 0 8385 0
[pid=8911] ppid=8905 vsize=280108 CPUtime=0 cores=1,3,5,7
/proc/8911/stat : 8911 (python) S 8905 8905 8511 0 -1 4202560 738 0 0 0 0 0 0 0 18 0 1 0 670564961 286830592 5814 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244368911 0 0 16781312 514 11217046656807567620 0 0 17 1 0 0 0
/proc/8911/statm: 70027 5814 355 364 0 5945 0
[pid=8913] ppid=8911 vsize=1018820 CPUtime=1.12 cores=1,3,5,7
/proc/8913/stat : 8913 (gnovelty+2) R 8911 8913 8913 0 -1 4202496 857 0 0 0 112 0 0 0 22 0 1 0 670564961 1043271680 595 33554432000 134512640 135663704 4292936624 18446744073709551615 134524553 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8913/statm: 254705 595 113 282 0 254421 0
[pid=8905/tid=8912] ppid=8903 vsize=289868 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8912/stat : 8912 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 2 0 670564961 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 2.81
Current children cumulated vsize (KiB) 1588796

[startup+6.3008 s]
/proc/loadavg: 4.01 4.07 4.10 5/505 8913
/proc/meminfo: memFree=20147668/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=289868 CPUtime=1.69 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 12110 933 1 0 159 10 0 0 19 0 2 0 670564764 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/8905/statm: 72467 7339 1877 364 0 8385 0
[pid=8911] ppid=8905 vsize=280108 CPUtime=0.02 cores=1,3,5,7
/proc/8911/stat : 8911 (python) S 8905 8905 8511 0 -1 4202560 885 0 0 0 1 1 0 0 18 0 1 0 670564961 286830592 5837 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244368911 0 0 16781312 514 11217046656807567620 0 0 17 7 0 0 0
/proc/8911/statm: 70027 5837 378 364 0 5945 0
[pid=8913] ppid=8911 vsize=1018820 CPUtime=4.32 cores=1,3,5,7
/proc/8913/stat : 8913 (gnovelty+2) R 8911 8913 8913 0 -1 4202496 857 0 0 0 432 0 0 0 25 0 1 0 670564961 1043271680 595 33554432000 134512640 135663704 4292936624 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8913/statm: 254705 595 113 282 0 254421 0
[pid=8905/tid=8912] ppid=8903 vsize=289868 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8912/stat : 8912 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 2 0 670564961 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 6.03
Current children cumulated vsize (KiB) 1588796

[startup+12.7007 s]
/proc/loadavg: 4.00 4.07 4.10 5/505 8913
/proc/meminfo: memFree=20144840/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=289868 CPUtime=1.69 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 12110 933 1 0 159 10 0 0 19 0 2 0 670564764 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/8905/statm: 72467 7339 1877 364 0 8385 0
[pid=8911] ppid=8905 vsize=280108 CPUtime=0.04 cores=1,3,5,7
/proc/8911/stat : 8911 (python) S 8905 8905 8511 0 -1 4202560 887 0 0 0 2 2 0 0 18 0 1 0 670564961 286830592 5837 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244368911 0 0 16781312 514 8777902701838041348 0 0 17 7 0 0 0
/proc/8911/statm: 70027 5837 378 364 0 5945 0
[pid=8913] ppid=8911 vsize=1018820 CPUtime=10.72 cores=1,3,5,7
/proc/8913/stat : 8913 (gnovelty+2) R 8911 8913 8913 0 -1 4202496 857 0 0 0 1072 0 0 0 25 0 1 0 670564961 1043271680 595 33554432000 134512640 135663704 4292936624 18446744073709551615 134544144 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8913/statm: 254705 595 113 282 0 254421 0
[pid=8905/tid=8912] ppid=8903 vsize=289868 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8912/stat : 8912 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 2 0 670564961 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 12.45
Current children cumulated vsize (KiB) 1588796

[startup+25.5004 s]
/proc/loadavg: 4.00 4.06 4.10 5/505 8913
/proc/meminfo: memFree=20143256/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=289868 CPUtime=1.69 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 12110 933 1 0 159 10 0 0 19 0 2 0 670564764 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/8905/statm: 72467 7339 1877 364 0 8385 0
[pid=8911] ppid=8905 vsize=280108 CPUtime=0.08 cores=1,3,5,7
/proc/8911/stat : 8911 (python) S 8905 8905 8511 0 -1 4202560 891 0 0 0 4 4 0 0 18 0 1 0 670564961 286830592 5837 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244368911 0 0 16781312 514 8777902701838041348 0 0 17 7 0 0 0
/proc/8911/statm: 70027 5837 378 364 0 5945 0
[pid=8913] ppid=8911 vsize=1018820 CPUtime=23.52 cores=1,3,5,7
/proc/8913/stat : 8913 (gnovelty+2) R 8911 8913 8913 0 -1 4202496 857 0 0 0 2352 0 0 0 25 0 1 0 670564961 1043271680 595 33554432000 134512640 135663704 4292936624 18446744073709551615 134525255 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8913/statm: 254705 595 113 282 0 254421 0
[pid=8905/tid=8912] ppid=8903 vsize=289868 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8912/stat : 8912 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 2 0 670564961 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 25.29
Current children cumulated vsize (KiB) 1588796

[startup+51.1009 s]
/proc/loadavg: 4.00 4.06 4.09 5/509 8918
/proc/meminfo: memFree=20131888/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=289868 CPUtime=1.69 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 12110 933 1 0 159 10 0 0 19 0 2 0 670564764 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/8905/statm: 72467 7339 1877 364 0 8385 0
[pid=8911] ppid=8905 vsize=280108 CPUtime=0.16 cores=1,3,5,7
/proc/8911/stat : 8911 (python) S 8905 8905 8511 0 -1 4202560 901 0 0 0 9 7 0 0 18 0 1 0 670564961 286830592 5837 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244368911 0 0 16781312 514 8777902701838041348 0 0 17 7 0 0 0
/proc/8911/statm: 70027 5837 378 364 0 5945 0
[pid=8913] ppid=8911 vsize=1018820 CPUtime=49.12 cores=1,3,5,7
/proc/8913/stat : 8913 (gnovelty+2) R 8911 8913 8913 0 -1 4202496 857 0 0 0 4912 0 0 0 25 0 1 0 670564961 1043271680 595 33554432000 134512640 135663704 4292936624 18446744073709551615 134524509 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8913/statm: 254705 595 113 282 0 254421 0
[pid=8905/tid=8912] ppid=8903 vsize=289868 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8912/stat : 8912 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 2 0 670564961 296824832 7339 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.97
Current children cumulated vsize (KiB) 1588796

[startup+102.318 s]
/proc/loadavg: 4.05 4.06 4.09 5/513 8923
/proc/meminfo: memFree=20112552/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=300124 CPUtime=1.73 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 12965 933 1 0 162 11 0 0 19 0 3 0 670564764 307326976 7346 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8905/statm: 75031 7346 1880 364 0 10946 0
[pid=8911] ppid=8905 vsize=290352 CPUtime=0.36 cores=1,3,5,7
/proc/8911/stat : 8911 (python) S 8905 8905 8511 0 -1 4202560 1073 0 0 0 19 17 0 0 19 0 2 0 670564961 297320448 5874 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8911/statm: 72588 5874 410 364 0 8506 0
[pid=8913] ppid=8911 vsize=1018820 CPUtime=97.42 cores=1,3,5,7
/proc/8913/stat : 8913 (gnovelty+2) T 8911 8913 8913 0 -1 4202496 857 0 0 0 9742 0 0 0 25 0 1 0 670564961 1043271680 595 33554432000 134512640 135663704 4292936624 18446744073709551615 134524686 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/8913/statm: 254705 595 113 282 0 254421 0
[pid=8911/tid=8920] ppid=8905 vsize=290352 CPUtime=0 cores=1,3,5,7
/proc/8911/task/8920/stat : 8920 (python) S 8905 8905 8511 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670574704 297320448 5874 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604459819332736 0 0 -1 3 0 0 0
[pid=8921] ppid=8905 vsize=290436 CPUtime=0 cores=1,3,5,7
/proc/8921/stat : 8921 (python) S 8905 8905 8511 0 -1 4202560 745 0 0 0 0 0 0 0 15 0 1 0 670574707 297406464 5812 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244368966 0 0 16781312 514 2966864731203141895 0 0 17 3 0 0 0
/proc/8921/statm: 72609 5812 349 364 0 8524 0
[pid=8923] ppid=8921 vsize=1018820 CPUtime=2.87 cores=1,3,5,7
/proc/8923/stat : 8923 (gnovelty+2) R 8921 8923 8923 0 -1 4202496 863 0 0 0 287 0 0 0 25 0 1 0 670574708 1043271680 596 33554432000 134512640 135663704 4287665344 18446744073709551615 134544098 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/8923/statm: 254705 596 113 282 0 254421 0
[pid=8905/tid=8912] ppid=8903 vsize=300124 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8912/stat : 8912 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 3 0 670564961 307326976 7346 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
[pid=8905/tid=8922] ppid=8903 vsize=300124 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8922/stat : 8922 (python) S 8903 8905 8511 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 3 0 670574707 307326976 7346 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 102.38
Current children cumulated vsize (KiB) 2918552

[startup+162.3 s]
/proc/loadavg: 4.09 4.08 4.09 5/517 8929
/proc/meminfo: memFree=20085936/32951124 swapFree=64205392/67111528
[pid=8905] ppid=8903 vsize=300124 CPUtime=1.73 cores=1,3,5,7
/proc/8905/stat : 8905 (python) S 8903 8905 8511 0 -1 4202496 12965 933 1 0 162 11 0 0 19 0 3 0 670564764 307326976 7346 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 2 18446744073709551615 0 0 17 3 0 0 0
/proc/8905/statm: 75031 7346 1880 364 0 10946 0
[pid=8911] ppid=8905 vsize=290352 CPUtime=0.36 cores=1,3,5,7
/proc/8911/stat : 8911 (python) S 8905 8905 8511 0 -1 4202560 1073 0 0 0 19 17 0 0 19 0 2 0 670564961 297320448 5874 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8911/statm: 72588 5874 410 364 0 8506 0
[pid=8913] ppid=8911 vsize=1018820 CPUtime=97.42 cores=1,3,5,7
/proc/8913/stat : 8913 (gnovelty+2) T 8911 8913 8913 0 -1 4202496 857 0 0 0 9742 0 0 0 25 0 1 0 670564961 1043271680 595 33554432000 134512640 135663704 4292936624 18446744073709551615 134524686 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/8913/statm: 254705 595 113 282 0 254421 0
[pid=8911/tid=8920] ppid=8905 vsize=290352 CPUtime=0 cores=1,3,5,7
/proc/8911/task/8920/stat : 8920 (python) S 8905 8905 8511 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670574704 297320448 5874 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604459819332736 0 0 -1 3 0 0 0
[pid=8921] ppid=8905 vsize=290436 CPUtime=0.21 cores=1,3,5,7

################
# More data... #
################

/proc/8945/statm: 104857 1176 59 146 0 103733 0
[pid=8943/tid=8952] ppid=8905 vsize=311048 CPUtime=0 cores=1,3,5,7
/proc/8943/task/8952/stat : 8952 (python) S 8905 8905 8511 0 -1 4202560 52 0 0 0 0 0 0 0 19 0 2 0 670604934 318513152 5892 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604465220815616 0 0 -1 1 0 0 0
[pid=8953] ppid=8905 vsize=321400 CPUtime=0.38 cores=1,3,5,7
/proc/8953/stat : 8953 (python) S 8905 8905 8511 0 -1 4202560 1101 0 0 0 19 19 0 0 19 0 2 0 670604938 329113600 5919 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8953/statm: 80350 5919 400 364 0 16256 0
[pid=8955] ppid=8953 vsize=419428 CPUtime=97.39 cores=1,3,5,7
/proc/8955/stat : 8955 (adaptg2wsat2009) T 8953 8955 8955 0 -1 4202496 1446 0 0 0 9739 0 0 0 25 0 1 0 670604938 429494272 1176 33554432000 134512640 135107411 4289922992 18446744073709551615 134532376 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/8955/statm: 104857 1176 59 146 0 103733 0
[pid=8953/tid=8963] ppid=8905 vsize=321400 CPUtime=0 cores=1,3,5,7
/proc/8953/task/8963/stat : 8963 (python) S 8905 8905 8511 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 670614678 329113600 5919 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604454934885440 0 0 -1 7 0 0 0
[pid=8964] ppid=8905 vsize=331760 CPUtime=0.39 cores=1,3,5,7
/proc/8964/stat : 8964 (python) S 8905 8905 8511 0 -1 4202560 1125 0 0 0 20 19 0 0 19 0 2 0 670614682 339722240 5947 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8964/statm: 82940 5947 400 364 0 18843 0
[pid=8966] ppid=8964 vsize=1018820 CPUtime=100.42 cores=1,3,5,7
/proc/8966/stat : 8966 (gnovelty+2) T 8964 8966 8966 0 -1 4202496 863 0 0 0 10042 0 0 0 25 0 1 0 670614682 1043271680 596 33554432000 134512640 135663704 4289587344 18446744073709551615 134545631 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/8966/statm: 254705 596 113 282 0 254421 0
[pid=8964/tid=8974] ppid=8905 vsize=331760 CPUtime=0 cores=1,3,5,7
/proc/8964/task/8974/stat : 8974 (python) S 8905 8905 8511 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670624725 339722240 5947 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604459571436352 0 0 -1 7 0 0 0
[pid=8975] ppid=8905 vsize=341836 CPUtime=0.39 cores=1,3,5,7
/proc/8975/stat : 8975 (python) S 8905 8905 8511 0 -1 4202560 1169 0 0 0 22 17 0 0 19 0 2 0 670624729 350040064 5983 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8975/statm: 85459 5983 406 364 0 21359 0
[pid=8977] ppid=8975 vsize=249192 CPUtime=98.39 cores=1,3,5,7
/proc/8977/stat : 8977 (FH) T 8975 8977 8977 0 -1 4202496 1326 0 0 0 9839 0 0 0 25 0 1 0 670624729 255172608 1059 33554432000 134512640 134936988 4293165376 18446744073709551615 134554976 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/8977/statm: 62298 1059 41 104 0 61802 0
[pid=8975/tid=8984] ppid=8905 vsize=341836 CPUtime=0 cores=1,3,5,7
/proc/8975/task/8984/stat : 8984 (python) S 8905 8905 8511 0 -1 4202560 94 0 0 0 0 0 0 0 19 0 2 0 670634568 350040064 5983 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604462422701120 0 0 -1 7 0 0 0
[pid=8985] ppid=8905 vsize=352332 CPUtime=0.39 cores=1,3,5,7
/proc/8985/stat : 8985 (python) S 8905 8905 8511 0 -1 4202560 1155 0 0 0 19 20 0 0 19 0 2 0 670634572 360787968 5987 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/8985/statm: 88083 5987 406 364 0 23980 0
[pid=8987] ppid=8985 vsize=419428 CPUtime=100.39 cores=1,3,5,7
/proc/8987/stat : 8987 (adaptg2wsat2009) T 8985 8987 8987 0 -1 4202496 1452 0 0 0 10039 0 0 0 25 0 1 0 670634572 429494272 1177 33554432000 134512640 135107411 4288619344 18446744073709551615 134532128 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/8987/statm: 104857 1177 59 146 0 103733 0
[pid=8985/tid=9004] ppid=8905 vsize=352332 CPUtime=0 cores=1,3,5,7
/proc/8985/task/9004/stat : 9004 (python) S 8905 8905 8511 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670644612 360787968 5987 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604460381246784 0 0 -1 7 0 0 0
[pid=9005] ppid=8905 vsize=362432 CPUtime=0.41 cores=1,3,5,7
/proc/9005/stat : 9005 (python) S 8905 8905 8511 0 -1 4202560 1135 0 0 0 23 18 0 0 19 0 2 0 670644616 371130368 6000 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9005/statm: 90608 6000 400 364 0 26502 0
[pid=9007] ppid=9005 vsize=419428 CPUtime=102.43 cores=1,3,5,7
/proc/9007/stat : 9007 (adaptg2wsat2009) T 9005 9007 9007 0 -1 4202496 1449 0 0 0 10243 0 0 0 25 0 1 0 670644616 429494272 1175 33554432000 134512640 135107411 4294179728 18446744073709551615 134531792 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9007/statm: 104857 1175 59 146 0 103733 0
[pid=9005/tid=9015] ppid=8905 vsize=362432 CPUtime=0 cores=1,3,5,7
/proc/9005/task/9015/stat : 9015 (python) S 8905 8905 8511 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 670654860 371130368 6000 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604460042648576 0 0 -1 7 0 0 0
[pid=9016] ppid=8905 vsize=372960 CPUtime=0.42 cores=1,3,5,7
/proc/9016/stat : 9016 (python) S 8905 8905 8511 0 -1 4202560 1131 0 0 0 21 21 0 0 19 0 2 0 670654864 381911040 6004 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9016/statm: 93240 6004 400 364 0 29131 0
[pid=9018] ppid=9016 vsize=249196 CPUtime=102.44 cores=1,3,5,7
/proc/9018/stat : 9018 (FH) T 9016 9018 9018 0 -1 4202496 1329 0 0 0 10244 0 0 0 25 0 1 0 670654865 255176704 1059 33554432000 134512640 134936988 4289902368 18446744073709551615 134545007 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9018/statm: 62299 1059 41 104 0 61803 0
[pid=9016/tid=9028] ppid=8905 vsize=372960 CPUtime=0 cores=1,3,5,7
/proc/9016/task/9028/stat : 9028 (python) S 8905 8905 8511 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 670665111 381911040 6004 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604460042647680 0 0 -1 7 0 0 0
[pid=9029] ppid=8905 vsize=383356 CPUtime=0.75 cores=1,3,5,7
/proc/9029/stat : 9029 (python) S 8905 8905 8511 0 -1 4202560 1143 0 0 0 41 34 0 0 19 0 2 0 670665114 392556544 6037 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9029/statm: 95839 6037 400 364 0 31727 0
[pid=9031] ppid=9029 vsize=268424 CPUtime=196.56 cores=1,3,5,7
/proc/9031/stat : 9031 (hybridGM3) T 9029 9031 9031 0 -1 4202496 5277 0 0 0 19656 0 0 0 25 0 1 0 670665115 274866176 5008 33554432000 134512640 135204385 4290850240 18446744073709551615 134538257 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/9031/statm: 67106 5008 63 169 0 66936 0
[pid=9029/tid=9042] ppid=8905 vsize=383356 CPUtime=0 cores=1,3,5,7
/proc/9029/task/9042/stat : 9042 (python) S 8905 8905 8511 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670684793 392556544 6037 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604460042646784 0 0 -1 7 0 0 0
[pid=9043] ppid=8905 vsize=393912 CPUtime=0.4 cores=1,3,5,7
/proc/9043/stat : 9043 (python) S 8905 8905 8511 0 -1 4202560 1130 0 0 0 21 19 0 0 16 0 2 0 670684797 403365888 6076 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9043/statm: 98478 6076 400 364 0 34363 0
[pid=9045] ppid=9043 vsize=1018820 CPUtime=98.41 cores=1,3,5,7
/proc/9045/stat : 9045 (gnovelty+2) T 9043 9045 9045 0 -1 4202496 867 0 0 0 9841 0 0 0 25 0 1 0 670684798 1043271680 595 33554432000 134512640 135663704 4294236512 18446744073709551615 134525286 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9045/statm: 254705 595 113 282 0 254421 0
[pid=9043/tid=9052] ppid=8905 vsize=393912 CPUtime=0 cores=1,3,5,7
/proc/9043/task/9052/stat : 9052 (python) S 8905 8905 8511 0 -1 4202560 48 0 0 0 0 0 0 0 16 0 2 0 670694642 403365888 6076 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604469952164800 0 0 -1 7 0 0 0
[pid=9053] ppid=8905 vsize=404016 CPUtime=0.79 cores=1,3,5,7
/proc/9053/stat : 9053 (python) S 8905 8905 8511 0 -1 4202560 1157 0 0 0 45 34 0 0 19 0 2 0 670694646 413712384 6155 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9053/statm: 101004 6155 400 364 0 36886 0
[pid=9055] ppid=9053 vsize=268424 CPUtime=195.85 cores=1,3,5,7
/proc/9055/stat : 9055 (hybridGM3) T 9053 9055 9055 0 -1 4202496 5280 0 0 0 19585 0 0 0 25 0 1 0 670694647 274866176 5010 33554432000 134512640 135204385 4290618608 18446744073709551615 134537939 0 0 16781312 0 18446744071562691360 0 0 17 1 0 0 0
/proc/9055/statm: 67106 5010 63 169 0 66936 0
[pid=9053/tid=9065] ppid=8905 vsize=404016 CPUtime=0 cores=1,3,5,7
/proc/9053/task/9065/stat : 9065 (python) S 8905 8905 8511 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670714233 413712384 6155 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604469952165696 0 0 -1 7 0 0 0
[pid=9066] ppid=8905 vsize=413940 CPUtime=0.83 cores=1,3,5,7
/proc/9066/stat : 9066 (python) S 8905 8905 8511 0 -1 4202560 1173 0 0 0 42 41 0 0 19 0 2 0 670714236 423874560 6158 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9066/statm: 103485 6158 399 364 0 39364 0
[pid=9068] ppid=9066 vsize=268420 CPUtime=197.91 cores=1,3,5,7
/proc/9068/stat : 9068 (hybridGM3) T 9066 9068 9068 0 -1 4202496 5281 0 0 0 19791 0 0 0 25 0 1 0 670714237 274862080 5009 33554432000 134512640 135204385 4288263376 18446744073709551615 134537792 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9068/statm: 67105 5009 63 169 0 66935 0
[pid=9066/tid=9079] ppid=8905 vsize=413940 CPUtime=0 cores=1,3,5,7
/proc/9066/task/9079/stat : 9079 (python) S 8905 8905 8511 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 670734029 423874560 6158 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604463045773504 0 0 -1 7 0 0 0
[pid=9080] ppid=8905 vsize=424560 CPUtime=6.66 cores=1,3,5,7
/proc/9080/stat : 9080 (python) S 8905 8905 8511 0 -1 4202560 1214 0 0 0 318 348 0 0 19 0 2 0 670734032 434749440 6170 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9080/statm: 106140 6170 407 364 0 42016 0
[pid=9082] ppid=9080 vsize=457664 CPUtime=1554.76 cores=1,3,5,7
/proc/9082/stat : 9082 (TNM) T 9080 9082 9082 0 -1 4202496 1313 0 0 0 155476 0 0 0 25 0 1 0 670734033 468647936 1037 33554432000 134512640 135008748 4289199904 18446744073709551615 134529414 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9082/statm: 114416 1037 46 122 0 113316 0
[pid=9080/tid=9132] ppid=8905 vsize=424560 CPUtime=0 cores=1,3,5,7
/proc/9080/task/9132/stat : 9132 (python) S 8905 8905 8511 0 -1 4202560 45 0 0 0 0 0 0 0 19 0 2 0 670889506 434749440 6170 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604470783338560 0 0 -1 7 0 0 0
[pid=9133] ppid=8905 vsize=434452 CPUtime=0.42 cores=1,3,5,7
/proc/9133/stat : 9133 (python) S 8905 8905 8511 0 -1 4202560 1179 0 0 0 23 19 0 0 19 0 2 0 670889508 444878848 6173 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9133/statm: 108613 6173 406 364 0 44486 0
[pid=9135] ppid=9133 vsize=419428 CPUtime=100.41 cores=1,3,5,7
/proc/9135/stat : 9135 (adaptg2wsat2009) T 9133 9135 9135 0 -1 4202496 1449 0 0 0 10041 0 0 0 25 0 1 0 670889509 429494272 1175 33554432000 134512640 135107411 4290222800 18446744073709551615 134531709 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9135/statm: 104857 1175 59 146 0 103733 0
[pid=9133/tid=9139] ppid=8905 vsize=434452 CPUtime=0 cores=1,3,5,7
/proc/9133/task/9139/stat : 9139 (python) S 8905 8905 8511 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 670899551 444878848 6173 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604463052158784 0 0 -1 7 0 0 0
[pid=9140] ppid=8905 vsize=445168 CPUtime=0.41 cores=1,3,5,7
/proc/9140/stat : 9140 (python) S 8905 8905 8511 0 -1 4202560 1174 0 0 0 21 20 0 0 19 0 2 0 670899553 455852032 6172 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256175563 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9140/statm: 111292 6172 401 364 0 47162 0
[pid=9142] ppid=9140 vsize=249192 CPUtime=97.42 cores=1,3,5,7
/proc/9142/stat : 9142 (FH) T 9140 9142 9142 0 -1 4202496 1333 0 0 0 9742 0 0 0 25 0 1 0 670899553 255172608 1058 33554432000 134512640 134936988 4288387456 18446744073709551615 134554229 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9142/statm: 62298 1058 41 104 0 61802 0
[pid=9140/tid=9145] ppid=8905 vsize=445168 CPUtime=0 cores=1,3,5,7
/proc/9140/task/9145/stat : 9145 (python) S 8905 8905 8511 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670909296 455852032 6172 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 514 18446604463858255168 0 0 -1 7 0 0 0
[pid=9146] ppid=8905 vsize=455560 CPUtime=0 cores=1,3,5,7
/proc/9146/stat : 9146 (python) S 8905 8905 8511 0 -1 4202560 837 0 0 0 0 0 0 0 19 0 1 0 670909298 466493440 6124 33554432000 4194304 5683644 140733503542496 18446744073709551615 228244368966 0 0 16781312 514 2747995671700176895 0 0 17 1 0 0 0
/proc/9146/statm: 113890 6124 350 364 0 49757 0
[pid=9148] ppid=9146 vsize=268420 CPUtime=0.87 cores=1,3,5,7
/proc/9148/stat : 9148 (hybridGM3) R 9146 9148 9148 0 -1 4202496 5282 0 0 0 87 0 0 0 23 0 1 0 670909299 274862080 5008 33554432000 134512640 135204385 4288601808 18446744073709551615 134537699 0 0 16781312 0 0 0 0 17 3 0 0 0
/proc/9148/statm: 67105 5008 63 169 0 66935 0
[pid=8905/tid=8912] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8912/stat : 8912 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 19 0 19 0 670564961 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
[pid=8905/tid=8922] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8922/stat : 8922 (python) S 8903 8905 8511 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 670574707 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 3 0 0 0
[pid=8905/tid=8933] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8933/stat : 8933 (python) S 8903 8905 8511 0 -1 4202560 24 933 0 0 0 0 0 0 20 0 19 0 670584651 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 3 0 0 0
[pid=8905/tid=8944] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8944/stat : 8944 (python) S 8903 8905 8511 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 670594690 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 7 0 0 0
[pid=8905/tid=8954] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8954/stat : 8954 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 670604938 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=8965] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8965/stat : 8965 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 670614682 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=8976] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8976/stat : 8976 (python) S 8903 8905 8511 0 -1 4202560 30 933 0 0 0 0 0 0 20 0 19 0 670624729 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=8986] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/8986/stat : 8986 (python) S 8903 8905 8511 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 19 0 670634572 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=9006] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9006/stat : 9006 (python) S 8903 8905 8511 0 -1 4202560 28 933 0 0 0 0 0 0 20 0 19 0 670644616 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 3 0 0 0
[pid=8905/tid=9017] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9017/stat : 9017 (python) S 8903 8905 8511 0 -1 4202560 31 933 0 0 0 0 0 0 20 0 19 0 670654864 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 3 0 0 0
[pid=8905/tid=9030] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9030/stat : 9030 (python) S 8903 8905 8511 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 19 0 670665115 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 3 0 0 0
[pid=8905/tid=9044] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9044/stat : 9044 (python) S 8903 8905 8511 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 19 0 670684797 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 3 0 0 0
[pid=8905/tid=9054] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9054/stat : 9054 (python) S 8903 8905 8511 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 670694646 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=9067] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9067/stat : 9067 (python) S 8903 8905 8511 0 -1 4202560 29 933 0 0 0 0 0 0 20 0 19 0 670714236 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=9081] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9081/stat : 9081 (python) S 8903 8905 8511 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 670734032 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=9134] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9134/stat : 9134 (python) S 8903 8905 8511 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 670889508 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=9141] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9141/stat : 9141 (python) S 8903 8905 8511 0 -1 4202560 26 933 0 0 0 0 0 0 20 0 19 0 670899553 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
[pid=8905/tid=9147] ppid=8903 vsize=465208 CPUtime=0 cores=1,3,5,7
/proc/8905/task/9147/stat : 9147 (python) S 8903 8905 8511 0 -1 4202560 27 933 0 0 0 0 0 0 20 0 19 0 670909298 476372992 7689 33554432000 4194304 5683644 140733503542496 18446744073709551615 228256172465 0 0 16781312 2 18446604461711012096 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3459.48
Current children cumulated vsize (KiB) 15966208

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 8905 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=138674
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=157975
# CPU time returned by wait4() is 2.29665
# while last known CPU time is 3459.48
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 3446.34
CPU time (s): 3459.48
CPU user time (s): 3452.31
CPU system time (s): 7.17
CPU usage (%): 100.381
Max. virtual memory (cumulated for all children) (KiB): 15966208

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.13867
system time used= 0.157975
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 29547
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2917
involuntary context switches= 53

runsolver used 27.6288 second user time and 60.1239 second system time

The end

Launcher Data 

Begin job on node136 at 2011-05-17 01:47:48
IDJOB=3386624
IDBENCH=82689
IDSOLVER=1872
FILE ID=node136/3386624-1305589668
RUNJOBID= node136-1305586144-8530
PBS_JOBID= 13350342
Free space on /tmp= 72900 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k7-r85-v250-c21250-S58287803-001-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386624-1305589668/watcher-3386624-1305589668 -o /tmp/evaluation-result-3386624-1305589668/solver-3386624-1305589668 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386624-1305589668.cnf -seed 1865723472 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0d5d355ade4a956f83626786ca6018b7
RANDOM SEED=1865723472

node136.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      20176392 kB
Buffers:         57880 kB
Cached:         763604 kB
SwapCached:      24924 kB
Active:        6005592 kB
Inactive:      6538132 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      20176392 kB
SwapTotal:    67111528 kB
SwapFree:     64205392 kB
Dirty:           32320 kB
Writeback:           0 kB
AnonPages:    11703708 kB
Mapped:          42700 kB
Slab:            90928 kB
PageTables:      71280 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 154607664 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72868 MiB
End job on node136 at 2011-05-17 02:45:16