Trace number 3386634

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed)? (MO) 3444.78 3433.73

General information on the benchmark

Namerandom/large/
unif-k5-r20-v1250-c25000-S871131876-087-UNKNOWN.cnf
MD5SUM637b75113dde2f4a15dbf158341be921
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark61.3907
Satisfiable
(Un)Satisfiability was proved
Number of variables1250
Number of clauses25000
Sum of the clauses size125000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 525000
Number of clauses of size over 50

Solver Data 

0.70/2.36	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.70/2.36	c INFO: solving HOME/instance-3386634-1305589701.cnf
0.70/2.36	c INFO: computing task features
0.77/2.49	c INFO: computed variable-clause graph statistics
0.77/2.49	c INFO: computed clause balance statistics
1.47/3.12	c INFO: computed variable balance statistics
1.77/3.46	c INFO: computed small-clause counts
1.77/3.46	c INFO: computed Horn-clause counts
1.77/3.49	c INFO: computed variable graph statistics
1.77/3.49	c INFO: collected features for HOME/instance-3386634-1305589701.cnf in 1.13 s
1.87/3.52	c INFO: running gnovelty+2@0 for 97 with 4998 remaining (b = 0.38)
101.60/102.95	c INFO: running gnovelty+2@0 for 97 with 4899 remaining (b = 0.14)
199.48/200.40	c INFO: running adaptg2wsat2009++@0 for 97 with 4802 remaining (b = 0.09)
299.23/299.81	c INFO: running adaptg2wsat2009++@0 for 97 with 4702 remaining (b = 0.07)
398.00/398.24	c INFO: running adaptg2wsat2009++@0 for 97 with 4604 remaining (b = 0.06)
495.71/495.67	c INFO: running gnovelty+2@0 for 97 with 4506 remaining (b = 0.05)
595.62/595.12	c INFO: running FH@0 for 97 with 4407 remaining (b = 0.04)
696.37/695.54	c INFO: running adaptg2wsat2009++@0 for 97 with 4306 remaining (b = 0.03)
794.11/792.97	c INFO: running gnovelty+2@99 for 97 with 4209 remaining (b = 0.03)
891.87/890.37	c INFO: running adaptg2wsat2009++@0 for 97 with 4112 remaining (b = 0.03)
992.67/990.81	c INFO: running FH@0 for 97 with 4011 remaining (b = 0.02)
1090.40/1088.26	c INFO: running adaptg2wsat2009++@0 for 97 with 3914 remaining (b = 0.02)
1191.08/1188.66	c INFO: running hybridGM3@0 for 194 with 3814 remaining (b = 0.04)
1387.70/1384.60	c INFO: running gnovelty+2@0 for 97 with 3618 remaining (b = 0.02)
1488.58/1485.05	c INFO: running hybridGM3@0 for 194 with 3517 remaining (b = 0.03)
1686.07/1681.95	c INFO: running hybridGM3@0 for 194 with 3321 remaining (b = 0.03)
1884.75/1879.90	c INFO: running TNM@0 for 1552 with 3123 remaining (b = 0.20)
3444.65/3433.56	c INFO: running FH@0 for 97 with 1569 remaining (b = 0.01)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3386634-1305589701/watcher-3386634-1305589701 -o /tmp/evaluation-result-3386634-1305589701/solver-3386634-1305589701 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386634-1305589701.cnf -seed 1662753417 -budget 5000 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.05 3.06 3.00 3/500 6711
/proc/meminfo: memFree=18114352/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=15908 CPUtime=0 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 943 0 0 0 0 0 0 0 17 0 1 0 670569737 16289792 821 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 2 0 0 0
/proc/6711/statm: 3977 821 357 364 0 464 0

[startup+0.0929641 s]
/proc/loadavg: 3.05 3.06 3.00 3/500 6711
/proc/meminfo: memFree=18114352/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=85072 CPUtime=0.05 cores=0,2,4,6
/proc/6711/stat : 6711 (python) R 6709 6711 6320 0 -1 4202496 2521 0 0 0 4 1 0 0 17 0 1 0 670569737 87113728 2249 33554432000 4194304 5683644 140736973467536 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/6711/statm: 21268 2253 707 364 0 1485 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 85072

[startup+0.100974 s]
/proc/loadavg: 3.05 3.06 3.00 3/500 6711
/proc/meminfo: memFree=18114352/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=93960 CPUtime=0.05 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 2691 0 0 0 4 1 0 0 17 0 1 0 670569737 96215040 2407 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647142245 0 2147483391 16781312 2 18446744071699109054 0 0 17 2 0 0 0
/proc/6711/statm: 23490 2407 764 364 0 1622 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 93960

[startup+0.300926 s]
/proc/loadavg: 3.05 3.06 3.00 3/500 6711
/proc/meminfo: memFree=18114352/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=135752 CPUtime=0.17 cores=0,2,4,6
/proc/6711/stat : 6711 (python) R 6709 6711 6320 0 -1 4202496 5971 931 1 0 13 4 0 0 17 0 1 0 670569737 139010048 4150 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647146128 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/6711/statm: 33938 4150 1032 364 0 3124 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 135752

[startup+0.700843 s]
/proc/loadavg: 3.05 3.06 3.00 3/500 6711
/proc/meminfo: memFree=18114352/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=246708 CPUtime=0.37 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 8269 931 1 0 27 10 0 0 16 0 1 0 670569737 252628992 6158 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 6 0 0 0
/proc/6711/statm: 61677 6158 1621 364 0 4679 0
Current children cumulated CPU time (s) 0.37
Current children cumulated vsize (KiB) 246708

[startup+1.50067 s]
/proc/loadavg: 3.05 3.06 3.00 4/501 6716
/proc/meminfo: memFree=18092768/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=256700 CPUtime=0.49 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 8440 931 1 0 29 20 0 0 15 0 1 0 670569737 262860800 6302 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647144448 0 2147483391 16781312 2 18446744071699109054 0 0 17 6 0 0 0
/proc/6711/statm: 64175 6302 1648 364 0 5054 0
Current children cumulated CPU time (s) 0.49
Current children cumulated vsize (KiB) 256700

[startup+3.10038 s]
/proc/loadavg: 3.12 3.07 3.01 4/501 6716
/proc/meminfo: memFree=18090784/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=285816 CPUtime=1.47 cores=0,2,4,6
/proc/6711/stat : 6711 (python) R 6709 6711 6320 0 -1 4202496 12068 931 1 0 108 39 0 0 16 0 1 0 670569737 292675584 8839 33554432000 4194304 5683644 140736973467536 18446744073709551615 47410026399006 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/6711/statm: 71454 8839 1826 364 0 7378 0
Current children cumulated CPU time (s) 1.47
Current children cumulated vsize (KiB) 285816
heavy processes:

[startup+6.3009 s]
/proc/loadavg: 3.12 3.07 3.01 6/504 6719
/proc/meminfo: memFree=18080608/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=292296 CPUtime=1.9 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 12441 931 1 0 150 40 0 0 19 0 2 0 670569737 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6711/statm: 73074 7947 1877 364 0 8992 0
[pid=6717] ppid=6711 vsize=282052 CPUtime=0 cores=0,2,4,6
/proc/6717/stat : 6717 (python) S 6711 6711 6320 0 -1 4202560 735 0 0 0 0 0 0 0 18 0 1 0 670570090 288821248 6420 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647166479 0 0 16781312 514 5550743719633715457 0 0 17 2 0 0 0
/proc/6717/statm: 70513 6420 354 364 0 6431 0
[pid=6719] ppid=6717 vsize=1018800 CPUtime=2.76 cores=0,2,4,6
/proc/6719/stat : 6719 (gnovelty+2) R 6717 6719 6719 0 -1 4202496 866 0 0 0 276 0 0 0 25 0 1 0 670570090 1043251200 602 33554432000 134512640 135663704 4294831344 18446744073709551615 135007681 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6719/statm: 254700 602 113 282 0 254416 0
[pid=6711/tid=6718] ppid=6709 vsize=292296 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6718/stat : 6718 (python) S 6709 6711 6320 0 -1 4202560 28 931 0 0 0 0 0 0 19 0 2 0 670570090 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4.66
Current children cumulated vsize (KiB) 1593148
heavy processes:

[startup+12.7008 s]
/proc/loadavg: 3.25 3.10 3.02 4/504 6719
/proc/meminfo: memFree=18078588/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=292296 CPUtime=1.9 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 12441 931 1 0 150 40 0 0 19 0 2 0 670569737 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6711/statm: 73074 7947 1877 364 0 8992 0
[pid=6717] ppid=6711 vsize=282052 CPUtime=0.02 cores=0,2,4,6
/proc/6717/stat : 6717 (python) S 6711 6711 6320 0 -1 4202560 883 0 0 0 1 1 0 0 18 0 1 0 670570090 288821248 6443 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647166479 0 0 16781312 514 5550743719633715457 0 0 17 2 0 0 0
/proc/6717/statm: 70513 6443 377 364 0 6431 0
[pid=6719] ppid=6717 vsize=1018800 CPUtime=9.16 cores=0,2,4,6
/proc/6719/stat : 6719 (gnovelty+2) R 6717 6719 6719 0 -1 4202496 866 0 0 0 916 0 0 0 25 0 1 0 670570090 1043251200 602 33554432000 134512640 135663704 4294831344 18446744073709551615 134545631 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6719/statm: 254700 602 113 282 0 254416 0
[pid=6711/tid=6718] ppid=6709 vsize=292296 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6718/stat : 6718 (python) S 6709 6711 6320 0 -1 4202560 28 931 0 0 0 0 0 0 19 0 2 0 670570090 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.08
Current children cumulated vsize (KiB) 1593148

[startup+25.5007 s]
/proc/loadavg: 3.21 3.10 3.02 4/504 6720
/proc/meminfo: memFree=18080828/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=292296 CPUtime=1.9 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 12441 931 1 0 150 40 0 0 19 0 2 0 670569737 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6711/statm: 73074 7947 1877 364 0 8992 0
[pid=6717] ppid=6711 vsize=282052 CPUtime=0.06 cores=0,2,4,6
/proc/6717/stat : 6717 (python) S 6711 6711 6320 0 -1 4202560 886 0 0 0 3 3 0 0 18 0 1 0 670570090 288821248 6443 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647166479 0 0 16781312 514 10093815810146402561 0 0 17 6 0 0 0
/proc/6717/statm: 70513 6443 377 364 0 6431 0
[pid=6719] ppid=6717 vsize=1018800 CPUtime=21.96 cores=0,2,4,6
/proc/6719/stat : 6719 (gnovelty+2) R 6717 6719 6719 0 -1 4202496 866 0 0 0 2196 0 0 0 25 0 1 0 670570090 1043251200 602 33554432000 134512640 135663704 4294831344 18446744073709551615 134524548 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6719/statm: 254700 602 113 282 0 254416 0
[pid=6711/tid=6718] ppid=6709 vsize=292296 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6718/stat : 6718 (python) S 6709 6711 6320 0 -1 4202560 28 931 0 0 0 0 0 0 19 0 2 0 670570090 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 23.92
Current children cumulated vsize (KiB) 1593148

[startup+51.1004 s]
/proc/loadavg: 3.14 3.09 3.01 4/504 6721
/proc/meminfo: memFree=18080680/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=292296 CPUtime=1.9 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 12441 931 1 0 150 40 0 0 19 0 2 0 670569737 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6711/statm: 73074 7947 1877 364 0 8992 0
[pid=6717] ppid=6711 vsize=282052 CPUtime=0.17 cores=0,2,4,6
/proc/6717/stat : 6717 (python) S 6711 6711 6320 0 -1 4202560 901 0 0 0 9 8 0 0 18 0 1 0 670570090 288821248 6443 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647166479 0 0 16781312 514 10093815810146402561 0 0 17 6 0 0 0
/proc/6717/statm: 70513 6443 377 364 0 6431 0
[pid=6719] ppid=6717 vsize=1018800 CPUtime=47.56 cores=0,2,4,6
/proc/6719/stat : 6719 (gnovelty+2) R 6717 6719 6719 0 -1 4202496 866 0 0 0 4756 0 0 0 25 0 1 0 670570090 1043251200 602 33554432000 134512640 135663704 4294831344 18446744073709551615 134545682 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6719/statm: 254700 602 113 282 0 254416 0
[pid=6711/tid=6718] ppid=6709 vsize=292296 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6718/stat : 6718 (python) S 6709 6711 6320 0 -1 4202560 28 931 0 0 0 0 0 0 19 0 2 0 670570090 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 49.63
Current children cumulated vsize (KiB) 1593148

[startup+102.318 s]
/proc/loadavg: 3.05 3.07 3.01 4/504 6722
/proc/meminfo: memFree=18080924/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=292296 CPUtime=1.9 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 12441 931 1 0 150 40 0 0 19 0 2 0 670569737 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6711/statm: 73074 7947 1877 364 0 8992 0
[pid=6717] ppid=6711 vsize=282052 CPUtime=0.34 cores=0,2,4,6
/proc/6717/stat : 6717 (python) S 6711 6711 6320 0 -1 4202560 918 0 0 0 19 15 0 0 18 0 1 0 670570090 288821248 6443 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647166479 0 0 16781312 514 10093815810146402561 0 0 17 2 0 0 0
/proc/6717/statm: 70513 6443 377 364 0 6431 0
[pid=6719] ppid=6717 vsize=1018800 CPUtime=98.78 cores=0,2,4,6
/proc/6719/stat : 6719 (gnovelty+2) R 6717 6719 6719 0 -1 4202496 866 0 0 0 9878 0 0 0 25 0 1 0 670570090 1043251200 602 33554432000 134512640 135663704 4294831344 18446744073709551615 134524498 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6719/statm: 254700 602 113 282 0 254416 0
[pid=6711/tid=6718] ppid=6709 vsize=292296 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6718/stat : 6718 (python) S 6709 6711 6320 0 -1 4202560 28 931 0 0 0 0 0 0 19 0 2 0 670570090 299311104 7947 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 101.02
Current children cumulated vsize (KiB) 1593148

[startup+162.301 s]
/proc/loadavg: 3.02 3.06 3.00 4/508 6727
/proc/meminfo: memFree=18074056/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=368088 CPUtime=1.93 cores=0,2,4,6
/proc/6711/stat : 6711 (python) S 6709 6711 6320 0 -1 4202496 13289 931 1 0 153 40 0 0 18 0 3 0 670569737 376922112 7955 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 2 18446744073709551615 0 0 17 2 0 0 0
/proc/6711/statm: 92022 7955 1880 364 0 27937 0
[pid=6717] ppid=6711 vsize=292296 CPUtime=0.36 cores=0,2,4,6
/proc/6717/stat : 6717 (python) S 6711 6711 6320 0 -1 4202560 1075 0 0 0 20 16 0 0 19 0 2 0 670570090 299311104 6481 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6717/statm: 73074 6481 410 364 0 8992 0
[pid=6719] ppid=6717 vsize=1018800 CPUtime=99.39 cores=0,2,4,6
/proc/6719/stat : 6719 (gnovelty+2) T 6717 6719 6719 0 -1 4202496 866 0 0 0 9939 0 0 0 25 0 1 0 670570090 1043251200 602 33554432000 134512640 135663704 4294831344 18446744073709551615 134525678 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6719/statm: 254700 602 113 282 0 254416 0
[pid=6717/tid=6723] ppid=6711 vsize=292296 CPUtime=0 cores=0,2,4,6
/proc/6717/task/6723/stat : 6723 (python) S 6711 6711 6320 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670580030 299311104 6481 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604449910109184 0 0 -1 6 0 0 0
[pid=6724] ppid=6711 vsize=292308 CPUtime=0.21 cores=0,2,4,6
/proc/6724/stat : 6724 (python) S 6711 6711 6320 0 -1 4202560 914 0 0 0 12 9 0 0 18 0 1 0 670580033 299323392 6441 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647166534 0 0 16781312 514 10093815810146402561 0 0 17 2 0 0 0
/proc/6724/statm: 73077 6441 371 364 0 8992 0
[pid=6726] ppid=6724 vsize=1018800 CPUtime=59.34 cores=0,2,4,6
/proc/6726/stat : 6726 (gnovelty+2) R 6724 6726 6726 0 -1 4202496 869 0 0 0 5934 0 0 0 25 0 1 0 670580033 1043251200 603 33554432000 134512640 135663704 4288574960 18446744073709551615 134525276 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6726/statm: 254700 603 113 282 0 254416 0
[pid=6711/tid=6718] ppid=6709 vsize=368088 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6718/stat : 6718 (python) S 6709 6711 6320 0 -1 4202560 28 931 0 0 0 0 0 0 19 0 3 0 670570090 376922112 7955 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
[pid=6711/tid=6725] ppid=6709 vsize=368088 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6725/stat : 6725 (python) S 6709 6711 6320 0 -1 4202560 25 931 0 0 0 0 0 0 20 0 3 0 670580033 376922112 7955 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 161.23
Current children cumulated vsize (KiB) 2990292

[startup+222.302 s]
/proc/loadavg: 3.00 3.04 3.00 4/512 6733
/proc/meminfo: memFree=18064952/32951132 swapFree=64496000/67111528
[pid=6711] ppid=6709 vsize=378344 CPUtime=1.96 cores=0,2,4,6

################
# More data... #
################

[pid=6733] ppid=6731 vsize=420484 CPUtime=99.36 cores=0,2,4,6
/proc/6733/stat : 6733 (adaptg2wsat2009) T 6731 6733 6733 0 -1 4202496 1746 0 0 0 9936 0 0 0 25 0 1 0 670589778 430575616 1492 33554432000 134512640 135107411 4291798128 18446744073709551615 134531411 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6733/statm: 105121 1492 59 146 0 103997 0
[pid=6731/tid=6736] ppid=6711 vsize=368100 CPUtime=0 cores=0,2,4,6
/proc/6731/task/6736/stat : 6736 (python) S 6711 6711 6320 0 -1 4202560 120 0 0 0 0 0 0 0 19 0 2 0 670599715 376934400 6648 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604448958592256 0 0 -1 6 0 0 0
[pid=6737] ppid=6711 vsize=378356 CPUtime=0.37 cores=0,2,4,6
/proc/6737/stat : 6737 (python) S 6711 6711 6320 0 -1 4202560 977 0 0 0 21 16 0 0 19 0 2 0 670599719 387436544 6657 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6737/statm: 94589 6657 400 364 0 30498 0
[pid=6739] ppid=6737 vsize=420484 CPUtime=98.37 cores=0,2,4,6
/proc/6739/stat : 6739 (adaptg2wsat2009) T 6737 6739 6739 0 -1 4202496 1752 0 0 0 9837 0 0 0 25 0 1 0 670599719 430575616 1493 33554432000 134512640 135107411 4287878624 18446744073709551615 134531781 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6739/statm: 105121 1493 59 146 0 103997 0
[pid=6737/tid=6743] ppid=6711 vsize=378356 CPUtime=0 cores=0,2,4,6
/proc/6737/task/6743/stat : 6743 (python) S 6711 6711 6320 0 -1 4202560 67 0 0 0 0 0 0 0 19 0 2 0 670609558 387436544 6657 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604445919887552 0 0 -1 6 0 0 0
[pid=6744] ppid=6711 vsize=388612 CPUtime=0.37 cores=0,2,4,6
/proc/6744/stat : 6744 (python) S 6711 6711 6320 0 -1 4202560 981 0 0 0 19 18 0 0 19 0 2 0 670609561 397938688 6688 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6744/statm: 97153 6688 400 364 0 33059 0
[pid=6746] ppid=6744 vsize=420484 CPUtime=97.39 cores=0,2,4,6
/proc/6746/stat : 6746 (adaptg2wsat2009) T 6744 6746 6746 0 -1 4202496 1752 0 0 0 9738 1 0 0 25 0 1 0 670609562 430575616 1493 33554432000 134512640 135107411 4290692672 18446744073709551615 134532304 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6746/statm: 105121 1493 59 146 0 103997 0
[pid=6744/tid=6783] ppid=6711 vsize=388612 CPUtime=0 cores=0,2,4,6
/proc/6744/task/6783/stat : 6783 (python) S 6711 6711 6320 0 -1 4202560 69 0 0 0 0 0 0 0 19 0 2 0 670619301 397938688 6688 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604449161000768 0 0 -1 6 0 0 0
[pid=6784] ppid=6711 vsize=398868 CPUtime=0.37 cores=0,2,4,6
/proc/6784/stat : 6784 (python) S 6711 6711 6320 0 -1 4202560 1092 0 0 0 19 18 0 0 19 0 2 0 670619305 408440832 6630 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/6784/statm: 99717 6630 400 364 0 35620 0
[pid=6786] ppid=6784 vsize=1018800 CPUtime=99.4 cores=0,2,4,6
/proc/6786/stat : 6786 (gnovelty+2) T 6784 6786 6786 0 -1 4202496 865 0 0 0 9940 0 0 0 25 0 1 0 670619305 1043251200 602 33554432000 134512640 135663704 4294823632 18446744073709551615 134545640 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6786/statm: 254700 602 113 282 0 254416 0
[pid=6784/tid=6794] ppid=6711 vsize=398868 CPUtime=0 cores=0,2,4,6
/proc/6784/task/6794/stat : 6794 (python) S 6711 6711 6320 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670629246 408440832 6630 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604452964831488 0 0 -1 2 0 0 0
[pid=6795] ppid=6711 vsize=409124 CPUtime=0.38 cores=0,2,4,6
/proc/6795/stat : 6795 (python) S 6711 6711 6320 0 -1 4202560 1177 0 0 0 23 15 0 0 19 0 2 0 670629250 418942976 6667 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6795/statm: 102281 6667 406 364 0 38181 0
[pid=6797] ppid=6795 vsize=250372 CPUtime=100.37 cores=0,2,4,6
/proc/6797/stat : 6797 (FH) T 6795 6797 6797 0 -1 4202496 1677 0 0 0 10037 0 0 0 25 0 1 0 670629250 256380928 1407 33554432000 134512640 134936988 4288616880 18446744073709551615 134553261 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6797/statm: 62593 1407 41 104 0 62097 0
[pid=6795/tid=6805] ppid=6711 vsize=409124 CPUtime=0 cores=0,2,4,6
/proc/6795/task/6805/stat : 6805 (python) S 6711 6711 6320 0 -1 4202560 105 0 0 0 0 0 0 0 19 0 2 0 670639288 418942976 6667 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604448734623680 0 0 -1 6 0 0 0
[pid=6806] ppid=6711 vsize=419600 CPUtime=0.36 cores=0,2,4,6
/proc/6806/stat : 6806 (python) S 6711 6711 6320 0 -1 4202560 1171 0 0 0 18 18 0 0 19 0 2 0 670639292 429670400 6699 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6806/statm: 104900 6699 406 364 0 40797 0
[pid=6808] ppid=6806 vsize=420484 CPUtime=97.38 cores=0,2,4,6
/proc/6808/stat : 6808 (adaptg2wsat2009) T 6806 6808 6808 0 -1 4202496 1769 0 0 0 9738 0 0 0 25 0 1 0 670639292 430575616 1493 33554432000 134512640 135107411 4294780736 18446744073709551615 134531613 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6808/statm: 105121 1493 59 146 0 103997 0
[pid=6806/tid=6824] ppid=6711 vsize=419600 CPUtime=0 cores=0,2,4,6
/proc/6806/task/6824/stat : 6824 (python) S 6711 6711 6320 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 670649031 429670400 6699 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604452685872192 0 0 -1 6 0 0 0
[pid=6832] ppid=6711 vsize=430228 CPUtime=0.4 cores=0,2,4,6
/proc/6832/stat : 6832 (python) S 6711 6711 6320 0 -1 4202560 1153 0 0 0 24 16 0 0 19 0 2 0 670658775 440553472 6790 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/6832/statm: 107557 6790 400 364 0 43451 0
[pid=6834] ppid=6832 vsize=420484 CPUtime=100.28 cores=0,2,4,6
/proc/6834/stat : 6834 (adaptg2wsat2009) T 6832 6834 6834 0 -1 4202496 1765 0 0 0 10028 0 0 0 25 0 1 0 670658775 430575616 1492 33554432000 134512640 135107411 4286893120 18446744073709551615 134531246 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/6834/statm: 105121 1492 59 146 0 103997 0
[pid=6832/tid=6841] ppid=6711 vsize=430228 CPUtime=0 cores=0,2,4,6
/proc/6832/task/6841/stat : 6841 (python) S 6711 6711 6320 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670668814 440553472 6790 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604440514710464 0 0 -1 6 0 0 0
[pid=6842] ppid=6711 vsize=440484 CPUtime=0.37 cores=0,2,4,6
/proc/6842/stat : 6842 (python) S 6711 6711 6320 0 -1 4202560 1165 0 0 0 21 16 0 0 19 0 2 0 670668818 451055616 6794 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/6842/statm: 110121 6794 400 364 0 46012 0
[pid=6844] ppid=6842 vsize=250372 CPUtime=97.4 cores=0,2,4,6
/proc/6844/stat : 6844 (FH) T 6842 6844 6844 0 -1 4202496 1677 0 0 0 9740 0 0 0 25 0 1 0 670668819 256380928 1407 33554432000 134512640 134936988 4293352128 18446744073709551615 134545320 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6844/statm: 62593 1407 41 104 0 62097 0
[pid=6842/tid=6851] ppid=6711 vsize=440484 CPUtime=0 cores=0,2,4,6
/proc/6842/task/6851/stat : 6851 (python) S 6711 6711 6320 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670678560 451055616 6794 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604445919890240 0 0 -1 2 0 0 0
[pid=6852] ppid=6711 vsize=450740 CPUtime=0.36 cores=0,2,4,6
/proc/6852/stat : 6852 (python) S 6711 6711 6320 0 -1 4202560 1144 0 0 0 18 18 0 0 19 0 2 0 670678564 461557760 6799 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/6852/statm: 112685 6799 400 364 0 48573 0
[pid=6854] ppid=6852 vsize=420484 CPUtime=100.24 cores=0,2,4,6
/proc/6854/stat : 6854 (adaptg2wsat2009) T 6852 6854 6854 0 -1 4202496 1766 0 0 0 10024 0 0 0 25 0 1 0 670678564 430575616 1492 33554432000 134512640 135107411 4288159104 18446744073709551615 134532135 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/6854/statm: 105121 1492 59 146 0 103997 0
[pid=6852/tid=6866] ppid=6711 vsize=450740 CPUtime=0 cores=0,2,4,6
/proc/6852/task/6866/stat : 6866 (python) S 6711 6711 6320 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 670688600 461557760 6799 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604453896682304 0 0 -1 2 0 0 0
[pid=6867] ppid=6711 vsize=460996 CPUtime=0.76 cores=0,2,4,6
/proc/6867/stat : 6867 (python) S 6711 6711 6320 0 -1 4202560 1173 0 0 0 42 34 0 0 19 0 2 0 670688604 472059904 6803 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6867/statm: 115249 6803 400 364 0 51134 0
[pid=6869] ppid=6867 vsize=268424 CPUtime=195.89 cores=0,2,4,6
/proc/6869/stat : 6869 (hybridGM3) T 6867 6869 6869 0 -1 4202496 5634 0 0 0 19588 1 0 0 25 0 1 0 670688604 274866176 5360 33554432000 134512640 135204385 4293281136 18446744073709551615 134537792 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6869/statm: 67106 5360 63 169 0 66936 0
[pid=6867/tid=6879] ppid=6711 vsize=460996 CPUtime=0 cores=0,2,4,6
/proc/6867/task/6879/stat : 6879 (python) S 6711 6711 6320 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670708194 472059904 6803 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604451991467008 0 0 -1 6 0 0 0
[pid=6880] ppid=6711 vsize=471252 CPUtime=0.39 cores=0,2,4,6
/proc/6880/stat : 6880 (python) S 6711 6711 6320 0 -1 4202560 1148 0 0 0 19 20 0 0 19 0 2 0 670708197 482562048 6820 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6880/statm: 117813 6820 400 364 0 53695 0
[pid=6882] ppid=6880 vsize=1018800 CPUtime=100.41 cores=0,2,4,6
/proc/6882/stat : 6882 (gnovelty+2) T 6880 6882 6882 0 -1 4202496 874 0 0 0 10041 0 0 0 25 0 1 0 670708198 1043251200 602 33554432000 134512640 135663704 4294149264 18446744073709551615 134524888 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6882/statm: 254700 602 113 282 0 254416 0
[pid=6880/tid=6890] ppid=6711 vsize=471252 CPUtime=0 cores=0,2,4,6
/proc/6880/task/6890/stat : 6890 (python) S 6711 6711 6320 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670718239 482562048 6820 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604447256039872 0 0 -1 6 0 0 0
[pid=6891] ppid=6711 vsize=481508 CPUtime=0.81 cores=0,2,4,6
/proc/6891/stat : 6891 (python) S 6711 6711 6320 0 -1 4202560 1183 0 0 0 45 36 0 0 19 0 2 0 670718243 493064192 6824 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/6891/statm: 120377 6824 400 364 0 56256 0
[pid=6893] ppid=6891 vsize=268428 CPUtime=196.62 cores=0,2,4,6
/proc/6893/stat : 6893 (hybridGM3) T 6891 6893 6893 0 -1 4202496 5631 0 0 0 19662 0 0 0 25 0 1 0 670718244 274870272 5360 33554432000 134512640 135204385 4294257296 18446744073709551615 134537717 0 0 16781312 0 18446744071562691360 0 0 17 6 0 0 0
/proc/6893/statm: 67107 5360 63 169 0 66937 0
[pid=6891/tid=6903] ppid=6711 vsize=481508 CPUtime=0 cores=0,2,4,6
/proc/6891/task/6903/stat : 6903 (python) S 6711 6711 6320 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 670737929 493064192 6824 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604449910108288 0 0 -1 6 0 0 0
[pid=6904] ppid=6711 vsize=491896 CPUtime=0.81 cores=0,2,4,6
/proc/6904/stat : 6904 (python) S 6711 6711 6320 0 -1 4202560 1229 0 0 0 44 37 0 0 19 0 2 0 670737933 503701504 6855 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6904/statm: 122974 6855 408 364 0 58850 0
[pid=6906] ppid=6904 vsize=268424 CPUtime=197.9 cores=0,2,4,6
/proc/6906/stat : 6906 (hybridGM3) T 6904 6906 6906 0 -1 4202496 5636 0 0 0 19790 0 0 0 25 0 1 0 670737933 274866176 5360 33554432000 134512640 135204385 4290701264 18446744073709551615 134537711 0 0 16781312 0 18446744071562691360 0 0 17 2 0 0 0
/proc/6906/statm: 67106 5360 63 169 0 66936 0
[pid=6904/tid=6916] ppid=6711 vsize=491896 CPUtime=0 cores=0,2,4,6
/proc/6904/task/6916/stat : 6916 (python) S 6711 6711 6320 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 670757724 503701504 6855 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604453857706304 0 0 -1 2 0 0 0
[pid=6917] ppid=6711 vsize=502496 CPUtime=6.18 cores=0,2,4,6
/proc/6917/stat : 6917 (python) S 6711 6711 6320 0 -1 4202560 1223 0 0 0 330 288 0 0 19 0 2 0 670757727 514555904 6943 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658973131 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/6917/statm: 125624 6943 406 364 0 61497 0
[pid=6919] ppid=6917 vsize=458844 CPUtime=1553.65 cores=0,2,4,6
/proc/6919/stat : 6919 (TNM) T 6917 6919 6919 0 -1 4202496 1660 0 0 0 155364 1 0 0 25 0 1 0 670757728 469856256 1382 33554432000 134512640 135008748 4291264432 18446744073709551615 134527722 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/6919/statm: 114711 1382 46 122 0 113611 0
[pid=6917/tid=6970] ppid=6711 vsize=502496 CPUtime=0 cores=0,2,4,6
/proc/6917/task/6970/stat : 6970 (python) S 6711 6711 6320 0 -1 4202560 50 0 0 0 0 0 0 0 19 0 2 0 670913092 514555904 6943 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 514 18446604452251711552 0 0 -1 6 0 0 0
[pid=6971] ppid=6711 vsize=512752 CPUtime=0 cores=0,2,4,6
/proc/6971/stat : 6971 (python) S 6711 6711 6320 0 -1 4202560 829 0 0 0 0 0 0 0 19 0 1 0 670913094 525058048 6885 33554432000 4194304 5683644 140736973467536 18446744073709551615 210647166534 0 0 16781312 514 1085834257177247743 0 0 17 6 0 0 0
/proc/6971/statm: 128188 6885 345 364 0 64058 0
[pid=6973] ppid=6971 vsize=247164 CPUtime=0.05 cores=0,2,4,6
/proc/6973/stat : 6973 (FH) R 6971 6973 6973 0 -1 4202496 1057 0 0 0 5 0 0 0 20 0 1 0 670913094 253095936 782 33554432000 134512640 134936988 4294400160 18446744073709551615 134523622 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/6973/statm: 61791 782 37 104 0 61295 0
[pid=6711/tid=6718] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6718/stat : 6718 (python) S 6709 6711 6320 0 -1 4202560 31 931 0 0 0 0 0 0 18 0 18 0 670570090 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
[pid=6711/tid=6725] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6725/stat : 6725 (python) S 6709 6711 6320 0 -1 4202560 25 931 0 0 0 0 0 0 20 0 18 0 670580033 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 6 0 0 0
[pid=6711/tid=6732] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6732/stat : 6732 (python) S 6709 6711 6320 0 -1 4202560 87 931 0 0 0 0 0 0 20 0 18 0 670589778 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6738] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6738/stat : 6738 (python) S 6709 6711 6320 0 -1 4202560 32 931 0 0 0 0 0 0 20 0 18 0 670599719 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6745] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6745/stat : 6745 (python) S 6709 6711 6320 0 -1 4202560 35 931 0 0 0 0 0 0 20 0 18 0 670609561 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6785] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6785/stat : 6785 (python) S 6709 6711 6320 0 -1 4202560 30 931 0 0 0 0 0 0 20 0 18 0 670619305 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6796] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6796/stat : 6796 (python) S 6709 6711 6320 0 -1 4202560 30 931 0 0 0 0 0 0 20 0 18 0 670629250 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6807] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6807/stat : 6807 (python) S 6709 6711 6320 0 -1 4202560 31 931 0 0 0 0 0 0 20 0 18 0 670639292 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
[pid=6711/tid=6833] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6833/stat : 6833 (python) S 6709 6711 6320 0 -1 4202560 32 931 0 0 0 0 0 0 20 0 18 0 670658775 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6843] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6843/stat : 6843 (python) S 6709 6711 6320 0 -1 4202560 30 931 0 0 0 0 0 0 20 0 18 0 670668819 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6853] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6853/stat : 6853 (python) S 6709 6711 6320 0 -1 4202560 24 931 0 0 0 0 0 0 20 0 18 0 670678564 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6868] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6868/stat : 6868 (python) S 6709 6711 6320 0 -1 4202560 29 931 0 0 0 0 0 0 20 0 18 0 670688604 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
[pid=6711/tid=6881] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6881/stat : 6881 (python) S 6709 6711 6320 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 18 0 670708198 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
[pid=6711/tid=6892] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6892/stat : 6892 (python) S 6709 6711 6320 0 -1 4202560 28 931 0 0 0 0 0 0 20 0 18 0 670718243 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 0 0 0 0
[pid=6711/tid=6905] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6905/stat : 6905 (python) S 6709 6711 6320 0 -1 4202560 27 931 0 0 0 0 0 0 20 0 18 0 670737933 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 6 0 0 0
[pid=6711/tid=6918] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6918/stat : 6918 (python) S 6709 6711 6320 0 -1 4202560 27 931 0 0 0 0 0 0 20 0 18 0 670757728 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
[pid=6711/tid=6972] ppid=6709 vsize=522996 CPUtime=0 cores=0,2,4,6
/proc/6711/task/6972/stat : 6972 (python) S 6709 6711 6320 0 -1 4202560 26 931 0 0 0 0 0 0 20 0 18 0 670913094 535547904 8454 33554432000 4194304 5683644 140736973467536 18446744073709551615 210658970033 0 0 16781312 2 18446604444122531072 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3444.78
Current children cumulated vsize (KiB) 16322744

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 6711 and gives
#  childrusage.ru_utime.tv_sec=2
#  childrusage.ru_utime.tv_usec=112678
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=439933
# CPU time returned by wait4() is 2.55261
# while last known CPU time is 3444.78
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 3433.73
CPU time (s): 3444.78
CPU user time (s): 3438.18
CPU system time (s): 6.6
CPU usage (%): 100.322
Max. virtual memory (cumulated for all children) (KiB): 16322744

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 2.11268
system time used= 0.439933
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 28661
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2925
involuntary context switches= 53

runsolver used 33.3549 second user time and 67.8727 second system time

The end

Launcher Data 

Begin job on node129 at 2011-05-17 01:48:21
IDJOB=3386634
IDBENCH=82699
IDSOLVER=1872
FILE ID=node129/3386634-1305589701
RUNJOBID= node129-1305586145-6338
PBS_JOBID= 13350349
Free space on /tmp= 73236 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed)
BENCH NAME= SAT11/random/large/unif-k5-r20-v1250-c25000-S871131876-087-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3386634-1305589701/watcher-3386634-1305589701 -o /tmp/evaluation-result-3386634-1305589701/solver-3386634-1305589701 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3386634-1305589701.cnf -seed 1662753417 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 637b75113dde2f4a15dbf158341be921
RANDOM SEED=1662753417

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      18114640 kB
Buffers:        296092 kB
Cached:        7104848 kB
SwapCached:      52664 kB
Active:        5570684 kB
Inactive:      8990044 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      18114640 kB
SwapTotal:    67111528 kB
SwapFree:     64496000 kB
Dirty:           32252 kB
Writeback:           0 kB
AnonPages:     7115968 kB
Mapped:          30624 kB
Slab:           144380 kB
PageTables:      62704 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS: 130313092 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73204 MiB
End job on node129 at 2011-05-17 02:45:37