Trace number 3484065

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-random (fixed, 1200 s timeout)? (TO) 1200.07 1199.25

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v25000-c105000-S109959541-056-UNKNOWN.cnf
MD5SUM5f909806405bfbe25b7db5afcabe8fc9
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark87.7936
Satisfiable
(Un)Satisfiability was proved
Number of variables24999
Number of clauses105000
Sum of the clauses size315000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3105000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.33/0.45	c INFO: loaded portfolio from HOME/etc/borg-mix+class.random.pickle
0.33/0.45	c INFO: solving HOME/instance-3484065-1307747026.cnf
0.33/0.45	c INFO: computing task features
0.71/0.82	c INFO: computed variable-clause graph statistics
0.71/0.84	c INFO: computed clause balance statistics
2.52/2.69	c INFO: computed variable balance statistics
4.12/4.21	c INFO: computed small-clause counts
4.12/4.22	c INFO: computed Horn-clause counts
4.92/5.05	c INFO: computed variable graph statistics
4.92/5.05	c INFO: collected features for HOME/instance-3484065-1307747026.cnf in 4.59 s
4.92/5.06	c INFO: running hybridGM3@0 for 97 with 1195 remaining (b = 0.37)
105.73/105.77	c INFO: running hybridGM3@0 for 97 with 1094 remaining (b = 0.24)
206.55/206.51	c INFO: running hybridGM3@0 for 97 with 994 remaining (b = 0.17)
307.36/307.26	c INFO: running hybridGM3@100 for 97 with 893 remaining (b = 0.13)
403.72/403.41	c INFO: running hybridGM3@100 for 97 with 797 remaining (b = 0.13)
500.87/500.55	c INFO: running TNM@0 for 97 with 700 remaining (b = 0.14)
601.07/600.70	c INFO: running hybridGM3@100 for 97 with 600 remaining (b = 0.12)
698.31/697.84	c INFO: running hybridGM3@0 for 97 with 503 remaining (b = 0.12)
798.15/797.59	c INFO: running hybridGM3@99 for 97 with 403 remaining (b = 0.09)
893.49/892.73	c INFO: running hybridGM3@0 for 97 with 308 remaining (b = 0.09)
994.22/993.48	c INFO: running hybridGM3@100 for 97 with 207 remaining (b = 0.07)
1089.45/1088.62	c INFO: running hybridGM3@0 for 112 with 112 remaining (b = 0.06)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 988)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3484065-1307747026/watcher-3484065-1307747026 -o /tmp/evaluation-result-3484065-1307747026/solver-3484065-1307747026 -C 1200 -W 1300 -M 7680 python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3484065-1307747026.cnf -seed 835850587 -budget 1200 

running on 2 cores: 5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1200 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1230 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 7864320 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 7915520 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.86 1.47 1.16 5/199 13759
/proc/meminfo: memFree=30346264/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=15384 CPUtime=0 cores=5,7
/proc/13759/stat : 13759 (python) R 13757 13759 13569 0 -1 4202496 822 0 0 0 0 0 0 0 25 0 1 0 886300586 15753216 705 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122709861 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13759/statm: 3846 705 334 364 0 333 0

[startup+0.0569291 s]
/proc/loadavg: 2.86 1.47 1.16 5/199 13759
/proc/meminfo: memFree=30346264/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=36508 CPUtime=0.04 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 1621 0 0 0 3 1 0 0 25 0 1 0 886300586 37384192 1450 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122712064 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/13759/statm: 9127 1450 452 364 0 944 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 36508

[startup+0.100922 s]
/proc/loadavg: 2.86 1.47 1.16 5/199 13759
/proc/meminfo: memFree=30346264/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=104616 CPUtime=0.07 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 3145 0 0 0 6 1 0 0 17 0 1 0 886300586 107126784 2822 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122712064 0 2147483391 16781312 2 18446744071699207358 0 0 17 7 0 0 0
/proc/13759/statm: 26154 2822 847 364 0 1902 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 104616

[startup+0.306954 s]
/proc/loadavg: 2.86 1.47 1.16 5/199 13759
/proc/meminfo: memFree=30346264/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=198920 CPUtime=0.23 cores=5,7
/proc/13759/stat : 13759 (python) R 13757 13759 13569 0 -1 4202496 7030 934 1 0 17 6 0 0 18 0 1 0 886300586 203694080 5054 33554432000 4194304 5683644 140734817021824 18446744073709551615 4620076 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13759/statm: 49730 5054 1466 364 0 3757 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 198920

[startup+0.700844 s]
/proc/loadavg: 2.86 1.47 1.16 5/199 13759
/proc/meminfo: memFree=30346264/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=295668 CPUtime=0.61 cores=5,7
/proc/13759/stat : 13759 (python) R 13757 13759 13569 0 -1 4202496 13425 934 1 0 52 9 0 0 18 0 1 0 886300586 302764032 11095 33554432000 4194304 5683644 140734817021824 18446744073709551615 47243492164925 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13759/statm: 73917 11095 1761 364 0 9841 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 295668

[startup+1.50071 s]
/proc/loadavg: 2.86 1.47 1.16 6/200 13764
/proc/meminfo: memFree=30301228/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=302956 CPUtime=1.42 cores=5,7
/proc/13759/stat : 13759 (python) R 13757 13759 13569 0 -1 4202496 18323 934 1 0 132 10 0 0 20 0 1 0 886300586 310226944 13124 33554432000 4194304 5683644 140734817021824 18446744073709551615 47243321565024 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13759/statm: 75739 13124 1826 364 0 11663 0
Current children cumulated CPU time (s) 1.42
Current children cumulated vsize (KiB) 302956

[startup+3.10041 s]
/proc/loadavg: 2.86 1.47 1.16 6/200 13764
/proc/meminfo: memFree=30300856/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=302956 CPUtime=3.02 cores=5,7
/proc/13759/stat : 13759 (python) R 13757 13759 13569 0 -1 4202496 18324 934 1 0 292 10 0 0 25 0 1 0 886300586 310226944 13125 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122400467 0 0 16781312 2 0 0 0 17 5 0 0 0
/proc/13759/statm: 75739 13125 1827 364 0 11663 0
Current children cumulated CPU time (s) 3.02
Current children cumulated vsize (KiB) 302956

[startup+6.3008 s]
/proc/loadavg: 3.11 1.55 1.19 6/195 13792
/proc/meminfo: memFree=30349212/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=311272 CPUtime=4.98 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 18607 934 1 0 488 10 0 0 25 0 2 0 886300586 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13759/statm: 77818 12681 1867 364 0 13736 0
[pid=13765] ppid=13759 vsize=301028 CPUtime=0 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 731 0 0 0 0 0 0 0 20 0 1 0 886301092 308252672 11163 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734095 0 0 16781312 514 4395637485422510079 0 0 17 7 0 0 0
/proc/13765/statm: 75257 11163 353 364 0 11175 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=1.23 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) R 13765 13767 13767 0 -1 4202496 7135 0 0 0 122 1 0 0 25 0 1 0 886301092 274870272 6869 33554432000 134512640 135204385 4287616416 18446744073709551615 134546086 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6869 64 169 0 66937 0
[pid=13759/tid=13766] ppid=13757 vsize=311272 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 2 0 886301092 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 6.21
Current children cumulated vsize (KiB) 880728

[startup+12.7007 s]
/proc/loadavg: 3.26 1.60 1.21 6/200 13811
/proc/meminfo: memFree=30223188/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=311272 CPUtime=4.98 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 18607 934 1 0 488 10 0 0 25 0 2 0 886300586 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13759/statm: 77818 12681 1867 364 0 13736 0
[pid=13765] ppid=13759 vsize=301028 CPUtime=0 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 871 0 0 0 0 0 0 0 18 0 1 0 886301092 308252672 11186 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734095 0 0 16781312 514 4395637485422477576 0 0 17 7 0 0 0
/proc/13765/statm: 75257 11186 376 364 0 11175 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=7.63 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) R 13765 13767 13767 0 -1 4202496 7135 0 0 0 762 1 0 0 25 0 1 0 886301092 274870272 6869 33554432000 134512640 135204385 4287616416 18446744073709551615 134546082 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6869 64 169 0 66937 0
[pid=13759/tid=13766] ppid=13757 vsize=311272 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 2 0 886301092 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.61
Current children cumulated vsize (KiB) 880728

[startup+25.5003 s]
/proc/loadavg: 3.65 1.77 1.27 6/203 13814
/proc/meminfo: memFree=30188228/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=311272 CPUtime=4.98 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 18607 934 1 0 488 10 0 0 25 0 2 0 886300586 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13759/statm: 77818 12681 1867 364 0 13736 0
[pid=13765] ppid=13759 vsize=301028 CPUtime=0.02 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 880 0 0 0 1 1 0 0 18 0 1 0 886301092 308252672 11186 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734095 0 0 16781312 514 12563090227712721160 0 0 17 7 0 0 0
/proc/13765/statm: 75257 11186 376 364 0 11175 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=20.43 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) R 13765 13767 13767 0 -1 4202496 7137 0 0 0 2042 1 0 0 25 0 1 0 886301092 274870272 6871 33554432000 134512640 135204385 4287616416 18446744073709551615 134547802 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6871 64 169 0 66937 0
[pid=13759/tid=13766] ppid=13757 vsize=311272 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 2 0 886301092 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 25.43
Current children cumulated vsize (KiB) 880728

[startup+51.1007 s]
/proc/loadavg: 4.30 2.07 1.38 6/203 13818
/proc/meminfo: memFree=30188108/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=311272 CPUtime=4.98 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 18607 934 1 0 488 10 0 0 25 0 2 0 886300586 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13759/statm: 77818 12681 1867 364 0 13736 0
[pid=13765] ppid=13759 vsize=301028 CPUtime=0.05 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 892 0 0 0 2 3 0 0 18 0 1 0 886301092 308252672 11186 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734095 0 0 16781312 514 12563090227712721160 0 0 17 7 0 0 0
/proc/13765/statm: 75257 11186 376 364 0 11175 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=46.03 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) R 13765 13767 13767 0 -1 4202496 7137 0 0 0 4602 1 0 0 25 0 1 0 886301092 274870272 6871 33554432000 134512640 135204385 4287616416 18446744073709551615 134536663 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6871 64 169 0 66937 0
[pid=13759/tid=13766] ppid=13757 vsize=311272 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 2 0 886301092 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 51.06
Current children cumulated vsize (KiB) 880728

[startup+102.307 s]
/proc/loadavg: 4.74 2.54 1.57 6/211 13827
/proc/meminfo: memFree=30146696/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=311272 CPUtime=4.98 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 18607 934 1 0 488 10 0 0 25 0 2 0 886300586 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13759/statm: 77818 12681 1867 364 0 13736 0
[pid=13765] ppid=13759 vsize=301028 CPUtime=0.12 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 917 0 0 0 6 6 0 0 18 0 1 0 886301092 308252672 11186 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734095 0 0 16781312 514 12563090227712721160 0 0 17 7 0 0 0
/proc/13765/statm: 75257 11186 376 364 0 11175 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=97.24 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) R 13765 13767 13767 0 -1 4202496 7137 0 0 0 9723 1 0 0 25 0 1 0 886301092 274870272 6871 33554432000 134512640 135204385 4287616416 18446744073709551615 134537452 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6871 64 169 0 66937 0
[pid=13759/tid=13766] ppid=13757 vsize=311272 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 2 0 886301092 318742528 12681 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 102.34
Current children cumulated vsize (KiB) 880728

[startup+162.301 s]
/proc/loadavg: 4.99 3.01 1.79 6/219 13837
/proc/meminfo: memFree=30061388/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=321528 CPUtime=4.99 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 19524 934 1 0 489 10 0 0 20 0 3 0 886300586 329244672 12688 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13759/statm: 80382 12688 1870 364 0 16297 0
[pid=13765] ppid=13759 vsize=311272 CPUtime=0.13 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 1163 0 0 0 7 6 0 0 19 0 2 0 886301092 318742528 11230 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13765/statm: 77818 11230 415 364 0 13736 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=100.7 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) T 13765 13767 13767 0 -1 4202496 7137 0 0 0 10069 1 0 0 25 0 1 0 886301092 274870272 6871 33554432000 134512640 135204385 4287616416 18446744073709551615 134536605 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6871 64 169 0 66937 0
[pid=13765/tid=13828] ppid=13759 vsize=311272 CPUtime=0 cores=5,7
/proc/13765/task/13828/stat : 13828 (python) S 13759 13759 13569 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 886311163 318742528 11230 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471116001536 0 0 -1 7 0 0 0
[pid=13829] ppid=13759 vsize=311284 CPUtime=0.06 cores=5,7
/proc/13829/stat : 13829 (python) S 13759 13759 13569 0 -1 4202560 925 0 0 0 3 3 0 0 15 0 1 0 886311164 318754816 11184 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734150 0 0 16781312 514 12563090227712721160 0 0 17 5 0 0 0
/proc/13829/statm: 77821 11184 370 364 0 13736 0
[pid=13831] ppid=13829 vsize=268424 CPUtime=56.45 cores=5,7
/proc/13831/stat : 13831 (hybridGM3) R 13829 13831 13831 0 -1 4202496 7136 0 0 0 5645 0 0 0 25 0 1 0 886311164 274866176 6873 33554432000 134512640 135204385 4293263888 18446744073709551615 134537322 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/13831/statm: 67106 6873 64 169 0 66936 0
[pid=13759/tid=13766] ppid=13757 vsize=321528 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 27 934 0 0 0 0 0 0 25 0 3 0 886301092 329244672 12688 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13830] ppid=13757 vsize=321528 CPUtime=0 cores=5,7
/proc/13759/task/13830/stat : 13830 (python) S 13757 13759 13569 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 3 0 886311164 329244672 12688 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 162.33
Current children cumulated vsize (KiB) 1480936

[startup+222.301 s]
/proc/loadavg: 5.04 3.39 2.00 6/235 13854
/proc/meminfo: memFree=29943548/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=331784 CPUtime=4.99 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 20387 934 1 0 489 10 0 0 20 0 4 0 886300586 339746816 12694 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 0
/proc/13759/statm: 82946 12694 1872 364 0 18858 0

################
# More data... #
################

[pid=13765] ppid=13759 vsize=311272 CPUtime=0.28 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 1223 0 0 0 15 13 0 0 18 0 2 0 886301092 318742528 11231 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13765/statm: 77818 11231 416 364 0 13736 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=196.84 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) T 13765 13767 13767 0 -1 4202496 7138 0 0 0 19683 1 0 0 25 0 1 0 886301092 274870272 6872 33554432000 134512640 135204385 4287616416 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6872 64 169 0 66937 0
[pid=13765/tid=13828] ppid=13759 vsize=311272 CPUtime=0 cores=5,7
/proc/13765/task/13828/stat : 13828 (python) S 13759 13759 13569 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 886311163 318742528 11231 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471116001536 0 0 -1 7 0 0 0
[pid=13829] ppid=13759 vsize=311284 CPUtime=0.28 cores=5,7
/proc/13829/stat : 13829 (python) S 13759 13759 13569 0 -1 4202560 1133 0 0 0 13 15 0 0 16 0 2 0 886311164 318754816 11214 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13829/statm: 77821 11214 399 364 0 13736 0
[pid=13831] ppid=13829 vsize=268424 CPUtime=197.64 cores=5,7
/proc/13831/stat : 13831 (hybridGM3) T 13829 13831 13831 0 -1 4202496 7136 0 0 0 19764 0 0 0 25 0 1 0 886311164 274866176 6873 33554432000 134512640 135204385 4293263888 18446744073709551615 134537357 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13831/statm: 67106 6873 64 169 0 66936 0
[pid=13829/tid=13847] ppid=13759 vsize=311284 CPUtime=0 cores=5,7
/proc/13829/task/13847/stat : 13847 (python) S 13759 13759 13569 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 886321236 318754816 11214 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471126098176 0 0 -1 5 0 0 0
[pid=13848] ppid=13759 vsize=321540 CPUtime=0.29 cores=5,7
/proc/13848/stat : 13848 (python) S 13759 13759 13569 0 -1 4202560 1131 0 0 0 17 12 0 0 15 0 2 0 886321237 329256960 11218 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13848/statm: 80385 11218 399 364 0 16297 0
[pid=13850] ppid=13848 vsize=268420 CPUtime=197.86 cores=5,7
/proc/13850/stat : 13850 (hybridGM3) T 13848 13850 13850 0 -1 4202496 7133 0 0 0 19786 0 0 0 25 0 1 0 886321238 274862080 6871 33554432000 134512640 135204385 4288187776 18446744073709551615 134536806 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13850/statm: 67105 6871 64 169 0 66935 0
[pid=13848/tid=13862] ppid=13759 vsize=321540 CPUtime=0 cores=5,7
/proc/13848/task/13862/stat : 13862 (python) S 13759 13759 13569 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 886331311 329256960 11218 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471098581248 0 0 -1 7 0 0 0
[pid=13881] ppid=13759 vsize=331796 CPUtime=0.14 cores=5,7
/proc/13881/stat : 13881 (python) S 13759 13759 13569 0 -1 4202560 1092 0 0 0 7 7 0 0 19 0 2 0 886350641 339759104 11222 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13881/statm: 82949 11222 399 364 0 18858 0
[pid=13883] ppid=13881 vsize=460952 CPUtime=100 cores=5,7
/proc/13883/stat : 13883 (TNM) T 13881 13883 13883 0 -1 4202496 4391 0 0 0 10000 0 0 0 25 0 1 0 886350641 472014848 4134 33554432000 134512640 135008748 4288804240 18446744073709551615 134524687 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13883/statm: 115238 4134 46 122 0 114138 0
[pid=13881/tid=13894] ppid=13759 vsize=331796 CPUtime=0 cores=5,7
/proc/13881/task/13894/stat : 13894 (python) S 13759 13759 13569 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 886360655 339759104 11222 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471126099968 0 0 -1 7 0 0 0
[pid=13952] ppid=13759 vsize=342052 CPUtime=0.3 cores=5,7
/proc/13952/stat : 13952 (python) S 13759 13759 13569 0 -1 4202560 1139 0 0 0 16 14 0 0 15 0 2 0 886370370 350261248 11226 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13952/statm: 85513 11226 399 364 0 21419 0
[pid=13954] ppid=13952 vsize=268420 CPUtime=194.86 cores=5,7
/proc/13954/stat : 13954 (hybridGM3) T 13952 13954 13954 0 -1 4202496 7134 0 0 0 19486 0 0 0 25 0 1 0 886370370 274862080 6873 33554432000 134512640 135204385 4292681840 18446744073709551615 134545992 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13954/statm: 67105 6873 64 169 0 66935 0
[pid=13952/tid=13966] ppid=13759 vsize=342052 CPUtime=0 cores=5,7
/proc/13952/task/13966/stat : 13966 (python) S 13759 13759 13569 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 886380345 350261248 11226 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604458875439872 0 0 -1 7 0 0 0
[pid=13977] ppid=13759 vsize=352308 CPUtime=0.29 cores=5,7
/proc/13977/stat : 13977 (python) S 13759 13759 13569 0 -1 4202560 1135 0 0 0 14 15 0 0 15 0 2 0 886389859 360763392 11230 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13977/statm: 88077 11230 399 364 0 23980 0
[pid=13979] ppid=13977 vsize=268424 CPUtime=195.63 cores=5,7
/proc/13979/stat : 13979 (hybridGM3) T 13977 13979 13979 0 -1 4202496 7136 0 0 0 19563 0 0 0 25 0 1 0 886389859 274866176 6876 33554432000 134512640 135204385 4291870816 18446744073709551615 134536744 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13979/statm: 67106 6876 64 169 0 66936 0
[pid=13977/tid=13991] ppid=13759 vsize=352308 CPUtime=0 cores=5,7
/proc/13977/task/13991/stat : 13991 (python) S 13759 13759 13569 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 886399933 360763392 11230 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604455945381888 0 0 -1 5 0 0 0
[pid=14002] ppid=13759 vsize=362564 CPUtime=0.14 cores=5,7
/proc/14002/stat : 14002 (python) S 13759 13759 13569 0 -1 4202560 949 0 0 0 6 8 0 0 15 0 1 0 886409448 371265536 11204 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734150 0 0 16781312 514 12563090227712721158 0 0 17 5 0 0 0
/proc/14002/statm: 90641 11204 370 364 0 26541 0
[pid=14004] ppid=14002 vsize=268428 CPUtime=93.55 cores=5,7
/proc/14004/stat : 14004 (hybridGM3) R 14002 14004 14004 0 -1 4202496 7134 0 0 0 9355 0 0 0 25 0 1 0 886409448 274870272 6874 33554432000 134512640 135204385 4292598304 18446744073709551615 134536744 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/14004/statm: 67107 6874 64 169 0 66937 0
[pid=13759/tid=13766] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 8 0 886301092 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13830] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13830/stat : 13830 (python) S 13757 13759 13569 0 -1 4202560 36 934 0 0 0 0 0 0 18 0 8 0 886311164 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13849] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13849/stat : 13849 (python) S 13757 13759 13569 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 8 0 886321237 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13882] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13882/stat : 13882 (python) S 13757 13759 13569 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 8 0 886350641 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13953] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13953/stat : 13953 (python) S 13757 13759 13569 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 8 0 886370370 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 7 0 0 0
[pid=13759/tid=13978] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13978/stat : 13978 (python) S 13757 13759 13569 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 8 0 886389859 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=14003] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/14003/stat : 14003 (python) S 13757 13759 13569 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 8 0 886409448 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1183.17
Current children cumulated vsize (KiB) 4777120



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1199.2 s]
/proc/loadavg: 4.00 4.08 3.47 5/270 14107
/proc/meminfo: memFree=30147052/32951124 swapFree=67051744/67111528
[pid=13759] ppid=13757 vsize=372808 CPUtime=5.07 cores=5,7
/proc/13759/stat : 13759 (python) S 13757 13759 13569 0 -1 4202496 24287 934 1 0 495 12 0 0 20 0 8 0 886300586 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 0
/proc/13759/statm: 93202 12726 1888 364 0 29102 0
[pid=13765] ppid=13759 vsize=311272 CPUtime=0.28 cores=5,7
/proc/13765/stat : 13765 (python) S 13759 13759 13569 0 -1 4202560 1223 0 0 0 15 13 0 0 18 0 2 0 886301092 318742528 11231 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13765/statm: 77818 11231 416 364 0 13736 0
[pid=13767] ppid=13765 vsize=268428 CPUtime=196.84 cores=5,7
/proc/13767/stat : 13767 (hybridGM3) T 13765 13767 13767 0 -1 4202496 7138 0 0 0 19683 1 0 0 25 0 1 0 886301092 274870272 6872 33554432000 134512640 135204385 4287616416 18446744073709551615 134536993 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13767/statm: 67107 6872 64 169 0 66937 0
[pid=13765/tid=13828] ppid=13759 vsize=311272 CPUtime=0 cores=5,7
/proc/13765/task/13828/stat : 13828 (python) S 13759 13759 13569 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 886311163 318742528 11231 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471116001536 0 0 -1 7 0 0 0
[pid=13829] ppid=13759 vsize=311284 CPUtime=0.28 cores=5,7
/proc/13829/stat : 13829 (python) S 13759 13759 13569 0 -1 4202560 1133 0 0 0 13 15 0 0 16 0 2 0 886311164 318754816 11214 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13829/statm: 77821 11214 399 364 0 13736 0
[pid=13831] ppid=13829 vsize=268424 CPUtime=197.64 cores=5,7
/proc/13831/stat : 13831 (hybridGM3) T 13829 13831 13831 0 -1 4202496 7136 0 0 0 19764 0 0 0 25 0 1 0 886311164 274866176 6873 33554432000 134512640 135204385 4293263888 18446744073709551615 134537357 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13831/statm: 67106 6873 64 169 0 66936 0
[pid=13829/tid=13847] ppid=13759 vsize=311284 CPUtime=0 cores=5,7
/proc/13829/task/13847/stat : 13847 (python) S 13759 13759 13569 0 -1 4202560 46 0 0 0 0 0 0 0 15 0 2 0 886321236 318754816 11214 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471126098176 0 0 -1 5 0 0 0
[pid=13848] ppid=13759 vsize=321540 CPUtime=0.29 cores=5,7
/proc/13848/stat : 13848 (python) S 13759 13759 13569 0 -1 4202560 1131 0 0 0 17 12 0 0 15 0 2 0 886321237 329256960 11218 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13848/statm: 80385 11218 399 364 0 16297 0
[pid=13850] ppid=13848 vsize=268420 CPUtime=197.86 cores=5,7
/proc/13850/stat : 13850 (hybridGM3) T 13848 13850 13850 0 -1 4202496 7133 0 0 0 19786 0 0 0 25 0 1 0 886321238 274862080 6871 33554432000 134512640 135204385 4288187776 18446744073709551615 134536806 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13850/statm: 67105 6871 64 169 0 66935 0
[pid=13848/tid=13862] ppid=13759 vsize=321540 CPUtime=0 cores=5,7
/proc/13848/task/13862/stat : 13862 (python) S 13759 13759 13569 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 886331311 329256960 11218 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471098581248 0 0 -1 7 0 0 0
[pid=13881] ppid=13759 vsize=331796 CPUtime=0.14 cores=5,7
/proc/13881/stat : 13881 (python) S 13759 13759 13569 0 -1 4202560 1092 0 0 0 7 7 0 0 19 0 2 0 886350641 339759104 11222 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13881/statm: 82949 11222 399 364 0 18858 0
[pid=13883] ppid=13881 vsize=460952 CPUtime=100 cores=5,7
/proc/13883/stat : 13883 (TNM) T 13881 13883 13883 0 -1 4202496 4391 0 0 0 10000 0 0 0 25 0 1 0 886350641 472014848 4134 33554432000 134512640 135008748 4288804240 18446744073709551615 134524687 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13883/statm: 115238 4134 46 122 0 114138 0
[pid=13881/tid=13894] ppid=13759 vsize=331796 CPUtime=0 cores=5,7
/proc/13881/task/13894/stat : 13894 (python) S 13759 13759 13569 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 886360655 339759104 11222 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604471126099968 0 0 -1 7 0 0 0
[pid=13952] ppid=13759 vsize=342052 CPUtime=0.3 cores=5,7
/proc/13952/stat : 13952 (python) S 13759 13759 13569 0 -1 4202560 1139 0 0 0 16 14 0 0 15 0 2 0 886370370 350261248 11226 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/13952/statm: 85513 11226 399 364 0 21419 0
[pid=13954] ppid=13952 vsize=268420 CPUtime=194.86 cores=5,7
/proc/13954/stat : 13954 (hybridGM3) T 13952 13954 13954 0 -1 4202496 7134 0 0 0 19486 0 0 0 25 0 1 0 886370370 274862080 6873 33554432000 134512640 135204385 4292681840 18446744073709551615 134545992 0 0 16781312 0 18446744071562691360 0 0 17 5 0 0 0
/proc/13954/statm: 67105 6873 64 169 0 66935 0
[pid=13952/tid=13966] ppid=13759 vsize=342052 CPUtime=0 cores=5,7
/proc/13952/task/13966/stat : 13966 (python) S 13759 13759 13569 0 -1 4202560 47 0 0 0 0 0 0 0 15 0 2 0 886380345 350261248 11226 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604458875439872 0 0 -1 7 0 0 0
[pid=13977] ppid=13759 vsize=352308 CPUtime=0.29 cores=5,7
/proc/13977/stat : 13977 (python) S 13759 13759 13569 0 -1 4202560 1135 0 0 0 14 15 0 0 15 0 2 0 886389859 360763392 11230 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134540747 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/13977/statm: 88077 11230 399 364 0 23980 0
[pid=13979] ppid=13977 vsize=268424 CPUtime=195.63 cores=5,7
/proc/13979/stat : 13979 (hybridGM3) T 13977 13979 13979 0 -1 4202496 7136 0 0 0 19563 0 0 0 25 0 1 0 886389859 274866176 6876 33554432000 134512640 135204385 4291870816 18446744073709551615 134536744 0 0 16781312 0 18446744071562691360 0 0 17 7 0 0 0
/proc/13979/statm: 67106 6876 64 169 0 66936 0
[pid=13977/tid=13991] ppid=13759 vsize=352308 CPUtime=0 cores=5,7
/proc/13977/task/13991/stat : 13991 (python) S 13759 13759 13569 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 886399933 360763392 11230 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 514 18446604455945381888 0 0 -1 5 0 0 0
[pid=14002] ppid=13759 vsize=362564 CPUtime=0.16 cores=5,7
/proc/14002/stat : 14002 (python) S 13759 13759 13569 0 -1 4202560 955 0 0 0 7 9 0 0 15 0 1 0 886409448 371265536 11204 33554432000 4194304 5683644 140734817021824 18446744073709551615 228122734150 0 0 16781312 514 12563090227712721158 0 0 17 5 0 0 0
/proc/14002/statm: 90641 11204 370 364 0 26541 0
[pid=14004] ppid=14002 vsize=268428 CPUtime=110.43 cores=5,7
/proc/14004/stat : 14004 (hybridGM3) R 14002 14004 14004 0 -1 4202496 7134 0 0 0 11043 0 0 0 25 0 1 0 886409448 274870272 6874 33554432000 134512640 135204385 4292598304 18446744073709551615 134536986 0 0 16781312 0 0 0 0 17 5 0 0 0
/proc/14004/statm: 67107 6874 64 169 0 66937 0
[pid=13759/tid=13766] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13766/stat : 13766 (python) S 13757 13759 13569 0 -1 4202560 31 934 0 0 0 0 0 0 18 0 8 0 886301092 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13830] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13830/stat : 13830 (python) S 13757 13759 13569 0 -1 4202560 36 934 0 0 0 0 0 0 18 0 8 0 886311164 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13849] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13849/stat : 13849 (python) S 13757 13759 13569 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 8 0 886321237 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13882] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13882/stat : 13882 (python) S 13757 13759 13569 0 -1 4202560 25 934 0 0 0 0 0 0 20 0 8 0 886350641 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=13953] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13953/stat : 13953 (python) S 13757 13759 13569 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 8 0 886370370 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 7 0 0 0
[pid=13759/tid=13978] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/13978/stat : 13978 (python) S 13757 13759 13569 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 8 0 886389859 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 5 0 0 0
[pid=13759/tid=14003] ppid=13757 vsize=372808 CPUtime=0 cores=5,7
/proc/13759/task/14003/stat : 14003 (python) S 13757 13759 13569 0 -1 4202560 31 934 0 0 0 0 0 0 20 0 8 0 886409448 381755392 12726 33554432000 4194304 5683644 140734817021824 18446744073709551615 228134537649 0 0 16781312 2 18446604457434509696 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1200.07
Current children cumulated vsize (KiB) 4777120

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 13759 and gives
#  childrusage.ru_utime.tv_sec=4
#  childrusage.ru_utime.tv_usec=955246
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=124981
# CPU time returned by wait4() is 5.08023
# while last known CPU time is 1200.07
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1199.25
CPU time (s): 1200.07
CPU user time (s): 1199.09
CPU system time (s): 0.98
CPU usage (%): 100.068
Max. virtual memory (cumulated for all children) (KiB): 4777120

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4.95525
system time used= 0.124981
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 25226
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1017
involuntary context switches= 47

runsolver used 2.78758 second user time and 5.70613 second system time

The end

Launcher Data 

Begin job on node138 at 2011-06-11 01:03:46
IDJOB=3484065
IDBENCH=82576
IDSOLVER=1975
FILE ID=node138/3484065-1307747026
RUNJOBID= node138-1307746989-13590
PBS_JOBID= 13550821
Free space on /tmp= 73984 MiB

SOLVER NAME= borg-sat 11.03.02-seq-random (fixed, 1200 s timeout)
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v25000-c105000-S109959541-056-UNKNOWN.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.random.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3484065-1307747026/watcher-3484065-1307747026 -o /tmp/evaluation-result-3484065-1307747026/solver-3484065-1307747026 -C 1200 -W 1300 -M 7680  python HOME/solve HOME/etc/borg-mix+class.random.pickle HOME/instance-3484065-1307747026.cnf -seed 835850587 -budget 1200

TIME LIMIT= 1200 seconds
MEMORY LIMIT= 7680 MiB
NBCORE= 2

MD5SUM BENCH= 5f909806405bfbe25b7db5afcabe8fc9
RANDOM SEED=835850587

node138.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30347064 kB
Buffers:        242792 kB
Cached:        1613272 kB
SwapCached:      13436 kB
Active:        1133180 kB
Inactive:      1270908 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30347064 kB
SwapTotal:    67111528 kB
SwapFree:     67051744 kB
Dirty:           81784 kB
Writeback:           0 kB
AnonPages:      545288 kB
Mapped:          16720 kB
Slab:           129924 kB
PageTables:       9524 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  3196896 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73948 MiB
End job on node138 at 2011-06-11 01:23:47