Trace number 3740903

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 501.16 860.778

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-153.opb
MD5SUM64b2e27e0cc69ae36d391a38c497bb88
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables55296
Total number of constraints1282690
Number of constraints which are clauses1152492
Number of constraints which are cardinality constraints (but not clauses)4030
Number of constraints which are nor clauses,nor cardinality constraints126168
Minimum length of a constraint2
Maximum length of a constraint756
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 757
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.41/1.29	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.41/1.29	c INFO: solving HOME/instance-3740903-1338720190.opb
9.47/10.36	c INFO: parsing took 8.89 s
24.76/25.69	c INFO: features took 15.29 CPU seconds
24.86/25.72	c INFO: running wbo1.4b-fixed@0 for 45 with 1775 remaining (b = 0.96)
72.31/73.20	c INFO: running pbct-0.1.2-linear@0 for 45 with 1728 remaining (b = 0.96)
102.52/278.26	c INFO: running wbo1.4a@0 for 45 with 1704 remaining (b = 0.95)
152.00/327.66	c INFO: running scip-2.0.1-clp@0 for 45 with 1655 remaining (b = 0.83)
199.67/375.27	c INFO: running scip-2.0.1-spx@0 for 45 with 1609 remaining (b = 0.68)
329.55/505.06	c INFO: wrote minion input file to HOME/tmpZyTAl5.minion
329.55/505.06	c INFO: running minion-0.12@0 for 45 with 1560 remaining (b = 0.55)
357.58/717.62	c INFO: running bsolo_pb10-l2@0 for 45 with 1459 remaining (b = 0.29)
404.95/765.00	c INFO: running clasp-1.3.7@0 for 45 with 1412 remaining (b = 0.17)
452.20/812.18	c INFO: running clasp-1.3.7@47 for 45 with 1365 remaining (b = 0.02)
500.48/860.34	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1317 remaining (b = 0.05)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3740903-1338720190/watcher-3740903-1338720190 -o /tmp/evaluation-result-3740903-1338720190/solver-3740903-1338720190 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740903-1338720190.opb -seed 1509199628 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.00 2.02 2.00 3/200 6299
/proc/meminfo: memFree=29568628/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/6299/stat : 6299 (python) D 6297 6299 2614 0 -1 4202496 634 0 0 0 0 0 0 0 20 0 1 0 338144169 15642624 527 33554432000 4194304 5683644 140737167963712 140737167949624 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 3819 527 312 364 0 261 0

[startup+0.097031 s]
/proc/loadavg: 2.00 2.02 2.00 3/200 6299
/proc/meminfo: memFree=29568628/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=19088 CPUtime=0.01 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 1375 0 0 0 1 0 0 0 20 0 1 0 338144169 19546112 1129 33554432000 4194304 5683644 140737167963712 140737167894216 4540219 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 4772 1129 430 364 0 696 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 19088

[startup+0.100225 s]
/proc/loadavg: 2.00 2.02 2.00 3/200 6299
/proc/meminfo: memFree=29568628/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=19088 CPUtime=0.01 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 1393 0 0 0 1 0 0 0 20 0 1 0 338144169 19546112 1147 33554432000 4194304 5683644 140737167963712 140737167894216 240564801428 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 4772 1147 430 364 0 696 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 19088

[startup+0.300238 s]
/proc/loadavg: 2.00 2.02 2.00 3/200 6299
/proc/meminfo: memFree=29568628/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=96896 CPUtime=0.11 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 3912 1006 1 0 9 2 0 0 20 0 1 0 338144169 99221504 2647 33554432000 4194304 5683644 140737167963712 140737167863704 240565175184 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 24224 2647 822 364 0 1715 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 96896

[startup+0.700183 s]
/proc/loadavg: 2.00 2.02 2.00 3/200 6299
/proc/meminfo: memFree=29568628/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=122348 CPUtime=0.22 cores=1,3,5,7
/proc/6299/stat : 6299 (python) D 6297 6299 2614 0 -1 4202496 5985 1006 1 0 16 6 0 0 20 0 1 0 338144169 125284352 3855 33554432000 4194304 5683644 140737167963712 140737167913304 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 30587 3855 1015 364 0 2830 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 122348

[startup+1.50024 s]
/proc/loadavg: 2.00 2.02 2.00 2/201 6304
/proc/meminfo: memFree=29552516/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=344096 CPUtime=0.66 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 11881 1006 1 0 51 15 0 0 20 0 1 0 338144169 352354304 22711 33554432000 4194304 5683644 140737167963712 140737167956664 240564801428 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 86024 22711 1811 364 0 21253 0
Current children cumulated CPU time (s) 0.66
Current children cumulated vsize (KiB) 344096

[startup+3.10023 s]
/proc/loadavg: 2.00 2.02 2.00 3/201 6304
/proc/meminfo: memFree=29463484/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=399696 CPUtime=2.26 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 25449 1006 1 0 208 18 0 0 20 0 1 0 338144169 409288704 36279 33554432000 4194304 5683644 140737167963712 140737167956328 4952164 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 99924 36281 1811 364 0 35153 0
Current children cumulated CPU time (s) 2.26
Current children cumulated vsize (KiB) 399696
heavy processes:

[startup+6.3002 s]
/proc/loadavg: 2.08 2.03 2.01 4/201 6304
/proc/meminfo: memFree=29329128/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=511216 CPUtime=5.47 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 52013 1006 1 0 523 24 0 0 20 0 1 0 338144169 523485184 63865 33554432000 4194304 5683644 140737167963712 140737167956328 240564801428 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 127804 63865 1811 364 0 63033 0
Current children cumulated CPU time (s) 5.47
Current children cumulated vsize (KiB) 511216

[startup+12.7072 s]
/proc/loadavg: 2.16 2.05 2.01 3/201 6304
/proc/meminfo: memFree=29201436/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=629308 CPUtime=11.87 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 77134 1006 1 0 1153 34 0 0 20 0 1 0 338144169 644411392 94078 33554432000 4194304 5683644 140737167963712 140737167950536 240572720933 0 0 16781312 33554434 0 0 0 17 3 0 0 0 0 0
/proc/6299/statm: 157327 94078 1846 364 0 92556 0
Current children cumulated CPU time (s) 11.87
Current children cumulated vsize (KiB) 629308

[startup+25.5002 s]
/proc/loadavg: 2.12 2.05 2.01 3/201 6304
/proc/meminfo: memFree=29192252/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=639328 CPUtime=24.66 cores=1,3,5,7
/proc/6299/stat : 6299 (python) R 6297 6299 2614 0 -1 4202496 77596 1006 1 0 2432 34 0 0 20 0 1 0 338144169 654671872 96584 33554432000 4194304 5683644 140737167963712 140737167956888 4324495 0 0 16781312 33554434 0 0 0 17 3 0 0 0 0 0
/proc/6299/statm: 159832 96584 1847 364 0 95061 0
Current children cumulated CPU time (s) 24.66
Current children cumulated vsize (KiB) 639328

[startup+51.1004 s]
/proc/loadavg: 2.08 2.04 2.01 3/204 6307
/proc/meminfo: memFree=29116212/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=648724 CPUtime=24.88 cores=1,3,5,7
/proc/6299/stat : 6299 (python) S 6297 6299 2614 0 -1 4202496 79268 1006 1 0 2453 35 0 0 20 0 2 0 338144169 664293376 80043 33554432000 4194304 5683644 140737167963712 140737167958544 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 162181 80043 1902 364 0 97404 0
[pid=6305] ppid=6299 vsize=572944 CPUtime=0.04 cores=1,3,5,7
/proc/6305/stat : 6305 (python) S 6299 6299 2614 0 -1 4202560 878 0 0 0 2 2 0 0 20 0 1 0 338146742 586694656 78498 33554432000 4194304 5683644 140737167963712 140737167954800 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/6305/statm: 143236 78498 363 364 0 78459 0
[pid=6307] ppid=6305 vsize=145108 CPUtime=25.35 cores=1,3,5,7
/proc/6307/stat : 6307 (wbo1.4b-fixed) R 6305 6307 6307 0 -1 4202496 49798 0 0 0 2480 55 0 0 20 0 1 0 338146743 148590592 34718 33554432000 134512640 135752315 4291488112 4291486104 134516982 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/6307/statm: 36277 34718 146 303 0 35972 0
[pid=6299/tid=6306] ppid=6297 vsize=648724 CPUtime=0 cores=1,3,5,7
/proc/6299/task/6306/stat : 6306 (python) S 6297 6299 2614 0 -1 4202560 26 1006 0 0 0 0 0 0 20 0 2 0 338146743 664293376 80043 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.27
Current children cumulated vsize (KiB) 1366776

[startup+102.301 s]
/proc/loadavg: 2.03 2.03 2.01 4/210 6313
/proc/meminfo: memFree=23871796/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=724516 CPUtime=24.91 cores=1,3,5,7
/proc/6299/stat : 6299 (python) S 6297 6299 2614 0 -1 4202496 79969 1006 1 0 2455 36 0 0 20 0 3 0 338144169 741904384 80052 33554432000 4194304 5683644 140737167963712 140737167958544 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 181129 80052 1905 364 0 116349 0
[pid=6305] ppid=6299 vsize=648724 CPUtime=0.08 cores=1,3,5,7
/proc/6305/stat : 6305 (python) S 6299 6299 2614 0 -1 4202560 1110 0 0 0 4 4 0 0 20 0 2 0 338146742 664293376 78538 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/6305/statm: 162181 78538 396 364 0 97404 0
[pid=6307] ppid=6305 vsize=161984 CPUtime=47.42 cores=1,3,5,7
/proc/6307/stat : 6307 (wbo1.4b-fixed) T 6305 6307 6307 0 -1 4202496 53999 0 0 0 4663 79 0 0 20 0 1 0 338146743 165871616 38919 33554432000 134512640 135752315 4291488112 4291486112 134611137 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/6307/statm: 40496 38919 146 303 0 40191 0
[pid=6305/tid=6308] ppid=6299 vsize=648724 CPUtime=0 cores=1,3,5,7
/proc/6305/task/6308/stat : 6308 (python) S 6299 6299 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338151488 664293376 78538 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6309] ppid=6299 vsize=648736 CPUtime=0.05 cores=1,3,5,7
/proc/6309/stat : 6309 (python) S 6299 6299 2614 0 -1 4202560 889 0 0 0 3 2 0 0 20 0 1 0 338151490 664305664 78499 33554432000 4194304 5683644 140737167963712 140737167954800 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/6309/statm: 162184 78499 358 364 0 97404 0
[pid=6311] ppid=6309 vsize=4189072 CPUtime=27.31 cores=1,3,5,7
/proc/6311/stat : 6311 (pbct-0.1.2-linu) R 6309 6311 6311 0 -1 4204032 978699 0 0 0 2477 254 0 0 20 0 1 0 338151491 4289609728 1018782 33554432000 134512640 136422188 4291694672 4291693148 135736970 0 0 16781312 16387 0 0 0 17 3 0 0 0 0 0
/proc/6311/statm: 1047268 1018782 239 467 0 1046799 0
[pid=6299/tid=6306] ppid=6297 vsize=724516 CPUtime=0 cores=1,3,5,7
/proc/6299/task/6306/stat : 6306 (python) S 6297 6299 2614 0 -1 4202560 26 1006 0 0 0 0 0 0 20 0 3 0 338146743 741904384 80052 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6299/tid=6310] ppid=6297 vsize=724516 CPUtime=0 cores=1,3,5,7
/proc/6299/task/6310/stat : 6310 (python) S 6297 6299 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 3 0 338151490 741904384 80052 33554432000 4194304 5683644 140737167963712 140566605267504 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 99.77
Current children cumulated vsize (KiB) 6373032

[startup+162.307 s]
/proc/loadavg: 3.99 2.58 2.19 2/209 6313
/proc/meminfo: memFree=16414912/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=724516 CPUtime=24.91 cores=1,3,5,7
/proc/6299/stat : 6299 (python) S 6297 6299 2614 0 -1 4202496 79969 1006 1 0 2455 36 0 0 20 0 3 0 338144169 741904384 80052 33554432000 4194304 5683644 140737167963712 140737167958544 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 181129 80052 1905 364 0 116349 0
[pid=6305] ppid=6299 vsize=648724 CPUtime=0.08 cores=1,3,5,7
/proc/6305/stat : 6305 (python) S 6299 6299 2614 0 -1 4202560 1110 0 0 0 4 4 0 0 20 0 2 0 338146742 664293376 78538 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/6305/statm: 162181 78538 396 364 0 97404 0
[pid=6307] ppid=6305 vsize=161984 CPUtime=47.42 cores=1,3,5,7
/proc/6307/stat : 6307 (wbo1.4b-fixed) T 6305 6307 6307 0 -1 4202496 53999 0 0 0 4663 79 0 0 20 0 1 0 338146743 165871616 38919 33554432000 134512640 135752315 4291488112 4291486112 134611137 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/6307/statm: 40496 38919 146 303 0 40191 0
[pid=6305/tid=6308] ppid=6299 vsize=648724 CPUtime=0 cores=1,3,5,7
/proc/6305/task/6308/stat : 6308 (python) S 6299 6299 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338151488 664293376 78538 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6309] ppid=6299 vsize=648736 CPUtime=0.15 cores=1,3,5,7
/proc/6309/stat : 6309 (python) S 6299 6299 2614 0 -1 4202560 920 0 0 0 8 7 0 0 20 0 1 0 338151490 664305664 78499 33554432000 4194304 5683644 140737167963712 140737167954800 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/6309/statm: 162184 78499 358 364 0 97404 0
[pid=6311] ppid=6309 vsize=4189072 CPUtime=29.57 cores=1,3,5,7
/proc/6311/stat : 6311 (pbct-0.1.2-linu) S 6309 6311 6311 0 -1 4204032 978699 0 0 0 2477 480 0 0 20 0 1 0 338151491 4289609728 1018782 33554432000 134512640 136422188 4291694672 4291693148 135736970 0 0 16781312 16387 18446744071580380219 0 0 17 5 0 0 0 0 0
/proc/6311/statm: 1047268 1018782 239 467 0 1046799 0
[pid=6299/tid=6306] ppid=6297 vsize=724516 CPUtime=0 cores=1,3,5,7
/proc/6299/task/6306/stat : 6306 (python) S 6297 6299 2614 0 -1 4202560 26 1006 0 0 0 0 0 0 20 0 3 0 338146743 741904384 80052 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6299/tid=6310] ppid=6297 vsize=724516 CPUtime=0 cores=1,3,5,7
/proc/6299/task/6310/stat : 6310 (python) S 6297 6299 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 3 0 338151490 741904384 80052 33554432000 4194304 5683644 140737167963712 140566605267504 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 102.13
Current children cumulated vsize (KiB) 6373032
heavy processes:
  %CPU=23 pid=6312 uid=0 cmd=

[startup+222.3 s]
/proc/loadavg: 4.85 3.08 2.38 2/209 6314
/proc/meminfo: memFree=16494760/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=724516 CPUtime=24.91 cores=1,3,5,7
/proc/6299/stat : 6299 (python) S 6297 6299 2614 0 -1 4202496 79969 1006 1 0 2455 36 0 0 20 0 3 0 338144169 741904384 80052 33554432000 4194304 5683644 140737167963712 140737167958544 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6299/statm: 181129 80052 1905 364 0 116349 0
[pid=6305] ppid=6299 vsize=648724 CPUtime=0.08 cores=1,3,5,7
/proc/6305/stat : 6305 (python) S 6299 6299 2614 0 -1 4202560 1110 0 0 0 4 4 0 0 20 0 2 0 338146742 664293376 78538 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/6305/statm: 162181 78538 396 364 0 97404 0
[pid=6307] ppid=6305 vsize=161984 CPUtime=47.42 cores=1,3,5,7
/proc/6307/stat : 6307 (wbo1.4b-fixed) T 6305 6307 6307 0 -1 4202496 53999 0 0 0 4663 79 0 0 20 0 1 0 338146743 165871616 38919 33554432000 134512640 135752315 4291488112 4291486112 134611137 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/6307/statm: 40496 38919 146 303 0 40191 0
[pid=6305/tid=6308] ppid=6299 vsize=648724 CPUtime=0 cores=1,3,5,7
/proc/6305/task/6308/stat : 6308 (python) S 6299 6299 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338151488 664293376 78538 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6309] ppid=6299 vsize=648736 CPUtime=0.26 cores=1,3,5,7
/proc/6309/stat : 6309 (python) S 6299 6299 2614 0 -1 4202560 935 0 0 0 15 11 0 0 20 0 1 0 338151490 664305664 78499 33554432000 4194304 5683644 140737167963712 140737167954800 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/6309/statm: 162184 78499 358 364 0 97404 0

################
# More data... #
################

[pid=6422/tid=6433] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6433/stat : 6433 (java) S 6420 6422 6422 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338230209 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181745542136 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6434] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6434/stat : 6434 (java) S 6420 6422 6422 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338230209 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181744489736 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6435] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6435/stat : 6435 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181743438224 240568555296 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6436] ppid=6420 vsize=9558408 CPUtime=0.13 cores=1,3,5,7
/proc/6422/task/6436/stat : 6436 (java) S 6420 6422 6422 0 -1 4202560 1320 0 0 0 13 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181742385848 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6437] ppid=6420 vsize=9558408 CPUtime=0.13 cores=1,3,5,7
/proc/6422/task/6437/stat : 6437 (java) S 6420 6422 6422 0 -1 4202560 1955 0 0 0 13 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181741333048 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6422/tid=6438] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6438/stat : 6438 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181740280952 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6422/tid=6439] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6439/stat : 6439 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181739228360 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6299/tid=6306] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6306/stat : 6306 (python) S 6297 6299 2614 0 -1 4202560 26 1361451 0 0 0 0 4798 1017 20 0 10 0 338146743 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6299/tid=6310] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6310/stat : 6310 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338151490 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566605267504 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6299/tid=6318] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6318/stat : 6318 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338171997 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566594777648 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6299/tid=6322] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6322/stat : 6322 (python) S 6297 6299 2614 0 -1 4202560 22 1361451 0 0 0 0 4798 1017 20 0 10 0 338176937 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566584287792 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=6299/tid=6326] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6326/stat : 6326 (python) S 6297 6299 2614 0 -1 4202560 20 1361451 0 0 0 0 4798 1017 20 0 10 0 338181698 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566573797936 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6299/tid=6334] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6334/stat : 6334 (python) S 6297 6299 2614 0 -1 4202560 19 1361451 0 0 0 0 4798 1017 20 0 10 0 338194676 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566494962224 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=6299/tid=6341] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6341/stat : 6341 (python) S 6297 6299 2614 0 -1 4202560 22 1361451 0 0 0 0 4798 1017 20 0 10 0 338215932 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566484472368 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6299/tid=6345] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6345/stat : 6345 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338220670 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566450299440 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6299/tid=6421] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6421/stat : 6421 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338230204 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566439797296 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 501.16
Current children cumulated vsize (KiB) 22581420
heavy processes:
  %CPU=96 pid=6418 uid=0 cmd=/usr/libexec/abrt-hook-ccpp /var/spool/abrt 6417 11 506 18446744073709551615 

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+860.701 s]
/proc/loadavg: 2.36 3.16 2.79 4/228 6439
/proc/meminfo: memFree=10988472/32873844 swapFree=7116/7116
[pid=6299] ppid=6297 vsize=1255060 CPUtime=161.62 cores=1,3,5,7
/proc/6299/stat : 6299 (python) S 6297 6299 2614 0 -1 4202496 84850 1361451 1 0 10283 64 4798 1017 20 0 10 0 338144169 1285181440 80616 33554432000 4194304 5683644 140737167963712 140737167958544 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/6299/statm: 313765 80616 1919 364 0 248964 0
[pid=6305] ppid=6299 vsize=648724 CPUtime=0.08 cores=1,3,5,7
/proc/6305/stat : 6305 (python) S 6299 6299 2614 0 -1 4202560 1110 0 0 0 4 4 0 0 20 0 2 0 338146742 664293376 78538 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/6305/statm: 162181 78538 396 364 0 97404 0
[pid=6307] ppid=6305 vsize=161984 CPUtime=47.42 cores=1,3,5,7
/proc/6307/stat : 6307 (wbo1.4b-fixed) T 6305 6307 6307 0 -1 4202496 53999 0 0 0 4663 79 0 0 20 0 1 0 338146743 165871616 38919 33554432000 134512640 135752315 4291488112 4291486112 134611137 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/6307/statm: 40496 38919 146 303 0 40191 0
[pid=6305/tid=6308] ppid=6299 vsize=648724 CPUtime=0 cores=1,3,5,7
/proc/6305/task/6308/stat : 6308 (python) S 6299 6299 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338151488 664293376 78538 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6317] ppid=6299 vsize=724528 CPUtime=0.09 cores=1,3,5,7
/proc/6317/stat : 6317 (python) S 6299 6299 2614 0 -1 4202560 1069 0 0 0 4 5 0 0 20 0 2 0 338171996 741916672 78531 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6317/statm: 181132 78531 383 364 0 116349 0
[pid=6319] ppid=6317 vsize=132716 CPUtime=49.35 cores=1,3,5,7
/proc/6319/stat : 6319 (wbo1.4a) T 6317 6319 6319 0 -1 4202496 46756 0 0 0 4859 76 0 0 20 0 1 0 338171997 135901184 31672 33554432000 134512640 135751879 4293412880 4293411056 134616334 0 0 16781312 3 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/6319/statm: 33179 31672 143 303 0 32874 0
[pid=6317/tid=6320] ppid=6299 vsize=724528 CPUtime=0 cores=1,3,5,7
/proc/6317/task/6320/stat : 6320 (python) S 6299 6299 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338176934 741916672 78531 33554432000 4194304 5683644 140737167963712 140566605267504 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6321] ppid=6299 vsize=800320 CPUtime=0.1 cores=1,3,5,7
/proc/6321/stat : 6321 (python) S 6299 6299 2614 0 -1 4202560 1067 0 0 0 5 5 0 0 20 0 2 0 338176936 819527680 78537 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6321/statm: 200080 78537 383 364 0 135294 0
[pid=6323] ppid=6321 vsize=1907028 CPUtime=47.56 cores=1,3,5,7
/proc/6323/stat : 6323 (scip-2.0.1.linu) T 6321 6323 6323 0 -1 4202496 465441 0 0 0 4577 179 0 0 20 0 1 0 338176937 1952796672 466263 33554432000 4194304 15243407 140733530596400 140733530592408 13277498 0 0 16781312 2 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/6323/statm: 476757 466263 703 2698 0 474052 0
[pid=6321/tid=6324] ppid=6299 vsize=800320 CPUtime=0 cores=1,3,5,7
/proc/6321/task/6324/stat : 6324 (python) S 6299 6299 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338181695 819527680 78537 33554432000 4194304 5683644 140737167963712 140566594777648 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6325] ppid=6299 vsize=876112 CPUtime=0.1 cores=1,3,5,7
/proc/6325/stat : 6325 (python) S 6299 6299 2614 0 -1 4202560 1091 0 0 0 5 5 0 0 20 0 2 0 338181697 897138688 78543 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6325/statm: 219028 78543 383 364 0 154239 0
[pid=6327] ppid=6325 vsize=2771168 CPUtime=51.37 cores=1,3,5,7
/proc/6327/stat : 6327 (scip-2.0.1.linu) T 6325 6327 6327 0 -1 4202496 540481 0 0 0 4924 213 0 0 20 0 1 0 338181698 2837676032 652438 33554432000 4194304 13684479 140734244033680 140734244029992 6046187 0 0 16781312 2 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/6327/statm: 692792 652438 703 2317 0 690470 0
[pid=6325/tid=6331] ppid=6299 vsize=876112 CPUtime=0 cores=1,3,5,7
/proc/6325/task/6331/stat : 6331 (python) S 6299 6299 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338186838 897138688 78543 33554432000 4194304 5683644 140737167963712 140566584287792 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6340] ppid=6299 vsize=1027696 CPUtime=0.09 cores=1,3,5,7
/proc/6340/stat : 6340 (python) S 6299 6299 2614 0 -1 4202560 1091 0 0 0 5 4 0 0 20 0 2 0 338215932 1052360704 79062 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6340/statm: 256924 79062 383 364 0 192129 0
[pid=6342] ppid=6340 vsize=320692 CPUtime=47.33 cores=1,3,5,7
/proc/6342/stat : 6342 (bsolo_pb10) T 6340 6342 6342 0 -1 4202496 92289 0 0 0 4700 33 0 0 20 0 1 0 338215933 328388608 77493 33554432000 134512640 137138936 4289904560 4289239472 134540164 0 0 16781312 16384 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/6342/statm: 80173 77493 184 642 0 79529 0
[pid=6340/tid=6343] ppid=6299 vsize=1027696 CPUtime=0 cores=1,3,5,7
/proc/6340/task/6343/stat : 6343 (python) S 6299 6299 2614 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 338220667 1052360704 79062 33554432000 4194304 5683644 140737167963712 140566494962224 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6344] ppid=6299 vsize=1103488 CPUtime=0.17 cores=1,3,5,7
/proc/6344/stat : 6344 (python) S 6299 6299 2614 0 -1 4202560 1185 0 0 0 10 7 0 0 20 0 2 0 338220670 1129971712 79075 33554432000 4194304 5683644 140737167963712 140737167955360 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/6344/statm: 275872 79075 389 364 0 211074 0
[pid=6346] ppid=6344 vsize=114216 CPUtime=95.25 cores=1,3,5,7
/proc/6346/stat : 6346 (clasp-1.3.7-x86) T 6344 6346 6346 0 -1 4202496 34025 0 0 0 9516 9 0 0 20 0 1 0 338220670 116957184 28023 33554432000 134512640 136139019 4289133408 4289132384 134883787 0 0 16781312 16386 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/6346/statm: 28554 28023 183 398 0 28154 0
[pid=6344/tid=6414] ppid=6299 vsize=1103488 CPUtime=0 cores=1,3,5,7
/proc/6344/task/6414/stat : 6414 (python) S 6299 6299 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338225385 1129971712 79075 33554432000 4194304 5683644 140737167963712 140566484472368 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6420] ppid=6299 vsize=1179280 CPUtime=0 cores=1,3,5,7
/proc/6420/stat : 6420 (python) S 6299 6299 2614 0 -1 4202560 831 0 0 0 0 0 0 0 20 0 1 0 338230204 1207582720 79033 33554432000 4194304 5683644 140737167963712 140737167954800 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 7 0 0 0 0 0
/proc/6420/statm: 294820 79033 342 364 0 230019 0
[pid=6422] ppid=6420 vsize=9558408 CPUtime=0.63 cores=1,3,5,7
/proc/6422/stat : 6422 (java) S 6420 6422 6422 0 -1 4202496 7119 0 1 0 60 3 0 0 20 0 18 0 338230204 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140737216331232 240568533053 0 0 0 16800975 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/6422/statm: 2389602 11060 2224 9 0 2380296 0
[pid=6422/tid=6423] ppid=6420 vsize=9558408 CPUtime=0.32 cores=1,3,5,7
/proc/6422/task/6423/stat : 6423 (java) R 6420 6422 6422 0 -1 4202560 2431 0 1 0 30 2 0 0 20 0 18 0 338230206 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140182258491264 140182112247471 0 4 0 16800975 0 0 0 -1 1 0 0 0 0 0
[pid=6422/tid=6424] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6424/stat : 6424 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140182230952232 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6425] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6425/stat : 6425 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140182101207208 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6422/tid=6426] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6426/stat : 6426 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140182100154920 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6422/tid=6427] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6427/stat : 6427 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140182099102120 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=6422/tid=6428] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6428/stat : 6428 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140182098049832 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6429] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6429/stat : 6429 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140182096997032 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=6422/tid=6430] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6430/stat : 6430 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181759847464 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6431] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6431/stat : 6431 (java) S 6420 6422 6422 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338230208 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181758794664 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6432] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6432/stat : 6432 (java) S 6420 6422 6422 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338230209 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181746596072 240568547241 0 0 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6433] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6433/stat : 6433 (java) S 6420 6422 6422 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338230209 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181745542136 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6434] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6434/stat : 6434 (java) S 6420 6422 6422 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338230209 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181744489736 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6435] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6435/stat : 6435 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181743438224 240568555296 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6436] ppid=6420 vsize=9558408 CPUtime=0.13 cores=1,3,5,7
/proc/6422/task/6436/stat : 6436 (java) S 6420 6422 6422 0 -1 4202560 1320 0 0 0 13 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181742385848 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6422/tid=6437] ppid=6420 vsize=9558408 CPUtime=0.13 cores=1,3,5,7
/proc/6422/task/6437/stat : 6437 (java) S 6420 6422 6422 0 -1 4202560 1955 0 0 0 13 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181741333048 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6422/tid=6438] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6438/stat : 6438 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181740280952 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6422/tid=6439] ppid=6420 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/6422/task/6439/stat : 6439 (java) S 6420 6422 6422 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338230212 9787809792 11060 33554432000 1073741824 1073778376 140737216340096 140181739228360 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6299/tid=6306] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6306/stat : 6306 (python) S 6297 6299 2614 0 -1 4202560 26 1361451 0 0 0 0 4798 1017 20 0 10 0 338146743 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566615757360 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6299/tid=6310] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6310/stat : 6310 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338151490 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566605267504 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=6299/tid=6318] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6318/stat : 6318 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338171997 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566594777648 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6299/tid=6322] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6322/stat : 6322 (python) S 6297 6299 2614 0 -1 4202560 22 1361451 0 0 0 0 4798 1017 20 0 10 0 338176937 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566584287792 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=6299/tid=6326] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6326/stat : 6326 (python) S 6297 6299 2614 0 -1 4202560 20 1361451 0 0 0 0 4798 1017 20 0 10 0 338181698 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566573797936 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=6299/tid=6334] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6334/stat : 6334 (python) S 6297 6299 2614 0 -1 4202560 19 1361451 0 0 0 0 4798 1017 20 0 10 0 338194676 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566494962224 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=6299/tid=6341] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6341/stat : 6341 (python) S 6297 6299 2614 0 -1 4202560 22 1361451 0 0 0 0 4798 1017 20 0 10 0 338215932 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566484472368 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6299/tid=6345] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6345/stat : 6345 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338220670 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566450299440 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=6299/tid=6421] ppid=6297 vsize=1255060 CPUtime=58.15 cores=1,3,5,7
/proc/6299/task/6421/stat : 6421 (python) S 6297 6299 2614 0 -1 4202560 21 1361451 0 0 0 0 4798 1017 20 0 10 0 338230204 1285181440 80616 33554432000 4194304 5683644 140737167963712 140566439797296 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 501.16
Current children cumulated vsize (KiB) 22581420

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 6299 and gives
#  childrusage.ru_utime.tv_sec=150
#  childrusage.ru_utime.tv_usec=825071
#  childrusage.ru_stime.tv_sec=10
#  childrusage.ru_stime.tv_usec=830353
# CPU time returned by wait4() is 161.655
# while last known CPU time is 501.16
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 860.778
CPU time (s): 501.16
CPU user time (s): 484.13
CPU system time (s): 17.03
CPU usage (%): 58.2217
Max. virtual memory (cumulated for all children) (KiB): 22581420

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 150.825
system time used= 10.8304
maximum resident set size= 4136852
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1446301
page faults= 1
swaps= 0
block input operations= 0
block output operations= 168664
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1992570
involuntary context switches= 304

runsolver used 3.41148 second user time and 7.2349 second system time

The end

Launcher Data 

Begin job on node146 at 2012-06-03 12:43:11
IDJOB=3740903
IDBENCH=89671
IDSOLVER=2327
FILE ID=node146/3740903-1338720190
RUNJOBID= node146-1338713700-2631
PBS_JOBID= 14636739
Free space on /tmp= 70748 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-153.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3740903-1338720190/watcher-3740903-1338720190 -o /tmp/evaluation-result-3740903-1338720190/solver-3740903-1338720190 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740903-1338720190.opb -seed 1509199628 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 64b2e27e0cc69ae36d391a38c497bb88
RANDOM SEED=1509199628

node146.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        29568636 kB
Buffers:          231336 kB
Cached:           669560 kB
SwapCached:         1792 kB
Active:          2214512 kB
Inactive:         295688 kB
Active(anon):    1610272 kB
Inactive(anon):     1748 kB
Active(file):     604240 kB
Inactive(file):   293940 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67099888 kB
Dirty:            158836 kB
Writeback:             0 kB
AnonPages:       1615732 kB
Mapped:            24436 kB
Shmem:                84 kB
Slab:             644784 kB
SReclaimable:      83004 kB
SUnreclaim:       561780 kB
KernelStack:        1640 kB
PageTables:        11608 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    2228504 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:     38912 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 61264 MiB
End job on node146 at 2012-06-03 12:57:35