Trace number 3740864

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 488.5 730.373

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-137.opb
MD5SUM395b6779bc0f993a1d1357b59c2e4536
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables48411
Total number of constraints1121517
Number of constraints which are clauses1007723
Number of constraints which are cardinality constraints (but not clauses)3523
Number of constraints which are nor clauses,nor cardinality constraints110271
Minimum length of a constraint2
Maximum length of a constraint690
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 691
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.41/1.30	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.41/1.30	c INFO: solving HOME/instance-3740864-1338720039.opb
8.51/9.41	c INFO: parsing took 7.95 s
21.61/22.59	c INFO: features took 13.15 CPU seconds
21.71/22.62	c INFO: running wbo1.4b-fixed@0 for 45 with 1778 remaining (b = 0.96)
70.57/71.46	c INFO: running pbct-0.1.2-linear@0 for 45 with 1730 remaining (b = 0.96)
104.79/225.18	c INFO: running wbo1.4a@0 for 45 with 1701 remaining (b = 0.95)
154.28/274.52	c INFO: running scip-2.0.1-clp@0 for 45 with 1653 remaining (b = 0.83)
203.04/323.26	c INFO: running scip-2.0.1-spx@0 for 45 with 1606 remaining (b = 0.68)
319.93/440.15	c INFO: wrote minion input file to HOME/tmp8dNbGE.minion
319.93/440.15	c INFO: running minion-0.12@0 for 45 with 1560 remaining (b = 0.55)
343.90/586.47	c INFO: running bsolo_pb10-l2@0 for 45 with 1471 remaining (b = 0.29)
391.76/634.27	c INFO: running clasp-1.3.7@0 for 45 with 1424 remaining (b = 0.17)
439.04/681.45	c INFO: running clasp-1.3.7@47 for 45 with 1377 remaining (b = 0.02)
487.30/729.60	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1329 remaining (b = 0.05)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3740864-1338720039/watcher-3740864-1338720039 -o /tmp/evaluation-result-3740864-1338720039/solver-3740864-1338720039 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740864-1338720039.opb -seed 184419883 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.94 1.99 1.99 3/201 5503
/proc/meminfo: memFree=25306268/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 672 0 0 0 0 0 0 0 20 0 1 0 338129041 15642624 559 33554432000 4194304 5683644 140734606903136 140734606871080 213419639696 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/5503/statm: 3819 559 324 364 0 261 0

[startup+0.0646971 s]
/proc/loadavg: 1.94 1.99 1.99 3/201 5503
/proc/meminfo: memFree=25306268/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=16596 CPUtime=0.01 cores=1,3,5,7
/proc/5503/stat : 5503 (python) D 5501 5503 3621 0 -1 4202496 1202 0 0 0 1 0 0 0 20 0 1 0 338129041 16994304 1001 33554432000 4194304 5683644 140734606903136 140734606885144 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/5503/statm: 4149 1001 422 364 0 591 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16596

[startup+0.100257 s]
/proc/loadavg: 1.94 1.99 1.99 3/201 5503
/proc/meminfo: memFree=25306268/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=19088 CPUtime=0.02 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 1398 0 0 0 2 0 0 0 20 0 1 0 338129041 19546112 1152 33554432000 4194304 5683644 140734606903136 140734606833640 4509097 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/5503/statm: 4772 1152 430 364 0 696 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19088

[startup+0.30023 s]
/proc/loadavg: 1.94 1.99 1.99 3/201 5503
/proc/meminfo: memFree=25306268/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=94456 CPUtime=0.07 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 3447 1008 1 0 5 2 0 0 20 0 1 0 338129041 96722944 2546 33554432000 4194304 5683644 140734606903136 140734606855608 5172114 0 0 16781312 2 0 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 23614 2546 820 364 0 1620 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 94456

[startup+0.700209 s]
/proc/loadavg: 1.94 1.99 1.99 3/201 5503
/proc/meminfo: memFree=25306268/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=122216 CPUtime=0.21 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 5952 1008 1 0 15 6 0 0 20 0 1 0 338129041 125149184 3837 33554432000 4194304 5683644 140734606903136 140734606844840 213419307725 0 0 16781312 2 0 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 30554 3837 1015 364 0 2797 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 122216

[startup+1.50021 s]
/proc/loadavg: 1.94 1.99 1.99 3/202 5508
/proc/meminfo: memFree=25292504/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=336124 CPUtime=0.61 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 11923 1008 1 0 48 13 0 0 20 0 1 0 338129041 344190976 20708 33554432000 4194304 5683644 140734606903136 140734606896664 4445115 0 0 16781312 2 0 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 84031 20708 1811 364 0 19260 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 336124

[startup+3.10018 s]
/proc/loadavg: 1.94 1.99 1.99 3/202 5508
/proc/meminfo: memFree=25212292/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=390236 CPUtime=2.21 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 25115 1008 1 0 206 15 0 0 20 0 1 0 338129041 399601664 33900 33554432000 4194304 5683644 140734606903136 140734606895752 213419265940 0 0 16781312 2 0 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 97559 33902 1811 364 0 32788 0
Current children cumulated CPU time (s) 2.21
Current children cumulated vsize (KiB) 390236
heavy processes:

[startup+6.30018 s]
/proc/loadavg: 1.94 1.99 1.99 3/202 5508
/proc/meminfo: memFree=25101072/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=501048 CPUtime=5.41 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 51506 1008 1 0 521 20 0 0 20 0 1 0 338129041 513073152 60802 33554432000 4194304 5683644 140734606903136 140734606895752 213419265940 0 0 16781312 2 0 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 125262 60802 1811 364 0 60491 0
Current children cumulated CPU time (s) 5.41
Current children cumulated vsize (KiB) 501048

[startup+12.7002 s]
/proc/loadavg: 1.95 1.99 1.99 3/202 5508
/proc/meminfo: memFree=24983620/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=585400 CPUtime=11.81 cores=1,3,5,7
/proc/5503/stat : 5503 (python) R 5501 5503 3621 0 -1 4202496 71772 1008 1 0 1152 29 0 0 20 0 1 0 338129041 599449600 83101 33554432000 4194304 5683644 140734606903136 140734606889960 140088687198509 0 0 16781312 33554434 0 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 146350 83101 1846 364 0 81579 0
Current children cumulated CPU time (s) 11.81
Current children cumulated vsize (KiB) 585400

[startup+25.5003 s]
/proc/loadavg: 1.96 1.99 1.99 3/204 5511
/proc/meminfo: memFree=24968092/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=611912 CPUtime=21.74 cores=1,3,5,7
/proc/5503/stat : 5503 (python) S 5501 5503 3621 0 -1 4202496 73667 1008 1 0 2143 31 0 0 20 0 2 0 338129041 626597888 70840 33554432000 4194304 5683644 140734606903136 140734606897968 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 152978 70840 1902 364 0 88201 0
[pid=5509] ppid=5503 vsize=536132 CPUtime=0 cores=1,3,5,7
/proc/5509/stat : 5509 (python) S 5503 5503 3621 0 -1 4202560 740 0 0 0 0 0 0 0 20 0 1 0 338131303 548999168 69272 33554432000 4194304 5683644 140734606903136 140734606894224 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/5509/statm: 134033 69272 340 364 0 69256 0
[pid=5511] ppid=5509 vsize=121412 CPUtime=2.85 cores=1,3,5,7
/proc/5511/stat : 5511 (wbo1.4b-fixed) R 5509 5511 5511 0 -1 4202496 27522 0 0 0 282 3 0 0 20 0 1 0 338131304 124325888 27242 33554432000 134512640 135752315 4290369904 4289319296 134564651 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/5511/statm: 30353 27242 10169 303 0 17624 0
[pid=5503/tid=5510] ppid=5501 vsize=611912 CPUtime=0 cores=1,3,5,7
/proc/5503/task/5510/stat : 5510 (python) S 5501 5503 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338131304 626597888 70840 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.59
Current children cumulated vsize (KiB) 1269456

[startup+51.1003 s]
/proc/loadavg: 1.97 1.99 1.99 3/204 5511
/proc/meminfo: memFree=24898400/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=611912 CPUtime=21.74 cores=1,3,5,7
/proc/5503/stat : 5503 (python) S 5501 5503 3621 0 -1 4202496 73667 1008 1 0 2143 31 0 0 20 0 2 0 338129041 626597888 70840 33554432000 4194304 5683644 140734606903136 140734606897968 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 152978 70840 1902 364 0 88201 0
[pid=5509] ppid=5503 vsize=536132 CPUtime=0.05 cores=1,3,5,7
/proc/5509/stat : 5509 (python) S 5503 5503 3621 0 -1 4202560 995 0 0 0 3 2 0 0 20 0 1 0 338131303 548999168 69300 33554432000 4194304 5683644 140734606903136 140734606894224 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/5509/statm: 134033 69300 368 364 0 69256 0
[pid=5511] ppid=5509 vsize=141036 CPUtime=28.45 cores=1,3,5,7
/proc/5511/stat : 5511 (wbo1.4b-fixed) R 5509 5511 5511 0 -1 4202496 46282 0 0 0 2791 54 0 0 20 0 1 0 338131304 144420864 33082 33554432000 134512640 135752315 4290369904 4290367904 134611730 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/5511/statm: 35259 33082 146 303 0 34954 0
[pid=5503/tid=5510] ppid=5501 vsize=611912 CPUtime=0 cores=1,3,5,7
/proc/5503/task/5510/stat : 5510 (python) S 5501 5503 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338131304 626597888 70840 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.24
Current children cumulated vsize (KiB) 1289080

[startup+102.308 s]
/proc/loadavg: 2.04 2.01 2.00 3/208 5516
/proc/meminfo: memFree=20915472/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=687704 CPUtime=21.76 cores=1,3,5,7
/proc/5503/stat : 5503 (python) S 5501 5503 3621 0 -1 4202496 74576 1008 1 0 2145 31 0 0 20 0 3 0 338129041 704208896 70849 33554432000 4194304 5683644 140734606903136 140734606897968 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 171926 70849 1905 364 0 107146 0
[pid=5509] ppid=5503 vsize=611912 CPUtime=0.08 cores=1,3,5,7
/proc/5509/stat : 5509 (python) S 5503 5503 3621 0 -1 4202560 1136 0 0 0 5 3 0 0 20 0 2 0 338131303 626597888 69335 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5509/statm: 152978 69335 396 364 0 88201 0
[pid=5511] ppid=5509 vsize=148948 CPUtime=48.79 cores=1,3,5,7
/proc/5511/stat : 5511 (wbo1.4b-fixed) T 5509 5511 5511 0 -1 4202496 48235 0 0 0 4773 106 0 0 20 0 1 0 338131304 152522752 35035 33554432000 134512640 135752315 4290369904 4290367904 134611369 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5511/statm: 37237 35035 146 303 0 36932 0
[pid=5509/tid=5512] ppid=5503 vsize=611912 CPUtime=0 cores=1,3,5,7
/proc/5509/task/5512/stat : 5512 (python) S 5503 5503 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338136185 626597888 69335 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5513] ppid=5503 vsize=611924 CPUtime=0.05 cores=1,3,5,7
/proc/5513/stat : 5513 (python) S 5503 5503 3621 0 -1 4202560 907 0 0 0 3 2 0 0 20 0 1 0 338136187 626610176 69296 33554432000 4194304 5683644 140734606903136 140734606894224 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/5513/statm: 152981 69296 358 364 0 88201 0
[pid=5515] ppid=5513 vsize=4158816 CPUtime=30.8 cores=1,3,5,7
/proc/5515/stat : 5515 (pbct-0.1.2-linu) S 5513 5515 5515 0 -1 4204032 925927 0 0 0 2904 176 0 0 20 0 1 0 338136188 4258627584 987650 33554432000 134512640 136422188 4291482912 4291480924 135736970 0 0 16781312 16387 18446744071580380219 0 0 17 3 0 0 0 0 0
/proc/5515/statm: 1039704 987650 249 467 0 1039235 0
[pid=5503/tid=5510] ppid=5501 vsize=687704 CPUtime=0 cores=1,3,5,7
/proc/5503/task/5510/stat : 5510 (python) S 5501 5503 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 3 0 338131304 704208896 70849 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5514] ppid=5501 vsize=687704 CPUtime=0 cores=1,3,5,7
/proc/5503/task/5514/stat : 5514 (python) S 5501 5503 3621 0 -1 4202560 21 1008 0 0 0 0 0 0 20 0 3 0 338136188 704208896 70849 33554432000 4194304 5683644 140734606903136 140088503444016 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.48
Current children cumulated vsize (KiB) 6219304

[startup+162.3 s]
/proc/loadavg: 3.97 2.55 2.19 2/213 5521
/proc/meminfo: memFree=12741936/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=687704 CPUtime=21.76 cores=1,3,5,7
/proc/5503/stat : 5503 (python) S 5501 5503 3621 0 -1 4202496 74576 1008 1 0 2145 31 0 0 20 0 3 0 338129041 704208896 70849 33554432000 4194304 5683644 140734606903136 140734606897968 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 171926 70849 1905 364 0 107146 0
[pid=5509] ppid=5503 vsize=611912 CPUtime=0.08 cores=1,3,5,7
/proc/5509/stat : 5509 (python) S 5503 5503 3621 0 -1 4202560 1136 0 0 0 5 3 0 0 20 0 2 0 338131303 626597888 69335 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5509/statm: 152978 69335 396 364 0 88201 0
[pid=5511] ppid=5509 vsize=148948 CPUtime=48.79 cores=1,3,5,7
/proc/5511/stat : 5511 (wbo1.4b-fixed) T 5509 5511 5511 0 -1 4202496 48235 0 0 0 4773 106 0 0 20 0 1 0 338131304 152522752 35035 33554432000 134512640 135752315 4290369904 4290367904 134611369 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5511/statm: 37237 35035 146 303 0 36932 0
[pid=5509/tid=5512] ppid=5503 vsize=611912 CPUtime=0 cores=1,3,5,7
/proc/5509/task/5512/stat : 5512 (python) S 5503 5503 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338136185 626597888 69335 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5513] ppid=5503 vsize=611924 CPUtime=0.15 cores=1,3,5,7
/proc/5513/stat : 5513 (python) S 5503 5503 3621 0 -1 4202560 942 0 0 0 9 6 0 0 20 0 1 0 338136187 626610176 69296 33554432000 4194304 5683644 140734606903136 140734606894224 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/5513/statm: 152981 69296 358 364 0 88201 0
[pid=5515] ppid=5513 vsize=4158816 CPUtime=33.73 cores=1,3,5,7
/proc/5515/stat : 5515 (pbct-0.1.2-linu) S 5513 5515 5515 0 -1 4204032 925927 0 0 0 2904 469 0 0 20 0 1 0 338136188 4258627584 987650 33554432000 134512640 136422188 4291482912 4291480924 135736970 0 0 16781312 16387 18446744071580380219 0 0 17 5 0 0 0 0 0
/proc/5515/statm: 1039704 987650 249 467 0 1039235 0
[pid=5503/tid=5510] ppid=5501 vsize=687704 CPUtime=0 cores=1,3,5,7
/proc/5503/task/5510/stat : 5510 (python) S 5501 5503 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 3 0 338131304 704208896 70849 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5514] ppid=5501 vsize=687704 CPUtime=0 cores=1,3,5,7
/proc/5503/task/5514/stat : 5514 (python) S 5501 5503 3621 0 -1 4202560 21 1008 0 0 0 0 0 0 20 0 3 0 338136188 704208896 70849 33554432000 4194304 5683644 140734606903136 140088503444016 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 104.51
Current children cumulated vsize (KiB) 6219304
heavy processes:
  %CPU=24 pid=5517 uid=0 cmd=

[startup+222.3 s]
/proc/loadavg: 4.74 3.04 2.37 2/212 5521
/proc/meminfo: memFree=12672588/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=687704 CPUtime=21.76 cores=1,3,5,7
/proc/5503/stat : 5503 (python) S 5501 5503 3621 0 -1 4202496 74576 1008 1 0 2145 31 0 0 20 0 3 0 338129041 704208896 70849 33554432000 4194304 5683644 140734606903136 140734606897968 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 6 0 0
/proc/5503/statm: 171926 70849 1905 364 0 107146 0
[pid=5509] ppid=5503 vsize=611912 CPUtime=0.08 cores=1,3,5,7
/proc/5509/stat : 5509 (python) S 5503 5503 3621 0 -1 4202560 1136 0 0 0 5 3 0 0 20 0 2 0 338131303 626597888 69335 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5509/statm: 152978 69335 396 364 0 88201 0

################
# More data... #
################

/proc/5558/task/5568/stat : 5568 (java) S 5556 5558 5558 0 -1 4202560 70 0 0 0 0 0 0 0 20 0 18 0 338202007 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912907434216 213423011753 0 0 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5569] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5569/stat : 5569 (java) S 5556 5558 5558 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 18 0 338202007 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912906380280 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5570] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5570/stat : 5570 (java) S 5556 5558 5558 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 18 0 338202007 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912905327880 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5571] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5571/stat : 5571 (java) S 5556 5558 5558 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912904276368 213423019808 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5572] ppid=5556 vsize=9558408 CPUtime=0.25 cores=1,3,5,7
/proc/5558/task/5572/stat : 5572 (java) S 5556 5558 5558 0 -1 4202560 2341 0 0 0 25 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912903223992 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5573] ppid=5556 vsize=9558408 CPUtime=0.23 cores=1,3,5,7
/proc/5558/task/5573/stat : 5573 (java) S 5556 5558 5558 0 -1 4202560 3248 0 0 0 23 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912902171192 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5558/tid=5574] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5574/stat : 5574 (java) S 5556 5558 5558 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912901119096 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5558/tid=5575] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5575/stat : 5575 (java) S 5556 5558 5558 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912900066504 213423011753 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5503/tid=5510] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5510/stat : 5510 (python) S 5501 5503 3621 0 -1 4202560 27 1328931 0 0 0 0 4925 897 20 0 10 0 338131304 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5514] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5514/stat : 5514 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338136188 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088503444016 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5525] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5525/stat : 5525 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338151559 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088492954160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5530] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5530/stat : 5530 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338156494 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088482464304 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5534] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5534/stat : 5534 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338161367 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088471974448 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5539] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5539/stat : 5539 (python) S 5501 5503 3621 0 -1 4202560 22 1328931 0 0 0 0 4925 897 20 0 10 0 338173057 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088461484592 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5503/tid=5548] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5548/stat : 5548 (python) S 5501 5503 3621 0 -1 4202560 23 1328931 0 0 0 0 4925 897 20 0 10 0 338187689 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088450994736 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5503/tid=5552] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5552/stat : 5552 (python) S 5501 5503 3621 0 -1 4202560 22 1328931 0 0 0 0 4925 897 20 0 10 0 338192469 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088440504880 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5503/tid=5557] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5557/stat : 5557 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338202002 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088430015024 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 488.5
Current children cumulated vsize (KiB) 22192988
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+730.301 s]
/proc/loadavg: 2.29 2.89 2.56 3/251 5575
/proc/meminfo: memFree=11115020/32873844 swapFree=6608/6608
[pid=5503] ppid=5501 vsize=1218248 CPUtime=149.02 cores=1,3,5,7
/proc/5503/stat : 5503 (python) S 5501 5503 3621 0 -1 4202496 81002 1328931 1 0 9020 60 4925 897 20 0 10 0 338129041 1247485952 70905 33554432000 4194304 5683644 140734606903136 140734606897968 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 6 0 0
/proc/5503/statm: 304562 70905 1919 364 0 239761 0
[pid=5509] ppid=5503 vsize=611912 CPUtime=0.08 cores=1,3,5,7
/proc/5509/stat : 5509 (python) S 5503 5503 3621 0 -1 4202560 1136 0 0 0 5 3 0 0 20 0 2 0 338131303 626597888 69335 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5509/statm: 152978 69335 396 364 0 88201 0
[pid=5511] ppid=5509 vsize=148948 CPUtime=48.79 cores=1,3,5,7
/proc/5511/stat : 5511 (wbo1.4b-fixed) T 5509 5511 5511 0 -1 4202496 48235 0 0 0 4773 106 0 0 20 0 1 0 338131304 152522752 35035 33554432000 134512640 135752315 4290369904 4290367904 134611369 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5511/statm: 37237 35035 146 303 0 36932 0
[pid=5509/tid=5512] ppid=5503 vsize=611912 CPUtime=0 cores=1,3,5,7
/proc/5509/task/5512/stat : 5512 (python) S 5503 5503 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338136185 626597888 69335 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5524] ppid=5503 vsize=687716 CPUtime=0.1 cores=1,3,5,7
/proc/5524/stat : 5524 (python) S 5503 5503 3621 0 -1 4202560 1103 0 0 0 5 5 0 0 20 0 2 0 338151559 704221184 69328 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5524/statm: 171929 69328 383 364 0 107146 0
[pid=5526] ppid=5524 vsize=126548 CPUtime=49.29 cores=1,3,5,7
/proc/5526/stat : 5526 (wbo1.4a) T 5524 5526 5526 0 -1 4202496 42689 0 0 0 4841 88 0 0 20 0 1 0 338151560 129585152 29487 33554432000 134512640 135751879 4294679088 4294676992 134601285 0 0 16781312 3 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/5526/statm: 31637 29487 143 303 0 31332 0
[pid=5524/tid=5528] ppid=5503 vsize=687716 CPUtime=0 cores=1,3,5,7
/proc/5524/task/5528/stat : 5528 (python) S 5503 5503 3621 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338156491 704221184 69328 33554432000 4194304 5683644 140734606903136 140088503444016 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5529] ppid=5503 vsize=763508 CPUtime=0.08 cores=1,3,5,7
/proc/5529/stat : 5529 (python) S 5503 5503 3621 0 -1 4202560 1101 0 0 0 4 4 0 0 20 0 2 0 338156493 781832192 69334 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5529/statm: 190877 69334 383 364 0 126091 0
[pid=5531] ppid=5529 vsize=2163468 CPUtime=48.68 cores=1,3,5,7
/proc/5531/stat : 5531 (scip-2.0.1.linu) T 5529 5531 5531 0 -1 4202496 478253 0 0 0 4694 174 0 0 20 0 1 0 338156494 2215391232 530174 33554432000 4194304 15243407 140735683272096 140735683268200 5489481 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5531/statm: 540867 530174 769 2698 0 538162 0
[pid=5529/tid=5532] ppid=5503 vsize=763508 CPUtime=0 cores=1,3,5,7
/proc/5529/task/5532/stat : 5532 (python) S 5503 5503 3621 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338161364 781832192 69334 33554432000 4194304 5683644 140734606903136 140088492954160 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5533] ppid=5503 vsize=839300 CPUtime=0.08 cores=1,3,5,7
/proc/5533/stat : 5533 (python) S 5503 5503 3621 0 -1 4202560 1120 0 0 0 4 4 0 0 20 0 2 0 338161367 859443200 69340 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5533/statm: 209825 69340 383 364 0 145036 0
[pid=5535] ppid=5533 vsize=2418352 CPUtime=47.96 cores=1,3,5,7
/proc/5535/stat : 5535 (scip-2.0.1.linu) T 5533 5535 5535 0 -1 4202496 479100 0 0 0 4620 176 0 0 20 0 1 0 338161368 2476392448 570767 33554432000 4194304 13684479 140735656103200 140735656099304 5688982 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5535/statm: 604588 570767 712 2317 0 602266 0
[pid=5533/tid=5536] ppid=5503 vsize=839300 CPUtime=0 cores=1,3,5,7
/proc/5533/task/5536/stat : 5536 (python) S 5503 5503 3621 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338166166 859443200 69340 33554432000 4194304 5683644 140734606903136 140088482464304 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5547] ppid=5503 vsize=990884 CPUtime=0.09 cores=1,3,5,7
/proc/5547/stat : 5547 (python) S 5503 5503 3621 0 -1 4202560 1142 0 0 0 4 5 0 0 20 0 2 0 338187688 1014665216 69352 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5547/statm: 247721 69352 383 364 0 182926 0
[pid=5549] ppid=5547 vsize=281616 CPUtime=47.73 cores=1,3,5,7
/proc/5549/stat : 5549 (bsolo_pb10) T 5547 5549 5549 0 -1 4202496 80645 0 0 0 4735 38 0 0 20 0 1 0 338187689 288374784 67741 33554432000 134512640 137138936 4294063664 4293481228 136570369 0 0 16781312 16384 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/5549/statm: 70404 67741 184 642 0 69760 0
[pid=5547/tid=5550] ppid=5503 vsize=990884 CPUtime=0 cores=1,3,5,7
/proc/5547/task/5550/stat : 5550 (python) S 5503 5503 3621 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338192466 1014665216 69352 33554432000 4194304 5683644 140734606903136 140088461484592 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5551] ppid=5503 vsize=1066676 CPUtime=0.16 cores=1,3,5,7
/proc/5551/stat : 5551 (python) S 5503 5503 3621 0 -1 4202560 1231 0 0 0 9 7 0 0 20 0 2 0 338192468 1092276224 69364 33554432000 4194304 5683644 140734606903136 140734606894784 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5551/statm: 266669 69364 389 364 0 201871 0
[pid=5553] ppid=5551 vsize=174936 CPUtime=95.25 cores=1,3,5,7
/proc/5553/stat : 5553 (clasp-1.3.7-x86) T 5551 5553 5553 0 -1 4202496 48981 0 0 0 9512 13 0 0 20 0 1 0 338192469 179134464 42836 33554432000 134512640 136139019 4286797584 4286796560 134883787 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5553/statm: 43734 42836 183 398 0 43334 0
[pid=5551/tid=5554] ppid=5503 vsize=1066676 CPUtime=0 cores=1,3,5,7
/proc/5551/task/5554/stat : 5554 (python) S 5503 5503 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338197183 1092276224 69364 33554432000 4194304 5683644 140734606903136 140088450994736 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5556] ppid=5503 vsize=1142468 CPUtime=0 cores=1,3,5,7
/proc/5556/stat : 5556 (python) S 5503 5503 3621 0 -1 4202560 869 0 0 0 0 0 0 0 20 0 1 0 338202001 1169887232 69322 33554432000 4194304 5683644 140734606903136 140734606894224 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/5556/statm: 285617 69322 342 364 0 220816 0
[pid=5558] ppid=5556 vsize=9558408 CPUtime=1.19 cores=1,3,5,7
/proc/5558/stat : 5558 (java) S 5556 5558 5558 0 -1 4202496 9589 0 1 0 114 5 0 0 20 0 18 0 338202002 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 140737079875856 213422997565 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5558/statm: 2389602 42214 2247 9 0 2380296 0
[pid=5558/tid=5559] ppid=5556 vsize=9558408 CPUtime=0.64 cores=1,3,5,7
/proc/5558/task/5559/stat : 5559 (java) R 5556 5558 5558 0 -1 4202560 2484 0 1 0 61 3 0 0 20 0 18 0 338202004 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139913419578920 139913274610425 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=5558/tid=5560] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5560/stat : 5560 (java) S 5556 5558 5558 0 -1 4202560 6 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139913263098152 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5561] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5561/stat : 5561 (java) S 5556 5558 5558 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139913262045352 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5558/tid=5562] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5562/stat : 5562 (java) S 5556 5558 5558 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139913260993064 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5558/tid=5563] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5563/stat : 5563 (java) S 5556 5558 5558 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139913259940264 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5558/tid=5564] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5564/stat : 5564 (java) S 5556 5558 5558 0 -1 4202560 4 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139913258887976 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5558/tid=5565] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5565/stat : 5565 (java) S 5556 5558 5558 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912921737896 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5558/tid=5566] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5566/stat : 5566 (java) S 5556 5558 5558 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912920685608 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5567] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5567/stat : 5567 (java) S 5556 5558 5558 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338202006 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912919632808 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5558/tid=5568] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5568/stat : 5568 (java) S 5556 5558 5558 0 -1 4202560 70 0 0 0 0 0 0 0 20 0 18 0 338202007 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912907434216 213423011753 0 0 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5569] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5569/stat : 5569 (java) S 5556 5558 5558 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 18 0 338202007 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912906380280 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5570] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5570/stat : 5570 (java) S 5556 5558 5558 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 18 0 338202007 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912905327880 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5571] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5571/stat : 5571 (java) S 5556 5558 5558 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912904276368 213423019808 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5572] ppid=5556 vsize=9558408 CPUtime=0.25 cores=1,3,5,7
/proc/5558/task/5572/stat : 5572 (java) S 5556 5558 5558 0 -1 4202560 2341 0 0 0 25 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912903223992 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5558/tid=5573] ppid=5556 vsize=9558408 CPUtime=0.23 cores=1,3,5,7
/proc/5558/task/5573/stat : 5573 (java) S 5556 5558 5558 0 -1 4202560 3248 0 0 0 23 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912902171192 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5558/tid=5574] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5574/stat : 5574 (java) S 5556 5558 5558 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912901119096 213423010876 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5558/tid=5575] ppid=5556 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/5558/task/5575/stat : 5575 (java) S 5556 5558 5558 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338202010 9787809792 42214 33554432000 1073741824 1073778376 140737079884720 139912900066504 213423011753 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5503/tid=5510] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5510/stat : 5510 (python) S 5501 5503 3621 0 -1 4202560 27 1328931 0 0 0 0 4925 897 20 0 10 0 338131304 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088513933872 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5514] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5514/stat : 5514 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338136188 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088503444016 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5525] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5525/stat : 5525 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338151559 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088492954160 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5530] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5530/stat : 5530 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338156494 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088482464304 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5534] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5534/stat : 5534 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338161367 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088471974448 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5503/tid=5539] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5539/stat : 5539 (python) S 5501 5503 3621 0 -1 4202560 22 1328931 0 0 0 0 4925 897 20 0 10 0 338173057 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088461484592 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5503/tid=5548] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5548/stat : 5548 (python) S 5501 5503 3621 0 -1 4202560 23 1328931 0 0 0 0 4925 897 20 0 10 0 338187689 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088450994736 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5503/tid=5552] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5552/stat : 5552 (python) S 5501 5503 3621 0 -1 4202560 22 1328931 0 0 0 0 4925 897 20 0 10 0 338192469 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088440504880 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5503/tid=5557] ppid=5501 vsize=1218248 CPUtime=58.22 cores=1,3,5,7
/proc/5503/task/5557/stat : 5557 (python) S 5501 5503 3621 0 -1 4202560 21 1328931 0 0 0 0 4925 897 20 0 10 0 338202002 1247485952 70905 33554432000 4194304 5683644 140734606903136 140088430015024 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 488.5
Current children cumulated vsize (KiB) 22192988

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5503 and gives
#  childrusage.ru_utime.tv_sec=139
#  childrusage.ru_utime.tv_usec=461798
#  childrusage.ru_stime.tv_sec=9
#  childrusage.ru_stime.tv_usec=597540
# CPU time returned by wait4() is 149.059
# while last known CPU time is 488.5
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 730.373
CPU time (s): 488.5
CPU user time (s): 472.65
CPU system time (s): 15.85
CPU usage (%): 66.8836
Max. virtual memory (cumulated for all children) (KiB): 22192988

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 139.462
system time used= 9.59754
maximum resident set size= 4133856
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1409933
page faults= 1
swaps= 0
block input operations= 0
block output operations= 147184
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2069452
involuntary context switches= 372

runsolver used 2.6416 second user time and 6.69598 second system time

The end

Launcher Data 

Begin job on node140 at 2012-06-03 12:40:40
IDJOB=3740864
IDBENCH=89649
IDSOLVER=2327
FILE ID=node140/3740864-1338720039
RUNJOBID= node140-1338713222-3638
PBS_JOBID= 14636725
Free space on /tmp= 70468 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-137.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3740864-1338720039/watcher-3740864-1338720039 -o /tmp/evaluation-result-3740864-1338720039/solver-3740864-1338720039 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3740864-1338720039.opb -seed 184419883 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 395b6779bc0f993a1d1357b59c2e4536
RANDOM SEED=184419883

node140.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.88
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        25306796 kB
Buffers:          459808 kB
Cached:          4618172 kB
SwapCached:         1664 kB
Active:          3238124 kB
Inactive:        3407564 kB
Active(anon):    1560508 kB
Inactive(anon):     9488 kB
Active(file):    1677616 kB
Inactive(file):  3398076 kB
Unevictable:        6608 kB
Mlocked:            6608 kB
SwapTotal:      67108856 kB
SwapFree:       67105156 kB
Dirty:            151376 kB
Writeback:             0 kB
AnonPages:       1572880 kB
Mapped:            25840 kB
Shmem:                92 kB
Slab:             774308 kB
SReclaimable:     213488 kB
SUnreclaim:       560820 kB
KernelStack:        1640 kB
PageTables:        11496 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    2207312 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:     34816 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 62468 MiB
End job on node140 at 2012-06-03 12:52:54