Trace number 3741176

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 491.88 824.175

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-122.opb
MD5SUM45533d52c524a74641c715d4320b890b
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables48411
Total number of constraints1121517
Number of constraints which are clauses1007723
Number of constraints which are cardinality constraints (but not clauses)3523
Number of constraints which are nor clauses,nor cardinality constraints110271
Minimum length of a constraint2
Maximum length of a constraint690
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 691
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.45/1.20	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.45/1.20	c INFO: solving HOME/instance-3741176-1338721474.opb
8.75/9.56	c INFO: parsing took 8.21 s
21.84/22.65	c INFO: features took 13.05 CPU seconds
21.84/22.68	c INFO: running wbo1.4b-fixed@0 for 45 with 1778 remaining (b = 0.96)
71.45/72.29	c INFO: running pbct-0.1.2-linear@0 for 45 with 1729 remaining (b = 0.96)
106.21/267.77	c INFO: running wbo1.4a@0 for 45 with 1700 remaining (b = 0.95)
154.99/316.49	c INFO: running scip-2.0.1-clp@0 for 45 with 1652 remaining (b = 0.83)
203.77/365.14	c INFO: running scip-2.0.1-spx@0 for 45 with 1606 remaining (b = 0.68)
324.02/485.38	c INFO: wrote minion input file to HOME/tmp9N8MMz.minion
324.02/485.38	c INFO: running minion-0.12@0 for 45 with 1559 remaining (b = 0.55)
348.31/680.85	c INFO: running bsolo_pb10-l2@0 for 45 with 1467 remaining (b = 0.29)
396.19/728.63	c INFO: running clasp-1.3.7@0 for 45 with 1420 remaining (b = 0.17)
443.35/775.80	c INFO: running clasp-1.3.7@47 for 45 with 1373 remaining (b = 0.02)
491.65/823.97	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1325 remaining (b = 0.05)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3741176-1338721474/watcher-3741176-1338721474 -o /tmp/evaluation-result-3741176-1338721474/solver-3741176-1338721474 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741176-1338721474.opb -seed 1824488862 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.13 3.16 3.04 2/195 6665
/proc/meminfo: memFree=26489152/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=15276 CPUtime=0 cores=0,2,4,6
/proc/6665/stat : 6665 (python) D 6663 6665 2614 0 -1 4202496 633 0 0 0 0 0 0 0 20 0 1 0 338272635 15642624 526 33554432000 4194304 5683644 140735961428720 140735961414632 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 3819 526 312 364 0 261 0

[startup+0.056965 s]
/proc/loadavg: 2.13 3.16 3.04 2/195 6665
/proc/meminfo: memFree=26489152/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=16588 CPUtime=0.01 cores=0,2,4,6
/proc/6665/stat : 6665 (python) D 6663 6665 2614 0 -1 4202496 1187 0 0 0 1 0 0 0 20 0 1 0 338272635 16986112 994 33554432000 4194304 5683644 140735961428720 140735961382936 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 4147 994 417 364 0 589 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16588

[startup+0.100214 s]
/proc/loadavg: 2.13 3.16 3.04 2/195 6665
/proc/meminfo: memFree=26489152/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=19080 CPUtime=0.02 cores=0,2,4,6
/proc/6665/stat : 6665 (python) R 6663 6665 2614 0 -1 4202496 1398 0 0 0 2 0 0 0 20 0 1 0 338272635 19537920 1152 33554432000 4194304 5683644 140735961428720 140735961359224 4870104 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 4770 1152 430 364 0 694 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19080

[startup+0.300207 s]
/proc/loadavg: 2.13 3.16 3.04 2/195 6665
/proc/meminfo: memFree=26489152/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=102044 CPUtime=0.12 cores=0,2,4,6
/proc/6665/stat : 6665 (python) D 6663 6665 2614 0 -1 4202496 4127 1012 1 0 9 3 0 0 20 0 1 0 338272635 104493056 2725 33554432000 4194304 5683644 140735961428720 140735961283480 240565173941 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 25511 2725 876 364 0 1721 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 102044

[startup+0.700225 s]
/proc/loadavg: 2.13 3.16 3.04 2/195 6665
/proc/meminfo: memFree=26489152/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=162436 CPUtime=0.24 cores=0,2,4,6
/proc/6665/stat : 6665 (python) D 6663 6665 2614 0 -1 4202496 6467 1012 1 0 16 8 0 0 20 0 1 0 338272635 166334464 4273 33554432000 4194304 5683644 140735961428720 140735961319224 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 40609 4273 1192 364 0 3254 0
Current children cumulated CPU time (s) 0.24
Current children cumulated vsize (KiB) 162436

[startup+1.50019 s]
/proc/loadavg: 2.20 3.16 3.04 3/196 6670
/proc/meminfo: memFree=26471652/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=339292 CPUtime=0.75 cores=0,2,4,6
/proc/6665/stat : 6665 (python) R 6663 6665 2614 0 -1 4202496 12176 1012 1 0 60 15 0 0 20 0 1 0 338272635 347435008 21473 33554432000 4194304 5683644 140735961428720 140735961421960 4571108 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 84823 21473 1811 364 0 20052 0
Current children cumulated CPU time (s) 0.75
Current children cumulated vsize (KiB) 339292

[startup+3.10019 s]
/proc/loadavg: 2.20 3.16 3.04 3/196 6670
/proc/meminfo: memFree=26390136/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=391600 CPUtime=2.35 cores=0,2,4,6
/proc/6665/stat : 6665 (python) R 6663 6665 2614 0 -1 4202496 24952 1012 1 0 217 18 0 0 20 0 1 0 338272635 400998400 34249 33554432000 4194304 5683644 140735961428720 140735961421336 240565487893 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 97900 34250 1811 364 0 33129 0
Current children cumulated CPU time (s) 2.35
Current children cumulated vsize (KiB) 391600
heavy processes:

[startup+6.30019 s]
/proc/loadavg: 2.26 3.16 3.04 3/196 6670
/proc/meminfo: memFree=26280148/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=493596 CPUtime=5.55 cores=0,2,4,6
/proc/6665/stat : 6665 (python) R 6663 6665 2614 0 -1 4202496 50527 1012 1 0 532 23 0 0 20 0 1 0 338272635 505442304 59824 33554432000 4194304 5683644 140735961428720 140735961421336 4445115 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 123399 59825 1811 364 0 58628 0
Current children cumulated CPU time (s) 5.55
Current children cumulated vsize (KiB) 493596

[startup+12.7002 s]
/proc/loadavg: 2.24 3.14 3.03 3/196 6670
/proc/meminfo: memFree=26155172/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=585396 CPUtime=11.95 cores=0,2,4,6
/proc/6665/stat : 6665 (python) R 6663 6665 2614 0 -1 4202496 70689 1012 1 0 1165 30 0 0 20 0 1 0 338272635 599445504 83101 33554432000 4194304 5683644 140735961428720 140735961415544 240564798679 0 0 16781312 33554434 0 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 146349 83101 1846 364 0 81578 0
Current children cumulated CPU time (s) 11.95
Current children cumulated vsize (KiB) 585396

[startup+25.5003 s]
/proc/loadavg: 2.19 3.08 3.01 3/199 6673
/proc/meminfo: memFree=26131124/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=611908 CPUtime=21.93 cores=0,2,4,6
/proc/6665/stat : 6665 (python) S 6663 6665 2614 0 -1 4202496 73086 1012 1 0 2161 32 0 0 20 0 2 0 338272635 626593792 71349 33554432000 4194304 5683644 140735961428720 140735961423552 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 152977 71349 1902 364 0 88200 0
[pid=6671] ppid=6665 vsize=536128 CPUtime=0 cores=0,2,4,6
/proc/6671/stat : 6671 (python) S 6665 6665 2614 0 -1 4202560 732 0 0 0 0 0 0 0 20 0 1 0 338274904 548995072 69272 33554432000 4194304 5683644 140735961428720 140735961419808 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6671/statm: 134032 69272 340 364 0 69255 0
[pid=6673] ppid=6671 vsize=116000 CPUtime=2.76 cores=0,2,4,6
/proc/6673/stat : 6673 (wbo1.4b-fixed) R 6671 6673 6673 0 -1 4202496 25080 0 0 0 271 5 0 0 20 0 1 0 338274904 118784000 24801 33554432000 134512640 135752315 4294107040 4293056492 135228687 0 0 16781312 3 0 0 0 17 0 0 0 3 0 0
/proc/6673/statm: 29000 24801 9236 303 0 16271 0
[pid=6665/tid=6672] ppid=6663 vsize=611908 CPUtime=0 cores=0,2,4,6
/proc/6665/task/6672/stat : 6672 (python) S 6663 6665 2614 0 -1 4202560 24 1012 0 0 0 0 0 0 20 0 2 0 338274904 626593792 71349 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 24.69
Current children cumulated vsize (KiB) 1264036

[startup+51.1004 s]
/proc/loadavg: 2.12 2.99 2.99 3/203 6677
/proc/meminfo: memFree=26018104/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=611908 CPUtime=21.93 cores=0,2,4,6
/proc/6665/stat : 6665 (python) S 6663 6665 2614 0 -1 4202496 73086 1012 1 0 2161 32 0 0 20 0 2 0 338272635 626593792 71349 33554432000 4194304 5683644 140735961428720 140735961423552 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 152977 71349 1902 364 0 88200 0
[pid=6671] ppid=6665 vsize=536128 CPUtime=0.03 cores=0,2,4,6
/proc/6671/stat : 6671 (python) S 6665 6665 2614 0 -1 4202560 895 0 0 0 2 1 0 0 20 0 1 0 338274904 548995072 69295 33554432000 4194304 5683644 140735961428720 140735961419808 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6671/statm: 134032 69295 363 364 0 69255 0
[pid=6673] ppid=6671 vsize=139724 CPUtime=28.35 cores=0,2,4,6
/proc/6673/stat : 6673 (wbo1.4b-fixed) R 6671 6673 6673 0 -1 4202496 45827 0 0 0 2792 43 0 0 20 0 1 0 338274904 143077376 33139 33554432000 134512640 135752315 4294107040 4294105116 135270849 0 0 16781312 3 0 0 0 17 0 0 0 3 0 0
/proc/6673/statm: 34931 33139 146 303 0 34626 0
[pid=6665/tid=6672] ppid=6663 vsize=611908 CPUtime=0 cores=0,2,4,6
/proc/6665/task/6672/stat : 6672 (python) S 6663 6665 2614 0 -1 4202560 24 1012 0 0 0 0 0 0 20 0 2 0 338274904 626593792 71349 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 50.31
Current children cumulated vsize (KiB) 1287760

[startup+102.301 s]
/proc/loadavg: 2.10 2.85 2.94 3/187 6750
/proc/meminfo: memFree=26712556/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=687700 CPUtime=21.96 cores=0,2,4,6
/proc/6665/stat : 6665 (python) S 6663 6665 2614 0 -1 4202496 73974 1012 1 0 2163 33 0 0 20 0 3 0 338272635 704204800 71866 33554432000 4194304 5683644 140735961428720 140735961423552 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 171925 71866 1905 364 0 107145 0
[pid=6671] ppid=6665 vsize=611908 CPUtime=0.08 cores=0,2,4,6
/proc/6671/stat : 6671 (python) S 6665 6665 2614 0 -1 4202560 1111 0 0 0 5 3 0 0 20 0 2 0 338274904 626593792 69843 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6671/statm: 152977 69843 396 364 0 88200 0
[pid=6673] ppid=6671 vsize=153332 CPUtime=49.52 cores=0,2,4,6
/proc/6673/stat : 6673 (wbo1.4b-fixed) T 6671 6673 6673 0 -1 4202496 49238 0 0 0 4893 59 0 0 20 0 1 0 338274904 157011968 36550 33554432000 134512640 135752315 4294107040 4294105040 134611367 0 0 16781312 3 18446744071579387992 0 0 17 0 0 0 3 0 0
/proc/6673/statm: 38333 36550 146 303 0 38028 0
[pid=6671/tid=6678] ppid=6665 vsize=611908 CPUtime=0 cores=0,2,4,6
/proc/6671/task/6678/stat : 6678 (python) S 6665 6665 2614 0 -1 4202560 40 0 0 0 0 0 0 0 20 0 2 0 338279862 626593792 69843 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6679] ppid=6665 vsize=611920 CPUtime=0.04 cores=0,2,4,6
/proc/6679/stat : 6679 (python) S 6665 6665 2614 0 -1 4202560 895 0 0 0 2 2 0 0 20 0 1 0 338279865 626606080 69805 33554432000 4194304 5683644 140735961428720 140735961419808 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6679/statm: 152980 69805 358 364 0 88200 0
[pid=6681] ppid=6679 vsize=4155400 CPUtime=29.99 cores=0,2,4,6
/proc/6681/stat : 6681 (pbct-0.1.2-linu) R 6679 6681 6681 0 -1 4202496 926215 0 0 0 2839 160 0 0 20 0 1 0 338279865 4255129600 987134 33554432000 134512640 136422188 4293190576 4293188996 135830671 0 0 16781312 16387 0 0 0 17 0 0 0 0 0 0
/proc/6681/statm: 1038850 987134 202 467 0 1038381 0
[pid=6665/tid=6672] ppid=6663 vsize=687700 CPUtime=0 cores=0,2,4,6
/proc/6665/task/6672/stat : 6672 (python) S 6663 6665 2614 0 -1 4202560 24 1012 0 0 0 0 0 0 20 0 3 0 338274904 704204800 71866 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6680] ppid=6663 vsize=687700 CPUtime=0 cores=0,2,4,6
/proc/6665/task/6680/stat : 6680 (python) S 6663 6665 2614 0 -1 4202560 19 1012 0 0 0 0 0 0 20 0 3 0 338279865 704204800 71866 33554432000 4194304 5683644 140735961428720 140594351432240 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 101.59
Current children cumulated vsize (KiB) 6220260

[startup+162.3 s]
/proc/loadavg: 3.81 3.19 3.04 2/193 6757
/proc/meminfo: memFree=18451384/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=687700 CPUtime=21.96 cores=0,2,4,6
/proc/6665/stat : 6665 (python) S 6663 6665 2614 0 -1 4202496 73974 1012 1 0 2163 33 0 0 20 0 3 0 338272635 704204800 71866 33554432000 4194304 5683644 140735961428720 140735961423552 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 171925 71866 1905 364 0 107145 0
[pid=6671] ppid=6665 vsize=611908 CPUtime=0.08 cores=0,2,4,6
/proc/6671/stat : 6671 (python) S 6665 6665 2614 0 -1 4202560 1111 0 0 0 5 3 0 0 20 0 2 0 338274904 626593792 69843 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6671/statm: 152977 69843 396 364 0 88200 0
[pid=6673] ppid=6671 vsize=153332 CPUtime=49.52 cores=0,2,4,6
/proc/6673/stat : 6673 (wbo1.4b-fixed) T 6671 6673 6673 0 -1 4202496 49238 0 0 0 4893 59 0 0 20 0 1 0 338274904 157011968 36550 33554432000 134512640 135752315 4294107040 4294105040 134611367 0 0 16781312 3 18446744071579387992 0 0 17 0 0 0 3 0 0
/proc/6673/statm: 38333 36550 146 303 0 38028 0
[pid=6671/tid=6678] ppid=6665 vsize=611908 CPUtime=0 cores=0,2,4,6
/proc/6671/task/6678/stat : 6678 (python) S 6665 6665 2614 0 -1 4202560 40 0 0 0 0 0 0 0 20 0 2 0 338279862 626593792 69843 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6679] ppid=6665 vsize=611920 CPUtime=0.15 cores=0,2,4,6
/proc/6679/stat : 6679 (python) S 6665 6665 2614 0 -1 4202560 929 0 0 0 8 7 0 0 20 0 1 0 338279865 626606080 69805 33554432000 4194304 5683644 140735961428720 140735961419808 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/6679/statm: 152980 69805 358 364 0 88200 0
[pid=6681] ppid=6679 vsize=4158472 CPUtime=34.16 cores=0,2,4,6
/proc/6681/stat : 6681 (pbct-0.1.2-linu) S 6679 6681 6681 0 -1 4204032 926912 0 0 0 2852 564 0 0 20 0 1 0 338279865 4258275328 987831 33554432000 134512640 136422188 4293190576 4293188588 135736970 0 0 16781312 16387 18446744071580380219 0 0 17 2 0 0 0 0 0
/proc/6681/statm: 1039618 987831 249 467 0 1039149 0
[pid=6665/tid=6672] ppid=6663 vsize=687700 CPUtime=0 cores=0,2,4,6
/proc/6665/task/6672/stat : 6672 (python) S 6663 6665 2614 0 -1 4202560 24 1012 0 0 0 0 0 0 20 0 3 0 338274904 704204800 71866 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6680] ppid=6663 vsize=687700 CPUtime=0 cores=0,2,4,6
/proc/6665/task/6680/stat : 6680 (python) S 6663 6665 2614 0 -1 4202560 19 1012 0 0 0 0 0 0 20 0 3 0 338279865 704204800 71866 33554432000 4194304 5683644 140735961428720 140594351432240 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 105.87
Current children cumulated vsize (KiB) 6223332
heavy processes:
  %CPU=25 pid=6751 uid=0 cmd=

[startup+222.307 s]
/proc/loadavg: 4.70 3.55 3.17 2/196 6761
/proc/meminfo: memFree=18426064/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=687700 CPUtime=21.96 cores=0,2,4,6
/proc/6665/stat : 6665 (python) S 6663 6665 2614 0 -1 4202496 73974 1012 1 0 2163 33 0 0 20 0 3 0 338272635 704204800 71866 33554432000 4194304 5683644 140735961428720 140735961423552 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6665/statm: 171925 71866 1905 364 0 107145 0
[pid=6671] ppid=6665 vsize=611908 CPUtime=0.08 cores=0,2,4,6
/proc/6671/stat : 6671 (python) S 6665 6665 2614 0 -1 4202560 1111 0 0 0 5 3 0 0 20 0 2 0 338274904 626593792 69843 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6671/statm: 152977 69843 396 364 0 88200 0

################
# More data... #
################

/proc/6935/task/6945/stat : 6945 (java) S 6933 6935 6935 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338355036 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405786011752 240568547241 0 0 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6946] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6946/stat : 6946 (java) S 6933 6935 6935 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338355037 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405784958072 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6947] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6947/stat : 6947 (java) S 6933 6935 6935 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338355037 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405783905416 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6948] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6948/stat : 6948 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405782854160 240568555296 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6949] ppid=6933 vsize=9558408 CPUtime=0.01 cores=0,2,4,6
/proc/6935/task/6949/stat : 6949 (java) S 6933 6935 6935 0 -1 4202560 332 0 0 0 1 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405781801528 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6950] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6950/stat : 6950 (java) S 6933 6935 6935 0 -1 4202560 511 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405780748984 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6935/tid=6951] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6951/stat : 6951 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405779696632 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6935/tid=6952] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6952/stat : 6952 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405778644296 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6665/tid=6672] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6672/stat : 6672 (python) S 6663 6665 2614 0 -1 4202560 24 1331003 0 0 0 0 4903 995 20 0 10 0 338274904 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6680] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6680/stat : 6680 (python) S 6663 6665 2614 0 -1 4202560 19 1331003 0 0 0 0 4903 995 20 0 10 0 338279865 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594351432240 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6769] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6769/stat : 6769 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338299414 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594340942384 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6777] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6777/stat : 6777 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338304286 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594330452528 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6784] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6784/stat : 6784 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338309151 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594319962672 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6880] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6880/stat : 6880 (python) S 6663 6665 2614 0 -1 4202560 22 1331003 0 0 0 0 4903 995 20 0 10 0 338321175 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594309472816 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6665/tid=6899] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6899/stat : 6899 (python) S 6663 6665 2614 0 -1 4202560 23 1331003 0 0 0 0 4903 995 20 0 10 0 338340721 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594298982960 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6907] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6907/stat : 6907 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338345499 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594288493104 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6665/tid=6934] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6934/stat : 6934 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338355033 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594278003248 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 491.88
Current children cumulated vsize (KiB) 23654060
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+824.101 s]
/proc/loadavg: 2.27 3.07 3.10 3/264 6952
/proc/meminfo: memFree=14355700/32873844 swapFree=7116/7116
[pid=6665] ppid=6663 vsize=1218244 CPUtime=152.4 cores=0,2,4,6
/proc/6665/stat : 6665 (python) S 6663 6665 2614 0 -1 4202496 80338 1331003 1 0 9285 57 4903 995 20 0 10 0 338272635 1247481856 71922 33554432000 4194304 5683644 140735961428720 140735961423552 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/6665/statm: 304561 71922 1919 364 0 239760 0
[pid=6671] ppid=6665 vsize=611908 CPUtime=0.08 cores=0,2,4,6
/proc/6671/stat : 6671 (python) S 6665 6665 2614 0 -1 4202560 1111 0 0 0 5 3 0 0 20 0 2 0 338274904 626593792 69843 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6671/statm: 152977 69843 396 364 0 88200 0
[pid=6673] ppid=6671 vsize=153332 CPUtime=49.52 cores=0,2,4,6
/proc/6673/stat : 6673 (wbo1.4b-fixed) T 6671 6673 6673 0 -1 4202496 49238 0 0 0 4893 59 0 0 20 0 1 0 338274904 157011968 36817 33554432000 134512640 135752315 4294107040 4294105040 134611367 0 0 16781312 3 18446744071579387992 0 0 17 0 0 0 3 0 0
/proc/6673/statm: 38333 36817 146 303 0 38028 0
[pid=6671/tid=6678] ppid=6665 vsize=611908 CPUtime=0 cores=0,2,4,6
/proc/6671/task/6678/stat : 6678 (python) S 6665 6665 2614 0 -1 4202560 40 0 0 0 0 0 0 0 20 0 2 0 338279862 626593792 69843 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6768] ppid=6665 vsize=687712 CPUtime=0.09 cores=0,2,4,6
/proc/6768/stat : 6768 (python) S 6665 6665 2614 0 -1 4202560 1086 0 0 0 4 5 0 0 20 0 2 0 338299413 704217088 70344 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6768/statm: 171928 70344 383 364 0 107145 0
[pid=6770] ppid=6768 vsize=134912 CPUtime=48.67 cores=0,2,4,6
/proc/6770/stat : 6770 (wbo1.4a) T 6768 6770 6770 0 -1 4202496 44640 0 0 0 4810 57 0 0 20 0 1 0 338299414 138149888 32185 33554432000 134512640 135751879 4294657120 4294655296 134616398 0 0 16781312 3 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6770/statm: 33728 32185 143 303 0 33423 0
[pid=6768/tid=6775] ppid=6665 vsize=687712 CPUtime=0 cores=0,2,4,6
/proc/6768/task/6775/stat : 6775 (python) S 6665 6665 2614 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338304283 704217088 70344 33554432000 4194304 5683644 140735961428720 140594351432240 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6776] ppid=6665 vsize=763504 CPUtime=0.09 cores=0,2,4,6
/proc/6776/stat : 6776 (python) S 6665 6665 2614 0 -1 4202560 1088 0 0 0 5 4 0 0 20 0 2 0 338304285 781828096 70351 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6776/statm: 190876 70351 383 364 0 126090 0
[pid=6778] ppid=6776 vsize=3020324 CPUtime=48.6 cores=0,2,4,6
/proc/6778/stat : 6778 (scip-2.0.1.linu) T 6776 6778 6778 0 -1 4202496 537789 0 0 0 4668 192 0 0 20 0 1 0 338304286 3092811776 651662 33554432000 4194304 15243407 140736094955600 140736094950744 7060909 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6778/statm: 755081 651662 875 2698 0 752376 0
[pid=6776/tid=6782] ppid=6665 vsize=763504 CPUtime=0 cores=0,2,4,6
/proc/6776/task/6782/stat : 6782 (python) S 6665 6665 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338309148 781828096 70351 33554432000 4194304 5683644 140735961428720 140594340942384 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6783] ppid=6665 vsize=839296 CPUtime=0.09 cores=0,2,4,6
/proc/6783/stat : 6783 (python) S 6665 6665 2614 0 -1 4202560 1107 0 0 0 4 5 0 0 20 0 2 0 338309150 859439104 70357 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6783/statm: 209824 70357 383 364 0 145035 0
[pid=6785] ppid=6783 vsize=3022540 CPUtime=48.93 cores=0,2,4,6
/proc/6785/stat : 6785 (scip-2.0.1.linu) T 6783 6785 6785 0 -1 4202496 501912 0 0 0 4706 187 0 0 20 0 1 0 338309151 3095080960 704434 33554432000 4194304 13684479 140737036617888 140737036612040 7266928 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6785/statm: 755635 704434 752 2317 0 753313 0
[pid=6783/tid=6807] ppid=6665 vsize=839296 CPUtime=0 cores=0,2,4,6
/proc/6783/task/6807/stat : 6807 (python) S 6665 6665 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338314046 859439104 70357 33554432000 4194304 5683644 140735961428720 140594330452528 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6898] ppid=6665 vsize=990880 CPUtime=0.09 cores=0,2,4,6
/proc/6898/stat : 6898 (python) S 6665 6665 2614 0 -1 4202560 1128 0 0 0 5 4 0 0 20 0 2 0 338340721 1014661120 70369 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6898/statm: 247720 70369 383 364 0 182925 0
[pid=6900] ppid=6898 vsize=281612 CPUtime=47.73 cores=0,2,4,6
/proc/6900/stat : 6900 (bsolo_pb10) T 6898 6900 6900 0 -1 4202496 80645 0 0 0 4739 34 0 0 20 0 1 0 338340721 288370688 67742 33554432000 134512640 137138936 4294360880 4293778400 134540051 0 0 16781312 16384 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/6900/statm: 70403 67742 184 642 0 69759 0
[pid=6898/tid=6905] ppid=6665 vsize=990880 CPUtime=0 cores=0,2,4,6
/proc/6898/task/6905/stat : 6905 (python) S 6665 6665 2614 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338345496 1014661120 70369 33554432000 4194304 5683644 140735961428720 140594309472816 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6906] ppid=6665 vsize=1066672 CPUtime=0.18 cores=0,2,4,6
/proc/6906/stat : 6906 (python) S 6665 6665 2614 0 -1 4202560 1210 0 0 0 10 8 0 0 20 0 2 0 338345498 1092272128 70381 33554432000 4194304 5683644 140735961428720 140735961420368 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6906/statm: 266668 70381 389 364 0 201870 0
[pid=6908] ppid=6906 vsize=162252 CPUtime=95.26 cores=0,2,4,6
/proc/6908/stat : 6908 (clasp-1.3.7-x86) T 6906 6908 6908 0 -1 4202496 46260 0 0 0 9514 12 0 0 20 0 1 0 338345499 166146048 39809 33554432000 134512640 136139019 4291358272 4291357252 134883821 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6908/statm: 40563 39809 183 398 0 40163 0
[pid=6906/tid=6911] ppid=6665 vsize=1066672 CPUtime=0 cores=0,2,4,6
/proc/6906/task/6911/stat : 6911 (python) S 6665 6665 2614 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338350213 1092272128 70381 33554432000 4194304 5683644 140735961428720 140594298982960 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6933] ppid=6665 vsize=1142464 CPUtime=0 cores=0,2,4,6
/proc/6933/stat : 6933 (python) S 6665 6665 2614 0 -1 4202560 867 0 0 0 0 0 0 0 20 0 1 0 338355033 1169883136 70339 33554432000 4194304 5683644 140735961428720 140735961419808 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/6933/statm: 285616 70339 342 364 0 220815 0
[pid=6935] ppid=6933 vsize=9558408 CPUtime=0.15 cores=0,2,4,6
/proc/6935/stat : 6935 (java) S 6933 6935 6935 0 -1 4202496 4684 0 1 0 14 1 0 0 20 0 18 0 338355033 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140737355787040 240568533053 0 0 0 16800975 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6935/statm: 2389602 6131 2152 9 0 2380296 0
[pid=6935/tid=6936] ppid=6933 vsize=9558408 CPUtime=0.1 cores=0,2,4,6
/proc/6935/task/6936/stat : 6936 (java) R 6933 6935 6935 0 -1 4202560 2418 0 1 0 10 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405961961400 140405786142133 0 4 0 16800975 0 0 0 -1 2 0 0 0 0 0
[pid=6935/tid=6937] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6937/stat : 6937 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405923461288 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6938] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6938/stat : 6938 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405922408744 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6935/tid=6939] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6939/stat : 6939 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405921356200 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6935/tid=6940] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6940/stat : 6940 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405920303656 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6935/tid=6941] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6941/stat : 6941 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405919251112 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6942] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6942/stat : 6942 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405918198568 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6935/tid=6943] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6943/stat : 6943 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405917146024 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6935/tid=6944] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6944/stat : 6944 (java) S 6933 6935 6935 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338355035 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405916093480 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6945] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6945/stat : 6945 (java) S 6933 6935 6935 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338355036 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405786011752 240568547241 0 0 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6946] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6946/stat : 6946 (java) S 6933 6935 6935 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338355037 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405784958072 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6947] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6947/stat : 6947 (java) S 6933 6935 6935 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338355037 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405783905416 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6948] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6948/stat : 6948 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405782854160 240568555296 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6949] ppid=6933 vsize=9558408 CPUtime=0.01 cores=0,2,4,6
/proc/6935/task/6949/stat : 6949 (java) S 6933 6935 6935 0 -1 4202560 332 0 0 0 1 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405781801528 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6935/tid=6950] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6950/stat : 6950 (java) S 6933 6935 6935 0 -1 4202560 511 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405780748984 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6935/tid=6951] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6951/stat : 6951 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405779696632 240568546364 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6935/tid=6952] ppid=6933 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/6935/task/6952/stat : 6952 (java) S 6933 6935 6935 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338355039 9787809792 6131 33554432000 1073741824 1073778376 140737355795904 140405778644296 240568547241 0 4 0 16800975 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6665/tid=6672] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6672/stat : 6672 (python) S 6663 6665 2614 0 -1 4202560 24 1331003 0 0 0 0 4903 995 20 0 10 0 338274904 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594361922096 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6680] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6680/stat : 6680 (python) S 6663 6665 2614 0 -1 4202560 19 1331003 0 0 0 0 4903 995 20 0 10 0 338279865 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594351432240 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6769] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6769/stat : 6769 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338299414 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594340942384 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6777] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6777/stat : 6777 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338304286 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594330452528 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6784] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6784/stat : 6784 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338309151 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594319962672 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6880] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6880/stat : 6880 (python) S 6663 6665 2614 0 -1 4202560 22 1331003 0 0 0 0 4903 995 20 0 10 0 338321175 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594309472816 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6665/tid=6899] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6899/stat : 6899 (python) S 6663 6665 2614 0 -1 4202560 23 1331003 0 0 0 0 4903 995 20 0 10 0 338340721 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594298982960 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6665/tid=6907] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6907/stat : 6907 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338345499 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594288493104 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6665/tid=6934] ppid=6663 vsize=1218244 CPUtime=58.98 cores=0,2,4,6
/proc/6665/task/6934/stat : 6934 (python) S 6663 6665 2614 0 -1 4202560 21 1331003 0 0 0 0 4903 995 20 0 10 0 338355033 1247481856 71922 33554432000 4194304 5683644 140735961428720 140594278003248 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 491.88
Current children cumulated vsize (KiB) 23654060

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 6665 and gives
#  childrusage.ru_utime.tv_sec=141
#  childrusage.ru_utime.tv_usec=883430
#  childrusage.ru_stime.tv_sec=10
#  childrusage.ru_stime.tv_usec=538397
# CPU time returned by wait4() is 152.422
# while last known CPU time is 491.88
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 824.175
CPU time (s): 491.88
CPU user time (s): 475.65
CPU system time (s): 16.23
CPU usage (%): 59.6815
Max. virtual memory (cumulated for all children) (KiB): 23654060

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 141.883
system time used= 10.5384
maximum resident set size= 4140556
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1411341
page faults= 1
swaps= 0
block input operations= 0
block output operations= 147184
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1957483
involuntary context switches= 311

runsolver used 2.81157 second user time and 6.73498 second system time

The end

Launcher Data 

Begin job on node146 at 2012-06-03 13:04:36
IDJOB=3741176
IDBENCH=89800
IDSOLVER=2327
FILE ID=node146/3741176-1338721474
RUNJOBID= node146-1338713700-2630
PBS_JOBID= 14636739
Free space on /tmp= 70740 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-122.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3741176-1338721474/watcher-3741176-1338721474 -o /tmp/evaluation-result-3741176-1338721474/solver-3741176-1338721474 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741176-1338721474.opb -seed 1824488862 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 45533d52c524a74641c715d4320b890b
RANDOM SEED=1824488862

node146.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        26489044 kB
Buffers:          260036 kB
Cached:           681564 kB
SwapCached:         1792 kB
Active:          5253132 kB
Inactive:         302712 kB
Active(anon):    4614628 kB
Inactive(anon):     1736 kB
Active(file):     638504 kB
Inactive(file):   300976 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67099888 kB
Dirty:            151412 kB
Writeback:             0 kB
AnonPages:       4620156 kB
Mapped:            19332 kB
Shmem:                72 kB
Slab:             674240 kB
SReclaimable:     112036 kB
SUnreclaim:       562204 kB
KernelStack:        1592 kB
PageTables:        17080 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    5323620 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:    157696 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 62752 MiB
End job on node146 at 2012-06-03 13:18:23