Trace number 3742096

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 615.22 612.865

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K93.opb
MD5SUM91412677f9785e1c6465ad7d3d170f82
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark510.634
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables6324
Total number of constraints6512
Number of constraints which are clauses6419
Number of constraints which are cardinality constraints (but not clauses)93
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint93
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 94
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.42/1.21	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.42/1.21	c INFO: solving HOME/instance-3742096-1338723239.opb
0.42/1.28	c INFO: parsing took 0.07 s
0.62/1.41	c INFO: features took 0.13 CPU seconds
0.62/1.43	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.68)
50.07/50.82	c INFO: running clasp-1.3.7@0 for 45 with 1750 remaining (b = 0.36)
97.22/97.99	c INFO: running scip-2.0.1-spx@0 for 45 with 1703 remaining (b = 0.06)
146.28/146.98	c INFO: running clasp-1.3.7@47 for 45 with 1655 remaining (b = 0.01)
192.17/192.72	c INFO: wrote minion input file to HOME/tmpSjYdWO.minion
192.17/192.72	c INFO: running minion-0.12@0 for 45 with 1609 remaining (b = 0.09)
239.61/240.10	c INFO: running wbo1.4a@0 for 45 with 1562 remaining (b = 0.02)
287.87/288.38	c INFO: running scip-2.0.1-clp@48 for 45 with 1514 remaining (b = 0.01)
334.34/334.70	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1468 remaining (b = 0.04)
384.24/382.24	c INFO: running clasp-1.3.7@92 for 136 with 1418 remaining (b = 0.02)
521.87/519.65	c INFO: running scip-2.0.1-spx@48 for 45 with 1281 remaining (b = 0.01)
567.15/564.80	c INFO: running bsolo_pb10-l3@0 for 45 with 1236 remaining (b = 0.02)
614.61/612.20	c INFO: running bsolo_pb10-l1@0 for 91 with 1188 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742096-1338723239/watcher-3742096-1338723239 -o /tmp/evaluation-result-3742096-1338723239/solver-3742096-1338723239 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742096-1338723239.opb -seed 99247840 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.98 2.04 2.21 3/217 2206
/proc/meminfo: memFree=25455492/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/2206/stat : 2206 (python) D 2204 2206 27000 0 -1 4202496 672 0 0 0 0 0 0 0 20 0 1 0 338451443 15642624 559 33554432000 4194304 5683644 140736514721504 140736514689448 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 3819 559 324 364 0 261 0

[startup+0.044531 s]
/proc/loadavg: 1.98 2.04 2.21 3/217 2206
/proc/meminfo: memFree=25455492/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=16440 CPUtime=0 cores=1,3,5,7
/proc/2206/stat : 2206 (python) R 2204 2206 27000 0 -1 4202496 1057 0 0 0 0 0 0 0 20 0 1 0 338451443 16834560 884 33554432000 4194304 5683644 140736514721504 140736514650504 4545378 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 4110 884 379 364 0 552 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 16440

[startup+0.100212 s]
/proc/loadavg: 1.98 2.04 2.21 3/217 2206
/proc/meminfo: memFree=25455492/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=19088 CPUtime=0.03 cores=1,3,5,7
/proc/2206/stat : 2206 (python) R 2204 2206 27000 0 -1 4202496 1399 0 0 0 2 1 0 0 20 0 1 0 338451443 19546112 1153 33554432000 4194304 5683644 140736514721504 140736514652008 4510807 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 4772 1153 430 364 0 696 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 19088

[startup+0.300272 s]
/proc/loadavg: 1.98 2.04 2.21 3/217 2206
/proc/meminfo: memFree=25455492/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=102112 CPUtime=0.11 cores=1,3,5,7
/proc/2206/stat : 2206 (python) R 2204 2206 27000 0 -1 4202496 4143 1010 1 0 8 3 0 0 20 0 1 0 338451443 104562688 2732 33554432000 4194304 5683644 140736514721504 140736514562472 219514004407 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 25520 2733 876 364 0 1730 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 102112

[startup+0.700217 s]
/proc/loadavg: 1.98 2.04 2.21 3/217 2206
/proc/meminfo: memFree=25455492/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=162444 CPUtime=0.22 cores=1,3,5,7
/proc/2206/stat : 2206 (python) D 2204 2206 27000 0 -1 4202496 6440 1010 1 0 15 7 0 0 20 0 1 0 338451443 166342656 4254 33554432000 4194304 5683644 140736514721504 140736514612008 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 40611 4254 1192 364 0 3256 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 162444

[startup+1.50013 s]
/proc/loadavg: 1.98 2.04 2.21 2/218 2211
/proc/meminfo: memFree=25434252/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=359724 CPUtime=0.65 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 11537 1010 1 0 53 12 0 0 20 0 2 0 338451443 368357376 7805 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 89931 7805 1901 364 0 25154 0
Current children cumulated CPU time (s) 0.65
Current children cumulated vsize (KiB) 359724

[startup+3.1004 s]
/proc/loadavg: 1.98 2.04 2.21 3/221 2214
/proc/meminfo: memFree=25356664/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=359724 CPUtime=0.65 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 11537 1010 1 0 53 12 0 0 20 0 2 0 338451443 368357376 7805 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 89931 7805 1901 364 0 25154 0
[pid=2212] ppid=2206 vsize=283944 CPUtime=0 cores=1,3,5,7
/proc/2212/stat : 2212 (python) S 2206 2206 27000 0 -1 4202560 764 0 0 0 0 0 0 0 20 0 1 0 338451587 290758656 6239 33554432000 4194304 5683644 140736514721504 140736514712592 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2212/statm: 70986 6239 341 364 0 6209 0
[pid=2214] ppid=2212 vsize=96188 CPUtime=1.64 cores=1,3,5,7
/proc/2214/stat : 2214 (scip-2.0.1.linu) R 2212 2214 2214 0 -1 4202496 21142 0 0 0 160 4 0 0 20 0 1 0 338451587 98496512 20542 33554432000 4194304 15243407 140733527380352 140733527376648 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2214/statm: 24047 20542 682 2698 0 21342 0
[pid=2206/tid=2213] ppid=2204 vsize=359724 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338451587 368357376 7805 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 2.29
Current children cumulated vsize (KiB) 739856
heavy processes:

[startup+6.30043 s]
/proc/loadavg: 2.06 2.06 2.21 3/221 2214
/proc/meminfo: memFree=25283384/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=359724 CPUtime=0.65 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 11537 1010 1 0 53 12 0 0 20 0 2 0 338451443 368357376 7805 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 89931 7805 1901 364 0 25154 0
[pid=2212] ppid=2206 vsize=283944 CPUtime=0 cores=1,3,5,7
/proc/2212/stat : 2212 (python) S 2206 2206 27000 0 -1 4202560 891 0 0 0 0 0 0 0 20 0 1 0 338451587 290758656 6262 33554432000 4194304 5683644 140736514721504 140736514712592 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2212/statm: 70986 6262 364 364 0 6209 0
[pid=2214] ppid=2212 vsize=159172 CPUtime=4.85 cores=1,3,5,7
/proc/2214/stat : 2214 (scip-2.0.1.linu) R 2212 2214 2214 0 -1 4202496 36819 0 0 0 474 11 0 0 20 0 1 0 338451587 162992128 36219 33554432000 4194304 15243407 140733527380352 140733527376648 4865653 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2214/statm: 39793 36219 686 2698 0 37088 0
[pid=2206/tid=2213] ppid=2204 vsize=359724 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338451587 368357376 7805 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 5.5
Current children cumulated vsize (KiB) 802840

[startup+12.7004 s]
/proc/loadavg: 2.05 2.06 2.21 3/221 2214
/proc/meminfo: memFree=25227424/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=359724 CPUtime=0.65 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 11537 1010 1 0 53 12 0 0 20 0 2 0 338451443 368357376 7805 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 89931 7805 1901 364 0 25154 0
[pid=2212] ppid=2206 vsize=283944 CPUtime=0.01 cores=1,3,5,7
/proc/2212/stat : 2212 (python) S 2206 2206 27000 0 -1 4202560 896 0 0 0 0 1 0 0 20 0 1 0 338451587 290758656 6262 33554432000 4194304 5683644 140736514721504 140736514712592 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2212/statm: 70986 6262 364 364 0 6209 0
[pid=2214] ppid=2212 vsize=212252 CPUtime=11.24 cores=1,3,5,7
/proc/2214/stat : 2214 (scip-2.0.1.linu) R 2212 2214 2214 0 -1 4202496 50073 0 0 0 1105 19 0 0 20 0 1 0 338451587 217346048 49473 33554432000 4194304 15243407 140733527380352 140733527376648 4880915 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2214/statm: 53063 49473 686 2698 0 50358 0
[pid=2206/tid=2213] ppid=2204 vsize=359724 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338451587 368357376 7805 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 11.9
Current children cumulated vsize (KiB) 855920

[startup+25.5072 s]
/proc/loadavg: 2.04 2.05 2.21 3/222 2215
/proc/meminfo: memFree=25187156/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=359724 CPUtime=0.65 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 11537 1010 1 0 53 12 0 0 20 0 2 0 338451443 368357376 7805 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 89931 7805 1901 364 0 25154 0
[pid=2212] ppid=2206 vsize=283944 CPUtime=0.04 cores=1,3,5,7
/proc/2212/stat : 2212 (python) S 2206 2206 27000 0 -1 4202560 1005 0 0 0 2 2 0 0 20 0 1 0 338451587 290758656 6267 33554432000 4194304 5683644 140736514721504 140736514712592 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2212/statm: 70986 6267 369 364 0 6209 0
[pid=2214] ppid=2212 vsize=272072 CPUtime=24.05 cores=1,3,5,7
/proc/2214/stat : 2214 (scip-2.0.1.linu) R 2212 2214 2214 0 -1 4202496 64700 0 0 0 2362 43 0 0 20 0 1 0 338451587 278601728 64066 33554432000 4194304 15243407 140733527380352 140733527376440 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2214/statm: 68018 64066 873 2698 0 65313 0
[pid=2206/tid=2213] ppid=2204 vsize=359724 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 2 0 338451587 368357376 7805 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 24.74
Current children cumulated vsize (KiB) 915740

[startup+51.1004 s]
/proc/loadavg: 2.03 2.05 2.20 3/222 2215
/proc/meminfo: memFree=25161180/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=435516 CPUtime=0.68 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 12411 1010 1 0 55 13 0 0 20 0 3 0 338451443 445968384 7830 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 108879 7830 1904 364 0 44099 0
[pid=2212] ppid=2206 vsize=359724 CPUtime=0.08 cores=1,3,5,7
/proc/2212/stat : 2212 (python) S 2206 2206 27000 0 -1 4202560 1157 0 0 0 5 3 0 0 20 0 2 0 338451587 368357376 6302 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2212/statm: 89931 6302 397 364 0 25154 0
[pid=2214] ppid=2212 vsize=296300 CPUtime=49.33 cores=1,3,5,7
/proc/2214/stat : 2214 (scip-2.0.1.linu) T 2212 2214 2214 0 -1 4202496 77446 0 0 0 4855 78 0 0 20 0 1 0 338451587 303411200 69667 33554432000 4194304 15243407 140733527380352 140733527375064 8135333 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/2214/statm: 74075 69667 898 2698 0 71370 0
[pid=2206/tid=2213] ppid=2204 vsize=435516 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 3 0 338451587 445968384 7830 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 50.09
Current children cumulated vsize (KiB) 1091540

[startup+102.301 s]
/proc/loadavg: 2.05 2.05 2.19 3/230 2223
/proc/meminfo: memFree=25010824/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=511308 CPUtime=0.7 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 13250 1010 1 0 57 13 0 0 20 0 4 0 338451443 523579392 7838 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 127827 7838 1906 364 0 63044 0
[pid=2212] ppid=2206 vsize=359724 CPUtime=0.08 cores=1,3,5,7
/proc/2212/stat : 2212 (python) S 2206 2206 27000 0 -1 4202560 1157 0 0 0 5 3 0 0 20 0 2 0 338451587 368357376 6302 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2212/statm: 89931 6302 397 364 0 25154 0
[pid=2214] ppid=2212 vsize=296300 CPUtime=49.33 cores=1,3,5,7
/proc/2214/stat : 2214 (scip-2.0.1.linu) T 2212 2214 2214 0 -1 4202496 77446 0 0 0 4855 78 0 0 20 0 1 0 338451587 303411200 69667 33554432000 4194304 15243407 140733527380352 140733527375064 8135333 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/2214/statm: 74075 69667 898 2698 0 71370 0
[pid=2212/tid=2216] ppid=2206 vsize=359724 CPUtime=0 cores=1,3,5,7
/proc/2212/task/2216/stat : 2216 (python) S 2206 2206 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338456523 368357376 6302 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2217] ppid=2206 vsize=359736 CPUtime=0.08 cores=1,3,5,7
/proc/2217/stat : 2217 (python) S 2206 2206 27000 0 -1 4202560 1080 0 0 0 5 3 0 0 20 0 2 0 338456526 368369664 6305 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2217/statm: 89934 6305 384 364 0 25154 0
[pid=2219] ppid=2217 vsize=16484 CPUtime=47.12 cores=1,3,5,7
/proc/2219/stat : 2219 (clasp-1.3.7-x86) T 2217 2219 2219 0 -1 4202496 4200 0 0 0 4711 1 0 0 20 0 1 0 338456526 16879616 3851 33554432000 134512640 136139019 4290857632 4290856496 134608891 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2219/statm: 4121 3851 182 398 0 3721 0
[pid=2217/tid=2220] ppid=2206 vsize=359736 CPUtime=0 cores=1,3,5,7
/proc/2217/task/2220/stat : 2220 (python) S 2206 2206 27000 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338461240 368369664 6305 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2221] ppid=2206 vsize=435528 CPUtime=0 cores=1,3,5,7
/proc/2221/stat : 2221 (python) S 2206 2206 27000 0 -1 4202560 920 0 0 0 0 0 0 0 20 0 1 0 338461242 445980672 6284 33554432000 4194304 5683644 140736514721504 140736514712592 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2221/statm: 108882 6284 358 364 0 44099 0
[pid=2223] ppid=2221 vsize=153860 CPUtime=4.3 cores=1,3,5,7
/proc/2223/stat : 2223 (scip-2.0.1.linu) R 2221 2223 2223 0 -1 4202496 35732 0 0 0 421 9 0 0 20 0 1 0 338461243 157552640 35116 33554432000 4194304 13684479 140733603312016 140733603308328 4552016 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2223/statm: 38465 35116 639 2317 0 36143 0
[pid=2206/tid=2213] ppid=2204 vsize=511308 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 27 1010 0 0 0 0 0 0 20 0 4 0 338451587 523579392 7838 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2206/tid=2218] ppid=2204 vsize=511308 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2218/stat : 2218 (python) S 2204 2206 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 4 0 338456526 523579392 7838 33554432000 4194304 5683644 140736514721504 140230454157872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2222] ppid=2204 vsize=511308 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2222/stat : 2222 (python) S 2204 2206 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 4 0 338461242 523579392 7838 33554432000 4194304 5683644 140736514721504 140230443668016 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.61
Current children cumulated vsize (KiB) 2132940

[startup+162.301 s]
/proc/loadavg: 2.09 2.06 2.18 3/231 2224
/proc/meminfo: memFree=24865448/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=511308 CPUtime=0.73 cores=1,3,5,7

################
# More data... #
################

[pid=2292/tid=2320] ppid=2206 vsize=663460 CPUtime=0 cores=1,3,5,7
/proc/2292/task/2320/stat : 2320 (python) S 2206 2206 27000 0 -1 4202560 41 0 0 0 0 0 0 0 20 0 2 0 338489665 679383040 6946 33554432000 4194304 5683644 140736514721504 140230212444720 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2331] ppid=2206 vsize=739252 CPUtime=0.02 cores=1,3,5,7
/proc/2331/stat : 2331 (python) S 2206 2206 27000 0 -1 4202560 1001 0 0 0 1 1 0 0 20 0 1 0 338507924 756994048 6927 33554432000 4194304 5683644 140736514721504 140736514712592 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2331/statm: 184813 6927 359 364 0 120018 0
[pid=2333] ppid=2331 vsize=55552 CPUtime=17.47 cores=1,3,5,7
/proc/2333/stat : 2333 (bsolo_pb10) R 2331 2333 2333 0 -1 4202496 11936 0 0 0 1739 8 0 0 20 0 1 0 338507925 56885248 11466 33554432000 134512640 137138936 4292554880 4292553304 134606839 0 0 16781312 16384 0 0 0 17 1 0 0 0 0 0
/proc/2333/statm: 13888 11466 192 642 0 13244 0
[pid=2206/tid=2213] ppid=2204 vsize=815032 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 32 1010 0 0 0 0 0 0 20 0 8 0 338451587 834592768 8488 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2206/tid=2218] ppid=2204 vsize=815032 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2218/stat : 2218 (python) S 2204 2206 27000 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 8 0 338456526 834592768 8488 33554432000 4194304 5683644 140736514721504 140230454157872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2206/tid=2222] ppid=2204 vsize=815032 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2222/stat : 2222 (python) S 2204 2206 27000 0 -1 4202560 25 1010 0 0 0 0 0 0 20 0 8 0 338461242 834592768 8488 33554432000 4194304 5683644 140736514721504 140230443668016 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2206/tid=2226] ppid=2204 vsize=815032 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2226/stat : 2226 (python) S 2204 2206 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338470716 834592768 8488 33554432000 4194304 5683644 140736514721504 140230433178160 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2234] ppid=2204 vsize=815032 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2234/stat : 2234 (python) S 2204 2206 27000 0 -1 4202560 19 1010 0 0 0 0 0 0 20 0 8 0 338475454 834592768 8488 33554432000 4194304 5683644 140736514721504 140230212444720 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2293] ppid=2204 vsize=815032 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2293/stat : 2293 (python) S 2204 2206 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 8 0 338484914 834592768 8488 33554432000 4194304 5683644 140736514721504 140230201954864 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2332] ppid=2204 vsize=815032 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2332/stat : 2332 (python) S 2204 2206 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 8 0 338507924 834592768 8488 33554432000 4194304 5683644 140736514721504 140230191465008 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 584.65
Current children cumulated vsize (KiB) 14949380



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+612.802 s]
/proc/loadavg: 2.00 2.03 2.12 3/267 2358
/proc/meminfo: memFree=23407196/32873648 swapFree=7120/7120
[pid=2206] ppid=2204 vsize=890824 CPUtime=1.51 cores=1,3,5,7
/proc/2206/stat : 2206 (python) S 2204 2206 27000 0 -1 4202496 18393 1010 1 0 136 15 0 0 20 0 9 0 338451443 912203776 8496 33554432000 4194304 5683644 140736514721504 140736514716336 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2206/statm: 222706 8496 1916 364 0 157908 0
[pid=2212] ppid=2206 vsize=359724 CPUtime=0.16 cores=1,3,5,7
/proc/2212/stat : 2212 (python) S 2206 2206 27000 0 -1 4202560 1200 0 0 0 9 7 0 0 20 0 2 0 338451587 368357376 6303 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2212/statm: 89931 6303 398 364 0 25154 0
[pid=2214] ppid=2212 vsize=299736 CPUtime=95.62 cores=1,3,5,7
/proc/2214/stat : 2214 (scip-2.0.1.linu) T 2212 2214 2214 0 -1 4202496 90894 0 0 0 9473 89 0 0 20 0 1 0 338451587 306929664 72168 33554432000 4194304 15243407 140733527380352 140733527375064 8118528 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/2214/statm: 74934 72168 946 2698 0 72229 0
[pid=2212/tid=2216] ppid=2206 vsize=359724 CPUtime=0 cores=1,3,5,7
/proc/2212/task/2216/stat : 2216 (python) S 2206 2206 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338456523 368357376 6303 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2217] ppid=2206 vsize=359736 CPUtime=0.41 cores=1,3,5,7
/proc/2217/stat : 2217 (python) S 2206 2206 27000 0 -1 4202560 1172 0 0 0 24 17 0 0 20 0 2 0 338456526 368369664 6305 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2217/statm: 89934 6305 384 364 0 25154 0
[pid=2219] ppid=2217 vsize=27560 CPUtime=229.61 cores=1,3,5,7
/proc/2219/stat : 2219 (clasp-1.3.7-x86) T 2217 2219 2219 0 -1 4202496 7048 0 0 0 22960 1 0 0 20 0 1 0 338456526 28221440 6646 33554432000 134512640 136139019 4290857632 4290856612 134883821 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2219/statm: 6890 6646 182 398 0 6490 0
[pid=2217/tid=2220] ppid=2206 vsize=359736 CPUtime=0 cores=1,3,5,7
/proc/2217/task/2220/stat : 2220 (python) S 2206 2206 27000 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338461240 368369664 6305 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2221] ppid=2206 vsize=435528 CPUtime=0.17 cores=1,3,5,7
/proc/2221/stat : 2221 (python) S 2206 2206 27000 0 -1 4202560 1149 0 0 0 10 7 0 0 20 0 2 0 338461242 445980672 6311 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2221/statm: 108882 6311 384 364 0 44099 0
[pid=2223] ppid=2221 vsize=279084 CPUtime=94.07 cores=1,3,5,7
/proc/2223/stat : 2223 (scip-2.0.1.linu) T 2221 2223 2223 0 -1 4202496 72978 0 0 0 9369 38 0 0 20 0 1 0 338461243 285782016 67512 33554432000 4194304 13684479 140733603312016 140733603303928 7010159 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2223/statm: 69771 67512 758 2317 0 67449 0
[pid=2221/tid=2224] ppid=2206 vsize=435528 CPUtime=0 cores=1,3,5,7
/proc/2221/task/2224/stat : 2224 (python) S 2206 2206 27000 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338466139 445980672 6311 33554432000 4194304 5683644 140736514721504 140230454157872 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2225] ppid=2206 vsize=511876 CPUtime=0.07 cores=1,3,5,7
/proc/2225/stat : 2225 (python) S 2206 2206 27000 0 -1 4202560 1137 0 0 0 4 3 0 0 20 0 2 0 338470716 524161024 6427 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2225/statm: 127969 6427 384 364 0 63183 0
[pid=2227] ppid=2225 vsize=87064 CPUtime=47.33 cores=1,3,5,7
/proc/2227/stat : 2227 (minion) T 2225 2227 2227 0 -1 4202496 8149 0 0 0 4732 1 0 0 20 0 1 0 338470716 89153536 14482 33554432000 134512640 165565024 4291334752 4291331920 134641486 0 0 16781312 8396802 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2227/statm: 21766 14482 336 7582 0 14181 0
[pid=2225/tid=2232] ppid=2206 vsize=511876 CPUtime=0 cores=1,3,5,7
/proc/2225/task/2232/stat : 2232 (python) S 2206 2206 27000 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338475452 524161024 6427 33554432000 4194304 5683644 140736514721504 140230443668016 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2233] ppid=2206 vsize=587668 CPUtime=0.07 cores=1,3,5,7
/proc/2233/stat : 2233 (python) S 2206 2206 27000 0 -1 4202560 1128 0 0 0 4 3 0 0 20 0 2 0 338475454 601772032 6433 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2233/statm: 146917 6433 384 364 0 82128 0
[pid=2235] ppid=2233 vsize=103136 CPUtime=48.24 cores=1,3,5,7
/proc/2235/stat : 2235 (wbo1.4a) T 2233 2235 2235 0 -1 4202496 25663 0 0 0 4779 45 0 0 20 0 1 0 338475454 105611264 25304 33554432000 134512640 135751879 4289200528 4289198704 134616514 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2235/statm: 25784 25304 143 303 0 25479 0
[pid=2233/tid=2287] ppid=2206 vsize=587668 CPUtime=0 cores=1,3,5,7
/proc/2233/task/2287/stat : 2287 (python) S 2206 2206 27000 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338480279 601772032 6433 33554432000 4194304 5683644 140736514721504 140230433178160 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2292] ppid=2206 vsize=663460 CPUtime=0.08 cores=1,3,5,7
/proc/2292/stat : 2292 (python) S 2206 2206 27000 0 -1 4202560 1150 0 0 0 5 3 0 0 20 0 2 0 338484914 679383040 6946 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2292/statm: 165865 6946 384 364 0 101073 0
[pid=2294] ppid=2292 vsize=9624972 CPUtime=49.84 cores=1,3,5,7
/proc/2294/stat : 2294 (java) T 2292 2294 2294 0 -1 4202496 20663 0 1 0 4952 32 0 0 20 0 19 0 338484914 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140736016030240 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2294/statm: 2406243 269477 2266 9 0 2396937 0
[pid=2294/tid=2295] ppid=2292 vsize=9624972 CPUtime=47.18 cores=1,3,5,7
/proc/2294/task/2295/stat : 2295 (java) T 2292 2294 2294 0 -1 4202560 3250 0 1 0 4702 16 0 0 20 0 19 0 338484915 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555256937584 140555103979838 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2296] ppid=2292 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2294/task/2296/stat : 2296 (java) T 2292 2294 2294 0 -1 4202560 80 0 0 0 6 2 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555229391144 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2297] ppid=2292 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/2294/task/2297/stat : 2297 (java) T 2292 2294 2294 0 -1 4202560 28 0 0 0 9 1 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555228338344 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2298] ppid=2292 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/2294/task/2298/stat : 2298 (java) T 2292 2294 2294 0 -1 4202560 84 0 0 0 9 2 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555227286056 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2299] ppid=2292 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/2294/task/2299/stat : 2299 (java) T 2292 2294 2294 0 -1 4202560 40 0 0 0 8 2 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555226233256 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2300] ppid=2292 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/2294/task/2300/stat : 2300 (java) T 2292 2294 2294 0 -1 4202560 36 0 0 0 9 1 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555225180968 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2301] ppid=2292 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/2294/task/2301/stat : 2301 (java) T 2292 2294 2294 0 -1 4202560 74 0 0 0 10 0 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555224128168 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2302] ppid=2292 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/2294/task/2302/stat : 2302 (java) T 2292 2294 2294 0 -1 4202560 30 0 0 0 9 2 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555223075880 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2303] ppid=2292 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/2294/task/2303/stat : 2303 (java) T 2292 2294 2294 0 -1 4202560 75 0 0 0 8 2 0 0 20 0 19 0 338484917 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140555222023080 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2304] ppid=2292 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2294/task/2304/stat : 2304 (java) T 2292 2294 2294 0 -1 4202560 439 0 0 0 0 0 0 0 20 0 19 0 338484918 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554610378984 219517335465 0 0 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2305] ppid=2292 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2294/task/2305/stat : 2305 (java) T 2292 2294 2294 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338484918 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554609325048 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2294/tid=2306] ppid=2292 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2294/task/2306/stat : 2306 (java) T 2292 2294 2294 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338484918 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554608272648 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2307] ppid=2292 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2294/task/2307/stat : 2307 (java) T 2292 2294 2294 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338484921 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554607221136 219517343520 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2308] ppid=2292 vsize=9624972 CPUtime=0.9 cores=1,3,5,7
/proc/2294/task/2308/stat : 2308 (java) T 2292 2294 2294 0 -1 4202560 5506 0 0 0 90 0 0 0 20 0 19 0 338484921 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554606168760 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2309] ppid=2292 vsize=9624972 CPUtime=0.78 cores=1,3,5,7
/proc/2294/task/2309/stat : 2309 (java) T 2292 2294 2294 0 -1 4202560 9601 0 0 0 76 2 0 0 20 0 19 0 338484921 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554605115960 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2294/tid=2310] ppid=2292 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2294/task/2310/stat : 2310 (java) T 2292 2294 2294 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338484921 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554604063864 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2294/tid=2311] ppid=2292 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/2294/task/2311/stat : 2311 (java) T 2292 2294 2294 0 -1 4202560 3 0 0 0 0 1 0 0 20 0 19 0 338484921 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554603011272 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2294/tid=2312] ppid=2292 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2294/task/2312/stat : 2312 (java) T 2292 2294 2294 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338484953 9855971328 269477 33554432000 1073741824 1073778376 140736016039104 140554601956264 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2292/tid=2320] ppid=2206 vsize=663460 CPUtime=0 cores=1,3,5,7
/proc/2292/task/2320/stat : 2320 (python) S 2206 2206 27000 0 -1 4202560 41 0 0 0 0 0 0 0 20 0 2 0 338489665 679383040 6946 33554432000 4194304 5683644 140736514721504 140230212444720 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2331] ppid=2206 vsize=739252 CPUtime=0.09 cores=1,3,5,7
/proc/2331/stat : 2331 (python) S 2206 2206 27000 0 -1 4202560 1170 0 0 0 5 4 0 0 20 0 2 0 338507924 756994048 6953 33554432000 4194304 5683644 140736514721504 140736514713152 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2331/statm: 184813 6953 384 364 0 120018 0
[pid=2333] ppid=2331 vsize=106764 CPUtime=47.35 cores=1,3,5,7
/proc/2333/stat : 2333 (bsolo_pb10) T 2331 2333 2333 0 -1 4202496 24731 0 0 0 4722 13 0 0 20 0 1 0 338507925 109326336 24261 33554432000 134512640 137138936 4292554880 4292553808 134542495 0 0 16781312 16384 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2333/statm: 26691 24261 192 642 0 26047 0
[pid=2331/tid=2355] ppid=2206 vsize=739252 CPUtime=0 cores=1,3,5,7
/proc/2331/task/2355/stat : 2355 (python) S 2206 2206 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338512662 756994048 6953 33554432000 4194304 5683644 140736514721504 140230201954864 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2356] ppid=2206 vsize=815044 CPUtime=0 cores=1,3,5,7
/proc/2356/stat : 2356 (python) S 2206 2206 27000 0 -1 4202560 858 0 0 0 0 0 0 0 20 0 1 0 338512664 834605056 6910 33554432000 4194304 5683644 140736514721504 140736514712592 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/2356/statm: 203761 6910 336 364 0 138963 0
[pid=2358] ppid=2356 vsize=13032 CPUtime=0.6 cores=1,3,5,7
/proc/2358/stat : 2358 (bsolo_pb10) R 2356 2358 2358 0 -1 4202496 1181 0 0 0 60 0 0 0 20 0 1 0 338512664 13344768 815 33554432000 134512640 137138936 4293885760 4293808572 136570369 0 0 16781312 16384 0 0 0 17 1 0 0 0 0 0
/proc/2358/statm: 3258 815 184 642 0 2614 0
[pid=2206/tid=2213] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2213/stat : 2213 (python) S 2204 2206 27000 0 -1 4202560 32 1010 0 0 0 0 0 0 20 0 9 0 338451587 912203776 8496 33554432000 4194304 5683644 140736514721504 140230464647728 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2206/tid=2218] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2218/stat : 2218 (python) S 2204 2206 27000 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 9 0 338456526 912203776 8496 33554432000 4194304 5683644 140736514721504 140230454157872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2206/tid=2222] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2222/stat : 2222 (python) S 2204 2206 27000 0 -1 4202560 25 1010 0 0 0 0 0 0 20 0 9 0 338461242 912203776 8496 33554432000 4194304 5683644 140736514721504 140230443668016 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2206/tid=2226] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2226/stat : 2226 (python) S 2204 2206 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338470716 912203776 8496 33554432000 4194304 5683644 140736514721504 140230433178160 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2234] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2234/stat : 2234 (python) S 2204 2206 27000 0 -1 4202560 19 1010 0 0 0 0 0 0 20 0 9 0 338475454 912203776 8496 33554432000 4194304 5683644 140736514721504 140230212444720 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2293] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2293/stat : 2293 (python) S 2204 2206 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338484914 912203776 8496 33554432000 4194304 5683644 140736514721504 140230201954864 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2332] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2332/stat : 2332 (python) S 2204 2206 27000 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 9 0 338507924 912203776 8496 33554432000 4194304 5683644 140736514721504 140230191465008 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2206/tid=2357] ppid=2204 vsize=890824 CPUtime=0 cores=1,3,5,7
/proc/2206/task/2357/stat : 2357 (python) S 2204 2206 27000 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338512664 912203776 8496 33554432000 4194304 5683644 140736514721504 140230180975152 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 615.22
Current children cumulated vsize (KiB) 15904460

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2206 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=369791
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=158975
# CPU time returned by wait4() is 1.52877
# while last known CPU time is 615.22
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 612.865
CPU time (s): 615.22
CPU user time (s): 612.44
CPU system time (s): 2.78
CPU usage (%): 100.384
Max. virtual memory (cumulated for all children) (KiB): 15904460

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.36979
system time used= 0.158975
maximum resident set size= 33984
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19910
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1544
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2089
involuntary context switches= 16

runsolver used 2.95155 second user time and 6.22405 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 13:34:00
IDJOB=3742096
IDBENCH=90131
IDSOLVER=2327
FILE ID=node114/3742096-1338723239
RUNJOBID= node114-1338713908-27017
PBS_JOBID= 14636748
Free space on /tmp= 70040 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K93.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742096-1338723239/watcher-3742096-1338723239 -o /tmp/evaluation-result-3742096-1338723239/solver-3742096-1338723239 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742096-1338723239.opb -seed 99247840 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 91412677f9785e1c6465ad7d3d170f82
RANDOM SEED=99247840

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        25455252 kB
Buffers:         1035472 kB
Cached:          3819308 kB
SwapCached:         2180 kB
Active:          4616036 kB
Inactive:        1773740 kB
Active(anon):    1519884 kB
Inactive(anon):    10732 kB
Active(file):    3096152 kB
Inactive(file):  1763008 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:            102212 kB
Writeback:             0 kB
AnonPages:       1545632 kB
Mapped:            34716 kB
Shmem:               184 kB
Slab:             870136 kB
SReclaimable:     312280 kB
SUnreclaim:       557856 kB
KernelStack:        1784 kB
PageTables:        11496 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    2404880 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:    872448 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 69940 MiB
End job on node114 at 2012-06-03 13:44:16