Trace number 3742616

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 1071.34 1068.17

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathBA/
normalized-BeauxArts_K65.opb
MD5SUMa4b90b74bc3d4958439bc17b9ac2ee03
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkUNSAT
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark312.531
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables3575
Total number of constraints3707
Number of constraints which are clauses3642
Number of constraints which are cardinality constraints (but not clauses)65
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint65
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 66
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.38/1.23	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.38/1.23	c INFO: solving HOME/instance-3742616-1338725324.opb
0.38/1.27	c INFO: parsing took 0.04 s
0.47/1.34	c INFO: features took 0.07 CPU seconds
0.47/1.37	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.72)
50.12/50.91	c INFO: running clasp-1.3.7@0 for 45 with 1750 remaining (b = 0.37)
97.28/98.06	c INFO: running scip-2.0.1-spx@0 for 45 with 1703 remaining (b = 0.07)
146.01/146.75	c INFO: running clasp-1.3.7@47 for 45 with 1655 remaining (b = 0.01)
191.57/192.22	c INFO: wrote minion input file to HOME/tmpclpebA.minion
191.57/192.22	c INFO: running minion-0.12@0 for 45 with 1610 remaining (b = 0.10)
238.81/239.49	c INFO: running pbct-0.1.2-linear@0 for 45 with 1562 remaining (b = 0.02)
290.18/290.70	c INFO: running scip-2.0.1-clp@49 for 45 with 1515 remaining (b = 0.01)
335.32/335.84	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1470 remaining (b = 0.05)
381.50/379.32	c INFO: running clasp-1.3.7@92 for 136 with 1424 remaining (b = 0.02)
519.14/516.72	c INFO: running scip-2.0.1-spx@48 for 45 with 1287 remaining (b = 0.01)
564.29/561.87	c INFO: running bsolo_pb10-l2@0 for 45 with 1242 remaining (b = 0.02)
612.05/609.52	c INFO: running scip-2.0.1-spx@93 for 410 with 1195 remaining (b = 0.06)
1023.36/1020.29	c INFO: running bsolo_pb10-l3@0 for 45 with 786 remaining (b = 0.01)
1071.13/1067.94	c INFO: running bsolo_pb10-l1@0 for 91 with 738 remaining (b = 0.01)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742616-1338725324/watcher-3742616-1338725324 -o /tmp/evaluation-result-3742616-1338725324/solver-3742616-1338725324 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742616-1338725324.opb -seed 63979392 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.92 1.98 1.99 3/209 9898
/proc/meminfo: memFree=27866704/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=15276 CPUtime=0 cores=0,2,4,6
/proc/9898/stat : 9898 (python) D 9896 9898 3958 0 -1 4202496 634 0 0 0 0 0 0 0 20 0 1 0 338658551 15642624 527 33554432000 4194304 5683644 140734773566144 140734773552056 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 3819 527 312 364 0 261 0

[startup+0.0591769 s]
/proc/loadavg: 1.92 1.98 1.99 3/209 9898
/proc/meminfo: memFree=27866704/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=16592 CPUtime=0.01 cores=0,2,4,6
/proc/9898/stat : 9898 (python) D 9896 9898 3958 0 -1 4202496 1187 0 0 0 1 0 0 0 20 0 1 0 338658551 16990208 994 33554432000 4194304 5683644 140734773566144 140734773520360 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 4148 994 417 364 0 590 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16592

[startup+0.100217 s]
/proc/loadavg: 1.92 1.98 1.99 3/209 9898
/proc/meminfo: memFree=27866704/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=19084 CPUtime=0.02 cores=0,2,4,6
/proc/9898/stat : 9898 (python) R 9896 9898 3958 0 -1 4202496 1395 0 0 0 2 0 0 0 20 0 1 0 338658551 19542016 1149 33554432000 4194304 5683644 140734773566144 140734773496648 4475920 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 4771 1149 430 364 0 695 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19084

[startup+0.300158 s]
/proc/loadavg: 1.92 1.98 1.99 3/209 9898
/proc/meminfo: memFree=27866704/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=96916 CPUtime=0.11 cores=0,2,4,6
/proc/9898/stat : 9898 (python) D 9896 9898 3958 0 -1 4202496 3958 1008 1 0 9 2 0 0 20 0 1 0 338658551 99241984 2652 33554432000 4194304 5683644 140734773566144 140734773434504 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 24229 2652 823 364 0 1720 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 96916

[startup+0.700117 s]
/proc/loadavg: 1.92 1.98 1.99 3/209 9898
/proc/meminfo: memFree=27866704/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=136268 CPUtime=0.21 cores=0,2,4,6
/proc/9898/stat : 9898 (python) D 9896 9898 3958 0 -1 4202496 6235 1008 1 0 15 6 0 0 20 0 1 0 338658551 139538432 4073 33554432000 4194304 5683644 140734773566144 140734773466136 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 34067 4073 1093 364 0 2953 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 136268

[startup+1.50014 s]
/proc/loadavg: 1.92 1.98 1.99 3/210 9903
/proc/meminfo: memFree=27849576/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=359044 CPUtime=0.54 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 10936 1008 1 0 45 9 0 0 20 0 2 0 338658551 367661056 7616 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 89761 7616 1900 364 0 24984 0
Current children cumulated CPU time (s) 0.54
Current children cumulated vsize (KiB) 359044

[startup+3.10026 s]
/proc/loadavg: 2.01 2.00 2.00 3/213 9906
/proc/meminfo: memFree=27784792/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=359044 CPUtime=0.54 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 10936 1008 1 0 45 9 0 0 20 0 2 0 338658551 367661056 7616 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 89761 7616 1900 364 0 24984 0
[pid=9904] ppid=9898 vsize=283264 CPUtime=0 cores=0,2,4,6
/proc/9904/stat : 9904 (python) S 9898 9898 3958 0 -1 4202560 768 0 0 0 0 0 0 0 20 0 1 0 338658689 290062336 6051 33554432000 4194304 5683644 140734773566144 140734773557232 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9904/statm: 70816 6051 341 364 0 6039 0
[pid=9906] ppid=9904 vsize=81988 CPUtime=1.7 cores=0,2,4,6
/proc/9906/stat : 9906 (scip-2.0.1.linu) R 9904 9906 9906 0 -1 4202496 17997 0 0 0 164 6 0 0 20 0 1 0 338658689 83955712 17397 33554432000 4194304 15243407 140735251730608 140735251726920 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9906/statm: 20497 17397 693 2698 0 17792 0
[pid=9898/tid=9905] ppid=9896 vsize=359044 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9905/stat : 9905 (python) S 9896 9898 3958 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 2 0 338658689 367661056 7616 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 2.24
Current children cumulated vsize (KiB) 724296
heavy processes:

[startup+6.30027 s]
/proc/loadavg: 2.01 2.00 2.00 3/213 9906
/proc/meminfo: memFree=27729292/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=359044 CPUtime=0.54 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 10936 1008 1 0 45 9 0 0 20 0 2 0 338658551 367661056 7616 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 89761 7616 1900 364 0 24984 0
[pid=9904] ppid=9898 vsize=283264 CPUtime=0 cores=0,2,4,6
/proc/9904/stat : 9904 (python) S 9898 9898 3958 0 -1 4202560 889 0 0 0 0 0 0 0 20 0 1 0 338658689 290062336 6074 33554432000 4194304 5683644 140734773566144 140734773557232 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9904/statm: 70816 6074 364 364 0 6039 0
[pid=9906] ppid=9904 vsize=120084 CPUtime=4.9 cores=0,2,4,6
/proc/9906/stat : 9906 (scip-2.0.1.linu) R 9904 9906 9906 0 -1 4202496 27493 0 0 0 478 12 0 0 20 0 1 0 338658689 122966016 26893 33554432000 4194304 15243407 140735251730608 140735251726904 4520069 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9906/statm: 30021 26893 693 2698 0 27316 0
[pid=9898/tid=9905] ppid=9896 vsize=359044 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9905/stat : 9905 (python) S 9896 9898 3958 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 2 0 338658689 367661056 7616 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 5.44
Current children cumulated vsize (KiB) 762392

[startup+12.7003 s]
/proc/loadavg: 2.00 2.00 2.00 3/213 9906
/proc/meminfo: memFree=27694248/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=359044 CPUtime=0.54 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 10936 1008 1 0 45 9 0 0 20 0 2 0 338658551 367661056 7616 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 89761 7616 1900 364 0 24984 0
[pid=9904] ppid=9898 vsize=283264 CPUtime=0.01 cores=0,2,4,6
/proc/9904/stat : 9904 (python) S 9898 9898 3958 0 -1 4202560 993 0 0 0 1 0 0 0 20 0 1 0 338658689 290062336 6079 33554432000 4194304 5683644 140734773566144 140734773557232 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9904/statm: 70816 6079 369 364 0 6039 0
[pid=9906] ppid=9904 vsize=141300 CPUtime=11.3 cores=0,2,4,6
/proc/9906/stat : 9906 (scip-2.0.1.linu) R 9904 9906 9906 0 -1 4202496 32576 0 0 0 1100 30 0 0 20 0 1 0 338658689 144691200 31952 33554432000 4194304 15243407 140735251730608 140735251726296 6388706 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9906/statm: 35325 31952 888 2698 0 32620 0
[pid=9898/tid=9905] ppid=9896 vsize=359044 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9905/stat : 9905 (python) S 9896 9898 3958 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 2 0 338658689 367661056 7616 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 11.85
Current children cumulated vsize (KiB) 783608

[startup+25.5003 s]
/proc/loadavg: 2.00 2.00 2.00 3/214 9907
/proc/meminfo: memFree=27698252/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=359044 CPUtime=0.54 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 10936 1008 1 0 45 9 0 0 20 0 2 0 338658551 367661056 7616 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 89761 7616 1900 364 0 24984 0
[pid=9904] ppid=9898 vsize=283264 CPUtime=0.03 cores=0,2,4,6
/proc/9904/stat : 9904 (python) S 9898 9898 3958 0 -1 4202560 1001 0 0 0 2 1 0 0 20 0 1 0 338658689 290062336 6079 33554432000 4194304 5683644 140734773566144 140734773557232 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9904/statm: 70816 6079 369 364 0 6039 0
[pid=9906] ppid=9904 vsize=147208 CPUtime=24.1 cores=0,2,4,6
/proc/9906/stat : 9906 (scip-2.0.1.linu) R 9904 9906 9906 0 -1 4202496 41814 0 0 0 2375 35 0 0 20 0 1 0 338658689 150740992 32530 33554432000 4194304 15243407 140735251730608 140735251725000 8113576 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9906/statm: 36802 32530 949 2698 0 34097 0
[pid=9898/tid=9905] ppid=9896 vsize=359044 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9905/stat : 9905 (python) S 9896 9898 3958 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 2 0 338658689 367661056 7616 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 24.67
Current children cumulated vsize (KiB) 789516

[startup+51.1004 s]
/proc/loadavg: 2.00 2.00 2.00 3/214 9907
/proc/meminfo: memFree=27696772/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=435340 CPUtime=0.58 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 11867 1008 1 0 48 10 0 0 20 0 3 0 338658551 445788160 8259 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 108835 8259 1903 364 0 44055 0
[pid=9904] ppid=9898 vsize=359044 CPUtime=0.08 cores=0,2,4,6
/proc/9904/stat : 9904 (python) S 9898 9898 3958 0 -1 4202560 1152 0 0 0 5 3 0 0 20 0 2 0 338658689 367661056 6114 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9904/statm: 89761 6114 397 364 0 24984 0
[pid=9906] ppid=9904 vsize=146716 CPUtime=49.49 cores=0,2,4,6
/proc/9906/stat : 9906 (scip-2.0.1.linu) T 9904 9906 9906 0 -1 4202496 59105 0 0 0 4907 42 0 0 20 0 1 0 338658689 150237184 32627 33554432000 4194304 15243407 140735251730608 140735251725000 8134546 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/9906/statm: 36679 32627 949 2698 0 33974 0
[pid=9898/tid=9905] ppid=9896 vsize=435340 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9905/stat : 9905 (python) S 9896 9898 3958 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 3 0 338658689 445788160 8259 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 50.15
Current children cumulated vsize (KiB) 941100

[startup+102.301 s]
/proc/loadavg: 2.11 2.03 2.01 3/192 9986
/proc/meminfo: memFree=31009284/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=511132 CPUtime=0.6 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 12797 1008 1 0 50 10 0 0 20 0 4 0 338658551 523399168 8267 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 127783 8267 1905 364 0 63000 0
[pid=9904] ppid=9898 vsize=359044 CPUtime=0.08 cores=0,2,4,6
/proc/9904/stat : 9904 (python) S 9898 9898 3958 0 -1 4202560 1152 0 0 0 5 3 0 0 20 0 2 0 338658689 367661056 6114 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9904/statm: 89761 6114 397 364 0 24984 0
[pid=9906] ppid=9904 vsize=146716 CPUtime=49.49 cores=0,2,4,6
/proc/9906/stat : 9906 (scip-2.0.1.linu) T 9904 9906 9906 0 -1 4202496 59105 0 0 0 4907 42 0 0 20 0 1 0 338658689 150237184 32627 33554432000 4194304 15243407 140735251730608 140735251725000 8134546 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/9906/statm: 36679 32627 949 2698 0 33974 0
[pid=9904/tid=9908] ppid=9898 vsize=359044 CPUtime=0 cores=0,2,4,6
/proc/9904/task/9908/stat : 9908 (python) S 9898 9898 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338663640 367661056 6114 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9909] ppid=9898 vsize=359560 CPUtime=0.07 cores=0,2,4,6
/proc/9909/stat : 9909 (python) S 9898 9898 3958 0 -1 4202560 1078 0 0 0 4 3 0 0 20 0 2 0 338663643 368189440 6227 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9909/statm: 89890 6227 384 364 0 25110 0
[pid=9911] ppid=9909 vsize=8932 CPUtime=47.11 cores=0,2,4,6
/proc/9911/stat : 9911 (clasp-1.3.7-x86) T 9909 9911 9911 0 -1 4202496 2255 0 0 0 4711 0 0 0 20 0 1 0 338663643 9146368 1977 33554432000 134512640 136139019 4290158224 4290157200 134883787 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/9911/statm: 2233 1977 182 398 0 1833 0
[pid=9909/tid=9983] ppid=9898 vsize=359560 CPUtime=0 cores=0,2,4,6
/proc/9909/task/9983/stat : 9983 (python) S 9898 9898 3958 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338668355 368189440 6227 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9984] ppid=9898 vsize=435352 CPUtime=0 cores=0,2,4,6
/proc/9984/stat : 9984 (python) S 9898 9898 3958 0 -1 4202560 924 0 0 0 0 0 0 0 20 0 1 0 338668358 445800448 6715 33554432000 4194304 5683644 140734773566144 140734773557232 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9984/statm: 108838 6715 359 364 0 44055 0
[pid=9986] ppid=9984 vsize=111716 CPUtime=4.21 cores=0,2,4,6
/proc/9986/stat : 9986 (scip-2.0.1.linu) R 9984 9986 9986 0 -1 4202496 25776 0 0 0 410 11 0 0 20 0 1 0 338668358 114397184 25161 33554432000 4194304 13684479 140735850646192 140735850642504 11876362 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9986/statm: 27929 25161 646 2317 0 25607 0
[pid=9898/tid=9905] ppid=9896 vsize=511132 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9905/stat : 9905 (python) S 9896 9898 3958 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 4 0 338658689 523399168 8267 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=9910] ppid=9896 vsize=511132 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9910/stat : 9910 (python) S 9896 9898 3958 0 -1 4202560 18 1008 0 0 0 0 0 0 20 0 4 0 338663643 523399168 8267 33554432000 4194304 5683644 140734773566144 140711651435056 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9898/tid=9985] ppid=9896 vsize=511132 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9985/stat : 9985 (python) S 9896 9898 3958 0 -1 4202560 21 1008 0 0 0 0 0 0 20 0 4 0 338668358 523399168 8267 33554432000 4194304 5683644 140734773566144 140711640945200 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 101.56
Current children cumulated vsize (KiB) 1932452

[startup+162.3 s]
/proc/loadavg: 2.04 2.02 2.00 3/192 10037
/proc/meminfo: memFree=30940696/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=511132 CPUtime=0.62 cores=0,2,4,6

################
# More data... #
################

[pid=9898/tid=9985] ppid=9896 vsize=890092 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9985/stat : 9985 (python) S 9896 9898 3958 0 -1 4202560 29 1008 0 0 0 0 0 0 20 0 9 0 338668358 911454208 8820 33554432000 4194304 5683644 140734773566144 140711640945200 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=10039] ppid=9896 vsize=890092 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10039/stat : 10039 (python) S 9896 9898 3958 0 -1 4202560 21 1008 0 0 0 0 0 0 20 0 9 0 338677773 911454208 8820 33554432000 4194304 5683644 140734773566144 140711630455344 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9898/tid=10093] ppid=9896 vsize=890092 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10093/stat : 10093 (python) S 9896 9898 3958 0 -1 4202560 21 1008 0 0 0 0 0 0 20 0 9 0 338682501 911454208 8820 33554432000 4194304 5683644 140734773566144 140711619965488 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9898/tid=10106] ppid=9896 vsize=890092 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10106/stat : 10106 (python) S 9896 9898 3958 0 -1 4202560 22 1008 0 0 0 0 0 0 20 0 9 0 338692136 911454208 8820 33554432000 4194304 5683644 140734773566144 140711609475632 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=10159] ppid=9896 vsize=890092 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10159/stat : 10159 (python) S 9896 9898 3958 0 -1 4202560 22 1008 0 0 0 0 0 0 20 0 9 0 338714739 911454208 8820 33554432000 4194304 5683644 140734773566144 140711598985776 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=10172] ppid=9896 vsize=890092 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10172/stat : 10172 (python) S 9896 9898 3958 0 -1 4202560 20 1008 0 0 0 0 0 0 20 0 9 0 338760581 911454208 8820 33554432000 4194304 5683644 140734773566144 140711248781872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1065.52
Current children cumulated vsize (KiB) 15638748



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1068.1 s]
/proc/loadavg: 2.00 2.01 2.00 3/277 10178
/proc/meminfo: memFree=27954356/32873648 swapFree=7628/7628
[pid=9898] ppid=9896 vsize=965884 CPUtime=1.16 cores=0,2,4,6
/proc/9898/stat : 9898 (python) S 9896 9898 3958 0 -1 4202496 18914 1008 1 0 103 13 0 0 20 0 10 0 338658551 989065216 8828 33554432000 4194304 5683644 140734773566144 140734773560976 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9898/statm: 241471 8828 1917 364 0 176670 0
[pid=9904] ppid=9898 vsize=359044 CPUtime=0.15 cores=0,2,4,6
/proc/9904/stat : 9904 (python) S 9898 9898 3958 0 -1 4202560 1189 0 0 0 9 6 0 0 20 0 2 0 338658689 367661056 6115 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9904/statm: 89761 6115 398 364 0 24984 0
[pid=9906] ppid=9904 vsize=145308 CPUtime=94.59 cores=0,2,4,6
/proc/9906/stat : 9906 (scip-2.0.1.linu) T 9904 9906 9906 0 -1 4202496 71244 0 0 0 9409 50 0 0 20 0 1 0 338658689 148795392 33239 33554432000 4194304 15243407 140735251730608 140735251725000 8134473 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/9906/statm: 36327 33239 950 2698 0 33622 0
[pid=9904/tid=9908] ppid=9898 vsize=359044 CPUtime=0 cores=0,2,4,6
/proc/9904/task/9908/stat : 9908 (python) S 9898 9898 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338663640 367661056 6115 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9909] ppid=9898 vsize=359560 CPUtime=0.39 cores=0,2,4,6
/proc/9909/stat : 9909 (python) S 9898 9898 3958 0 -1 4202560 1171 0 0 0 20 19 0 0 20 0 2 0 338663643 368189440 6227 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9909/statm: 89890 6227 384 364 0 25110 0
[pid=9911] ppid=9909 vsize=13924 CPUtime=229.58 cores=0,2,4,6
/proc/9911/stat : 9911 (clasp-1.3.7-x86) T 9909 9911 9911 0 -1 4202496 3524 0 0 0 22957 1 0 0 20 0 1 0 338663643 14258176 3235 33554432000 134512640 136139019 4290158224 4290156848 134599025 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/9911/statm: 3481 3235 182 398 0 3081 0
[pid=9909/tid=9983] ppid=9898 vsize=359560 CPUtime=0 cores=0,2,4,6
/proc/9909/task/9983/stat : 9983 (python) S 9898 9898 3958 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338668355 368189440 6227 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9984] ppid=9898 vsize=435352 CPUtime=0.9 cores=0,2,4,6
/proc/9984/stat : 9984 (python) S 9898 9898 3958 0 -1 4202560 1195 0 0 0 50 40 0 0 20 0 2 0 338668358 445800448 6740 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9984/statm: 108838 6740 384 364 0 44055 0
[pid=9986] ppid=9984 vsize=163208 CPUtime=504.39 cores=0,2,4,6
/proc/9986/stat : 9986 (scip-2.0.1.linu) T 9984 9986 9986 0 -1 4202496 39461 0 0 0 50202 237 0 0 20 0 1 0 338668358 167124992 38848 33554432000 4194304 13684479 140735850646192 140735850640040 7361269 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/9986/statm: 40802 38848 815 2317 0 38480 0
[pid=9984/tid=10037] ppid=9898 vsize=435352 CPUtime=0 cores=0,2,4,6
/proc/9984/task/10037/stat : 10037 (python) S 9898 9898 3958 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338673224 445800448 6740 33554432000 4194304 5683644 140734773566144 140711651435056 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=10038] ppid=9898 vsize=511144 CPUtime=0.07 cores=0,2,4,6
/proc/10038/stat : 10038 (python) S 9898 9898 3958 0 -1 4202560 1124 0 0 0 4 3 0 0 20 0 2 0 338677773 523411456 6747 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/10038/statm: 127786 6747 384 364 0 63000 0
[pid=10040] ppid=10038 vsize=79056 CPUtime=47.23 cores=0,2,4,6
/proc/10040/stat : 10040 (minion) T 10038 10040 10040 0 -1 4202496 5807 0 0 0 4719 4 0 0 20 0 1 0 338677774 80953344 12467 33554432000 134512640 165565024 4293918608 4293915856 142764968 0 0 16781312 8396802 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/10040/statm: 19764 12467 336 7582 0 12179 0
[pid=10038/tid=10091] ppid=9898 vsize=511144 CPUtime=0 cores=0,2,4,6
/proc/10038/task/10091/stat : 10091 (python) S 9898 9898 3958 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338682498 523411456 6747 33554432000 4194304 5683644 140734773566144 140711640945200 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=10092] ppid=9898 vsize=586936 CPUtime=0.08 cores=0,2,4,6
/proc/10092/stat : 10092 (python) S 9898 9898 3958 0 -1 4202560 1114 0 0 0 5 3 0 0 20 0 2 0 338682501 601022464 6753 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/10092/statm: 146734 6753 384 364 0 81945 0
[pid=10094] ppid=10092 vsize=47788 CPUtime=51.16 cores=0,2,4,6
/proc/10094/stat : 10094 (pbct-0.1.2-linu) T 10092 10094 10094 0 -1 4202496 12073 0 0 0 4680 436 0 0 20 0 1 0 338682501 48934912 11516 33554432000 134512640 136422188 4288613104 4288610624 135006571 0 0 16781312 16387 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/10094/statm: 11947 11516 214 467 0 11478 0
[pid=10092/tid=10100] ppid=9898 vsize=586936 CPUtime=0 cores=0,2,4,6
/proc/10092/task/10100/stat : 10100 (python) S 9898 9898 3958 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338687619 601022464 6753 33554432000 4194304 5683644 140734773566144 140711630455344 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=10105] ppid=9898 vsize=662728 CPUtime=0.07 cores=0,2,4,6
/proc/10105/stat : 10105 (python) S 9898 9898 3958 0 -1 4202560 1135 0 0 0 4 3 0 0 20 0 2 0 338692136 678633472 6764 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/10105/statm: 165682 6764 384 364 0 100890 0
[pid=10107] ppid=10105 vsize=9624972 CPUtime=46.05 cores=0,2,4,6
/proc/10107/stat : 10107 (java) T 10105 10107 10107 0 -1 4202496 23315 0 1 0 4573 32 0 0 20 0 19 0 338692136 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140735784013840 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/10107/statm: 2406243 281286 2267 9 0 2396937 0
[pid=10107/tid=10108] ppid=10105 vsize=9624972 CPUtime=43.03 cores=0,2,4,6
/proc/10107/task/10108/stat : 10108 (java) T 10105 10107 10107 0 -1 4202560 3564 0 1 0 4287 16 0 0 20 0 19 0 338692137 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355826871392 140355656343971 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10109] ppid=10105 vsize=9624972 CPUtime=0.15 cores=0,2,4,6
/proc/10107/task/10109/stat : 10109 (java) T 10105 10107 10107 0 -1 4202560 154 0 0 0 13 2 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355788364456 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10110] ppid=10105 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/10107/task/10110/stat : 10110 (java) T 10105 10107 10107 0 -1 4202560 94 0 0 0 10 2 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355787311912 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10111] ppid=10105 vsize=9624972 CPUtime=0.13 cores=0,2,4,6
/proc/10107/task/10111/stat : 10111 (java) T 10105 10107 10107 0 -1 4202560 45 0 0 0 11 2 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355786259368 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10112] ppid=10105 vsize=9624972 CPUtime=0.15 cores=0,2,4,6
/proc/10107/task/10112/stat : 10112 (java) T 10105 10107 10107 0 -1 4202560 180 0 0 0 13 2 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355785206824 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10113] ppid=10105 vsize=9624972 CPUtime=0.11 cores=0,2,4,6
/proc/10107/task/10113/stat : 10113 (java) T 10105 10107 10107 0 -1 4202560 44 0 0 0 11 0 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355784153256 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10114] ppid=10105 vsize=9624972 CPUtime=0.15 cores=0,2,4,6
/proc/10107/task/10114/stat : 10114 (java) T 10105 10107 10107 0 -1 4202560 35 0 0 0 13 2 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355783100712 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10115] ppid=10105 vsize=9624972 CPUtime=0.16 cores=0,2,4,6
/proc/10107/task/10115/stat : 10115 (java) T 10105 10107 10107 0 -1 4202560 118 0 0 0 14 2 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355782048168 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=10107/tid=10116] ppid=10105 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/10107/task/10116/stat : 10116 (java) T 10105 10107 10107 0 -1 4202560 69 0 0 0 10 2 0 0 20 0 19 0 338692138 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355780995624 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10117] ppid=10105 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/10107/task/10117/stat : 10117 (java) T 10105 10107 10107 0 -1 4202560 577 0 0 0 0 0 0 0 20 0 19 0 338692139 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355650427496 217413892009 0 0 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=10107/tid=10118] ppid=10105 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/10107/task/10118/stat : 10118 (java) T 10105 10107 10107 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338692140 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355649373816 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10119] ppid=10105 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/10107/task/10119/stat : 10119 (java) T 10105 10107 10107 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338692140 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355648321160 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10120] ppid=10105 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/10107/task/10120/stat : 10120 (java) T 10105 10107 10107 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338692142 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355647269904 217413900064 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=10107/tid=10121] ppid=10105 vsize=9624972 CPUtime=1.04 cores=0,2,4,6
/proc/10107/task/10121/stat : 10121 (java) T 10105 10107 10107 0 -1 4202560 6682 0 0 0 102 2 0 0 20 0 19 0 338692142 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355646216248 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10122] ppid=10105 vsize=9624972 CPUtime=0.75 cores=0,2,4,6
/proc/10107/task/10122/stat : 10122 (java) T 10105 10107 10107 0 -1 4202560 10327 0 0 0 73 2 0 0 20 0 19 0 338692142 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355645163704 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10123] ppid=10105 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/10107/task/10123/stat : 10123 (java) T 10105 10107 10107 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338692142 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355644111352 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10107/tid=10124] ppid=10105 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/10107/task/10124/stat : 10124 (java) T 10105 10107 10107 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338692142 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355643059016 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=10107/tid=10125] ppid=10105 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/10107/task/10125/stat : 10125 (java) T 10105 10107 10107 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338692170 9855971328 281286 33554432000 1073741824 1073778376 140735784022704 140355642004776 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=10105/tid=10127] ppid=9898 vsize=662728 CPUtime=0 cores=0,2,4,6
/proc/10105/task/10127/stat : 10127 (python) S 9898 9898 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338696481 678633472 6764 33554432000 4194304 5683644 140734773566144 140711619965488 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=10158] ppid=9898 vsize=738520 CPUtime=0.07 cores=0,2,4,6
/proc/10158/stat : 10158 (python) S 9898 9898 3958 0 -1 4202560 1156 0 0 0 4 3 0 0 20 0 2 0 338714739 756244480 6770 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/10158/statm: 184630 6770 384 364 0 119835 0
[pid=10160] ppid=10158 vsize=124772 CPUtime=47.59 cores=0,2,4,6
/proc/10160/stat : 10160 (bsolo_pb10) T 10158 10160 10160 0 -1 4202496 29096 0 0 0 4742 17 0 0 20 0 1 0 338714739 127766528 28761 33554432000 134512640 137138936 4294783248 4294782176 134542199 0 0 16781312 16384 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/10160/statm: 31193 28761 192 642 0 30549 0
[pid=10158/tid=10162] ppid=9898 vsize=738520 CPUtime=0 cores=0,2,4,6
/proc/10158/task/10162/stat : 10162 (python) S 9898 9898 3958 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338719501 756244480 6770 33554432000 4194304 5683644 140734773566144 140711609475632 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=10171] ppid=9898 vsize=814312 CPUtime=0.09 cores=0,2,4,6
/proc/10171/stat : 10171 (python) S 9898 9898 3958 0 -1 4202560 1213 0 0 0 5 4 0 0 20 0 2 0 338760580 833855488 6782 33554432000 4194304 5683644 140734773566144 140734773557792 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/10171/statm: 203578 6782 390 364 0 138780 0
[pid=10173] ppid=10171 vsize=86944 CPUtime=47.61 cores=0,2,4,6
/proc/10173/stat : 10173 (bsolo_pb10) T 10171 10173 10173 0 -1 4202496 19636 0 0 0 4754 7 0 0 20 0 1 0 338760581 89030656 19300 33554432000 134512640 137138936 4289844416 4289843344 134542286 0 0 16781312 16384 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/10173/statm: 21736 19300 192 642 0 21092 0
[pid=10171/tid=10175] ppid=9898 vsize=814312 CPUtime=0 cores=0,2,4,6
/proc/10171/task/10175/stat : 10175 (python) S 9898 9898 3958 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338765343 833855488 6782 33554432000 4194304 5683644 140734773566144 140711598985776 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=10176] ppid=9898 vsize=890104 CPUtime=0 cores=0,2,4,6
/proc/10176/stat : 10176 (python) S 9898 9898 3958 0 -1 4202560 886 0 0 0 0 0 0 0 20 0 1 0 338765345 911466496 7248 33554432000 4194304 5683644 140734773566144 140734773557232 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 4 0 0 0 0 0
/proc/10176/statm: 222526 7248 343 364 0 157725 0
[pid=10178] ppid=10176 vsize=12032 CPUtime=0.16 cores=0,2,4,6
/proc/10178/stat : 10178 (bsolo_pb10) R 10176 10178 10178 0 -1 4202496 880 0 0 0 16 0 0 0 20 0 1 0 338765345 12320768 556 33554432000 134512640 137138936 4289453520 4289409308 136570369 0 0 16781312 16384 0 0 0 17 0 0 0 0 0 0
/proc/10178/statm: 3008 556 184 642 0 2364 0
[pid=9898/tid=9905] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9905/stat : 9905 (python) S 9896 9898 3958 0 -1 4202560 31 1008 0 0 0 0 0 0 20 0 10 0 338658689 989065216 8828 33554432000 4194304 5683644 140734773566144 140711739277872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=9910] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9910/stat : 9910 (python) S 9896 9898 3958 0 -1 4202560 23 1008 0 0 0 0 0 0 20 0 10 0 338663643 989065216 8828 33554432000 4194304 5683644 140734773566144 140711651435056 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9898/tid=9985] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/9985/stat : 9985 (python) S 9896 9898 3958 0 -1 4202560 29 1008 0 0 0 0 0 0 20 0 10 0 338668358 989065216 8828 33554432000 4194304 5683644 140734773566144 140711640945200 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=10039] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10039/stat : 10039 (python) S 9896 9898 3958 0 -1 4202560 21 1008 0 0 0 0 0 0 20 0 10 0 338677773 989065216 8828 33554432000 4194304 5683644 140734773566144 140711630455344 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9898/tid=10093] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10093/stat : 10093 (python) S 9896 9898 3958 0 -1 4202560 21 1008 0 0 0 0 0 0 20 0 10 0 338682501 989065216 8828 33554432000 4194304 5683644 140734773566144 140711619965488 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9898/tid=10106] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10106/stat : 10106 (python) S 9896 9898 3958 0 -1 4202560 22 1008 0 0 0 0 0 0 20 0 10 0 338692136 989065216 8828 33554432000 4194304 5683644 140734773566144 140711609475632 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=10159] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10159/stat : 10159 (python) S 9896 9898 3958 0 -1 4202560 22 1008 0 0 0 0 0 0 20 0 10 0 338714739 989065216 8828 33554432000 4194304 5683644 140734773566144 140711598985776 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=10172] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10172/stat : 10172 (python) S 9896 9898 3958 0 -1 4202560 20 1008 0 0 0 0 0 0 20 0 10 0 338760581 989065216 8828 33554432000 4194304 5683644 140734773566144 140711248781872 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9898/tid=10177] ppid=9896 vsize=965884 CPUtime=0 cores=0,2,4,6
/proc/9898/task/10177/stat : 10177 (python) S 9896 9898 3958 0 -1 4202560 20 1008 0 0 0 0 0 0 20 0 10 0 338765345 989065216 8828 33554432000 4194304 5683644 140734773566144 140711238292016 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1071.34
Current children cumulated vsize (KiB) 16621588

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 9898 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=46840
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=138978
# CPU time returned by wait4() is 1.18582
# while last known CPU time is 1071.34
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1068.17
CPU time (s): 1071.34
CPU user time (s): 1062.56
CPU system time (s): 8.78
CPU usage (%): 100.297
Max. virtual memory (cumulated for all children) (KiB): 16621588

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.04684
system time used= 0.138978
maximum resident set size= 35312
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 20388
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1056
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2116
involuntary context switches= 27

runsolver used 5.37018 second user time and 12.1232 second system time

The end

Launcher Data 

Begin job on node122 at 2012-06-03 14:08:44
IDJOB=3742616
IDBENCH=90217
IDSOLVER=2327
FILE ID=node122/3742616-1338725324
RUNJOBID= node122-1338713392-3974
PBS_JOBID= 14636732
Free space on /tmp= 71084 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathBA/normalized-BeauxArts_K65.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742616-1338725324/watcher-3742616-1338725324 -o /tmp/evaluation-result-3742616-1338725324/solver-3742616-1338725324 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742616-1338725324.opb -seed 63979392 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a4b90b74bc3d4958439bc17b9ac2ee03
RANDOM SEED=63979392

node122.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5438.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        27866580 kB
Buffers:          213672 kB
Cached:           377108 kB
SwapCached:         1832 kB
Active:          4113988 kB
Inactive:          42332 kB
Active(anon):    3561016 kB
Inactive(anon):     7108 kB
Active(file):     552972 kB
Inactive(file):    35224 kB
Unevictable:        7628 kB
Mlocked:            7628 kB
SwapTotal:      67108856 kB
SwapFree:       67100168 kB
Dirty:            102124 kB
Writeback:            16 kB
AnonPages:       3584080 kB
Mapped:            29388 kB
Shmem:                60 kB
Slab:             688712 kB
SReclaimable:     131380 kB
SUnreclaim:       557332 kB
KernelStack:        1712 kB
PageTables:        14196 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    4967440 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2105344 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70984 MiB
End job on node122 at 2012-06-03 14:26:36