Trace number 3742005

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 611.38 609.464

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K106.opb
MD5SUMafe4117ba13dcfbcd06fd88e1353402a
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark143.74
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables7208
Total number of constraints7422
Number of constraints which are clauses7316
Number of constraints which are cardinality constraints (but not clauses)106
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint106
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 107
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.35/1.27	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.35/1.27	c INFO: solving HOME/instance-3742005-1338722627.opb
0.42/1.35	c INFO: parsing took 0.08 s
0.52/1.49	c INFO: features took 0.15 CPU seconds
0.62/1.52	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.67)
51.26/52.16	c INFO: running clasp-1.3.7@0 for 45 with 1749 remaining (b = 0.36)
98.52/99.32	c INFO: running scip-2.0.1-spx@0 for 45 with 1702 remaining (b = 0.06)
147.48/148.28	c INFO: running clasp-1.3.7@47 for 45 with 1654 remaining (b = 0.01)
193.37/194.09	c INFO: wrote minion input file to HOME/tmpOsE3w1.minion
193.37/194.09	c INFO: running minion-0.12@0 for 45 with 1608 remaining (b = 0.09)
240.92/241.50	c INFO: running wbo1.4a@0 for 45 with 1560 remaining (b = 0.02)
288.34/288.95	c INFO: running scip-2.0.1-clp@49 for 45 with 1514 remaining (b = 0.01)
331.11/331.64	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1471 remaining (b = 0.03)
380.84/379.20	c INFO: running clasp-1.3.7@92 for 136 with 1422 remaining (b = 0.02)
518.46/516.62	c INFO: running scip-2.0.1-spx@48 for 45 with 1285 remaining (b = 0.01)
563.62/561.78	c INFO: running bsolo_pb10-l3@0 for 45 with 1240 remaining (b = 0.02)
611.28/609.38	c INFO: running bsolo_pb10-l1@0 for 91 with 1192 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742005-1338722627/watcher-3742005-1338722627 -o /tmp/evaluation-result-3742005-1338722627/solver-3742005-1338722627 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742005-1338722627.opb -seed 2139194399 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.15 2.13 2.40 2/217 2014
/proc/meminfo: memFree=24989336/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/2014/stat : 2014 (python) D 2012 2014 27000 0 -1 4202496 671 0 0 0 0 0 0 0 20 0 1 0 338390148 15642624 559 33554432000 4194304 5683644 140736701114928 140736701082872 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 3819 559 324 364 0 261 0

[startup+0.0956939 s]
/proc/loadavg: 2.15 2.13 2.40 2/217 2014
/proc/meminfo: memFree=24989336/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=19080 CPUtime=0.01 cores=1,3,5,7
/proc/2014/stat : 2014 (python) R 2012 2014 27000 0 -1 4202496 1383 0 0 0 1 0 0 0 20 0 1 0 338390148 19537920 1138 33554432000 4194304 5683644 140736701114928 140736701045432 219513589652 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 4770 1139 430 364 0 694 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 19080

[startup+0.100167 s]
/proc/loadavg: 2.15 2.13 2.40 2/217 2014
/proc/meminfo: memFree=24989336/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=19080 CPUtime=0.02 cores=1,3,5,7
/proc/2014/stat : 2014 (python) R 2012 2014 27000 0 -1 4202496 1393 0 0 0 2 0 0 0 20 0 1 0 338390148 19537920 1148 33554432000 4194304 5683644 140736701114928 140736701045432 4620161 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 4770 1148 430 364 0 694 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19080

[startup+0.300252 s]
/proc/loadavg: 2.15 2.13 2.40 2/217 2014
/proc/meminfo: memFree=24989336/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=96892 CPUtime=0.1 cores=1,3,5,7
/proc/2014/stat : 2014 (python) D 2012 2014 27000 0 -1 4202496 3947 1013 1 0 8 2 0 0 20 0 1 0 338390148 99217408 2648 33554432000 4194304 5683644 140736701114928 140736701001320 219513962165 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 24223 2648 823 364 0 1714 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 96892

[startup+0.700227 s]
/proc/loadavg: 2.15 2.13 2.40 2/217 2014
/proc/meminfo: memFree=24989336/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=130880 CPUtime=0.2 cores=1,3,5,7
/proc/2014/stat : 2014 (python) D 2012 2014 27000 0 -1 4202496 6122 1013 1 0 14 6 0 0 20 0 1 0 338390148 134021120 3981 33554432000 4194304 5683644 140736701114928 140736701019160 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 32720 3981 1073 364 0 2862 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 130880

[startup+1.50023 s]
/proc/loadavg: 2.14 2.13 2.40 3/218 2019
/proc/meminfo: memFree=24973068/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=284420 CPUtime=0.62 cores=1,3,5,7
/proc/2014/stat : 2014 (python) D 2012 2014 27000 0 -1 4202496 11229 1013 1 0 52 10 0 0 20 0 1 0 338390148 291246080 7870 33554432000 4194304 5683644 140736701114928 140736701078712 219513963408 0 0 16781312 33554434 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 71105 7870 1859 364 0 6334 0
Current children cumulated CPU time (s) 0.62
Current children cumulated vsize (KiB) 284420

[startup+3.10038 s]
/proc/loadavg: 2.14 2.13 2.40 3/221 2022
/proc/meminfo: memFree=24879416/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=360224 CPUtime=0.64 cores=1,3,5,7
/proc/2014/stat : 2014 (python) S 2012 2014 27000 0 -1 4202496 11487 1013 1 0 54 10 0 0 20 0 2 0 338390148 368869376 7910 33554432000 4194304 5683644 140736701114928 140736701109760 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 90056 7910 1893 364 0 25279 0
[pid=2020] ppid=2014 vsize=284444 CPUtime=0 cores=1,3,5,7
/proc/2020/stat : 2020 (python) S 2014 2014 27000 0 -1 4202560 764 0 0 0 0 0 0 0 20 0 1 0 338390301 291270656 6352 33554432000 4194304 5683644 140736701114928 140736701106016 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2020/statm: 71111 6352 341 364 0 6334 0
[pid=2022] ppid=2020 vsize=115544 CPUtime=1.56 cores=1,3,5,7
/proc/2022/stat : 2022 (scip-2.0.1.linu) R 2020 2022 2022 0 -1 4202496 24805 0 0 0 151 5 0 0 20 0 1 0 338390301 118317056 24203 33554432000 4194304 15243407 140737334936400 140737334931784 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2022/statm: 28886 24203 682 2698 0 26181 0
[pid=2014/tid=2021] ppid=2012 vsize=360224 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 27 1013 0 0 0 0 0 0 20 0 2 0 338390301 368869376 7910 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 2.2
Current children cumulated vsize (KiB) 760212
heavy processes:

[startup+6.30036 s]
/proc/loadavg: 2.13 2.13 2.39 3/221 2022
/proc/meminfo: memFree=24815140/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=360224 CPUtime=0.64 cores=1,3,5,7
/proc/2014/stat : 2014 (python) S 2012 2014 27000 0 -1 4202496 11487 1013 1 0 54 10 0 0 20 0 2 0 338390148 368869376 7910 33554432000 4194304 5683644 140736701114928 140736701109760 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 90056 7910 1893 364 0 25279 0
[pid=2020] ppid=2014 vsize=284444 CPUtime=0 cores=1,3,5,7
/proc/2020/stat : 2020 (python) S 2014 2014 27000 0 -1 4202560 897 0 0 0 0 0 0 0 20 0 1 0 338390301 291270656 6375 33554432000 4194304 5683644 140736701114928 140736701106016 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2020/statm: 71111 6375 364 364 0 6334 0
[pid=2022] ppid=2020 vsize=173432 CPUtime=4.76 cores=1,3,5,7
/proc/2022/stat : 2022 (scip-2.0.1.linu) R 2020 2022 2022 0 -1 4202496 39257 0 0 0 465 11 0 0 20 0 1 0 338390301 177594368 38655 33554432000 4194304 15243407 140737334936400 140737334932712 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2022/statm: 43358 38655 686 2698 0 40653 0
[pid=2014/tid=2021] ppid=2012 vsize=360224 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 27 1013 0 0 0 0 0 0 20 0 2 0 338390301 368869376 7910 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 5.4
Current children cumulated vsize (KiB) 818100

[startup+12.7004 s]
/proc/loadavg: 2.12 2.13 2.39 3/221 2022
/proc/meminfo: memFree=24746636/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=360224 CPUtime=0.64 cores=1,3,5,7
/proc/2014/stat : 2014 (python) S 2012 2014 27000 0 -1 4202496 11487 1013 1 0 54 10 0 0 20 0 2 0 338390148 368869376 7910 33554432000 4194304 5683644 140736701114928 140736701109760 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 90056 7910 1893 364 0 25279 0
[pid=2020] ppid=2014 vsize=284444 CPUtime=0 cores=1,3,5,7
/proc/2020/stat : 2020 (python) S 2014 2014 27000 0 -1 4202560 901 0 0 0 0 0 0 0 20 0 1 0 338390301 291270656 6375 33554432000 4194304 5683644 140736701114928 140736701106016 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2020/statm: 71111 6375 364 364 0 6334 0
[pid=2022] ppid=2020 vsize=231644 CPUtime=11.16 cores=1,3,5,7
/proc/2022/stat : 2022 (scip-2.0.1.linu) R 2020 2022 2022 0 -1 4202496 53747 0 0 0 1096 20 0 0 20 0 1 0 338390301 237203456 53145 33554432000 4194304 15243407 140737334936400 140737334932712 13229099 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2022/statm: 57911 53145 686 2698 0 55206 0
[pid=2014/tid=2021] ppid=2012 vsize=360224 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 27 1013 0 0 0 0 0 0 20 0 2 0 338390301 368869376 7910 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.8
Current children cumulated vsize (KiB) 876312

[startup+25.5005 s]
/proc/loadavg: 2.09 2.12 2.38 3/222 2023
/proc/meminfo: memFree=24637152/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=360224 CPUtime=0.64 cores=1,3,5,7
/proc/2014/stat : 2014 (python) S 2012 2014 27000 0 -1 4202496 11487 1013 1 0 54 10 0 0 20 0 2 0 338390148 368869376 7910 33554432000 4194304 5683644 140736701114928 140736701109760 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 90056 7910 1893 364 0 25279 0
[pid=2020] ppid=2014 vsize=284444 CPUtime=0.03 cores=1,3,5,7
/proc/2020/stat : 2020 (python) S 2014 2014 27000 0 -1 4202560 1001 0 0 0 2 1 0 0 20 0 1 0 338390301 291270656 6380 33554432000 4194304 5683644 140736701114928 140736701106016 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2020/statm: 71111 6380 369 364 0 6334 0
[pid=2022] ppid=2020 vsize=322272 CPUtime=23.96 cores=1,3,5,7
/proc/2022/stat : 2022 (scip-2.0.1.linu) R 2020 2022 2022 0 -1 4202496 76093 0 0 0 2360 36 0 0 20 0 1 0 338390301 330006528 75465 33554432000 4194304 15243407 140737334936400 140737334932488 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2022/statm: 80568 75465 873 2698 0 77863 0
[pid=2014/tid=2021] ppid=2012 vsize=360224 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 27 1013 0 0 0 0 0 0 20 0 2 0 338390301 368869376 7910 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.63
Current children cumulated vsize (KiB) 966940

[startup+51.1004 s]
/proc/loadavg: 2.06 2.11 2.37 3/222 2023
/proc/meminfo: memFree=24608508/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=360224 CPUtime=0.64 cores=1,3,5,7
/proc/2014/stat : 2014 (python) S 2012 2014 27000 0 -1 4202496 11487 1013 1 0 54 10 0 0 20 0 2 0 338390148 368869376 7910 33554432000 4194304 5683644 140736701114928 140736701109760 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 90056 7910 1893 364 0 25279 0
[pid=2020] ppid=2014 vsize=284444 CPUtime=0.07 cores=1,3,5,7
/proc/2020/stat : 2020 (python) S 2014 2014 27000 0 -1 4202560 1019 0 0 0 4 3 0 0 20 0 1 0 338390301 291270656 6380 33554432000 4194304 5683644 140736701114928 140736701106016 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2020/statm: 71111 6380 369 364 0 6334 0
[pid=2022] ppid=2020 vsize=350848 CPUtime=49.55 cores=1,3,5,7
/proc/2022/stat : 2022 (scip-2.0.1.linu) R 2020 2022 2022 0 -1 4202496 81950 0 0 0 4868 87 0 0 20 0 1 0 338390301 359268352 82344 33554432000 4194304 15243407 140737334936400 140737334931112 13354455 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2022/statm: 87712 82344 898 2698 0 85007 0
[pid=2014/tid=2021] ppid=2012 vsize=360224 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 27 1013 0 0 0 0 0 0 20 0 2 0 338390301 368869376 7910 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.26
Current children cumulated vsize (KiB) 995516

[startup+102.301 s]
/proc/loadavg: 2.06 2.10 2.35 3/230 2031
/proc/meminfo: memFree=24448112/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=511808 CPUtime=0.7 cores=1,3,5,7
/proc/2014/stat : 2014 (python) S 2012 2014 27000 0 -1 4202496 13284 1013 1 0 59 11 0 0 20 0 4 0 338390148 524091392 7935 33554432000 4194304 5683644 140736701114928 140736701109760 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 127952 7935 1906 364 0 63169 0
[pid=2020] ppid=2014 vsize=360224 CPUtime=0.07 cores=1,3,5,7
/proc/2020/stat : 2020 (python) S 2014 2014 27000 0 -1 4202560 1155 0 0 0 4 3 0 0 20 0 2 0 338390301 368869376 6415 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2020/statm: 90056 6415 397 364 0 25279 0
[pid=2022] ppid=2020 vsize=350848 CPUtime=50.59 cores=1,3,5,7
/proc/2022/stat : 2022 (scip-2.0.1.linu) T 2020 2022 2022 0 -1 4202496 81972 0 0 0 4972 87 0 0 20 0 1 0 338390301 359268352 82366 33554432000 4194304 15243407 140737334936400 140737334931112 8113519 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2022/statm: 87712 82366 898 2698 0 85007 0
[pid=2020/tid=2024] ppid=2014 vsize=360224 CPUtime=0 cores=1,3,5,7
/proc/2020/task/2024/stat : 2024 (python) S 2014 2014 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338395362 368869376 6415 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2025] ppid=2014 vsize=360236 CPUtime=0.07 cores=1,3,5,7
/proc/2025/stat : 2025 (python) S 2014 2014 27000 0 -1 4202560 1081 0 0 0 5 2 0 0 20 0 2 0 338395365 368881664 6402 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/2025/statm: 90059 6402 384 364 0 25279 0
[pid=2027] ppid=2025 vsize=18116 CPUtime=47.12 cores=1,3,5,7
/proc/2027/stat : 2027 (clasp-1.3.7-x86) T 2025 2027 2027 0 -1 4202496 4637 0 0 0 4711 1 0 0 20 0 1 0 338395365 18550784 4274 33554432000 134512640 136139019 4290572256 4290570880 134599831 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2027/statm: 4529 4274 182 398 0 4129 0
[pid=2025/tid=2028] ppid=2014 vsize=360236 CPUtime=0 cores=1,3,5,7
/proc/2025/task/2028/stat : 2028 (python) S 2014 2014 27000 0 -1 4202560 53 0 0 0 0 0 0 0 20 0 2 0 338400079 368881664 6402 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2029] ppid=2014 vsize=436028 CPUtime=0 cores=1,3,5,7
/proc/2029/stat : 2029 (python) S 2014 2014 27000 0 -1 4202560 804 0 0 0 0 0 0 0 20 0 1 0 338400081 446492672 6359 33554432000 4194304 5683644 140736701114928 140736701106016 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2029/statm: 109007 6359 336 364 0 44224 0
[pid=2031] ppid=2029 vsize=144428 CPUtime=2.96 cores=1,3,5,7
/proc/2031/stat : 2031 (scip-2.0.1.linu) R 2029 2031 2031 0 -1 4202496 32279 0 0 0 289 7 0 0 20 0 1 0 338400082 147894272 31662 33554432000 4194304 13684479 140734171913264 140734171909560 11876362 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2031/statm: 36107 31662 635 2317 0 33785 0
[pid=2014/tid=2021] ppid=2012 vsize=511808 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 27 1013 0 0 0 0 0 0 20 0 4 0 338390301 524091392 7935 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2026] ppid=2012 vsize=511808 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2026/stat : 2026 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 4 0 338395365 524091392 7935 33554432000 4194304 5683644 140736701114928 140483652022832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2030] ppid=2012 vsize=511808 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2030/stat : 2030 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 4 0 338400081 524091392 7935 33554432000 4194304 5683644 140736701114928 140483641532976 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.51
Current children cumulated vsize (KiB) 2181688

[startup+162.301 s]
/proc/loadavg: 2.02 2.08 2.33 3/232 2033
/proc/meminfo: memFree=24255680/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=511808 CPUtime=0.72 cores=1,3,5,7

################
# More data... #
################

/proc/2142/stat : 2142 (bsolo_pb10) T 2140 2142 2142 0 -1 4202496 25039 0 0 0 4741 14 0 0 20 0 1 0 338446328 110452736 24530 33554432000 134512640 137138936 4290247968 4290246896 134542770 0 0 16781312 16384 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2142/statm: 26966 24530 192 642 0 26322 0
[pid=2140/tid=2164] ppid=2014 vsize=739700 CPUtime=0 cores=1,3,5,7
/proc/2140/task/2164/stat : 2164 (python) S 2014 2014 27000 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338451085 757452800 7066 33554432000 4194304 5683644 140736701114928 140483403698736 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2165] ppid=2014 vsize=815492 CPUtime=0 cores=1,3,5,7
/proc/2165/stat : 2165 (python) S 2014 2014 27000 0 -1 4202560 848 0 0 0 0 0 0 0 20 0 1 0 338451087 835063808 7023 33554432000 4194304 5683644 140736701114928 140736701106016 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/2165/statm: 203873 7023 336 364 0 139075 0
[pid=2167] ppid=2165 vsize=12356 CPUtime=0.01 cores=1,3,5,7
/proc/2167/stat : 2167 (bsolo_pb10) R 2165 2167 2167 0 -1 4202496 859 0 0 0 1 0 0 0 20 0 1 0 338451087 12652544 592 33554432000 134512640 137138936 4287369744 4287367960 136506715 0 0 16781312 16384 0 0 0 17 1 0 0 0 0 0
/proc/2167/statm: 3089 592 241 642 0 2332 0
[pid=2014/tid=2021] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 32 1013 0 0 0 0 0 0 20 0 9 0 338390301 912662528 8609 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2014/tid=2026] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2026/stat : 2026 (python) S 2012 2014 27000 0 -1 4202560 30 1013 0 0 0 0 0 0 20 0 9 0 338395365 912662528 8609 33554432000 4194304 5683644 140736701114928 140483652022832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2014/tid=2030] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2030/stat : 2030 (python) S 2012 2014 27000 0 -1 4202560 25 1013 0 0 0 0 0 0 20 0 9 0 338400081 912662528 8609 33554432000 4194304 5683644 140736701114928 140483641532976 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2014/tid=2035] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2035/stat : 2035 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 9 0 338409558 912662528 8609 33554432000 4194304 5683644 140736701114928 140483631043120 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2043] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2043/stat : 2043 (python) S 2012 2014 27000 0 -1 4202560 19 1013 0 0 0 0 0 0 20 0 9 0 338414299 912662528 8609 33554432000 4194304 5683644 140736701114928 140483414188592 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2102] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2102/stat : 2102 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 9 0 338423313 912662528 8609 33554432000 4194304 5683644 140736701114928 140483403698736 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2141] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2141/stat : 2141 (python) S 2012 2014 27000 0 -1 4202560 22 1013 0 0 0 0 0 0 20 0 9 0 338446327 912662528 8609 33554432000 4194304 5683644 140736701114928 140483393208880 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2166] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2166/stat : 2166 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 9 0 338451087 912662528 8609 33554432000 4194304 5683644 140736701114928 140483382719024 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 611.38
Current children cumulated vsize (KiB) 16060592

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+609.402 s]
/proc/loadavg: 2.06 2.06 2.22 3/267 2167
/proc/meminfo: memFree=22713204/32873648 swapFree=7120/7120
[pid=2014] ppid=2012 vsize=891272 CPUtime=1.56 cores=1,3,5,7
/proc/2014/stat : 2014 (python) S 2012 2014 27000 0 -1 4202496 18471 1013 1 0 143 13 0 0 20 0 9 0 338390148 912662528 8609 33554432000 4194304 5683644 140736701114928 140736701109760 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2014/statm: 222818 8609 1916 364 0 158020 0
[pid=2020] ppid=2014 vsize=360224 CPUtime=0.15 cores=1,3,5,7
/proc/2020/stat : 2020 (python) S 2014 2014 27000 0 -1 4202560 1192 0 0 0 9 6 0 0 20 0 2 0 338390301 368869376 6416 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2020/statm: 90056 6416 398 364 0 25279 0
[pid=2022] ppid=2020 vsize=365324 CPUtime=93.24 cores=1,3,5,7
/proc/2022/stat : 2022 (scip-2.0.1.linu) T 2020 2022 2022 0 -1 4202496 107645 0 0 0 9228 96 0 0 20 0 1 0 338390301 374091776 88781 33554432000 4194304 15243407 140737334936400 140737334930792 8128837 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2022/statm: 91331 88781 946 2698 0 88626 0
[pid=2020/tid=2024] ppid=2014 vsize=360224 CPUtime=0 cores=1,3,5,7
/proc/2020/task/2024/stat : 2024 (python) S 2014 2014 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338395362 368869376 6416 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2025] ppid=2014 vsize=360236 CPUtime=0.41 cores=1,3,5,7
/proc/2025/stat : 2025 (python) S 2014 2014 27000 0 -1 4202560 1177 0 0 0 26 15 0 0 20 0 2 0 338395365 368881664 6402 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2025/statm: 90059 6402 384 364 0 25279 0
[pid=2027] ppid=2025 vsize=34764 CPUtime=229.63 cores=1,3,5,7
/proc/2027/stat : 2027 (clasp-1.3.7-x86) T 2025 2027 2027 0 -1 4202496 8779 0 0 0 22962 1 0 0 20 0 1 0 338395365 35598336 8419 33554432000 134512640 136139019 4290572256 4290571120 134608194 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2027/statm: 8691 8419 182 398 0 8291 0
[pid=2025/tid=2028] ppid=2014 vsize=360236 CPUtime=0 cores=1,3,5,7
/proc/2025/task/2028/stat : 2028 (python) S 2014 2014 27000 0 -1 4202560 53 0 0 0 0 0 0 0 20 0 2 0 338400079 368881664 6402 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=2029] ppid=2014 vsize=436028 CPUtime=0.18 cores=1,3,5,7
/proc/2029/stat : 2029 (python) S 2014 2014 27000 0 -1 4202560 1154 0 0 0 11 7 0 0 20 0 2 0 338400081 446492672 6408 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2029/statm: 109007 6408 384 364 0 44224 0
[pid=2031] ppid=2029 vsize=319832 CPUtime=94.03 cores=1,3,5,7
/proc/2031/stat : 2031 (scip-2.0.1.linu) T 2029 2031 2031 0 -1 4202496 82521 0 0 0 9350 53 0 0 20 0 1 0 338400082 327507968 75915 33554432000 4194304 13684479 140734171913264 140734171909352 11876362 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2031/statm: 79958 75915 775 2317 0 77636 0
[pid=2029/tid=2032] ppid=2014 vsize=436028 CPUtime=0 cores=1,3,5,7
/proc/2029/task/2032/stat : 2032 (python) S 2014 2014 27000 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338404974 446492672 6408 33554432000 4194304 5683644 140736701114928 140483652022832 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2034] ppid=2014 vsize=511820 CPUtime=0.07 cores=1,3,5,7
/proc/2034/stat : 2034 (python) S 2014 2014 27000 0 -1 4202560 1135 0 0 0 4 3 0 0 20 0 2 0 338409558 524103680 6414 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2034/statm: 127955 6414 384 364 0 63169 0
[pid=2036] ppid=2034 vsize=93632 CPUtime=47.36 cores=1,3,5,7
/proc/2036/stat : 2036 (minion) T 2034 2036 2036 0 -1 4202496 9821 0 0 0 4735 1 0 0 20 0 1 0 338409558 95879168 16102 33554432000 134512640 165565024 4289017008 4289014240 159737071 0 0 16781312 8396802 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2036/statm: 23408 16102 336 7582 0 15823 0
[pid=2034/tid=2041] ppid=2014 vsize=511820 CPUtime=0 cores=1,3,5,7
/proc/2034/task/2041/stat : 2041 (python) S 2014 2014 27000 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338414296 524103680 6414 33554432000 4194304 5683644 140736701114928 140483641532976 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2042] ppid=2014 vsize=588116 CPUtime=0.07 cores=1,3,5,7
/proc/2042/stat : 2042 (python) S 2014 2014 27000 0 -1 4202560 1145 0 0 0 4 3 0 0 20 0 2 0 338414299 602230784 6546 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2042/statm: 147029 6546 384 364 0 82240 0
[pid=2044] ppid=2042 vsize=135052 CPUtime=47.4 cores=1,3,5,7
/proc/2044/stat : 2044 (wbo1.4a) T 2042 2044 2044 0 -1 4202496 33684 0 0 0 4666 74 0 0 20 0 1 0 338414299 138293248 33310 33554432000 134512640 135751879 4289999248 4289997424 134616776 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2044/statm: 33763 33310 143 303 0 33458 0
[pid=2042/tid=2096] ppid=2014 vsize=588116 CPUtime=0 cores=1,3,5,7
/proc/2042/task/2096/stat : 2096 (python) S 2014 2014 27000 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338419042 602230784 6546 33554432000 4194304 5683644 140736701114928 140483631043120 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2101] ppid=2014 vsize=663908 CPUtime=0.08 cores=1,3,5,7
/proc/2101/stat : 2101 (python) S 2014 2014 27000 0 -1 4202560 1142 0 0 0 4 4 0 0 20 0 2 0 338423313 679841792 7059 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2101/statm: 165977 7059 384 364 0 101185 0
[pid=2103] ppid=2101 vsize=9624972 CPUtime=49.56 cores=1,3,5,7
/proc/2103/stat : 2103 (java) T 2101 2103 2103 0 -1 4202496 14106 0 1 0 4916 40 0 0 20 0 19 0 338423313 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140735351057760 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2103/statm: 2406243 423396 2267 9 0 2396937 0
[pid=2103/tid=2104] ppid=2101 vsize=9624972 CPUtime=47.24 cores=1,3,5,7
/proc/2103/task/2104/stat : 2104 (java) T 2101 2103 2103 0 -1 4202560 3503 0 1 0 4699 25 0 0 20 0 19 0 338423314 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140199107529856 140198957278105 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2103/tid=2105] ppid=2101 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/2103/task/2105/stat : 2105 (java) T 2101 2103 2103 0 -1 4202560 106 0 0 0 8 3 0 0 20 0 19 0 338423315 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140199079983272 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2106] ppid=2101 vsize=9624972 CPUtime=0.13 cores=1,3,5,7
/proc/2103/task/2106/stat : 2106 (java) T 2101 2103 2103 0 -1 4202560 148 0 0 0 11 2 0 0 20 0 19 0 338423315 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140199078930728 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2103/tid=2107] ppid=2101 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2103/task/2107/stat : 2107 (java) T 2101 2103 2103 0 -1 4202560 64 0 0 0 6 2 0 0 20 0 19 0 338423315 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140199077878184 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2103/tid=2108] ppid=2101 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/2103/task/2108/stat : 2108 (java) T 2101 2103 2103 0 -1 4202560 52 0 0 0 8 2 0 0 20 0 19 0 338423315 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140199076825640 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2109] ppid=2101 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/2103/task/2109/stat : 2109 (java) T 2101 2103 2103 0 -1 4202560 124 0 0 0 7 2 0 0 20 0 19 0 338423315 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140199075773096 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2110] ppid=2101 vsize=9624972 CPUtime=0.06 cores=1,3,5,7
/proc/2103/task/2110/stat : 2110 (java) T 2101 2103 2103 0 -1 4202560 84 0 0 0 4 2 0 0 20 0 19 0 338423315 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198945532712 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2111] ppid=2101 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/2103/task/2111/stat : 2111 (java) T 2101 2103 2103 0 -1 4202560 111 0 0 0 7 2 0 0 20 0 19 0 338423316 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198944480168 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2112] ppid=2101 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2103/task/2112/stat : 2112 (java) T 2101 2103 2103 0 -1 4202560 76 0 0 0 8 0 0 0 20 0 19 0 338423316 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198943427624 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2103/tid=2113] ppid=2101 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2103/task/2113/stat : 2113 (java) T 2101 2103 2103 0 -1 4202560 741 0 0 0 0 0 0 0 20 0 19 0 338423317 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198460479592 219517335465 0 0 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2103/tid=2114] ppid=2101 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2103/task/2114/stat : 2114 (java) T 2101 2103 2103 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338423317 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198459425912 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2103/tid=2115] ppid=2101 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2103/task/2115/stat : 2115 (java) T 2101 2103 2103 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338423317 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198458373256 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2116] ppid=2101 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2103/task/2116/stat : 2116 (java) T 2101 2103 2103 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338423319 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198457322000 219517343520 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=2103/tid=2117] ppid=2101 vsize=9624972 CPUtime=0.84 cores=1,3,5,7
/proc/2103/task/2117/stat : 2117 (java) T 2101 2103 2103 0 -1 4202560 3387 0 0 0 84 0 0 0 20 0 19 0 338423319 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198456269368 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2103/tid=2118] ppid=2101 vsize=9624972 CPUtime=0.6 cores=1,3,5,7
/proc/2103/task/2118/stat : 2118 (java) T 2101 2103 2103 0 -1 4202560 4286 0 0 0 58 2 0 0 20 0 19 0 338423319 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198455216824 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2119] ppid=2101 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2103/task/2119/stat : 2119 (java) T 2101 2103 2103 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338423320 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198454164472 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2103/tid=2120] ppid=2101 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/2103/task/2120/stat : 2120 (java) T 2101 2103 2103 0 -1 4202560 3 0 0 0 1 0 0 0 20 0 19 0 338423320 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198453112136 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2103/tid=2121] ppid=2101 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2103/task/2121/stat : 2121 (java) T 2101 2103 2103 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338423354 9855971328 423396 33554432000 1073741824 1073778376 140735351066624 140198452056872 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2101/tid=2129] ppid=2014 vsize=663908 CPUtime=0 cores=1,3,5,7
/proc/2101/task/2129/stat : 2129 (python) S 2014 2014 27000 0 -1 4202560 41 0 0 0 0 0 0 0 20 0 2 0 338428066 679841792 7059 33554432000 4194304 5683644 140736701114928 140483414188592 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2140] ppid=2014 vsize=739700 CPUtime=0.08 cores=1,3,5,7
/proc/2140/stat : 2140 (python) S 2014 2014 27000 0 -1 4202560 1174 0 0 0 5 3 0 0 20 0 2 0 338446327 757452800 7066 33554432000 4194304 5683644 140736701114928 140736701106576 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2140/statm: 184925 7066 384 364 0 120130 0
[pid=2142] ppid=2140 vsize=107864 CPUtime=47.55 cores=1,3,5,7
/proc/2142/stat : 2142 (bsolo_pb10) T 2140 2142 2142 0 -1 4202496 25039 0 0 0 4741 14 0 0 20 0 1 0 338446328 110452736 24530 33554432000 134512640 137138936 4290247968 4290246896 134542770 0 0 16781312 16384 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2142/statm: 26966 24530 192 642 0 26322 0
[pid=2140/tid=2164] ppid=2014 vsize=739700 CPUtime=0 cores=1,3,5,7
/proc/2140/task/2164/stat : 2164 (python) S 2014 2014 27000 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338451085 757452800 7066 33554432000 4194304 5683644 140736701114928 140483403698736 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2165] ppid=2014 vsize=815492 CPUtime=0 cores=1,3,5,7
/proc/2165/stat : 2165 (python) S 2014 2014 27000 0 -1 4202560 848 0 0 0 0 0 0 0 20 0 1 0 338451087 835063808 7023 33554432000 4194304 5683644 140736701114928 140736701106016 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/2165/statm: 203873 7023 336 364 0 139075 0
[pid=2167] ppid=2165 vsize=12356 CPUtime=0.01 cores=1,3,5,7
/proc/2167/stat : 2167 (bsolo_pb10) R 2165 2167 2167 0 -1 4202496 859 0 0 0 1 0 0 0 20 0 1 0 338451087 12652544 592 33554432000 134512640 137138936 4287369744 4287367960 136506715 0 0 16781312 16384 0 0 0 17 1 0 0 0 0 0
/proc/2167/statm: 3089 592 241 642 0 2332 0
[pid=2014/tid=2021] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2021/stat : 2021 (python) S 2012 2014 27000 0 -1 4202560 32 1013 0 0 0 0 0 0 20 0 9 0 338390301 912662528 8609 33554432000 4194304 5683644 140736701114928 140483662512688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2014/tid=2026] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2026/stat : 2026 (python) S 2012 2014 27000 0 -1 4202560 30 1013 0 0 0 0 0 0 20 0 9 0 338395365 912662528 8609 33554432000 4194304 5683644 140736701114928 140483652022832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2014/tid=2030] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2030/stat : 2030 (python) S 2012 2014 27000 0 -1 4202560 25 1013 0 0 0 0 0 0 20 0 9 0 338400081 912662528 8609 33554432000 4194304 5683644 140736701114928 140483641532976 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2014/tid=2035] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2035/stat : 2035 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 9 0 338409558 912662528 8609 33554432000 4194304 5683644 140736701114928 140483631043120 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2043] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2043/stat : 2043 (python) S 2012 2014 27000 0 -1 4202560 19 1013 0 0 0 0 0 0 20 0 9 0 338414299 912662528 8609 33554432000 4194304 5683644 140736701114928 140483414188592 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2102] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2102/stat : 2102 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 9 0 338423313 912662528 8609 33554432000 4194304 5683644 140736701114928 140483403698736 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2141] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2141/stat : 2141 (python) S 2012 2014 27000 0 -1 4202560 22 1013 0 0 0 0 0 0 20 0 9 0 338446327 912662528 8609 33554432000 4194304 5683644 140736701114928 140483393208880 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2014/tid=2166] ppid=2012 vsize=891272 CPUtime=0 cores=1,3,5,7
/proc/2014/task/2166/stat : 2166 (python) S 2012 2014 27000 0 -1 4202560 21 1013 0 0 0 0 0 0 20 0 9 0 338451087 912662528 8609 33554432000 4194304 5683644 140736701114928 140483382719024 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 611.38
Current children cumulated vsize (KiB) 16060592

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2014 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=444780
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=140978
# CPU time returned by wait4() is 1.58576
# while last known CPU time is 611.38
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 609.464
CPU time (s): 611.38
CPU user time (s): 608.05
CPU system time (s): 3.33
CPU usage (%): 100.314
Max. virtual memory (cumulated for all children) (KiB): 16060592

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.44478
system time used= 0.140978
maximum resident set size= 34436
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 20012
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1696
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2094
involuntary context switches= 14

runsolver used 2.53161 second user time and 6.42602 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 13:23:47
IDJOB=3742005
IDBENCH=90124
IDSOLVER=2327
FILE ID=node114/3742005-1338722627
RUNJOBID= node114-1338713908-27017
PBS_JOBID= 14636748
Free space on /tmp= 70040 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K106.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742005-1338722627/watcher-3742005-1338722627 -o /tmp/evaluation-result-3742005-1338722627/solver-3742005-1338722627 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742005-1338722627.opb -seed 2139194399 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= afe4117ba13dcfbcd06fd88e1353402a
RANDOM SEED=2139194399

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        24989476 kB
Buffers:         1033432 kB
Cached:          3819472 kB
SwapCached:         2180 kB
Active:          5086232 kB
Inactive:        1774112 kB
Active(anon):    1991636 kB
Inactive(anon):    11240 kB
Active(file):    3094596 kB
Inactive(file):  1762872 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:            102100 kB
Writeback:             0 kB
AnonPages:       2012080 kB
Mapped:            34820 kB
Shmem:               184 kB
Slab:             870272 kB
SReclaimable:     312428 kB
SUnreclaim:       557844 kB
KernelStack:        1776 kB
PageTables:        12548 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    3435580 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1124352 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 69940 MiB
End job on node114 at 2012-06-03 13:33:59