Trace number 3742070

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 609.97 607.565

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K98.opb
MD5SUM0d56c20200c8726f988a44319a68d2e3
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark165.396
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables6664
Total number of constraints6862
Number of constraints which are clauses6764
Number of constraints which are cardinality constraints (but not clauses)98
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint98
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 99
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.45/1.20	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.45/1.20	c INFO: solving HOME/instance-3742070-1338723184.opb
0.45/1.27	c INFO: parsing took 0.07 s
0.65/1.44	c INFO: features took 0.17 CPU seconds
0.65/1.47	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.68)
48.80/49.52	c INFO: running clasp-1.3.7@0 for 45 with 1752 remaining (b = 0.36)
95.93/96.68	c INFO: running scip-2.0.1-spx@0 for 45 with 1705 remaining (b = 0.06)
142.70/143.31	c INFO: running clasp-1.3.7@47 for 45 with 1658 remaining (b = 0.01)
188.45/189.08	c INFO: wrote minion input file to HOME/tmpauuXsR.minion
188.45/189.08	c INFO: running minion-0.12@0 for 45 with 1613 remaining (b = 0.09)
235.90/236.48	c INFO: running wbo1.4a@0 for 45 with 1565 remaining (b = 0.02)
282.26/282.77	c INFO: running scip-2.0.1-clp@47 for 45 with 1520 remaining (b = 0.01)
329.06/329.41	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1473 remaining (b = 0.03)
375.00/372.97	c INFO: running clasp-1.3.7@92 for 136 with 1427 remaining (b = 0.02)
512.62/510.38	c INFO: running scip-2.0.1-spx@46 for 45 with 1290 remaining (b = 0.01)
561.91/559.55	c INFO: running bsolo_pb10-l3@0 for 45 with 1241 remaining (b = 0.02)
609.45/607.04	c INFO: running bsolo_pb10-l1@0 for 91 with 1194 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742070-1338723184/watcher-3742070-1338723184 -o /tmp/evaluation-result-3742070-1338723184/solver-3742070-1338723184 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742070-1338723184.opb -seed 608775160 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.99 2.07 2.41 3/195 7257
/proc/meminfo: memFree=30509340/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/7257/stat : 7257 (python) D 7255 7257 2614 0 -1 4202496 632 0 0 0 0 0 0 0 20 0 1 0 338443488 15642624 526 33554432000 4194304 5683644 140736069155296 140736069141208 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 3819 526 312 364 0 261 0

[startup+0.0874241 s]
/proc/loadavg: 1.99 2.07 2.41 3/195 7257
/proc/meminfo: memFree=30509340/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=19112 CPUtime=0.02 cores=1,3,5,7
/proc/7257/stat : 7257 (python) R 7255 7257 2614 0 -1 4202496 1360 0 0 0 1 1 0 0 20 0 1 0 338443488 19570688 1120 33554432000 4194304 5683644 140736069155296 140736069099528 240564801428 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 4778 1124 430 364 0 702 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19112

[startup+0.100314 s]
/proc/loadavg: 1.99 2.07 2.41 3/195 7257
/proc/meminfo: memFree=30509340/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=19080 CPUtime=0.03 cores=1,3,5,7
/proc/7257/stat : 7257 (python) R 7255 7257 2614 0 -1 4202496 1397 0 0 0 2 1 0 0 20 0 1 0 338443488 19537920 1152 33554432000 4194304 5683644 140736069155296 140736069085800 4318016 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 4770 1152 430 364 0 694 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 19080

[startup+0.300267 s]
/proc/loadavg: 1.99 2.07 2.41 3/195 7257
/proc/meminfo: memFree=30509340/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=102076 CPUtime=0.11 cores=1,3,5,7
/proc/7257/stat : 7257 (python) D 7255 7257 2614 0 -1 4202496 4135 1010 1 0 8 3 0 0 20 0 1 0 338443488 104525824 2732 33554432000 4194304 5683644 140736069155296 140736068996264 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 25519 2732 876 364 0 1729 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 102076

[startup+0.700257 s]
/proc/loadavg: 1.99 2.07 2.41 3/195 7257
/proc/meminfo: memFree=30509340/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=166944 CPUtime=0.24 cores=1,3,5,7
/proc/7257/stat : 7257 (python) D 7255 7257 2614 0 -1 4202496 6514 1010 1 0 15 9 0 0 20 0 1 0 338443488 170950656 4318 33554432000 4194304 5683644 140736069155296 140736069055288 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 41736 4318 1215 364 0 3289 0
Current children cumulated CPU time (s) 0.24
Current children cumulated vsize (KiB) 166944

[startup+1.50024 s]
/proc/loadavg: 1.99 2.07 2.41 2/196 7262
/proc/meminfo: memFree=30486624/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=359928 CPUtime=0.72 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 11535 1010 1 0 57 15 0 0 20 0 2 0 338443488 368566272 7857 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 89982 7857 1901 364 0 25205 0
Current children cumulated CPU time (s) 0.72
Current children cumulated vsize (KiB) 359928

[startup+3.10046 s]
/proc/loadavg: 2.15 2.10 2.42 3/199 7265
/proc/meminfo: memFree=30400544/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=359928 CPUtime=0.72 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 11535 1010 1 0 57 15 0 0 20 0 2 0 338443488 368566272 7857 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 89982 7857 1901 364 0 25205 0
[pid=7263] ppid=7257 vsize=284148 CPUtime=0 cores=1,3,5,7
/proc/7263/stat : 7263 (python) S 7257 7257 2614 0 -1 4202560 770 0 0 0 0 0 0 0 20 0 1 0 338443636 290967552 6291 33554432000 4194304 5683644 140736069155296 140736069146384 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7263/statm: 71037 6291 341 364 0 6260 0
[pid=7265] ppid=7263 vsize=102452 CPUtime=1.6 cores=1,3,5,7
/proc/7265/stat : 7265 (scip-2.0.1.linu) R 7263 7265 7265 0 -1 4202496 22702 0 0 0 155 5 0 0 20 0 1 0 338443636 104910848 22100 33554432000 4194304 15243407 140735801569456 140735801565752 4518818 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7265/statm: 25613 22100 682 2698 0 22908 0
[pid=7257/tid=7264] ppid=7255 vsize=359928 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338443636 368566272 7857 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 2.32
Current children cumulated vsize (KiB) 746528
heavy processes:

[startup+6.30044 s]
/proc/loadavg: 2.15 2.10 2.42 3/199 7265
/proc/meminfo: memFree=30340164/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=359928 CPUtime=0.72 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 11535 1010 1 0 57 15 0 0 20 0 2 0 338443488 368566272 7857 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 89982 7857 1901 364 0 25205 0
[pid=7263] ppid=7257 vsize=284148 CPUtime=0 cores=1,3,5,7
/proc/7263/stat : 7263 (python) S 7257 7257 2614 0 -1 4202560 898 0 0 0 0 0 0 0 20 0 1 0 338443636 290967552 6314 33554432000 4194304 5683644 140736069155296 140736069146384 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7263/statm: 71037 6314 364 364 0 6260 0
[pid=7265] ppid=7263 vsize=154856 CPUtime=4.8 cores=1,3,5,7
/proc/7265/stat : 7265 (scip-2.0.1.linu) R 7263 7265 7265 0 -1 4202496 35822 0 0 0 469 11 0 0 20 0 1 0 338443636 158572544 35220 33554432000 4194304 15243407 140735801569456 140735801565752 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7265/statm: 38714 35220 686 2698 0 36009 0
[pid=7257/tid=7264] ppid=7255 vsize=359928 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338443636 368566272 7857 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 5.52
Current children cumulated vsize (KiB) 798932

[startup+12.7004 s]
/proc/loadavg: 2.13 2.09 2.42 3/199 7265
/proc/meminfo: memFree=30279924/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=359928 CPUtime=0.72 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 11535 1010 1 0 57 15 0 0 20 0 2 0 338443488 368566272 7857 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 89982 7857 1901 364 0 25205 0
[pid=7263] ppid=7257 vsize=284148 CPUtime=0 cores=1,3,5,7
/proc/7263/stat : 7263 (python) S 7257 7257 2614 0 -1 4202560 906 0 0 0 0 0 0 0 20 0 1 0 338443636 290967552 6314 33554432000 4194304 5683644 140736069155296 140736069146384 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7263/statm: 71037 6314 364 364 0 6260 0
[pid=7265] ppid=7263 vsize=202024 CPUtime=11.2 cores=1,3,5,7
/proc/7265/stat : 7265 (scip-2.0.1.linu) R 7263 7265 7265 0 -1 4202496 47531 0 0 0 1102 18 0 0 20 0 1 0 338443636 206872576 46929 33554432000 4194304 15243407 140735801569456 140735801565752 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7265/statm: 50506 46929 686 2698 0 47801 0
[pid=7257/tid=7264] ppid=7255 vsize=359928 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338443636 368566272 7857 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.92
Current children cumulated vsize (KiB) 846100

[startup+25.5004 s]
/proc/loadavg: 2.11 2.09 2.41 3/199 7265
/proc/meminfo: memFree=30181600/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=359928 CPUtime=0.72 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 11535 1010 1 0 57 15 0 0 20 0 2 0 338443488 368566272 7857 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 89982 7857 1901 364 0 25205 0
[pid=7263] ppid=7257 vsize=284148 CPUtime=0.04 cores=1,3,5,7
/proc/7263/stat : 7263 (python) S 7257 7257 2614 0 -1 4202560 1013 0 0 0 2 2 0 0 20 0 1 0 338443636 290967552 6319 33554432000 4194304 5683644 140736069155296 140736069146384 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7263/statm: 71037 6319 369 364 0 6260 0
[pid=7265] ppid=7263 vsize=286692 CPUtime=24 cores=1,3,5,7
/proc/7265/stat : 7265 (scip-2.0.1.linu) R 7263 7265 7265 0 -1 4202496 68402 0 0 0 2362 38 0 0 20 0 1 0 338443636 293572608 67775 33554432000 4194304 15243407 140735801569456 140735801565544 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7265/statm: 71673 67775 873 2698 0 68968 0
[pid=7257/tid=7264] ppid=7255 vsize=359928 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 2 0 338443636 368566272 7857 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.76
Current children cumulated vsize (KiB) 930768

[startup+51.1004 s]
/proc/loadavg: 2.19 2.11 2.41 3/204 7270
/proc/meminfo: memFree=30140504/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=436276 CPUtime=0.75 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 12467 1010 1 0 59 16 0 0 20 0 3 0 338443488 446746624 7992 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 109069 7992 1904 364 0 44289 0
[pid=7263] ppid=7257 vsize=359928 CPUtime=0.08 cores=1,3,5,7
/proc/7263/stat : 7263 (python) S 7257 7257 2614 0 -1 4202560 1169 0 0 0 5 3 0 0 20 0 2 0 338443636 368566272 6354 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7263/statm: 89982 6354 397 364 0 25205 0
[pid=7265] ppid=7263 vsize=310772 CPUtime=47.99 cores=1,3,5,7
/proc/7265/stat : 7265 (scip-2.0.1.linu) T 7263 7265 7265 0 -1 4202496 74455 0 0 0 4709 90 0 0 20 0 1 0 338443636 318230528 73826 33554432000 4194304 15243407 140735801569456 140735801564168 8137700 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7265/statm: 77693 73826 898 2698 0 74988 0
[pid=7263/tid=7267] ppid=7257 vsize=359928 CPUtime=0 cores=1,3,5,7
/proc/7263/task/7267/stat : 7267 (python) S 7257 7257 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338448438 368566272 6354 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7268] ppid=7257 vsize=360496 CPUtime=0 cores=1,3,5,7
/proc/7268/stat : 7268 (python) S 7257 7257 2614 0 -1 4202560 768 0 0 0 0 0 0 0 20 0 1 0 338448441 369147904 6418 33554432000 4194304 5683644 140736069155296 140736069146384 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7268/statm: 90124 6418 336 364 0 25344 0
[pid=7270] ppid=7268 vsize=5140 CPUtime=1.57 cores=1,3,5,7
/proc/7270/stat : 7270 (clasp-1.3.7-x86) R 7268 7270 7270 0 -1 4202496 1384 0 0 0 157 0 0 0 20 0 1 0 338448441 5263360 1028 33554432000 134512640 136139019 4287407296 4287406272 134883391 0 0 16781312 16386 0 0 0 17 1 0 0 0 0 0
/proc/7270/statm: 1285 1028 182 398 0 885 0
[pid=7257/tid=7264] ppid=7255 vsize=436276 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 3 0 338443636 446746624 7992 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7269] ppid=7255 vsize=436276 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7269/stat : 7269 (python) S 7255 7257 2614 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 3 0 338448441 446746624 7992 33554432000 4194304 5683644 140736069155296 139937579401776 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.39
Current children cumulated vsize (KiB) 1472612

[startup+102.301 s]
/proc/loadavg: 2.08 2.09 2.39 3/229 7295
/proc/meminfo: memFree=29221896/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=512068 CPUtime=0.77 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 13381 1010 1 0 61 16 0 0 20 0 4 0 338443488 524357632 8000 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 128017 8000 1906 364 0 63234 0
[pid=7263] ppid=7257 vsize=359928 CPUtime=0.08 cores=1,3,5,7
/proc/7263/stat : 7263 (python) S 7257 7257 2614 0 -1 4202560 1169 0 0 0 5 3 0 0 20 0 2 0 338443636 368566272 6354 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7263/statm: 89982 6354 397 364 0 25205 0
[pid=7265] ppid=7263 vsize=310772 CPUtime=47.99 cores=1,3,5,7
/proc/7265/stat : 7265 (scip-2.0.1.linu) T 7263 7265 7265 0 -1 4202496 74455 0 0 0 4709 90 0 0 20 0 1 0 338443636 318230528 73826 33554432000 4194304 15243407 140735801569456 140735801564168 8137700 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7265/statm: 77693 73826 898 2698 0 74988 0
[pid=7263/tid=7267] ppid=7257 vsize=359928 CPUtime=0 cores=1,3,5,7
/proc/7263/task/7267/stat : 7267 (python) S 7257 7257 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338448438 368566272 6354 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7268] ppid=7257 vsize=360496 CPUtime=0.07 cores=1,3,5,7
/proc/7268/stat : 7268 (python) S 7257 7257 2614 0 -1 4202560 1091 0 0 0 4 3 0 0 20 0 2 0 338448441 369147904 6467 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7268/statm: 90124 6467 384 364 0 25344 0
[pid=7270] ppid=7268 vsize=16756 CPUtime=47.11 cores=1,3,5,7
/proc/7270/stat : 7270 (clasp-1.3.7-x86) T 7268 7270 7270 0 -1 4202496 4286 0 0 0 4710 1 0 0 20 0 1 0 338448441 17158144 3930 33554432000 134512640 136139019 4287407296 4287406016 134875024 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7270/statm: 4189 3930 182 398 0 3789 0
[pid=7268/tid=7292] ppid=7257 vsize=360496 CPUtime=0 cores=1,3,5,7
/proc/7268/task/7292/stat : 7292 (python) S 7257 7257 2614 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338453154 369147904 6467 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7293] ppid=7257 vsize=436288 CPUtime=0 cores=1,3,5,7
/proc/7293/stat : 7293 (python) S 7257 7257 2614 0 -1 4202560 925 0 0 0 0 0 0 0 20 0 1 0 338453156 446758912 6447 33554432000 4194304 5683644 140736069155296 140736069146384 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/7293/statm: 109072 6447 359 364 0 44289 0
[pid=7295] ppid=7293 vsize=164072 CPUtime=5.6 cores=1,3,5,7
/proc/7295/stat : 7295 (scip-2.0.1.linu) R 7293 7295 7295 0 -1 4202496 38288 0 0 0 549 11 0 0 20 0 1 0 338453157 168009728 37672 33554432000 4194304 13684479 140737183970816 140737183967128 11876362 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/7295/statm: 41018 37672 639 2317 0 38696 0
[pid=7257/tid=7264] ppid=7255 vsize=512068 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 26 1010 0 0 0 0 0 0 20 0 4 0 338443636 524357632 8000 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7269] ppid=7255 vsize=512068 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/7397/stat : 7397 (bsolo_pb10) T 7395 7397 7397 0 -1 4202496 27260 0 0 0 4731 13 0 0 20 0 1 0 338499444 119603200 26777 33554432000 134512640 137138936 4292645296 4292644224 134542508 0 0 16781312 16384 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7397/statm: 29200 26777 192 642 0 28556 0
[pid=7395/tid=7401] ppid=7257 vsize=739456 CPUtime=0 cores=1,3,5,7
/proc/7395/task/7401/stat : 7401 (python) S 7257 7257 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338504191 757202944 7005 33554432000 4194304 5683644 140736069155296 139937328382512 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7402] ppid=7257 vsize=815248 CPUtime=0 cores=1,3,5,7
/proc/7402/stat : 7402 (python) S 7257 7257 2614 0 -1 4202560 851 0 0 0 0 0 0 0 20 0 1 0 338504193 834813952 6962 33554432000 4194304 5683644 140736069155296 140736069146384 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/7402/statm: 203812 6962 336 364 0 139014 0
[pid=7404] ppid=7402 vsize=13252 CPUtime=0.46 cores=1,3,5,7
/proc/7404/stat : 7404 (bsolo_pb10) R 7402 7404 7404 0 -1 4202496 1214 0 0 0 46 0 0 0 20 0 1 0 338504193 13570048 843 33554432000 134512640 137138936 4294539344 4294458076 136570369 0 0 16781312 16384 0 0 0 17 1 0 0 0 0 0
/proc/7404/statm: 3313 843 184 642 0 2669 0
[pid=7257/tid=7264] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 31 1010 0 0 0 0 0 0 20 0 9 0 338443636 912412672 8548 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7257/tid=7269] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7269/stat : 7269 (python) S 7255 7257 2614 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 9 0 338448441 912412672 8548 33554432000 4194304 5683644 140736069155296 139937579401776 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7257/tid=7294] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7294/stat : 7294 (python) S 7255 7257 2614 0 -1 4202560 25 1010 0 0 0 0 0 0 20 0 9 0 338453157 912412672 8548 33554432000 4194304 5683644 140736069155296 139937568911920 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7257/tid=7299] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7299/stat : 7299 (python) S 7255 7257 2614 0 -1 4202560 19 1010 0 0 0 0 0 0 20 0 9 0 338462397 912412672 8548 33554432000 4194304 5683644 140736069155296 139937349362224 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7303] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7303/stat : 7303 (python) S 7255 7257 2614 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 9 0 338467137 912412672 8548 33554432000 4194304 5683644 140736069155296 139937338872368 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7311] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7311/stat : 7311 (python) S 7255 7257 2614 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338476430 912412672 8548 33554432000 4194304 5683644 140736069155296 139937328382512 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7396] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7396/stat : 7396 (python) S 7255 7257 2614 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 9 0 338499444 912412672 8548 33554432000 4194304 5683644 140736069155296 139937317892656 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7403] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7403/stat : 7403 (python) S 7255 7257 2614 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338504193 912412672 8548 33554432000 4194304 5683644 140736069155296 139937307402800 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 609.97
Current children cumulated vsize (KiB) 15959988

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+607.501 s]
/proc/loadavg: 2.08 2.05 2.23 3/241 7404
/proc/meminfo: memFree=28800824/32873844 swapFree=7116/7116
[pid=7257] ppid=7255 vsize=891028 CPUtime=1.61 cores=1,3,5,7
/proc/7257/stat : 7257 (python) S 7255 7257 2614 0 -1 4202496 18427 1010 1 0 142 19 0 0 20 0 9 0 338443488 912412672 8548 33554432000 4194304 5683644 140736069155296 140736069150128 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7257/statm: 222757 8548 1916 364 0 157959 0
[pid=7263] ppid=7257 vsize=359928 CPUtime=0.17 cores=1,3,5,7
/proc/7263/stat : 7263 (python) S 7257 7257 2614 0 -1 4202560 1200 0 0 0 9 8 0 0 20 0 2 0 338443636 368566272 6355 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7263/statm: 89982 6355 398 364 0 25205 0
[pid=7265] ppid=7263 vsize=321996 CPUtime=94.6 cores=1,3,5,7
/proc/7265/stat : 7265 (scip-2.0.1.linu) T 7263 7265 7265 0 -1 4202496 89106 0 0 0 9358 102 0 0 20 0 1 0 338443636 329723904 75643 33554432000 4194304 15243407 140735801569456 140735801563848 8136077 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7265/statm: 80499 75643 946 2698 0 77794 0
[pid=7263/tid=7267] ppid=7257 vsize=359928 CPUtime=0 cores=1,3,5,7
/proc/7263/task/7267/stat : 7267 (python) S 7257 7257 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338448438 368566272 6355 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7268] ppid=7257 vsize=360496 CPUtime=0.41 cores=1,3,5,7
/proc/7268/stat : 7268 (python) S 7257 7257 2614 0 -1 4202560 1188 0 0 0 24 17 0 0 20 0 2 0 338448441 369147904 6467 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7268/statm: 90124 6467 384 364 0 25344 0
[pid=7270] ppid=7268 vsize=33260 CPUtime=229.59 cores=1,3,5,7
/proc/7270/stat : 7270 (clasp-1.3.7-x86) T 7268 7270 7270 0 -1 4202496 8403 0 0 0 22957 2 0 0 20 0 1 0 338448441 34058240 8047 33554432000 134512640 136139019 4287407296 4287406272 134876516 0 0 16781312 16386 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/7270/statm: 8315 8047 182 398 0 7915 0
[pid=7268/tid=7292] ppid=7257 vsize=360496 CPUtime=0 cores=1,3,5,7
/proc/7268/task/7292/stat : 7292 (python) S 7257 7257 2614 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338453154 369147904 6467 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7293] ppid=7257 vsize=436288 CPUtime=0.17 cores=1,3,5,7
/proc/7293/stat : 7293 (python) S 7257 7257 2614 0 -1 4202560 1156 0 0 0 9 8 0 0 20 0 2 0 338453156 446758912 6473 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7293/statm: 109072 6473 384 364 0 44289 0
[pid=7295] ppid=7293 vsize=283920 CPUtime=95.72 cores=1,3,5,7
/proc/7295/stat : 7295 (scip-2.0.1.linu) T 7293 7295 7295 0 -1 4202496 74672 0 0 0 9515 57 0 0 20 0 1 0 338453157 290734080 67885 33554432000 4194304 13684479 140737183970816 140737183966904 11876362 0 0 16781312 2 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/7295/statm: 70980 67885 775 2317 0 68658 0
[pid=7293/tid=7297] ppid=7257 vsize=436288 CPUtime=0 cores=1,3,5,7
/proc/7293/task/7297/stat : 7297 (python) S 7257 7257 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338457817 446758912 6473 33554432000 4194304 5683644 140736069155296 139937579401776 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7298] ppid=7257 vsize=512080 CPUtime=0.08 cores=1,3,5,7
/proc/7298/stat : 7298 (python) S 7257 7257 2614 0 -1 4202560 1138 0 0 0 4 4 0 0 20 0 2 0 338462397 524369920 6479 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7298/statm: 128020 6479 384 364 0 63234 0
[pid=7300] ppid=7298 vsize=89480 CPUtime=47.35 cores=1,3,5,7
/proc/7300/stat : 7300 (minion) T 7298 7300 7300 0 -1 4202496 8842 0 0 0 4734 1 0 0 20 0 1 0 338462397 91627520 15073 33554432000 134512640 165565024 4291371680 4291368928 159737071 0 0 16781312 8396802 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7300/statm: 22370 15073 336 7582 0 14785 0
[pid=7298/tid=7301] ppid=7257 vsize=512080 CPUtime=0 cores=1,3,5,7
/proc/7298/task/7301/stat : 7301 (python) S 7257 7257 2614 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338467134 524369920 6479 33554432000 4194304 5683644 140736069155296 139937568911920 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7302] ppid=7257 vsize=587872 CPUtime=0.08 cores=1,3,5,7
/proc/7302/stat : 7302 (python) S 7257 7257 2614 0 -1 4202560 1143 0 0 0 5 3 0 0 20 0 2 0 338467137 601980928 6992 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7302/statm: 146968 6992 384 364 0 82179 0
[pid=7304] ppid=7302 vsize=110248 CPUtime=46.24 cores=1,3,5,7
/proc/7304/stat : 7304 (wbo1.4a) T 7302 7304 7304 0 -1 4202496 27444 0 0 0 4583 41 0 0 20 0 1 0 338467137 112893952 27079 33554432000 134512640 135751879 4294269568 4294267184 134605088 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7304/statm: 27562 27079 143 303 0 27257 0
[pid=7302/tid=7306] ppid=7257 vsize=587872 CPUtime=0 cores=1,3,5,7
/proc/7302/task/7306/stat : 7306 (python) S 7257 7257 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338471763 601980928 6992 33554432000 4194304 5683644 140736069155296 139937349362224 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7310] ppid=7257 vsize=663664 CPUtime=0.08 cores=1,3,5,7
/proc/7310/stat : 7310 (python) S 7257 7257 2614 0 -1 4202560 1155 0 0 0 4 4 0 0 20 0 2 0 338476429 679591936 6999 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/7310/statm: 165916 6999 384 364 0 101124 0
[pid=7312] ppid=7310 vsize=9624972 CPUtime=45.89 cores=1,3,5,7
/proc/7312/stat : 7312 (java) T 7310 7312 7312 0 -1 4202496 17499 0 1 0 4564 25 0 0 20 0 19 0 338476430 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140735035614240 240568533053 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7312/statm: 2406243 182646 2266 9 0 2396937 0
[pid=7312/tid=7313] ppid=7310 vsize=9624972 CPUtime=42.99 cores=1,3,5,7
/proc/7312/task/7313/stat : 7313 (java) T 7310 7312 7312 0 -1 4202560 4163 0 1 0 4287 12 0 0 20 0 19 0 338476431 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140327346490672 140327202240724 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7312/tid=7314] ppid=7310 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/7312/task/7314/stat : 7314 (java) T 7310 7312 7312 0 -1 4202560 33 0 0 0 11 0 0 0 20 0 19 0 338476432 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140327190404136 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7312/tid=7315] ppid=7310 vsize=9624972 CPUtime=0.13 cores=1,3,5,7
/proc/7312/task/7315/stat : 7315 (java) T 7310 7312 7312 0 -1 4202560 18 0 0 0 13 0 0 0 20 0 19 0 338476432 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140327189351336 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7312/tid=7316] ppid=7310 vsize=9624972 CPUtime=0.15 cores=1,3,5,7
/proc/7312/task/7316/stat : 7316 (java) T 7310 7312 7312 0 -1 4202560 28 0 0 0 15 0 0 0 20 0 19 0 338476432 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140327188298536 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7312/tid=7317] ppid=7310 vsize=9624972 CPUtime=0.13 cores=1,3,5,7
/proc/7312/task/7317/stat : 7317 (java) T 7310 7312 7312 0 -1 4202560 26 0 0 0 13 0 0 0 20 0 19 0 338476432 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140327187245736 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7312/tid=7318] ppid=7310 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/7312/task/7318/stat : 7318 (java) T 7310 7312 7312 0 -1 4202560 24 0 0 0 10 0 0 0 20 0 19 0 338476432 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140327186192936 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7312/tid=7319] ppid=7310 vsize=9624972 CPUtime=0.14 cores=1,3,5,7
/proc/7312/task/7319/stat : 7319 (java) T 7310 7312 7312 0 -1 4202560 24 0 0 0 14 0 0 0 20 0 19 0 338476432 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326849210792 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7312/tid=7320] ppid=7310 vsize=9624972 CPUtime=0.12 cores=1,3,5,7
/proc/7312/task/7320/stat : 7320 (java) T 7310 7312 7312 0 -1 4202560 28 0 0 0 11 1 0 0 20 0 19 0 338476433 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326848157992 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7312/tid=7321] ppid=7310 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/7312/task/7321/stat : 7321 (java) T 7310 7312 7312 0 -1 4202560 26 0 0 0 9 2 0 0 20 0 19 0 338476433 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326847105192 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7312/tid=7322] ppid=7310 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7312/task/7322/stat : 7322 (java) T 7310 7312 7312 0 -1 4202560 326 0 0 0 0 0 0 0 20 0 19 0 338476434 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326834908136 240568547241 0 0 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7312/tid=7323] ppid=7310 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7312/task/7323/stat : 7323 (java) T 7310 7312 7312 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338476434 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326833854200 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7312/tid=7324] ppid=7310 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7312/task/7324/stat : 7324 (java) T 7310 7312 7312 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338476434 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326832801288 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7312/tid=7325] ppid=7310 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7312/task/7325/stat : 7325 (java) T 7310 7312 7312 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338476437 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326831749776 240568555296 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=7312/tid=7326] ppid=7310 vsize=9624972 CPUtime=0.76 cores=1,3,5,7
/proc/7312/task/7326/stat : 7326 (java) T 7310 7312 7312 0 -1 4202560 6518 0 0 0 76 0 0 0 20 0 19 0 338476437 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326830696888 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=7312/tid=7327] ppid=7310 vsize=9624972 CPUtime=0.88 cores=1,3,5,7
/proc/7312/task/7327/stat : 7327 (java) T 7310 7312 7312 0 -1 4202560 4857 0 0 0 86 2 0 0 20 0 19 0 338476437 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326829644088 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7312/tid=7328] ppid=7310 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7312/task/7328/stat : 7328 (java) T 7310 7312 7312 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338476437 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326828591480 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7312/tid=7329] ppid=7310 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/7312/task/7329/stat : 7329 (java) T 7310 7312 7312 0 -1 4202560 3 0 0 0 0 1 0 0 20 0 19 0 338476437 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326827538888 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=7312/tid=7330] ppid=7310 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/7312/task/7330/stat : 7330 (java) T 7310 7312 7312 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338476471 9855971328 182646 33554432000 1073741824 1073778376 140735035623104 140326826456744 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=7310/tid=7335] ppid=7257 vsize=663664 CPUtime=0 cores=1,3,5,7
/proc/7310/task/7335/stat : 7335 (python) S 7257 7257 2614 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338480783 679591936 6999 33554432000 4194304 5683644 140736069155296 139937338872368 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7395] ppid=7257 vsize=739456 CPUtime=0.08 cores=1,3,5,7
/proc/7395/stat : 7395 (python) S 7257 7257 2614 0 -1 4202560 1182 0 0 0 4 4 0 0 20 0 2 0 338499444 757202944 7005 33554432000 4194304 5683644 140736069155296 140736069146944 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/7395/statm: 184864 7005 384 364 0 120069 0
[pid=7397] ppid=7395 vsize=116800 CPUtime=47.44 cores=1,3,5,7
/proc/7397/stat : 7397 (bsolo_pb10) T 7395 7397 7397 0 -1 4202496 27260 0 0 0 4731 13 0 0 20 0 1 0 338499444 119603200 26777 33554432000 134512640 137138936 4292645296 4292644224 134542508 0 0 16781312 16384 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/7397/statm: 29200 26777 192 642 0 28556 0
[pid=7395/tid=7401] ppid=7257 vsize=739456 CPUtime=0 cores=1,3,5,7
/proc/7395/task/7401/stat : 7401 (python) S 7257 7257 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338504191 757202944 7005 33554432000 4194304 5683644 140736069155296 139937328382512 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7402] ppid=7257 vsize=815248 CPUtime=0 cores=1,3,5,7
/proc/7402/stat : 7402 (python) S 7257 7257 2614 0 -1 4202560 851 0 0 0 0 0 0 0 20 0 1 0 338504193 834813952 6962 33554432000 4194304 5683644 140736069155296 140736069146384 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/7402/statm: 203812 6962 336 364 0 139014 0
[pid=7404] ppid=7402 vsize=13252 CPUtime=0.46 cores=1,3,5,7
/proc/7404/stat : 7404 (bsolo_pb10) R 7402 7404 7404 0 -1 4202496 1214 0 0 0 46 0 0 0 20 0 1 0 338504193 13570048 843 33554432000 134512640 137138936 4294539344 4294458076 136570369 0 0 16781312 16384 0 0 0 17 1 0 0 0 0 0
/proc/7404/statm: 3313 843 184 642 0 2669 0
[pid=7257/tid=7264] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7264/stat : 7264 (python) S 7255 7257 2614 0 -1 4202560 31 1010 0 0 0 0 0 0 20 0 9 0 338443636 912412672 8548 33554432000 4194304 5683644 140736069155296 139937589891632 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7257/tid=7269] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7269/stat : 7269 (python) S 7255 7257 2614 0 -1 4202560 30 1010 0 0 0 0 0 0 20 0 9 0 338448441 912412672 8548 33554432000 4194304 5683644 140736069155296 139937579401776 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=7257/tid=7294] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7294/stat : 7294 (python) S 7255 7257 2614 0 -1 4202560 25 1010 0 0 0 0 0 0 20 0 9 0 338453157 912412672 8548 33554432000 4194304 5683644 140736069155296 139937568911920 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=7257/tid=7299] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7299/stat : 7299 (python) S 7255 7257 2614 0 -1 4202560 19 1010 0 0 0 0 0 0 20 0 9 0 338462397 912412672 8548 33554432000 4194304 5683644 140736069155296 139937349362224 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7303] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7303/stat : 7303 (python) S 7255 7257 2614 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 9 0 338467137 912412672 8548 33554432000 4194304 5683644 140736069155296 139937338872368 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7311] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7311/stat : 7311 (python) S 7255 7257 2614 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338476430 912412672 8548 33554432000 4194304 5683644 140736069155296 139937328382512 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7396] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7396/stat : 7396 (python) S 7255 7257 2614 0 -1 4202560 22 1010 0 0 0 0 0 0 20 0 9 0 338499444 912412672 8548 33554432000 4194304 5683644 140736069155296 139937317892656 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=7257/tid=7403] ppid=7255 vsize=891028 CPUtime=0 cores=1,3,5,7
/proc/7257/task/7403/stat : 7403 (python) S 7255 7257 2614 0 -1 4202560 21 1010 0 0 0 0 0 0 20 0 9 0 338504193 912412672 8548 33554432000 4194304 5683644 140736069155296 139937307402800 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 609.97
Current children cumulated vsize (KiB) 15959988

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7257 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=430782
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=195970
# CPU time returned by wait4() is 1.62675
# while last known CPU time is 609.97
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 607.565
CPU time (s): 609.97
CPU user time (s): 606.89
CPU system time (s): 3.08
CPU usage (%): 100.396
Max. virtual memory (cumulated for all children) (KiB): 15959988

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.43078
system time used= 0.19597
maximum resident set size= 34192
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19945
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1600
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2083
involuntary context switches= 13

runsolver used 2.70159 second user time and 6.584 second system time

The end

Launcher Data 

Begin job on node146 at 2012-06-03 13:33:04
IDJOB=3742070
IDBENCH=90129
IDSOLVER=2327
FILE ID=node146/3742070-1338723184
RUNJOBID= node146-1338713700-2631
PBS_JOBID= 14636739
Free space on /tmp= 70808 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K98.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742070-1338723184/watcher-3742070-1338723184 -o /tmp/evaluation-result-3742070-1338723184/solver-3742070-1338723184 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742070-1338723184.opb -seed 608775160 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0d56c20200c8726f988a44319a68d2e3
RANDOM SEED=608775160

node146.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        30509852 kB
Buffers:          282232 kB
Cached:           613164 kB
SwapCached:         1792 kB
Active:          1232104 kB
Inactive:         303416 kB
Active(anon):     640984 kB
Inactive(anon):     1736 kB
Active(file):     591120 kB
Inactive(file):   301680 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67099888 kB
Dirty:            102100 kB
Writeback:             0 kB
AnonPages:        646940 kB
Mapped:            24112 kB
Shmem:                72 kB
Slab:             682792 kB
SReclaimable:     121120 kB
SUnreclaim:       561672 kB
KernelStack:        1600 kB
PageTables:         8688 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    1057232 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:     36864 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70708 MiB
End job on node146 at 2012-06-03 13:43:15