Trace number 3742148

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 613.74 611.662

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K96.opb
MD5SUM12d577f059d1da283bb935f5e106dee7
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark568.277
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables6528
Total number of constraints6722
Number of constraints which are clauses6626
Number of constraints which are cardinality constraints (but not clauses)96
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint96
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 97
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.39/1.23	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.39/1.24	c INFO: solving HOME/instance-3742148-1338723627.opb
0.48/1.31	c INFO: parsing took 0.07 s
0.58/1.44	c INFO: features took 0.13 CPU seconds
0.65/1.64	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.68)
48.22/49.01	c INFO: running clasp-1.3.7@0 for 45 with 1752 remaining (b = 0.36)
95.36/96.17	c INFO: running scip-2.0.1-spx@0 for 45 with 1705 remaining (b = 0.06)
141.82/142.54	c INFO: running clasp-1.3.7@47 for 45 with 1659 remaining (b = 0.01)
187.57/188.27	c INFO: wrote minion input file to HOME/tmpDinH8Y.minion
187.57/188.27	c INFO: running minion-0.12@0 for 45 with 1614 remaining (b = 0.09)
235.00/235.65	c INFO: running wbo1.4a@0 for 45 with 1566 remaining (b = 0.02)
284.45/285.10	c INFO: running scip-2.0.1-clp@46 for 45 with 1517 remaining (b = 0.01)
329.32/329.89	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1472 remaining (b = 0.03)
379.16/377.44	c INFO: running clasp-1.3.7@92 for 136 with 1423 remaining (b = 0.02)
516.79/514.86	c INFO: running scip-2.0.1-spx@45 for 45 with 1286 remaining (b = 0.01)
566.09/564.02	c INFO: running bsolo_pb10-l3@0 for 45 with 1237 remaining (b = 0.02)
613.53/611.46	c INFO: running bsolo_pb10-l1@0 for 91 with 1189 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742148-1338723627/watcher-3742148-1338723627 -o /tmp/evaluation-result-3742148-1338723627/solver-3742148-1338723627 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742148-1338723627.opb -seed 2098139013 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.87 1.98 2.16 3/184 8957
/proc/meminfo: memFree=31115828/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/8957/stat : 8957 (python) R 8955 8957 3958 0 -1 4202496 624 0 0 0 0 0 0 0 20 0 1 0 338488822 15642624 517 33554432000 4194304 5683644 140736952290704 140736952277480 217410146196 0 0 16781312 2 0 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 3819 521 308 364 0 261 0

[startup+0.067633 s]
/proc/loadavg: 1.87 1.98 2.16 3/184 8957
/proc/meminfo: memFree=31115828/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=16592 CPUtime=0.01 cores=1,3,5,7
/proc/8957/stat : 8957 (python) D 8955 8957 3958 0 -1 4202496 1205 0 0 0 1 0 0 0 20 0 1 0 338488822 16990208 1002 33554432000 4194304 5683644 140736952290704 140736952258984 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 4148 1002 422 364 0 590 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16592

[startup+0.100233 s]
/proc/loadavg: 1.87 1.98 2.16 3/184 8957
/proc/meminfo: memFree=31115828/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=19084 CPUtime=0.02 cores=1,3,5,7
/proc/8957/stat : 8957 (python) R 8955 8957 3958 0 -1 4202496 1397 0 0 0 2 0 0 0 20 0 1 0 338488822 19542016 1151 33554432000 4194304 5683644 140736952290704 140736952221208 4867961 0 0 16781312 2 0 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 4771 1151 430 364 0 695 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19084

[startup+0.30021 s]
/proc/loadavg: 1.87 1.98 2.16 3/184 8957
/proc/meminfo: memFree=31115828/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=102048 CPUtime=0.11 cores=1,3,5,7
/proc/8957/stat : 8957 (python) D 8955 8957 3958 0 -1 4202496 4106 1009 1 0 8 3 0 0 20 0 1 0 338488822 104497152 2725 33554432000 4194304 5683644 140736952290704 140736952159064 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 25512 2725 876 364 0 1722 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 102048

[startup+0.700185 s]
/proc/loadavg: 1.87 1.98 2.16 3/184 8957
/proc/meminfo: memFree=31115828/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=138348 CPUtime=0.22 cores=1,3,5,7
/proc/8957/stat : 8957 (python) D 8955 8957 3958 0 -1 4202496 6251 1009 1 0 15 7 0 0 20 0 1 0 338488822 141668352 4086 33554432000 4194304 5683644 140736952290704 140736952190696 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 34587 4086 1098 364 0 2953 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 138348

[startup+1.50013 s]
/proc/loadavg: 1.87 1.98 2.16 3/185 8962
/proc/meminfo: memFree=31098824/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=359880 CPUtime=0.65 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 11378 1009 1 0 53 12 0 0 20 0 2 0 338488822 368517120 7832 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 89970 7832 1901 364 0 25193 0
Current children cumulated CPU time (s) 0.65
Current children cumulated vsize (KiB) 359880

[startup+3.1003 s]
/proc/loadavg: 1.87 1.98 2.16 3/188 8965
/proc/meminfo: memFree=31020812/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=359880 CPUtime=0.65 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 11378 1009 1 0 53 12 0 0 20 0 2 0 338488822 368517120 7832 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 89970 7832 1901 364 0 25193 0
[pid=8963] ppid=8957 vsize=284100 CPUtime=0 cores=1,3,5,7
/proc/8963/stat : 8963 (python) S 8957 8957 3958 0 -1 4202560 766 0 0 0 0 0 0 0 20 0 1 0 338488969 290918400 6266 33554432000 4194304 5683644 140736952290704 140736952281792 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8963/statm: 71025 6266 341 364 0 6248 0
[pid=8965] ppid=8963 vsize=101204 CPUtime=1.62 cores=1,3,5,7
/proc/8965/stat : 8965 (scip-2.0.1.linu) R 8963 8965 8965 0 -1 4202496 22443 0 0 0 157 5 0 0 20 0 1 0 338488969 103632896 21842 33554432000 4194304 15243407 140734144276384 140734144272056 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8965/statm: 25301 21842 683 2698 0 22596 0
[pid=8957/tid=8964] ppid=8955 vsize=359880 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338488969 368517120 7832 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 2.27
Current children cumulated vsize (KiB) 745184
heavy processes:

[startup+6.30037 s]
/proc/loadavg: 1.88 1.98 2.16 3/188 8965
/proc/meminfo: memFree=30953696/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=359880 CPUtime=0.65 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 11378 1009 1 0 53 12 0 0 20 0 2 0 338488822 368517120 7832 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 89970 7832 1901 364 0 25193 0
[pid=8963] ppid=8957 vsize=284100 CPUtime=0 cores=1,3,5,7
/proc/8963/stat : 8963 (python) S 8957 8957 3958 0 -1 4202560 891 0 0 0 0 0 0 0 20 0 1 0 338488969 290918400 6289 33554432000 4194304 5683644 140736952290704 140736952281792 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8963/statm: 71025 6289 364 364 0 6248 0
[pid=8965] ppid=8963 vsize=163720 CPUtime=4.81 cores=1,3,5,7
/proc/8965/stat : 8965 (scip-2.0.1.linu) R 8963 8965 8965 0 -1 4202496 38008 0 0 0 469 12 0 0 20 0 1 0 338488969 167649280 37407 33554432000 4194304 15243407 140734144276384 140734144272696 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8965/statm: 40930 37407 687 2698 0 38225 0
[pid=8957/tid=8964] ppid=8955 vsize=359880 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338488969 368517120 7832 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 5.46
Current children cumulated vsize (KiB) 807700

[startup+12.7004 s]
/proc/loadavg: 1.89 1.98 2.16 3/188 8965
/proc/meminfo: memFree=30894488/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=359880 CPUtime=0.65 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 11378 1009 1 0 53 12 0 0 20 0 2 0 338488822 368517120 7832 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 89970 7832 1901 364 0 25193 0
[pid=8963] ppid=8957 vsize=284100 CPUtime=0 cores=1,3,5,7
/proc/8963/stat : 8963 (python) S 8957 8957 3958 0 -1 4202560 895 0 0 0 0 0 0 0 20 0 1 0 338488969 290918400 6289 33554432000 4194304 5683644 140736952290704 140736952281792 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8963/statm: 71025 6289 364 364 0 6248 0
[pid=8965] ppid=8963 vsize=214736 CPUtime=11.21 cores=1,3,5,7
/proc/8965/stat : 8965 (scip-2.0.1.linu) R 8963 8965 8965 0 -1 4202496 50880 0 0 0 1100 21 0 0 20 0 1 0 338488969 219889664 50279 33554432000 4194304 15243407 140734144276384 140734144272696 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8965/statm: 53684 50279 687 2698 0 50979 0
[pid=8957/tid=8964] ppid=8955 vsize=359880 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338488969 368517120 7832 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 11.86
Current children cumulated vsize (KiB) 858716

[startup+25.5004 s]
/proc/loadavg: 1.99 2.00 2.16 3/192 8969
/proc/meminfo: memFree=30666868/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=359880 CPUtime=0.65 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 11378 1009 1 0 53 12 0 0 20 0 2 0 338488822 368517120 7832 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8957/statm: 89970 7832 1901 364 0 25193 0
[pid=8963] ppid=8957 vsize=284100 CPUtime=0.03 cores=1,3,5,7
/proc/8963/stat : 8963 (python) S 8957 8957 3958 0 -1 4202560 1005 0 0 0 2 1 0 0 20 0 1 0 338488969 290918400 6294 33554432000 4194304 5683644 140736952290704 140736952281792 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8963/statm: 71025 6294 369 364 0 6248 0
[pid=8965] ppid=8963 vsize=282688 CPUtime=24.01 cores=1,3,5,7
/proc/8965/stat : 8965 (scip-2.0.1.linu) R 8963 8965 8965 0 -1 4202496 67282 0 0 0 2355 46 0 0 20 0 1 0 338488969 289472512 66655 33554432000 4194304 15243407 140734144276384 140734144272472 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8965/statm: 70672 66655 875 2698 0 67967 0
[pid=8957/tid=8964] ppid=8955 vsize=359880 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 2 0 338488969 368517120 7832 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 24.69
Current children cumulated vsize (KiB) 926668

[startup+51.1004 s]
/proc/loadavg: 1.99 2.00 2.16 3/195 8973
/proc/meminfo: memFree=30505900/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=435672 CPUtime=0.67 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 12277 1009 1 0 55 12 0 0 20 0 3 0 338488822 446128128 7857 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8957/statm: 108918 7857 1904 364 0 44138 0
[pid=8963] ppid=8957 vsize=359880 CPUtime=0.07 cores=1,3,5,7
/proc/8963/stat : 8963 (python) S 8957 8957 3958 0 -1 4202560 1155 0 0 0 4 3 0 0 20 0 2 0 338488969 368517120 6329 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8963/statm: 89970 6329 397 364 0 25193 0
[pid=8965] ppid=8963 vsize=308952 CPUtime=47.5 cores=1,3,5,7
/proc/8965/stat : 8965 (scip-2.0.1.linu) T 8963 8965 8965 0 -1 4202496 75174 0 0 0 4669 81 0 0 20 0 1 0 338488969 316366848 73121 33554432000 4194304 15243407 140734144276384 140734144271096 13354455 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/8965/statm: 77238 73121 900 2698 0 74533 0
[pid=8963/tid=8970] ppid=8957 vsize=359880 CPUtime=0 cores=1,3,5,7
/proc/8963/task/8970/stat : 8970 (python) S 8957 8957 3958 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338493721 368517120 6329 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8971] ppid=8957 vsize=359892 CPUtime=0 cores=1,3,5,7
/proc/8971/stat : 8971 (python) S 8957 8957 3958 0 -1 4202560 762 0 0 0 0 0 0 0 20 0 1 0 338493724 368529408 6283 33554432000 4194304 5683644 140736952290704 140736952281792 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8971/statm: 89973 6283 336 364 0 25193 0
[pid=8973] ppid=8971 vsize=4616 CPUtime=2.07 cores=1,3,5,7
/proc/8973/stat : 8973 (clasp-1.3.7-x86) R 8971 8973 8973 0 -1 4202496 1262 0 0 0 207 0 0 0 20 0 1 0 338493724 4726784 909 33554432000 134512640 136139019 4287718832 4287717808 134876521 0 0 16781312 16386 0 0 0 17 1 0 0 0 0 0
/proc/8973/statm: 1154 909 182 398 0 754 0
[pid=8957/tid=8964] ppid=8955 vsize=435672 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 3 0 338488969 446128128 7857 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8957/tid=8972] ppid=8955 vsize=435672 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8972/stat : 8972 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 3 0 338493724 446128128 7857 33554432000 4194304 5683644 140736952290704 140573219205680 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 50.31
Current children cumulated vsize (KiB) 1469012

[startup+102.301 s]
/proc/loadavg: 1.99 2.00 2.15 3/200 8978
/proc/meminfo: memFree=30309984/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=511464 CPUtime=0.7 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 13214 1009 1 0 57 13 0 0 20 0 4 0 338488822 523739136 8373 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8957/statm: 127866 8373 1906 364 0 63083 0
[pid=8963] ppid=8957 vsize=359880 CPUtime=0.07 cores=1,3,5,7
/proc/8963/stat : 8963 (python) S 8957 8957 3958 0 -1 4202560 1155 0 0 0 4 3 0 0 20 0 2 0 338488969 368517120 6329 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8963/statm: 89970 6329 397 364 0 25193 0
[pid=8965] ppid=8963 vsize=308952 CPUtime=47.5 cores=1,3,5,7
/proc/8965/stat : 8965 (scip-2.0.1.linu) T 8963 8965 8965 0 -1 4202496 75174 0 0 0 4669 81 0 0 20 0 1 0 338488969 316366848 73121 33554432000 4194304 15243407 140734144276384 140734144271096 13354455 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/8965/statm: 77238 73121 900 2698 0 74533 0
[pid=8963/tid=8970] ppid=8957 vsize=359880 CPUtime=0 cores=1,3,5,7
/proc/8963/task/8970/stat : 8970 (python) S 8957 8957 3958 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338493721 368517120 6329 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8971] ppid=8957 vsize=359892 CPUtime=0.07 cores=1,3,5,7
/proc/8971/stat : 8971 (python) S 8957 8957 3958 0 -1 4202560 1069 0 0 0 4 3 0 0 20 0 2 0 338493724 368529408 6332 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8971/statm: 89973 6332 384 364 0 25193 0
[pid=8973] ppid=8971 vsize=16892 CPUtime=47.12 cores=1,3,5,7
/proc/8973/stat : 8973 (clasp-1.3.7-x86) T 8971 8973 8973 0 -1 4202496 4335 0 0 0 4711 1 0 0 20 0 1 0 338493724 17297408 3982 33554432000 134512640 136139019 4287718832 4287717808 134883721 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8973/statm: 4223 3982 182 398 0 3823 0
[pid=8971/tid=8975] ppid=8957 vsize=359892 CPUtime=0 cores=1,3,5,7
/proc/8971/task/8975/stat : 8975 (python) S 8957 8957 3958 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338498437 368529408 6332 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8976] ppid=8957 vsize=435684 CPUtime=0 cores=1,3,5,7
/proc/8976/stat : 8976 (python) S 8957 8957 3958 0 -1 4202560 917 0 0 0 0 0 0 0 20 0 1 0 338498439 446140416 6312 33554432000 4194304 5683644 140736952290704 140736952281792 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8976/statm: 108921 6312 359 364 0 44138 0
[pid=8978] ppid=8976 vsize=174036 CPUtime=6.11 cores=1,3,5,7
/proc/8978/stat : 8978 (scip-2.0.1.linu) R 8976 8978 8978 0 -1 4202496 40861 0 0 0 597 14 0 0 20 0 1 0 338498440 178212864 40245 33554432000 4194304 13684479 140734902248496 140734902244792 11827963 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8978/statm: 43509 40245 640 2317 0 41187 0
[pid=8957/tid=8964] ppid=8955 vsize=511464 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 27 1009 0 0 0 0 0 0 20 0 4 0 338488969 523739136 8373 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8957/tid=8972] ppid=8955 vsize=511464 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

/proc/9098/stat : 9098 (bsolo_pb10) T 9096 9098 9098 0 -1 4202496 21804 0 0 0 4727 12 0 0 20 0 1 0 338545225 97333248 21326 33554432000 134512640 137138936 4288074288 4288073216 134542484 0 0 16781312 16384 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/9098/statm: 23763 21326 192 642 0 23119 0
[pid=9096/tid=9106] ppid=8957 vsize=739420 CPUtime=0 cores=1,3,5,7
/proc/9096/task/9106/stat : 9106 (python) S 8957 8957 3958 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338549966 757166080 6996 33554432000 4194304 5683644 140736952290704 140573107270192 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=9107] ppid=8957 vsize=815212 CPUtime=0 cores=1,3,5,7
/proc/9107/stat : 9107 (python) S 8957 8957 3958 0 -1 4202560 854 0 0 0 0 0 0 0 20 0 1 0 338549968 834777088 6953 33554432000 4194304 5683644 140736952290704 140736952281792 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/9107/statm: 203803 6953 336 364 0 139005 0
[pid=9109] ppid=9107 vsize=13088 CPUtime=0.14 cores=1,3,5,7
/proc/9109/stat : 9109 (bsolo_pb10) R 9107 9109 9109 0 -1 4202496 1199 0 0 0 14 0 0 0 20 0 1 0 338549968 13402112 830 33554432000 134512640 137138936 4289078496 4288998860 136570369 0 0 16781312 16384 0 0 0 17 5 0 0 0 0 0
/proc/9109/statm: 3272 830 184 642 0 2628 0
[pid=8957/tid=8964] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 32 1009 0 0 0 0 0 0 20 0 9 0 338488969 912375808 8539 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8957/tid=8972] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8972/stat : 8972 (python) S 8955 8957 3958 0 -1 4202560 30 1009 0 0 0 0 0 0 20 0 9 0 338493724 912375808 8539 33554432000 4194304 5683644 140736952290704 140573219205680 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8957/tid=8977] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8977/stat : 8977 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338498440 912375808 8539 33554432000 4194304 5683644 140736952290704 140573138739760 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8957/tid=8989] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8989/stat : 8989 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338507650 912375808 8539 33554432000 4194304 5683644 140736952290704 140573128249904 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8957/tid=8994] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8994/stat : 8994 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338512388 912375808 8539 33554432000 4194304 5683644 140736952290704 140573117760048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8957/tid=9020] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/9020/stat : 9020 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338521812 912375808 8539 33554432000 4194304 5683644 140736952290704 140573107270192 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8957/tid=9097] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/9097/stat : 9097 (python) S 8955 8957 3958 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338545225 912375808 8539 33554432000 4194304 5683644 140736952290704 140573096780336 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8957/tid=9108] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/9108/stat : 9108 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338549968 912375808 8539 33554432000 4194304 5683644 140736952290704 140573086290480 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 613.74
Current children cumulated vsize (KiB) 15939620

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+611.601 s]
/proc/loadavg: 1.99 2.00 2.07 3/234 9109
/proc/meminfo: memFree=28565620/32873648 swapFree=7628/7628
[pid=8957] ppid=8955 vsize=890992 CPUtime=1.51 cores=1,3,5,7
/proc/8957/stat : 8957 (python) S 8955 8957 3958 0 -1 4202496 18313 1009 1 0 137 14 0 0 20 0 9 0 338488822 912375808 8539 33554432000 4194304 5683644 140736952290704 140736952285536 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/8957/statm: 222748 8539 1916 364 0 157950 0
[pid=8963] ppid=8957 vsize=359880 CPUtime=0.16 cores=1,3,5,7
/proc/8963/stat : 8963 (python) S 8957 8957 3958 0 -1 4202560 1188 0 0 0 10 6 0 0 20 0 2 0 338488969 368517120 6330 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8963/statm: 89970 6330 398 364 0 25193 0
[pid=8965] ppid=8963 vsize=319668 CPUtime=92.26 cores=1,3,5,7
/proc/8965/stat : 8965 (scip-2.0.1.linu) T 8963 8965 8965 0 -1 4202496 92411 0 0 0 9137 89 0 0 20 0 1 0 338488969 327340032 77299 33554432000 4194304 15243407 140734144276384 140734144270776 8113827 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/8965/statm: 79917 77299 947 2698 0 77212 0
[pid=8963/tid=8970] ppid=8957 vsize=359880 CPUtime=0 cores=1,3,5,7
/proc/8963/task/8970/stat : 8970 (python) S 8957 8957 3958 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338493721 368517120 6330 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8971] ppid=8957 vsize=359892 CPUtime=0.41 cores=1,3,5,7
/proc/8971/stat : 8971 (python) S 8957 8957 3958 0 -1 4202560 1166 0 0 0 23 18 0 0 20 0 2 0 338493724 368529408 6332 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8971/statm: 89973 6332 384 364 0 25193 0
[pid=8973] ppid=8971 vsize=37228 CPUtime=229.59 cores=1,3,5,7
/proc/8973/stat : 8973 (clasp-1.3.7-x86) T 8971 8973 8973 0 -1 4202496 9408 0 0 0 22957 2 0 0 20 0 1 0 338493724 38121472 9066 33554432000 134512640 136139019 4287718832 4287717808 134882965 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8973/statm: 9307 9066 182 398 0 8907 0
[pid=8971/tid=8975] ppid=8957 vsize=359892 CPUtime=0 cores=1,3,5,7
/proc/8971/task/8975/stat : 8975 (python) S 8957 8957 3958 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338498437 368529408 6332 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8976] ppid=8957 vsize=435684 CPUtime=0.16 cores=1,3,5,7
/proc/8976/stat : 8976 (python) S 8957 8957 3958 0 -1 4202560 1144 0 0 0 9 7 0 0 20 0 2 0 338498439 446140416 6338 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8976/statm: 108921 6338 384 364 0 44138 0
[pid=8978] ppid=8976 vsize=296428 CPUtime=95.44 cores=1,3,5,7
/proc/8978/stat : 8978 (scip-2.0.1.linu) T 8976 8978 8978 0 -1 4202496 74700 0 0 0 9501 43 0 0 20 0 1 0 338498440 303542272 71077 33554432000 4194304 13684479 140734902248496 140734902240360 7390754 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8978/statm: 74107 71077 760 2317 0 71785 0
[pid=8976/tid=8983] ppid=8957 vsize=435684 CPUtime=0 cores=1,3,5,7
/proc/8976/task/8983/stat : 8983 (python) S 8957 8957 3958 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338503073 446140416 6338 33554432000 4194304 5683644 140736952290704 140573219205680 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8988] ppid=8957 vsize=512044 CPUtime=0.06 cores=1,3,5,7
/proc/8988/stat : 8988 (python) S 8957 8957 3958 0 -1 4202560 1121 0 0 0 4 2 0 0 20 0 2 0 338507650 524333056 6977 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8988/statm: 128011 6977 384 364 0 63225 0
[pid=8990] ppid=8988 vsize=88632 CPUtime=47.32 cores=1,3,5,7
/proc/8990/stat : 8990 (minion) T 8988 8990 8990 0 -1 4202496 8523 0 0 0 4730 2 0 0 20 0 1 0 338507650 90759168 14842 33554432000 134512640 165565024 4288112480 4288109728 142764988 0 0 16781312 8396802 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/8990/statm: 22158 14842 336 7582 0 14573 0
[pid=8988/tid=8992] ppid=8957 vsize=512044 CPUtime=0 cores=1,3,5,7
/proc/8988/task/8992/stat : 8992 (python) S 8957 8957 3958 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 338512385 524333056 6977 33554432000 4194304 5683644 140736952290704 140573138739760 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8993] ppid=8957 vsize=587836 CPUtime=0.08 cores=1,3,5,7
/proc/8993/stat : 8993 (python) S 8957 8957 3958 0 -1 4202560 1143 0 0 0 5 3 0 0 20 0 2 0 338512388 601944064 6984 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8993/statm: 146959 6984 384 364 0 82170 0
[pid=8995] ppid=8993 vsize=99964 CPUtime=49.4 cores=1,3,5,7
/proc/8995/stat : 8995 (wbo1.4a) T 8993 8995 8995 0 -1 4202496 24893 0 0 0 4894 46 0 0 20 0 1 0 338512388 102363136 24530 33554432000 134512640 135751879 4291410784 4291408400 134605024 0 0 16781312 3 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/8995/statm: 24991 24530 143 303 0 24686 0
[pid=8993/tid=9017] ppid=8957 vsize=587836 CPUtime=0 cores=1,3,5,7
/proc/8993/task/9017/stat : 9017 (python) S 8957 8957 3958 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338517329 601944064 6984 33554432000 4194304 5683644 140736952290704 140573128249904 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=9019] ppid=8957 vsize=663628 CPUtime=0.08 cores=1,3,5,7
/proc/9019/stat : 9019 (python) S 8957 8957 3958 0 -1 4202560 1156 0 0 0 4 4 0 0 20 0 2 0 338521811 679555072 6990 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/9019/statm: 165907 6990 384 364 0 101115 0
[pid=9021] ppid=9019 vsize=9624972 CPUtime=49.67 cores=1,3,5,7
/proc/9021/stat : 9021 (java) T 9019 9021 9021 0 -1 4202496 18718 0 1 0 4933 34 0 0 20 0 19 0 338521812 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 140735556945280 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/9021/statm: 2406243 344727 2267 9 0 2396937 0
[pid=9021/tid=9022] ppid=9019 vsize=9624972 CPUtime=47.2 cores=1,3,5,7
/proc/9021/task/9022/stat : 9022 (java) T 9019 9021 9021 0 -1 4202560 3417 0 1 0 4698 22 0 0 20 0 19 0 338521813 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825775881312 139825630882964 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=9021/tid=9023] ppid=9019 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/9021/task/9023/stat : 9023 (java) T 9019 9021 9021 0 -1 4202560 81 0 0 0 9 0 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825618921896 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=9021/tid=9024] ppid=9019 vsize=9624972 CPUtime=0.13 cores=1,3,5,7
/proc/9021/task/9024/stat : 9024 (java) T 9019 9021 9021 0 -1 4202560 50 0 0 0 11 2 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825617869352 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=9021/tid=9025] ppid=9019 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/9021/task/9025/stat : 9025 (java) T 9019 9021 9021 0 -1 4202560 27 0 0 0 10 0 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825616816296 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=9021/tid=9026] ppid=9019 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/9021/task/9026/stat : 9026 (java) T 9019 9021 9021 0 -1 4202560 77 0 0 0 8 2 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825615763752 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=9021/tid=9027] ppid=9019 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/9021/task/9027/stat : 9027 (java) T 9019 9021 9021 0 -1 4202560 38 0 0 0 9 0 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825614711720 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=9021/tid=9028] ppid=9019 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/9021/task/9028/stat : 9028 (java) T 9019 9021 9021 0 -1 4202560 78 0 0 0 8 2 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825277561896 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=9021/tid=9029] ppid=9019 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/9021/task/9029/stat : 9029 (java) T 9019 9021 9021 0 -1 4202560 74 0 0 0 9 2 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825276508840 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=9021/tid=9030] ppid=9019 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/9021/task/9030/stat : 9030 (java) T 9019 9021 9021 0 -1 4202560 23 0 0 0 8 0 0 0 20 0 19 0 338521814 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825275456296 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=9021/tid=9031] ppid=9019 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/9021/task/9031/stat : 9031 (java) T 9019 9021 9021 0 -1 4202560 591 0 0 0 0 0 0 0 20 0 19 0 338521815 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825263257960 217413892009 0 0 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=9021/tid=9032] ppid=9019 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/9021/task/9032/stat : 9032 (java) T 9019 9021 9021 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338521816 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825262204280 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=9021/tid=9033] ppid=9019 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/9021/task/9033/stat : 9033 (java) T 9019 9021 9021 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 19 0 338521816 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825261151112 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=9021/tid=9034] ppid=9019 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/9021/task/9034/stat : 9034 (java) T 9019 9021 9021 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338521818 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825260099856 217413900064 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=9021/tid=9035] ppid=9019 vsize=9624972 CPUtime=0.79 cores=1,3,5,7
/proc/9021/task/9035/stat : 9035 (java) T 9019 9021 9021 0 -1 4202560 7375 0 0 0 77 2 0 0 20 0 19 0 338521818 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825259047736 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 5 0 0 0 0 0
[pid=9021/tid=9036] ppid=9019 vsize=9624972 CPUtime=0.72 cores=1,3,5,7
/proc/9021/task/9036/stat : 9036 (java) T 9019 9021 9021 0 -1 4202560 5466 0 0 0 72 0 0 0 20 0 19 0 338521818 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825257995192 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 7 0 0 0 0 0
[pid=9021/tid=9037] ppid=9019 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/9021/task/9037/stat : 9037 (java) T 9019 9021 9021 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338521818 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825256942328 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=9021/tid=9038] ppid=9019 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/9021/task/9038/stat : 9038 (java) T 9019 9021 9021 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338521818 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825255889992 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=9021/tid=9039] ppid=9019 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/9021/task/9039/stat : 9039 (java) T 9019 9021 9021 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338521852 9855971328 344727 33554432000 1073741824 1073778376 140735556954144 139825254806568 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=9019/tid=9040] ppid=8957 vsize=663628 CPUtime=0 cores=1,3,5,7
/proc/9019/task/9040/stat : 9040 (python) S 8957 8957 3958 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338526564 679555072 6990 33554432000 4194304 5683644 140736952290704 140573117760048 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=9096] ppid=8957 vsize=739420 CPUtime=0.07 cores=1,3,5,7
/proc/9096/stat : 9096 (python) S 8957 8957 3958 0 -1 4202560 1167 0 0 0 4 3 0 0 20 0 2 0 338545225 757166080 6996 33554432000 4194304 5683644 140736952290704 140736952282352 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/9096/statm: 184855 6996 384 364 0 120060 0
[pid=9098] ppid=9096 vsize=95052 CPUtime=47.39 cores=1,3,5,7
/proc/9098/stat : 9098 (bsolo_pb10) T 9096 9098 9098 0 -1 4202496 21804 0 0 0 4727 12 0 0 20 0 1 0 338545225 97333248 21326 33554432000 134512640 137138936 4288074288 4288073216 134542484 0 0 16781312 16384 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/9098/statm: 23763 21326 192 642 0 23119 0
[pid=9096/tid=9106] ppid=8957 vsize=739420 CPUtime=0 cores=1,3,5,7
/proc/9096/task/9106/stat : 9106 (python) S 8957 8957 3958 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338549966 757166080 6996 33554432000 4194304 5683644 140736952290704 140573107270192 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=9107] ppid=8957 vsize=815212 CPUtime=0 cores=1,3,5,7
/proc/9107/stat : 9107 (python) S 8957 8957 3958 0 -1 4202560 854 0 0 0 0 0 0 0 20 0 1 0 338549968 834777088 6953 33554432000 4194304 5683644 140736952290704 140736952281792 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/9107/statm: 203803 6953 336 364 0 139005 0
[pid=9109] ppid=9107 vsize=13088 CPUtime=0.14 cores=1,3,5,7
/proc/9109/stat : 9109 (bsolo_pb10) R 9107 9109 9109 0 -1 4202496 1199 0 0 0 14 0 0 0 20 0 1 0 338549968 13402112 830 33554432000 134512640 137138936 4289078496 4288998860 136570369 0 0 16781312 16384 0 0 0 17 5 0 0 0 0 0
/proc/9109/statm: 3272 830 184 642 0 2628 0
[pid=8957/tid=8964] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8964/stat : 8964 (python) S 8955 8957 3958 0 -1 4202560 32 1009 0 0 0 0 0 0 20 0 9 0 338488969 912375808 8539 33554432000 4194304 5683644 140736952290704 140573229695536 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8957/tid=8972] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8972/stat : 8972 (python) S 8955 8957 3958 0 -1 4202560 30 1009 0 0 0 0 0 0 20 0 9 0 338493724 912375808 8539 33554432000 4194304 5683644 140736952290704 140573219205680 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8957/tid=8977] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8977/stat : 8977 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338498440 912375808 8539 33554432000 4194304 5683644 140736952290704 140573138739760 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8957/tid=8989] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8989/stat : 8989 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338507650 912375808 8539 33554432000 4194304 5683644 140736952290704 140573128249904 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8957/tid=8994] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/8994/stat : 8994 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338512388 912375808 8539 33554432000 4194304 5683644 140736952290704 140573117760048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8957/tid=9020] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/9020/stat : 9020 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338521812 912375808 8539 33554432000 4194304 5683644 140736952290704 140573107270192 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8957/tid=9097] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/9097/stat : 9097 (python) S 8955 8957 3958 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338545225 912375808 8539 33554432000 4194304 5683644 140736952290704 140573096780336 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8957/tid=9108] ppid=8955 vsize=890992 CPUtime=0 cores=1,3,5,7
/proc/8957/task/9108/stat : 9108 (python) S 8955 8957 3958 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338549968 912375808 8539 33554432000 4194304 5683644 140736952290704 140573086290480 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 613.74
Current children cumulated vsize (KiB) 15939620

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 8957 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=372791
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=149977
# CPU time returned by wait4() is 1.52277
# while last known CPU time is 613.74
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 611.662
CPU time (s): 613.74
CPU user time (s): 610.89
CPU system time (s): 2.85
CPU usage (%): 100.34
Max. virtual memory (cumulated for all children) (KiB): 15939620

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.37279
system time used= 0.149977
maximum resident set size= 34156
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19322
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1584
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2112
involuntary context switches= 12

runsolver used 2.67259 second user time and 6.17506 second system time

The end

Launcher Data 

Begin job on node122 at 2012-06-03 13:40:27
IDJOB=3742148
IDBENCH=90135
IDSOLVER=2327
FILE ID=node122/3742148-1338723627
RUNJOBID= node122-1338713392-3975
PBS_JOBID= 14636732
Free space on /tmp= 71088 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K96.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742148-1338723627/watcher-3742148-1338723627 -o /tmp/evaluation-result-3742148-1338723627/solver-3742148-1338723627 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742148-1338723627.opb -seed 2098139013 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 12d577f059d1da283bb935f5e106dee7
RANDOM SEED=2098139013

node122.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5438.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        31116472 kB
Buffers:          204940 kB
Cached:           374104 kB
SwapCached:         1804 kB
Active:           887696 kB
Inactive:          38700 kB
Active(anon):     342988 kB
Inactive(anon):     7052 kB
Active(file):     544708 kB
Inactive(file):    31648 kB
Unevictable:        7628 kB
Mlocked:            7628 kB
SwapTotal:      67108856 kB
SwapFree:       67100164 kB
Dirty:            102116 kB
Writeback:             0 kB
AnonPages:        353792 kB
Mapped:            19424 kB
Shmem:                36 kB
Slab:             688784 kB
SReclaimable:     131368 kB
SUnreclaim:       557416 kB
KernelStack:        1504 kB
PageTables:         6396 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:     576904 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70984 MiB
End job on node122 at 2012-06-03 13:50:42