Trace number 3742031

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 662 659.658

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K81.opb
MD5SUM07ddcd712bd0834d3fad6b8ca9adfb4d
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables5508
Total number of constraints5672
Number of constraints which are clauses5591
Number of constraints which are cardinality constraints (but not clauses)81
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint81
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 82
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.46/1.30	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.46/1.30	c INFO: solving HOME/instance-3742031-1338722924.opb
0.46/1.36	c INFO: parsing took 0.06 s
0.56/1.48	c INFO: features took 0.12 CPU seconds
0.66/1.51	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.69)
47.60/48.42	c INFO: running clasp-1.3.7@0 for 45 with 1753 remaining (b = 0.36)
94.73/95.58	c INFO: running scip-2.0.1-spx@0 for 45 with 1706 remaining (b = 0.06)
144.39/145.11	c INFO: running clasp-1.3.7@47 for 45 with 1656 remaining (b = 0.01)
190.06/190.78	c INFO: wrote minion input file to HOME/tmp4H7gzN.minion
190.06/190.78	c INFO: running minion-0.12@0 for 45 with 1611 remaining (b = 0.09)
237.51/238.13	c INFO: running wbo1.4a@0 for 45 with 1564 remaining (b = 0.02)
287.48/288.07	c INFO: running scip-2.0.1-clp@46 for 45 with 1514 remaining (b = 0.01)
333.84/334.32	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1468 remaining (b = 0.04)
383.82/381.86	c INFO: running clasp-1.3.7@92 for 136 with 1418 remaining (b = 0.02)
521.43/519.28	c INFO: running scip-2.0.1-spx@49 for 45 with 1281 remaining (b = 0.01)
566.70/564.45	c INFO: running bsolo_pb10-l3@0 for 45 with 1236 remaining (b = 0.02)
613.94/611.68	c INFO: running bsolo_pb10-l1@0 for 91 with 1189 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742031-1338722924/watcher-3742031-1338722924 -o /tmp/evaluation-result-3742031-1338722924/solver-3742031-1338722924 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742031-1338722924.opb -seed 1509199628 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.92 2.11 2.54 3/217 7183
/proc/meminfo: memFree=28616832/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=15276 CPUtime=0 cores=0,2,4,6
/proc/7183/stat : 7183 (python) R 7181 7183 2614 0 -1 4202496 632 0 0 0 0 0 0 0 20 0 1 0 338417416 15642624 526 33554432000 4194304 5683644 140734693950576 140734693932248 240565175184 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 3819 527 312 364 0 261 0

[startup+0.060723 s]
/proc/loadavg: 1.92 2.11 2.54 3/217 7183
/proc/meminfo: memFree=28616832/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=16628 CPUtime=0.01 cores=0,2,4,6
/proc/7183/stat : 7183 (python) D 7181 7183 2614 0 -1 4202496 1183 0 0 0 1 0 0 0 20 0 1 0 338417416 17027072 993 33554432000 4194304 5683644 140734693950576 140734693933944 240565173941 0 0 16781312 2 18446744072100609316 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 4157 993 416 364 0 599 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16628

[startup+0.100294 s]
/proc/loadavg: 1.92 2.11 2.54 3/217 7183
/proc/meminfo: memFree=28616832/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=19088 CPUtime=0.02 cores=0,2,4,6
/proc/7183/stat : 7183 (python) R 7181 7183 2614 0 -1 4202496 1389 0 0 0 1 1 0 0 20 0 1 0 338417416 19546112 1144 33554432000 4194304 5683644 140734693950576 140734693881080 240564801428 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 4772 1144 430 364 0 696 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19088

[startup+0.300283 s]
/proc/loadavg: 1.92 2.11 2.54 3/217 7183
/proc/meminfo: memFree=28616832/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=96896 CPUtime=0.12 cores=0,2,4,6
/proc/7183/stat : 7183 (python) D 7181 7183 2614 0 -1 4202496 3899 1006 1 0 8 4 0 0 20 0 1 0 338417416 99221504 2647 33554432000 4194304 5683644 140734693950576 140734693868600 240565173941 0 0 16781312 2 18446744072100609316 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 24224 2647 822 364 0 1715 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 96896

[startup+0.700231 s]
/proc/loadavg: 1.92 2.11 2.54 3/217 7183
/proc/meminfo: memFree=28616832/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=122348 CPUtime=0.25 cores=0,2,4,6
/proc/7183/stat : 7183 (python) D 7181 7183 2614 0 -1 4202496 5962 1006 1 0 15 10 0 0 20 0 1 0 338417416 125284352 3841 33554432000 4194304 5683644 140734693950576 140734693875464 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 30587 3841 1015 364 0 2830 0
Current children cumulated CPU time (s) 0.25
Current children cumulated vsize (KiB) 122348

[startup+1.50025 s]
/proc/loadavg: 1.92 2.11 2.54 3/218 7188
/proc/meminfo: memFree=28592140/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=283600 CPUtime=0.66 cores=0,2,4,6
/proc/7183/stat : 7183 (python) R 7181 7183 2614 0 -1 4202496 10979 1006 1 0 50 16 0 0 20 0 1 0 338417416 290406400 7695 33554432000 4194304 5683644 140734693950576 140734693930360 240572799832 0 0 16781312 33554434 0 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 70900 7695 1883 364 0 6126 0
Current children cumulated CPU time (s) 0.66
Current children cumulated vsize (KiB) 283600

[startup+3.10038 s]
/proc/loadavg: 1.92 2.11 2.54 3/221 7191
/proc/meminfo: memFree=28517212/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=359392 CPUtime=0.67 cores=0,2,4,6
/proc/7183/stat : 7183 (python) S 7181 7183 2614 0 -1 4202496 11205 1006 1 0 51 16 0 0 20 0 2 0 338417416 368017408 7718 33554432000 4194304 5683644 140734693950576 140734693945408 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 89848 7718 1900 364 0 25071 0
[pid=7189] ppid=7183 vsize=283612 CPUtime=0 cores=0,2,4,6
/proc/7189/stat : 7189 (python) S 7183 7183 2614 0 -1 4202560 762 0 0 0 0 0 0 0 20 0 1 0 338417567 290418688 6153 33554432000 4194304 5683644 140734693950576 140734693941664 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/7189/statm: 70903 6153 341 364 0 6126 0
[pid=7191] ppid=7189 vsize=103412 CPUtime=1.57 cores=0,2,4,6
/proc/7191/stat : 7191 (scip-2.0.1.linu) R 7189 7191 7191 0 -1 4202496 22499 0 0 0 152 5 0 0 20 0 1 0 338417567 105893888 21900 33554432000 4194304 15243407 140737059688160 140737059684456 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/7191/statm: 25853 21900 682 2698 0 23148 0
[pid=7183/tid=7190] ppid=7181 vsize=359392 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 27 1006 0 0 0 0 0 0 20 0 2 0 338417567 368017408 7718 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 2.24
Current children cumulated vsize (KiB) 746416
heavy processes:

[startup+6.30035 s]
/proc/loadavg: 2.01 2.13 2.54 3/221 7191
/proc/meminfo: memFree=28461040/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=359392 CPUtime=0.67 cores=0,2,4,6
/proc/7183/stat : 7183 (python) S 7181 7183 2614 0 -1 4202496 11205 1006 1 0 51 16 0 0 20 0 2 0 338417416 368017408 7718 33554432000 4194304 5683644 140734693950576 140734693945408 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 89848 7718 1900 364 0 25071 0
[pid=7189] ppid=7183 vsize=283612 CPUtime=0 cores=0,2,4,6
/proc/7189/stat : 7189 (python) S 7183 7183 2614 0 -1 4202560 888 0 0 0 0 0 0 0 20 0 1 0 338417567 290418688 6176 33554432000 4194304 5683644 140734693950576 140734693941664 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/7189/statm: 70903 6176 364 364 0 6126 0
[pid=7191] ppid=7189 vsize=145008 CPUtime=4.77 cores=0,2,4,6
/proc/7191/stat : 7191 (scip-2.0.1.linu) R 7189 7191 7191 0 -1 4202496 32887 0 0 0 467 10 0 0 20 0 1 0 338417567 148488192 32288 33554432000 4194304 15243407 140737059688160 140737059684456 4865653 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/7191/statm: 36252 32288 686 2698 0 33547 0
[pid=7183/tid=7190] ppid=7181 vsize=359392 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 27 1006 0 0 0 0 0 0 20 0 2 0 338417567 368017408 7718 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 5.44
Current children cumulated vsize (KiB) 788012

[startup+12.7004 s]
/proc/loadavg: 2.01 2.13 2.54 3/221 7191
/proc/meminfo: memFree=28407180/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=359392 CPUtime=0.67 cores=0,2,4,6
/proc/7183/stat : 7183 (python) S 7181 7183 2614 0 -1 4202496 11205 1006 1 0 51 16 0 0 20 0 2 0 338417416 368017408 7718 33554432000 4194304 5683644 140734693950576 140734693945408 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 89848 7718 1900 364 0 25071 0
[pid=7189] ppid=7183 vsize=283612 CPUtime=0 cores=0,2,4,6
/proc/7189/stat : 7189 (python) S 7183 7183 2614 0 -1 4202560 895 0 0 0 0 0 0 0 20 0 1 0 338417567 290418688 6176 33554432000 4194304 5683644 140734693950576 140734693941664 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/7189/statm: 70903 6176 364 364 0 6126 0
[pid=7191] ppid=7189 vsize=196460 CPUtime=11.17 cores=0,2,4,6
/proc/7191/stat : 7191 (scip-2.0.1.linu) R 7189 7191 7191 0 -1 4202496 45859 0 0 0 1097 20 0 0 20 0 1 0 338417567 201175040 45260 33554432000 4194304 15243407 140737059688160 140737059684472 4880950 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/7191/statm: 49115 45260 686 2698 0 46410 0
[pid=7183/tid=7190] ppid=7181 vsize=359392 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 27 1006 0 0 0 0 0 0 20 0 2 0 338417567 368017408 7718 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 11.84
Current children cumulated vsize (KiB) 839464

[startup+25.5004 s]
/proc/loadavg: 2.00 2.12 2.53 3/221 7191
/proc/meminfo: memFree=28367044/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=359392 CPUtime=0.67 cores=0,2,4,6
/proc/7183/stat : 7183 (python) S 7181 7183 2614 0 -1 4202496 11205 1006 1 0 51 16 0 0 20 0 2 0 338417416 368017408 7718 33554432000 4194304 5683644 140734693950576 140734693945408 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 89848 7718 1900 364 0 25071 0
[pid=7189] ppid=7183 vsize=283612 CPUtime=0.03 cores=0,2,4,6
/proc/7189/stat : 7189 (python) S 7183 7183 2614 0 -1 4202560 1004 0 0 0 2 1 0 0 20 0 1 0 338417567 290418688 6181 33554432000 4194304 5683644 140734693950576 140734693941664 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/7189/statm: 70903 6181 369 364 0 6126 0
[pid=7191] ppid=7189 vsize=235524 CPUtime=23.97 cores=0,2,4,6
/proc/7191/stat : 7191 (scip-2.0.1.linu) R 7189 7191 7191 0 -1 4202496 55151 0 0 0 2347 50 0 0 20 0 1 0 338417567 241176576 54514 33554432000 4194304 15243407 140737059688160 140737059684248 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/7191/statm: 58881 54514 874 2698 0 56176 0
[pid=7183/tid=7190] ppid=7181 vsize=359392 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 27 1006 0 0 0 0 0 0 20 0 2 0 338417567 368017408 7718 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 24.67
Current children cumulated vsize (KiB) 878528

[startup+51.1004 s]
/proc/loadavg: 2.00 2.11 2.52 3/226 7196
/proc/meminfo: memFree=28341476/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=435184 CPUtime=0.69 cores=0,2,4,6
/proc/7183/stat : 7183 (python) S 7181 7183 2614 0 -1 4202496 12162 1006 1 0 53 16 0 0 20 0 3 0 338417416 445628416 7727 33554432000 4194304 5683644 140734693950576 140734693945408 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 108796 7727 1903 364 0 44016 0
[pid=7189] ppid=7183 vsize=359392 CPUtime=0.06 cores=0,2,4,6
/proc/7189/stat : 7189 (python) S 7183 7183 2614 0 -1 4202560 1152 0 0 0 4 2 0 0 20 0 2 0 338417567 368017408 6216 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7189/statm: 89848 6216 397 364 0 25071 0
[pid=7191] ppid=7189 vsize=256488 CPUtime=46.86 cores=0,2,4,6
/proc/7191/stat : 7191 (scip-2.0.1.linu) T 7189 7191 7191 0 -1 4202496 60286 0 0 0 4617 69 0 0 20 0 1 0 338417567 262643712 59647 33554432000 4194304 15243407 140737059688160 140737059682872 8112832 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/7191/statm: 64122 59647 899 2698 0 61417 0
[pid=7189/tid=7193] ppid=7183 vsize=359392 CPUtime=0 cores=0,2,4,6
/proc/7189/task/7193/stat : 7193 (python) S 7183 7183 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338422256 368017408 6216 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7194] ppid=7183 vsize=359404 CPUtime=0 cores=0,2,4,6
/proc/7194/stat : 7194 (python) S 7183 7183 2614 0 -1 4202560 777 0 0 0 0 0 0 0 20 0 1 0 338422258 368029696 6154 33554432000 4194304 5683644 140734693950576 140734693941664 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/7194/statm: 89851 6154 336 364 0 25071 0
[pid=7196] ppid=7194 vsize=5472 CPUtime=2.67 cores=0,2,4,6
/proc/7196/stat : 7196 (clasp-1.3.7-x86) R 7194 7196 7196 0 -1 4202496 1448 0 0 0 267 0 0 0 20 0 1 0 338422258 5603328 1111 33554432000 134512640 136139019 4294358752 4294357616 134608416 0 0 16781312 16386 0 0 0 17 0 0 0 0 0 0
/proc/7196/statm: 1368 1111 182 398 0 968 0
[pid=7183/tid=7190] ppid=7181 vsize=435184 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 27 1006 0 0 0 0 0 0 20 0 3 0 338417567 445628416 7727 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=7183/tid=7195] ppid=7181 vsize=435184 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7195/stat : 7195 (python) S 7181 7183 2614 0 -1 4202560 22 1006 0 0 0 0 0 0 20 0 3 0 338422258 445628416 7727 33554432000 4194304 5683644 140734693950576 139709259936304 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 50.28
Current children cumulated vsize (KiB) 1415940

[startup+102.301 s]
/proc/loadavg: 2.00 2.09 2.49 3/231 7203
/proc/meminfo: memFree=28172232/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=511524 CPUtime=0.72 cores=0,2,4,6
/proc/7183/stat : 7183 (python) S 7181 7183 2614 0 -1 4202496 13114 1006 1 0 56 16 0 0 20 0 4 0 338417416 523800576 8369 33554432000 4194304 5683644 140734693950576 140734693945408 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7183/statm: 127881 8369 1905 364 0 63098 0
[pid=7189] ppid=7183 vsize=359392 CPUtime=0.06 cores=0,2,4,6
/proc/7189/stat : 7189 (python) S 7183 7183 2614 0 -1 4202560 1152 0 0 0 4 2 0 0 20 0 2 0 338417567 368017408 6216 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7189/statm: 89848 6216 397 364 0 25071 0
[pid=7191] ppid=7189 vsize=256488 CPUtime=46.86 cores=0,2,4,6
/proc/7191/stat : 7191 (scip-2.0.1.linu) T 7189 7191 7191 0 -1 4202496 60286 0 0 0 4617 69 0 0 20 0 1 0 338417567 262643712 59647 33554432000 4194304 15243407 140737059688160 140737059682872 8112832 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/7191/statm: 64122 59647 899 2698 0 61417 0
[pid=7189/tid=7193] ppid=7183 vsize=359392 CPUtime=0 cores=0,2,4,6
/proc/7189/task/7193/stat : 7193 (python) S 7183 7183 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338422256 368017408 6216 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7194] ppid=7183 vsize=359404 CPUtime=0.07 cores=0,2,4,6
/proc/7194/stat : 7194 (python) S 7183 7183 2614 0 -1 4202560 1092 0 0 0 4 3 0 0 20 0 2 0 338422258 368029696 6203 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7194/statm: 89851 6203 384 364 0 25071 0
[pid=7196] ppid=7194 vsize=11944 CPUtime=47.11 cores=0,2,4,6
/proc/7196/stat : 7196 (clasp-1.3.7-x86) T 7194 7196 7196 0 -1 4202496 3082 0 0 0 4711 0 0 0 20 0 1 0 338422258 12230656 2745 33554432000 134512640 136139019 4294358752 4294357616 134608331 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/7196/statm: 2986 2745 182 398 0 2586 0
[pid=7194/tid=7200] ppid=7183 vsize=359404 CPUtime=0 cores=0,2,4,6
/proc/7194/task/7200/stat : 7200 (python) S 7183 7183 2614 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338426971 368029696 6203 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7201] ppid=7183 vsize=435744 CPUtime=0 cores=0,2,4,6
/proc/7201/stat : 7201 (python) S 7183 7183 2614 0 -1 4202560 930 0 0 0 0 0 0 0 20 0 1 0 338426974 446201856 6309 33554432000 4194304 5683644 140734693950576 140734693941664 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/7201/statm: 108936 6309 359 364 0 44153 0
[pid=7203] ppid=7201 vsize=164624 CPUtime=6.71 cores=0,2,4,6
/proc/7203/stat : 7203 (scip-2.0.1.linu) R 7201 7203 7203 0 -1 4202496 37923 0 0 0 657 14 0 0 20 0 1 0 338426974 168574976 37308 33554432000 4194304 13684479 140733271888496 140733271884808 6144939 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/7203/statm: 41156 37308 639 2317 0 38834 0
[pid=7183/tid=7190] ppid=7181 vsize=511524 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 27 1006 0 0 0 0 0 0 20 0 4 0 338417567 523800576 8369 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=7183/tid=7195] ppid=7181 vsize=511524 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

/proc/7309/stat : 7309 (bsolo_pb10) T 7307 7309 7309 0 -1 4202496 10986 0 0 0 4714 3 0 0 20 0 1 0 338473861 53215232 10553 33554432000 134512640 137138936 4290212608 4290211320 136504124 0 0 16781312 16384 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/7309/statm: 12992 10553 192 642 0 12348 0
[pid=7307/tid=7331] ppid=7183 vsize=738912 CPUtime=0 cores=0,2,4,6
/proc/7307/task/7331/stat : 7331 (python) S 7183 7183 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338478582 756645888 6867 33554432000 4194304 5683644 140734693950576 139709147755056 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7332] ppid=7183 vsize=814704 CPUtime=0.08 cores=0,2,4,6
/proc/7332/stat : 7332 (python) S 7183 7183 2614 0 -1 4202560 1091 0 0 0 5 3 0 0 20 0 1 0 338478584 834256896 6853 33554432000 4194304 5683644 140734693950576 140734693941664 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/7332/statm: 203676 6853 365 364 0 138878 0
[pid=7334] ppid=7332 vsize=131168 CPUtime=47.88 cores=0,2,4,6
/proc/7334/stat : 7334 (bsolo_pb10) R 7332 7334 7334 0 -1 4202496 30781 0 0 0 4763 25 0 0 20 0 1 0 338478585 134316032 30350 33554432000 134512640 137138936 4290347872 4290346800 134543241 0 0 16781312 16384 0 0 0 17 0 0 0 0 0 0
/proc/7334/statm: 32792 30350 191 642 0 32148 0
[pid=7183/tid=7190] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 32 1006 0 0 0 0 0 0 20 0 9 0 338417567 911855616 8409 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=7183/tid=7195] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7195/stat : 7195 (python) S 7181 7183 2614 0 -1 4202560 31 1006 0 0 0 0 0 0 20 0 9 0 338422258 911855616 8409 33554432000 4194304 5683644 140734693950576 139709259936304 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=7183/tid=7202] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7202/stat : 7202 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338426974 911855616 8409 33554432000 4194304 5683644 140734693950576 139709179224624 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7183/tid=7206] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7206/stat : 7206 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338436495 911855616 8409 33554432000 4194304 5683644 140734693950576 139709168734768 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7183/tid=7214] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7214/stat : 7214 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338441230 911855616 8409 33554432000 4194304 5683644 140734693950576 139709158244912 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7183/tid=7272] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7272/stat : 7272 (python) S 7181 7183 2614 0 -1 4202560 22 1006 0 0 0 0 0 0 20 0 9 0 338450848 911855616 8409 33554432000 4194304 5683644 140734693950576 139709147755056 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=7183/tid=7308] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7308/stat : 7308 (python) S 7181 7183 2614 0 -1 4202560 22 1006 0 0 0 0 0 0 20 0 9 0 338473861 911855616 8409 33554432000 4194304 5683644 140734693950576 139709137265200 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7183/tid=7333] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7333/stat : 7333 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338478585 911855616 8409 33554432000 4194304 5683644 140734693950576 139709126775344 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 662
Current children cumulated vsize (KiB) 15872040

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+659.602 s]
/proc/loadavg: 2.11 2.08 2.29 3/268 7335
/proc/meminfo: memFree=27877516/32873844 swapFree=7116/7116
[pid=7183] ppid=7181 vsize=890484 CPUtime=1.47 cores=0,2,4,6
/proc/7183/stat : 7183 (python) S 7181 7183 2614 0 -1 4202496 18107 1006 1 0 129 18 0 0 20 0 9 0 338417416 911855616 8409 33554432000 4194304 5683644 140734693950576 140734693945408 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/7183/statm: 222621 8409 1915 364 0 157823 0
[pid=7189] ppid=7183 vsize=359392 CPUtime=0.14 cores=0,2,4,6
/proc/7189/stat : 7189 (python) S 7183 7183 2614 0 -1 4202560 1189 0 0 0 9 5 0 0 20 0 2 0 338417567 368017408 6217 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7189/statm: 89848 6217 398 364 0 25071 0
[pid=7191] ppid=7189 vsize=262920 CPUtime=93.07 cores=0,2,4,6
/proc/7191/stat : 7191 (scip-2.0.1.linu) T 7189 7191 7191 0 -1 4202496 87812 0 0 0 9226 81 0 0 20 0 1 0 338417567 269230080 63303 33554432000 4194304 15243407 140737059688160 140737059682552 8129135 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/7191/statm: 65730 63303 947 2698 0 63025 0
[pid=7189/tid=7193] ppid=7183 vsize=359392 CPUtime=0 cores=0,2,4,6
/proc/7189/task/7193/stat : 7193 (python) S 7183 7183 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338422256 368017408 6217 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7194] ppid=7183 vsize=359404 CPUtime=0.4 cores=0,2,4,6
/proc/7194/stat : 7194 (python) S 7183 7183 2614 0 -1 4202560 1183 0 0 0 24 16 0 0 20 0 2 0 338422258 368029696 6203 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/7194/statm: 89851 6203 384 364 0 25071 0
[pid=7196] ppid=7194 vsize=24228 CPUtime=229.59 cores=0,2,4,6
/proc/7196/stat : 7196 (clasp-1.3.7-x86) T 7194 7196 7196 0 -1 4202496 6138 0 0 0 22958 1 0 0 20 0 1 0 338422258 24809472 5793 33554432000 134512640 136139019 4294358752 4294357376 134599005 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/7196/statm: 6057 5793 182 398 0 5657 0
[pid=7194/tid=7200] ppid=7183 vsize=359404 CPUtime=0 cores=0,2,4,6
/proc/7194/task/7200/stat : 7200 (python) S 7183 7183 2614 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338426971 368029696 6203 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7201] ppid=7183 vsize=435744 CPUtime=0.18 cores=0,2,4,6
/proc/7201/stat : 7201 (python) S 7183 7183 2614 0 -1 4202560 1158 0 0 0 10 8 0 0 20 0 2 0 338426974 446201856 6335 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/7201/statm: 108936 6335 384 364 0 44153 0
[pid=7203] ppid=7201 vsize=233636 CPUtime=94.61 cores=0,2,4,6
/proc/7203/stat : 7203 (scip-2.0.1.linu) T 7201 7203 7203 0 -1 4202496 60483 0 0 0 9395 66 0 0 20 0 1 0 338426974 239243264 56559 33554432000 4194304 13684479 140733271888496 140733271884584 11876362 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/7203/statm: 58409 56559 776 2317 0 56087 0
[pid=7201/tid=7204] ppid=7183 vsize=435744 CPUtime=0 cores=0,2,4,6
/proc/7201/task/7204/stat : 7204 (python) S 7183 7183 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338431924 446201856 6335 33554432000 4194304 5683644 140734693950576 139709259936304 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7205] ppid=7183 vsize=511536 CPUtime=0.07 cores=0,2,4,6
/proc/7205/stat : 7205 (python) S 7183 7183 2614 0 -1 4202560 1126 0 0 0 4 3 0 0 20 0 2 0 338436494 523812864 6848 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/7205/statm: 127884 6848 384 364 0 63098 0
[pid=7207] ppid=7205 vsize=81872 CPUtime=47.3 cores=0,2,4,6
/proc/7207/stat : 7207 (minion) T 7205 7207 7207 0 -1 4202496 6604 0 0 0 4729 1 0 0 20 0 1 0 338436495 83836928 13171 33554432000 134512640 165565024 4288994464 4288991712 142765004 0 0 16781312 8396802 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/7207/statm: 20468 13171 336 7582 0 12883 0
[pid=7205/tid=7212] ppid=7183 vsize=511536 CPUtime=0 cores=0,2,4,6
/proc/7205/task/7212/stat : 7212 (python) S 7183 7183 2614 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338441227 523812864 6848 33554432000 4194304 5683644 140734693950576 139709179224624 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7213] ppid=7183 vsize=587328 CPUtime=0.08 cores=0,2,4,6
/proc/7213/stat : 7213 (python) S 7183 7183 2614 0 -1 4202560 1130 0 0 0 5 3 0 0 20 0 2 0 338441229 601423872 6855 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/7213/statm: 146832 6855 384 364 0 82043 0
[pid=7215] ppid=7213 vsize=100652 CPUtime=49.89 cores=0,2,4,6
/proc/7215/stat : 7215 (wbo1.4a) T 7213 7215 7215 0 -1 4202496 25033 0 0 0 4942 47 0 0 20 0 1 0 338441230 103067648 24688 33554432000 134512640 135751879 4289386112 4289384288 134616480 0 0 16781312 3 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/7215/statm: 25163 24688 143 303 0 24858 0
[pid=7213/tid=7266] ppid=7183 vsize=587328 CPUtime=0 cores=0,2,4,6
/proc/7213/task/7266/stat : 7266 (python) S 7183 7183 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338446220 601423872 6855 33554432000 4194304 5683644 140734693950576 139709168734768 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7271] ppid=7183 vsize=663120 CPUtime=0.08 cores=0,2,4,6
/proc/7271/stat : 7271 (python) S 7183 7183 2614 0 -1 4202560 1142 0 0 0 5 3 0 0 20 0 2 0 338450848 679034880 6861 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/7271/statm: 165780 6861 384 364 0 100988 0
[pid=7273] ppid=7271 vsize=9624972 CPUtime=49.91 cores=0,2,4,6
/proc/7273/stat : 7273 (java) T 7271 7273 7273 0 -1 4202496 21571 0 1 0 4965 26 0 0 20 0 19 0 338450848 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140737488195840 240568533053 0 0 0 16800975 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7273/statm: 2406243 211608 2266 9 0 2396937 0
[pid=7273/tid=7274] ppid=7271 vsize=9624972 CPUtime=46.94 cores=0,2,4,6
/proc/7273/task/7274/stat : 7274 (java) T 7271 7273 7273 0 -1 4202560 3040 0 1 0 4683 11 0 0 20 0 19 0 338450850 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121063964680 140120909621373 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7275] ppid=7271 vsize=9624972 CPUtime=0.13 cores=0,2,4,6
/proc/7273/task/7275/stat : 7275 (java) T 7271 7273 7273 0 -1 4202560 36 0 0 0 11 2 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121036417960 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=7273/tid=7276] ppid=7271 vsize=9624972 CPUtime=0.13 cores=0,2,4,6
/proc/7273/task/7276/stat : 7276 (java) T 7271 7273 7273 0 -1 4202560 130 0 0 0 13 0 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121035365416 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7277] ppid=7271 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/7273/task/7277/stat : 7277 (java) T 7271 7273 7273 0 -1 4202560 24 0 0 0 12 0 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121034312360 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=7273/tid=7278] ppid=7271 vsize=9624972 CPUtime=0.14 cores=0,2,4,6
/proc/7273/task/7278/stat : 7278 (java) T 7271 7273 7273 0 -1 4202560 31 0 0 0 12 2 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121033259816 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7279] ppid=7271 vsize=9624972 CPUtime=0.15 cores=0,2,4,6
/proc/7273/task/7279/stat : 7279 (java) T 7271 7273 7273 0 -1 4202560 99 0 0 0 13 2 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121032206760 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7280] ppid=7271 vsize=9624972 CPUtime=0.16 cores=0,2,4,6
/proc/7273/task/7280/stat : 7280 (java) T 7271 7273 7273 0 -1 4202560 40 0 0 0 14 2 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121031154216 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=7273/tid=7281] ppid=7271 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/7273/task/7281/stat : 7281 (java) T 7271 7273 7273 0 -1 4202560 81 0 0 0 10 2 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121030101160 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7282] ppid=7271 vsize=9624972 CPUtime=0.14 cores=0,2,4,6
/proc/7273/task/7282/stat : 7282 (java) T 7271 7273 7273 0 -1 4202560 99 0 0 0 12 2 0 0 20 0 19 0 338450851 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140121029048616 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7283] ppid=7271 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/7273/task/7283/stat : 7283 (java) T 7271 7273 7273 0 -1 4202560 433 0 0 0 0 0 0 0 20 0 19 0 338450852 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120418134888 240568547241 0 0 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=7273/tid=7284] ppid=7271 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/7273/task/7284/stat : 7284 (java) T 7271 7273 7273 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338450853 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120417081208 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=7273/tid=7285] ppid=7271 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/7273/task/7285/stat : 7285 (java) T 7271 7273 7273 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338450853 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120416028040 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=7273/tid=7286] ppid=7271 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/7273/task/7286/stat : 7286 (java) T 7271 7273 7273 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338450855 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120414976784 240568555296 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=7273/tid=7287] ppid=7271 vsize=9624972 CPUtime=0.77 cores=0,2,4,6
/proc/7273/task/7287/stat : 7287 (java) T 7271 7273 7273 0 -1 4202560 7239 0 0 0 75 2 0 0 20 0 19 0 338450855 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120413923640 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=7273/tid=7288] ppid=7271 vsize=9624972 CPUtime=0.86 cores=0,2,4,6
/proc/7273/task/7288/stat : 7288 (java) T 7271 7273 7273 0 -1 4202560 8898 0 0 0 86 0 0 0 20 0 19 0 338450855 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120412871096 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7289] ppid=7271 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/7273/task/7289/stat : 7289 (java) T 7271 7273 7273 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338450855 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120411818232 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7273/tid=7290] ppid=7271 vsize=9624972 CPUtime=0.01 cores=0,2,4,6
/proc/7273/task/7290/stat : 7290 (java) T 7271 7273 7273 0 -1 4202560 3 0 0 0 1 0 0 0 20 0 19 0 338450856 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120410765896 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=7273/tid=7291] ppid=7271 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/7273/task/7291/stat : 7291 (java) T 7271 7273 7273 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338450888 9855971328 211608 33554432000 1073741824 1073778376 140737488204704 140120409712168 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=7271/tid=7296] ppid=7183 vsize=663120 CPUtime=0 cores=0,2,4,6
/proc/7271/task/7296/stat : 7296 (python) S 7183 7183 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338455600 679034880 6861 33554432000 4194304 5683644 140734693950576 139709158244912 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7307] ppid=7183 vsize=738912 CPUtime=0.08 cores=0,2,4,6
/proc/7307/stat : 7307 (python) S 7183 7183 2614 0 -1 4202560 1163 0 0 0 6 2 0 0 20 0 2 0 338473861 756645888 6867 33554432000 4194304 5683644 140734693950576 140734693942224 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/7307/statm: 184728 6867 384 364 0 119933 0
[pid=7309] ppid=7307 vsize=51968 CPUtime=47.17 cores=0,2,4,6
/proc/7309/stat : 7309 (bsolo_pb10) T 7307 7309 7309 0 -1 4202496 10986 0 0 0 4714 3 0 0 20 0 1 0 338473861 53215232 10553 33554432000 134512640 137138936 4290212608 4290211320 136504124 0 0 16781312 16384 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/7309/statm: 12992 10553 192 642 0 12348 0
[pid=7307/tid=7331] ppid=7183 vsize=738912 CPUtime=0 cores=0,2,4,6
/proc/7307/task/7331/stat : 7331 (python) S 7183 7183 2614 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338478582 756645888 6867 33554432000 4194304 5683644 140734693950576 139709147755056 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7332] ppid=7183 vsize=814704 CPUtime=0.08 cores=0,2,4,6
/proc/7332/stat : 7332 (python) S 7183 7183 2614 0 -1 4202560 1091 0 0 0 5 3 0 0 20 0 1 0 338478584 834256896 6853 33554432000 4194304 5683644 140734693950576 140734693941664 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/7332/statm: 203676 6853 365 364 0 138878 0
[pid=7334] ppid=7332 vsize=131168 CPUtime=47.88 cores=0,2,4,6
/proc/7334/stat : 7334 (bsolo_pb10) R 7332 7334 7334 0 -1 4202496 30781 0 0 0 4763 25 0 0 20 0 1 0 338478585 134316032 30350 33554432000 134512640 137138936 4290347872 4290346800 134543241 0 0 16781312 16384 0 0 0 17 0 0 0 0 0 0
/proc/7334/statm: 32792 30350 191 642 0 32148 0
[pid=7183/tid=7190] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7190/stat : 7190 (python) S 7181 7183 2614 0 -1 4202560 32 1006 0 0 0 0 0 0 20 0 9 0 338417567 911855616 8409 33554432000 4194304 5683644 140734693950576 139709270426160 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=7183/tid=7195] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7195/stat : 7195 (python) S 7181 7183 2614 0 -1 4202560 31 1006 0 0 0 0 0 0 20 0 9 0 338422258 911855616 8409 33554432000 4194304 5683644 140734693950576 139709259936304 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=7183/tid=7202] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7202/stat : 7202 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338426974 911855616 8409 33554432000 4194304 5683644 140734693950576 139709179224624 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7183/tid=7206] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7206/stat : 7206 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338436495 911855616 8409 33554432000 4194304 5683644 140734693950576 139709168734768 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7183/tid=7214] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7214/stat : 7214 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338441230 911855616 8409 33554432000 4194304 5683644 140734693950576 139709158244912 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=7183/tid=7272] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7272/stat : 7272 (python) S 7181 7183 2614 0 -1 4202560 22 1006 0 0 0 0 0 0 20 0 9 0 338450848 911855616 8409 33554432000 4194304 5683644 140734693950576 139709147755056 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=7183/tid=7308] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7308/stat : 7308 (python) S 7181 7183 2614 0 -1 4202560 22 1006 0 0 0 0 0 0 20 0 9 0 338473861 911855616 8409 33554432000 4194304 5683644 140734693950576 139709137265200 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=7183/tid=7333] ppid=7181 vsize=890484 CPUtime=0 cores=0,2,4,6
/proc/7183/task/7333/stat : 7333 (python) S 7181 7183 2614 0 -1 4202560 21 1006 0 0 0 0 0 0 20 0 9 0 338478585 911855616 8409 33554432000 4194304 5683644 140734693950576 139709126775344 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 662
Current children cumulated vsize (KiB) 15872040

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 7183 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=294803
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=189971
# CPU time returned by wait4() is 1.48477
# while last known CPU time is 662
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 659.658
CPU time (s): 662
CPU user time (s): 658.89
CPU system time (s): 3.11
CPU usage (%): 100.355
Max. virtual memory (cumulated for all children) (KiB): 15872040

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.2948
system time used= 0.189971
maximum resident set size= 33636
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19113
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1400
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2090
involuntary context switches= 20

runsolver used 3.01254 second user time and 6.90395 second system time

The end

Launcher Data 

Begin job on node146 at 2012-06-03 13:28:44
IDJOB=3742031
IDBENCH=90126
IDSOLVER=2327
FILE ID=node146/3742031-1338722924
RUNJOBID= node146-1338713700-2630
PBS_JOBID= 14636739
Free space on /tmp= 70808 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K81.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742031-1338722924/watcher-3742031-1338722924 -o /tmp/evaluation-result-3742031-1338722924/solver-3742031-1338722924 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742031-1338722924.opb -seed 1509199628 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 07ddcd712bd0834d3fad6b8ca9adfb4d
RANDOM SEED=1509199628

node146.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        28617468 kB
Buffers:          281172 kB
Cached:           613268 kB
SwapCached:         1792 kB
Active:          3101812 kB
Inactive:         303680 kB
Active(anon):    2511536 kB
Inactive(anon):     1748 kB
Active(file):     590276 kB
Inactive(file):   301932 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67099888 kB
Dirty:            102052 kB
Writeback:             0 kB
AnonPages:       2535432 kB
Mapped:            32884 kB
Shmem:                84 kB
Slab:             683044 kB
SReclaimable:     121132 kB
SUnreclaim:       561912 kB
KernelStack:        1768 kB
PageTables:        13312 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    3361676 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1705984 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70708 MiB
End job on node146 at 2012-06-03 13:39:47