Trace number 3741966

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 654.25 715.464

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K92.opb
MD5SUM21cc97db69195cd3dcf69a6d797764c1
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark900.986
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables6256
Total number of constraints6442
Number of constraints which are clauses6350
Number of constraints which are cardinality constraints (but not clauses)92
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint92
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 93
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.40/1.32	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.40/1.32	c INFO: solving HOME/instance-3741966-1338722471.opb
0.40/1.39	c INFO: parsing took 0.07 s
0.60/1.53	c INFO: features took 0.13 CPU seconds
0.60/1.56	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.68)
51.02/51.91	c INFO: running clasp-1.3.7@0 for 45 with 1749 remaining (b = 0.36)
98.14/99.06	c INFO: running scip-2.0.1-spx@0 for 45 with 1702 remaining (b = 0.06)
148.11/148.97	c INFO: running clasp-1.3.7@47 for 45 with 1653 remaining (b = 0.01)
199.89/201.25	c INFO: wrote minion input file to HOME/tmpAPNd3N.minion
199.89/201.25	c INFO: running minion-0.12@0 for 45 with 1601 remaining (b = 0.09)
251.80/253.43	c INFO: running wbo1.4a@0 for 45 with 1549 remaining (b = 0.02)
251.85/331.23	c INFO: running scip-2.0.1-clp@49 for 45 with 1472 remaining (b = 0.01)
365.86/382.79	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1421 remaining (b = 0.04)
430.99/492.57	c INFO: running clasp-1.3.7@98 for 136 with 1372 remaining (b = 0.02)
560.62/621.98	c INFO: running scip-2.0.1-spx@49 for 45 with 1242 remaining (b = 0.01)
605.91/667.13	c INFO: running bsolo_pb10-l3@0 for 45 with 1197 remaining (b = 0.02)
653.36/714.59	c INFO: running bsolo_pb10-l1@0 for 91 with 1150 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3741966-1338722471/watcher-3741966-1338722471 -o /tmp/evaluation-result-3741966-1338722471/solver-3741966-1338722471 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741966-1338722471.opb -seed 952373340 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.00 2.03 2.23 3/218 6278
/proc/meminfo: memFree=24884964/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=15276 CPUtime=0 cores=0,2,4,6
/proc/6278/stat : 6278 (python) D 6276 6278 3621 0 -1 4202496 633 0 0 0 0 0 0 0 20 0 1 0 338372126 15642624 527 33554432000 4194304 5683644 140734370940320 140734370926232 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6278/statm: 3819 527 312 364 0 261 0

[startup+0.0154629 s]
/proc/loadavg: 2.00 2.03 2.23 3/218 6278
/proc/meminfo: memFree=24884964/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=15380 CPUtime=0 cores=0,2,4,6
/proc/6278/stat : 6278 (python) D 6276 6278 3621 0 -1 4202496 757 0 0 0 0 0 0 0 20 0 1 0 338372126 15749120 631 33554432000 4194304 5683644 140734370940320 140734370885048 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6278/statm: 3845 631 334 364 0 287 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15380

[startup+0.100246 s]
/proc/loadavg: 2.00 2.03 2.23 3/218 6278
/proc/meminfo: memFree=24884964/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=19088 CPUtime=0.02 cores=0,2,4,6
/proc/6278/stat : 6278 (python) R 6276 6278 3621 0 -1 4202496 1392 0 0 0 2 0 0 0 20 0 1 0 338372126 19546112 1147 33554432000 4194304 5683644 140734370940320 140734370870824 4522534 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6278/statm: 4772 1147 430 364 0 696 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19088

[startup+0.300246 s]
/proc/loadavg: 2.00 2.03 2.23 3/218 6278
/proc/meminfo: memFree=24884964/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=96516 CPUtime=0.11 cores=0,2,4,6
/proc/6278/stat : 6278 (python) D 6276 6278 3621 0 -1 4202496 3736 1009 1 0 9 2 0 0 20 0 1 0 338372126 98832384 2549 33554432000 4194304 5683644 140734370940320 140734370876184 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/6278/statm: 24129 2549 822 364 0 1620 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 96516

[startup+0.700266 s]
/proc/loadavg: 2.00 2.03 2.23 3/218 6278
/proc/meminfo: memFree=24884964/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=122380 CPUtime=0.21 cores=0,2,4,6
/proc/6278/stat : 6278 (python) R 6276 6278 3621 0 -1 4202496 5970 1009 1 0 15 6 0 0 20 0 1 0 338372126 125317120 3848 33554432000 4194304 5683644 140734370940320 140734370855720 213419308235 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6278/statm: 30595 3849 1015 364 0 2838 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 122380

[startup+1.50019 s]
/proc/loadavg: 2.00 2.03 2.23 2/219 6283
/proc/meminfo: memFree=24867696/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=283752 CPUtime=0.6 cores=0,2,4,6
/proc/6278/stat : 6278 (python) R 6276 6278 3621 0 -1 4202496 11006 1009 1 0 49 11 0 0 20 0 1 0 338372126 290562048 7703 33554432000 4194304 5683644 140734370940320 140734370932216 139784114281016 0 0 16781312 33554434 0 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 70938 7703 1847 364 0 6167 0
Current children cumulated CPU time (s) 0.6
Current children cumulated vsize (KiB) 283752

[startup+3.10036 s]
/proc/loadavg: 2.00 2.03 2.23 3/222 6286
/proc/meminfo: memFree=24788436/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=359708 CPUtime=0.65 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6276 6278 3621 0 -1 4202496 11329 1009 1 0 54 11 0 0 20 0 2 0 338372126 368340992 7800 33554432000 4194304 5683644 140734370940320 140734370935152 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 89927 7800 1901 364 0 25150 0
[pid=6284] ppid=6278 vsize=283928 CPUtime=0 cores=0,2,4,6
/proc/6284/stat : 6284 (python) S 6278 6278 3621 0 -1 4202560 765 0 0 0 0 0 0 0 20 0 1 0 338372282 290742272 6234 33554432000 4194304 5683644 140734370940320 140734370931408 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6284/statm: 70982 6234 341 364 0 6205 0
[pid=6286] ppid=6284 vsize=92612 CPUtime=1.53 cores=0,2,4,6
/proc/6286/stat : 6286 (scip-2.0.1.linu) R 6284 6286 6286 0 -1 4202496 20122 0 0 0 149 4 0 0 20 0 1 0 338372282 94834688 19520 33554432000 4194304 15243407 140735901751376 140735901746760 4520304 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6286/statm: 23153 19520 682 2698 0 20448 0
[pid=6278/tid=6285] ppid=6276 vsize=359708 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338372282 368340992 7800 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 2.18
Current children cumulated vsize (KiB) 736248
heavy processes:

[startup+6.30035 s]
/proc/loadavg: 2.00 2.03 2.22 3/222 6286
/proc/meminfo: memFree=24727312/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=359708 CPUtime=0.65 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6276 6278 3621 0 -1 4202496 11329 1009 1 0 54 11 0 0 20 0 2 0 338372126 368340992 7800 33554432000 4194304 5683644 140734370940320 140734370935152 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 89927 7800 1901 364 0 25150 0
[pid=6284] ppid=6278 vsize=283928 CPUtime=0 cores=0,2,4,6
/proc/6284/stat : 6284 (python) S 6278 6278 3621 0 -1 4202560 897 0 0 0 0 0 0 0 20 0 1 0 338372282 290742272 6257 33554432000 4194304 5683644 140734370940320 140734370931408 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6284/statm: 70982 6257 364 364 0 6205 0
[pid=6286] ppid=6284 vsize=156644 CPUtime=4.72 cores=0,2,4,6
/proc/6286/stat : 6286 (scip-2.0.1.linu) R 6284 6286 6286 0 -1 4202496 36107 0 0 0 463 9 0 0 20 0 1 0 338372282 160403456 35505 33554432000 4194304 15243407 140735901751376 140735901747688 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6286/statm: 39161 35505 686 2698 0 36456 0
[pid=6278/tid=6285] ppid=6276 vsize=359708 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338372282 368340992 7800 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 5.37
Current children cumulated vsize (KiB) 800280

[startup+12.7004 s]
/proc/loadavg: 2.00 2.03 2.22 3/222 6286
/proc/meminfo: memFree=24670668/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=359708 CPUtime=0.65 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6276 6278 3621 0 -1 4202496 11329 1009 1 0 54 11 0 0 20 0 2 0 338372126 368340992 7800 33554432000 4194304 5683644 140734370940320 140734370935152 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 89927 7800 1901 364 0 25150 0
[pid=6284] ppid=6278 vsize=283928 CPUtime=0 cores=0,2,4,6
/proc/6284/stat : 6284 (python) S 6278 6278 3621 0 -1 4202560 902 0 0 0 0 0 0 0 20 0 1 0 338372282 290742272 6257 33554432000 4194304 5683644 140734370940320 140734370931408 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6284/statm: 70982 6257 364 364 0 6205 0
[pid=6286] ppid=6284 vsize=208704 CPUtime=11.12 cores=0,2,4,6
/proc/6286/stat : 6286 (scip-2.0.1.linu) R 6284 6286 6286 0 -1 4202496 48945 0 0 0 1094 18 0 0 20 0 1 0 338372282 213712896 48343 33554432000 4194304 15243407 140735901751376 140735901747688 13277498 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/6286/statm: 52176 48343 686 2698 0 49471 0
[pid=6278/tid=6285] ppid=6276 vsize=359708 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338372282 368340992 7800 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 11.77
Current children cumulated vsize (KiB) 852340

[startup+25.5003 s]
/proc/loadavg: 2.00 2.03 2.22 3/222 6286
/proc/meminfo: memFree=24608988/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=359708 CPUtime=0.65 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6276 6278 3621 0 -1 4202496 11329 1009 1 0 54 11 0 0 20 0 2 0 338372126 368340992 7800 33554432000 4194304 5683644 140734370940320 140734370935152 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 89927 7800 1901 364 0 25150 0
[pid=6284] ppid=6278 vsize=283928 CPUtime=0.02 cores=0,2,4,6
/proc/6284/stat : 6284 (python) S 6278 6278 3621 0 -1 4202560 1008 0 0 0 1 1 0 0 20 0 1 0 338372282 290742272 6262 33554432000 4194304 5683644 140734370940320 140734370931408 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6284/statm: 70982 6262 369 364 0 6205 0
[pid=6286] ppid=6284 vsize=270964 CPUtime=23.91 cores=0,2,4,6
/proc/6286/stat : 6286 (scip-2.0.1.linu) R 6284 6286 6286 0 -1 4202496 64177 0 0 0 2348 43 0 0 20 0 1 0 338372282 277467136 63537 33554432000 4194304 15243407 140735901751376 140735901747464 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6286/statm: 67741 63537 873 2698 0 65036 0
[pid=6278/tid=6285] ppid=6276 vsize=359708 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338372282 368340992 7800 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 24.58
Current children cumulated vsize (KiB) 914600

[startup+51.1003 s]
/proc/loadavg: 2.00 2.02 2.21 3/222 6286
/proc/meminfo: memFree=24583968/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=359708 CPUtime=0.65 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6276 6278 3621 0 -1 4202496 11329 1009 1 0 54 11 0 0 20 0 2 0 338372126 368340992 7800 33554432000 4194304 5683644 140734370940320 140734370935152 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 89927 7800 1901 364 0 25150 0
[pid=6284] ppid=6278 vsize=283928 CPUtime=0.06 cores=0,2,4,6
/proc/6284/stat : 6284 (python) S 6278 6278 3621 0 -1 4202560 1028 0 0 0 3 3 0 0 20 0 1 0 338372282 290742272 6262 33554432000 4194304 5683644 140734370940320 140734370931408 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6284/statm: 70982 6262 369 364 0 6205 0
[pid=6286] ppid=6284 vsize=295536 CPUtime=49.5 cores=0,2,4,6
/proc/6286/stat : 6286 (scip-2.0.1.linu) R 6284 6286 6286 0 -1 4202496 69652 0 0 0 4874 76 0 0 20 0 1 0 338372282 302628864 69522 33554432000 4194304 15243407 140735901751376 140735901746088 8113576 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6286/statm: 73884 69522 898 2698 0 71179 0
[pid=6278/tid=6285] ppid=6276 vsize=359708 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338372282 368340992 7800 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 50.21
Current children cumulated vsize (KiB) 939172

[startup+102.301 s]
/proc/loadavg: 2.00 2.02 2.20 3/233 6298
/proc/meminfo: memFree=24413824/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=511292 CPUtime=0.71 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6276 6278 3621 0 -1 4202496 13050 1009 1 0 59 12 0 0 20 0 4 0 338372126 523563008 7833 33554432000 4194304 5683644 140734370940320 140734370935152 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 127823 7833 1906 364 0 63040 0
[pid=6284] ppid=6278 vsize=359708 CPUtime=0.07 cores=0,2,4,6
/proc/6284/stat : 6284 (python) S 6278 6278 3621 0 -1 4202560 1160 0 0 0 4 3 0 0 20 0 2 0 338372282 368340992 6297 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6284/statm: 89927 6297 397 364 0 25150 0
[pid=6286] ppid=6284 vsize=295536 CPUtime=50.28 cores=0,2,4,6
/proc/6286/stat : 6286 (scip-2.0.1.linu) T 6284 6286 6286 0 -1 4202496 69654 0 0 0 4952 76 0 0 20 0 1 0 338372282 302628864 69524 33554432000 4194304 15243407 140735901751376 140735901746088 8055530 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6286/statm: 73884 69524 898 2698 0 71179 0
[pid=6284/tid=6287] ppid=6278 vsize=359708 CPUtime=0 cores=0,2,4,6
/proc/6284/task/6287/stat : 6287 (python) S 6278 6278 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338377314 368340992 6297 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6288] ppid=6278 vsize=359720 CPUtime=0.07 cores=0,2,4,6
/proc/6288/stat : 6288 (python) S 6278 6278 3621 0 -1 4202560 1081 0 0 0 4 3 0 0 20 0 2 0 338377317 368353280 6284 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6288/statm: 89930 6284 384 364 0 25150 0
[pid=6290] ppid=6288 vsize=17388 CPUtime=47.09 cores=0,2,4,6
/proc/6290/stat : 6290 (clasp-1.3.7-x86) T 6288 6290 6290 0 -1 4202496 4430 0 0 0 4709 0 0 0 20 0 1 0 338377317 17805312 4084 33554432000 134512640 136139019 4290281216 4290280192 134883434 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6290/statm: 4347 4084 182 398 0 3947 0
[pid=6288/tid=6295] ppid=6278 vsize=359720 CPUtime=0 cores=0,2,4,6
/proc/6288/task/6295/stat : 6295 (python) S 6278 6278 3621 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338382030 368353280 6284 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6296] ppid=6278 vsize=435512 CPUtime=0 cores=0,2,4,6
/proc/6296/stat : 6296 (python) S 6278 6278 3621 0 -1 4202560 805 0 0 0 0 0 0 0 20 0 1 0 338382032 445964288 6257 33554432000 4194304 5683644 140734370940320 140734370931408 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6296/statm: 108878 6257 336 364 0 44095 0
[pid=6298] ppid=6296 vsize=131392 CPUtime=3.22 cores=0,2,4,6
/proc/6298/stat : 6298 (scip-2.0.1.linu) R 6296 6298 6298 0 -1 4202496 30036 0 0 0 313 9 0 0 20 0 1 0 338382032 134545408 29421 33554432000 4194304 13684479 140734944693872 140734944690184 11876362 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/6298/statm: 32848 29421 635 2317 0 30526 0
[pid=6278/tid=6285] ppid=6276 vsize=511292 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 4 0 338372282 523563008 7833 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6289] ppid=6276 vsize=511292 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6289/stat : 6289 (python) S 6276 6278 3621 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 4 0 338377317 523563008 7833 33554432000 4194304 5683644 140734370940320 139783928263216 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6297] ppid=6276 vsize=511292 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6297/stat : 6297 (python) S 6276 6278 3621 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 4 0 338382032 523563008 7833 33554432000 4194304 5683644 140734370940320 139783917773360 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 101.44
Current children cumulated vsize (KiB) 2110548

[startup+185.72 s]
/proc/loadavg: 1.92 2.00 2.18 3/189 6347
/proc/meminfo: memFree=26730456/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=511292 CPUtime=0.74 cores=0,2,4,6

################
# More data... #
################

/proc/6427/stat : 6427 (bsolo_pb10) T 6425 6427 6427 0 -1 4202496 21654 0 0 0 4729 11 0 0 20 0 1 0 338438840 96743424 21186 33554432000 134512640 137138936 4288170016 4288168944 134542808 0 0 16781312 16384 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6427/statm: 23619 21186 192 642 0 22975 0
[pid=6425/tid=6428] ppid=6278 vsize=739232 CPUtime=0 cores=0,2,4,6
/proc/6425/task/6428/stat : 6428 (python) S 6278 6278 3621 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338443583 756973568 6440 33554432000 4194304 5683644 140734370940320 139783886303792 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6429] ppid=6278 vsize=815024 CPUtime=0 cores=0,2,4,6
/proc/6429/stat : 6429 (python) S 6278 6278 3621 0 -1 4202560 852 0 0 0 0 0 0 0 20 0 1 0 338443585 834584576 6905 33554432000 4194304 5683644 140734370940320 140734370931408 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6429/statm: 203756 6905 336 364 0 138958 0
[pid=6431] ppid=6429 vsize=13072 CPUtime=0.8 cores=0,2,4,6
/proc/6431/stat : 6431 (bsolo_pb10) R 6429 6431 6431 0 -1 4202496 1171 0 0 0 80 0 0 0 20 0 1 0 338443586 13385728 807 33554432000 134512640 137138936 4292817248 4292740472 136504080 0 0 16781312 16384 0 0 0 17 0 0 0 0 0 0
/proc/6431/statm: 3268 807 184 642 0 2624 0
[pid=6278/tid=6285] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 31 1009 0 0 0 0 0 0 20 0 9 0 338372282 912183296 8491 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6289] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6289/stat : 6289 (python) S 6276 6278 3621 0 -1 4202560 30 1009 0 0 0 0 0 0 20 0 9 0 338377317 912183296 8491 33554432000 4194304 5683644 140734370940320 139783928263216 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6297] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6297/stat : 6297 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 9 0 338382032 912183296 8491 33554432000 4194304 5683644 140734370940320 139783917773360 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6356] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6356/stat : 6356 (python) S 6276 6278 3621 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338392252 912183296 8491 33554432000 4194304 5683644 140734370940320 139783907283504 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6278/tid=6361] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6361/stat : 6361 (python) S 6276 6278 3621 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338397469 912183296 8491 33554432000 4194304 5683644 140734370940320 139783896793648 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6278/tid=6370] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6370/stat : 6370 (python) S 6276 6278 3621 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338410405 912183296 8491 33554432000 4194304 5683644 140734370940320 139783886303792 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6278/tid=6426] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6426/stat : 6426 (python) S 6276 6278 3621 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338438840 912183296 8491 33554432000 4194304 5683644 140734370940320 139783535845936 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6278/tid=6430] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6430/stat : 6430 (python) S 6276 6278 3621 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338443586 912183296 8491 33554432000 4194304 5683644 140734370940320 139783525356080 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 654.25
Current children cumulated vsize (KiB) 15951160

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+715.401 s]
/proc/loadavg: 2.19 3.24 3.01 3/268 6431
/proc/meminfo: memFree=21851388/32873844 swapFree=6608/6608
[pid=6278] ppid=6276 vsize=890804 CPUtime=1.49 cores=0,2,4,6
/proc/6278/stat : 6278 (python) S 6276 6278 3621 0 -1 4202496 18154 1009 1 0 135 14 0 0 20 0 9 0 338372126 912183296 8491 33554432000 4194304 5683644 140734370940320 140734370935152 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 4 0 0
/proc/6278/statm: 222701 8491 1916 364 0 157903 0
[pid=6284] ppid=6278 vsize=359708 CPUtime=0.12 cores=0,2,4,6
/proc/6284/stat : 6284 (python) S 6278 6278 3621 0 -1 4202560 1186 0 0 0 7 5 0 0 20 0 2 0 338372282 368340992 6298 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/6284/statm: 89927 6298 398 364 0 25150 0
[pid=6286] ppid=6284 vsize=305900 CPUtime=101.79 cores=0,2,4,6
/proc/6286/stat : 6286 (scip-2.0.1.linu) T 6284 6286 6286 0 -1 4202496 90362 0 0 0 10093 86 0 0 20 0 1 0 338372282 313241600 73778 33554432000 4194304 15243407 140735901751376 140735901745768 8134448 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6286/statm: 76475 73778 946 2698 0 73770 0
[pid=6284/tid=6287] ppid=6278 vsize=359708 CPUtime=0 cores=0,2,4,6
/proc/6284/task/6287/stat : 6287 (python) S 6278 6278 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338377314 368340992 6298 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6288] ppid=6278 vsize=359720 CPUtime=0.34 cores=0,2,4,6
/proc/6288/stat : 6288 (python) S 6278 6278 3621 0 -1 4202560 1169 0 0 0 19 15 0 0 20 0 2 0 338377317 368353280 6284 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6288/statm: 89930 6284 384 364 0 25150 0
[pid=6290] ppid=6288 vsize=27948 CPUtime=228.14 cores=0,2,4,6
/proc/6290/stat : 6290 (clasp-1.3.7-x86) T 6288 6290 6290 0 -1 4202496 7073 0 0 0 22813 1 0 0 20 0 1 0 338377317 28618752 6727 33554432000 134512640 136139019 4290281216 4290280080 134608203 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6290/statm: 6987 6727 182 398 0 6587 0
[pid=6288/tid=6295] ppid=6278 vsize=359720 CPUtime=0 cores=0,2,4,6
/proc/6288/task/6295/stat : 6295 (python) S 6278 6278 3621 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338382030 368353280 6284 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6296] ppid=6278 vsize=435512 CPUtime=0.17 cores=0,2,4,6
/proc/6296/stat : 6296 (python) S 6278 6278 3621 0 -1 4202560 1153 0 0 0 10 7 0 0 20 0 2 0 338382032 445964288 6306 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6296/statm: 108878 6306 384 364 0 44095 0
[pid=6298] ppid=6296 vsize=271856 CPUtime=94.98 cores=0,2,4,6
/proc/6298/stat : 6298 (scip-2.0.1.linu) T 6296 6298 6298 0 -1 4202496 71755 0 0 0 9427 71 0 0 20 0 1 0 338382032 278380544 65554 33554432000 4194304 13684479 140734944693872 140734944689960 11876362 0 0 16781312 2 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/6298/statm: 67964 65554 775 2317 0 65642 0
[pid=6296/tid=6324] ppid=6278 vsize=435512 CPUtime=0 cores=0,2,4,6
/proc/6296/task/6324/stat : 6324 (python) S 6278 6278 3621 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 338387020 445964288 6306 33554432000 4194304 5683644 140734370940320 139783928263216 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6355] ppid=6278 vsize=511856 CPUtime=0.03 cores=0,2,4,6
/proc/6355/stat : 6355 (python) S 6278 6278 3621 0 -1 4202560 1105 0 0 0 1 2 0 0 20 0 2 0 338392251 524140544 6422 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6355/statm: 127964 6422 384 364 0 63178 0
[pid=6357] ppid=6355 vsize=86728 CPUtime=51.32 cores=0,2,4,6
/proc/6357/stat : 6357 (minion) T 6355 6357 6357 0 -1 4202496 8098 0 0 0 5131 1 0 0 20 0 1 0 338392252 88809472 14362 33554432000 134512640 165565024 4290067104 4290064336 159737071 0 0 16781312 8396802 18446744071579387992 0 0 17 6 0 0 0 0 0
/proc/6357/statm: 21682 14362 336 7582 0 14097 0
[pid=6355/tid=6358] ppid=6278 vsize=511856 CPUtime=0 cores=0,2,4,6
/proc/6355/task/6358/stat : 6358 (python) S 6278 6278 3621 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338397466 524140544 6422 33554432000 4194304 5683644 140734370940320 139783917773360 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6360] ppid=6278 vsize=587648 CPUtime=0.04 cores=0,2,4,6
/proc/6360/stat : 6360 (python) S 6278 6278 3621 0 -1 4202560 1116 0 0 0 1 3 0 0 20 0 2 0 338397469 601751552 6428 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/6360/statm: 146912 6428 384 364 0 82123 0
[pid=6364] ppid=6360 vsize=163264 CPUtime=77.75 cores=0,2,4,6
/proc/6364/stat : 6364 (wbo1.4a) T 6360 6364 6364 0 -1 4202496 40675 0 0 0 7712 63 0 0 20 0 1 0 338397469 167182336 40319 33554432000 134512640 135751879 4293635952 4293634128 134616543 0 0 16781312 3 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/6364/statm: 40816 40319 143 303 0 40511 0
[pid=6360/tid=6366] ppid=6278 vsize=587648 CPUtime=0 cores=0,2,4,6
/proc/6360/task/6366/stat : 6366 (python) S 6278 6278 3621 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338405247 601751552 6428 33554432000 4194304 5683644 140734370940320 139783907283504 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6369] ppid=6278 vsize=663440 CPUtime=0.1 cores=0,2,4,6
/proc/6369/stat : 6369 (python) S 6278 6278 3621 0 -1 4202560 1151 0 0 0 5 5 0 0 20 0 2 0 338410405 679362560 6443 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6369/statm: 165860 6443 393 364 0 101068 0
[pid=6371] ppid=6369 vsize=9624972 CPUtime=49.7 cores=0,2,4,6
/proc/6371/stat : 6371 (java) T 6369 6371 6371 0 -1 4202496 24244 0 1 0 4900 70 0 0 20 0 19 0 338410406 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 140733684535520 213422997565 0 0 0 16800975 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6371/statm: 2406243 687971 2266 9 0 2396937 0
[pid=6371/tid=6372] ppid=6369 vsize=9624972 CPUtime=48.57 cores=0,2,4,6
/proc/6371/task/6372/stat : 6372 (java) T 6369 6371 6371 0 -1 4202560 4071 0 1 0 4811 46 0 0 20 0 19 0 338410407 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754922626160 139754763519732 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6371/tid=6373] ppid=6369 vsize=9624972 CPUtime=0.07 cores=0,2,4,6
/proc/6371/task/6373/stat : 6373 (java) T 6369 6371 6371 0 -1 4202560 298 0 0 0 5 2 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754884119208 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6374] ppid=6369 vsize=9624972 CPUtime=0.09 cores=0,2,4,6
/proc/6371/task/6374/stat : 6374 (java) T 6369 6371 6371 0 -1 4202560 117 0 0 0 6 3 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754883066664 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6371/tid=6375] ppid=6369 vsize=9624972 CPUtime=0.05 cores=0,2,4,6
/proc/6371/task/6375/stat : 6375 (java) T 6369 6371 6371 0 -1 4202560 126 0 0 0 4 1 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754882014120 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6376] ppid=6369 vsize=9624972 CPUtime=0.1 cores=0,2,4,6
/proc/6371/task/6376/stat : 6376 (java) T 6369 6371 6371 0 -1 4202560 169 0 0 0 7 3 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754762923048 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6371/tid=6377] ppid=6369 vsize=9624972 CPUtime=0.04 cores=0,2,4,6
/proc/6371/task/6377/stat : 6377 (java) T 6369 6371 6371 0 -1 4202560 20 0 0 0 4 0 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754761869480 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6378] ppid=6369 vsize=9624972 CPUtime=0.05 cores=0,2,4,6
/proc/6371/task/6378/stat : 6378 (java) T 6369 6371 6371 0 -1 4202560 14 0 0 0 5 0 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754760816936 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6379] ppid=6369 vsize=9624972 CPUtime=0.09 cores=0,2,4,6
/proc/6371/task/6379/stat : 6379 (java) T 6369 6371 6371 0 -1 4202560 134 0 0 0 7 2 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754759764392 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6380] ppid=6369 vsize=9624972 CPUtime=0.05 cores=0,2,4,6
/proc/6371/task/6380/stat : 6380 (java) T 6369 6371 6371 0 -1 4202560 22 0 0 0 4 1 0 0 20 0 19 0 338410408 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754758711848 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6371/tid=6381] ppid=6369 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6371/task/6381/stat : 6381 (java) T 6369 6371 6371 0 -1 4202560 1267 0 0 0 0 0 0 0 20 0 19 0 338410409 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754276383336 213423011753 0 0 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6371/tid=6382] ppid=6369 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6371/task/6382/stat : 6382 (java) T 6369 6371 6371 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338410409 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754275329656 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6383] ppid=6369 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6371/task/6383/stat : 6383 (java) T 6369 6371 6371 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338410409 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754274277000 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6384] ppid=6369 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6371/task/6384/stat : 6384 (java) T 6369 6371 6371 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338410411 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754273225744 213423019808 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6385] ppid=6369 vsize=9624972 CPUtime=0.82 cores=0,2,4,6
/proc/6371/task/6385/stat : 6385 (java) T 6369 6371 6371 0 -1 4202560 8487 0 0 0 77 5 0 0 20 0 19 0 338410411 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754272172088 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6371/tid=6386] ppid=6369 vsize=9624972 CPUtime=0.85 cores=0,2,4,6
/proc/6371/task/6386/stat : 6386 (java) T 6369 6371 6371 0 -1 4202560 8099 0 0 0 79 6 0 0 20 0 19 0 338410412 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754271119544 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6371/tid=6387] ppid=6369 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6371/task/6387/stat : 6387 (java) T 6369 6371 6371 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338410412 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754270067192 213423010876 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=6371/tid=6388] ppid=6369 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6371/task/6388/stat : 6388 (java) T 6369 6371 6371 0 -1 4202560 5 0 0 0 0 0 0 0 20 0 19 0 338410412 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754269014856 213423011753 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=6371/tid=6389] ppid=6369 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/6371/task/6389/stat : 6389 (java) T 6369 6371 6371 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338410443 9855971328 687971 33554432000 1073741824 1073778376 140733684544384 139754267960616 213423011753 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=6369/tid=6394] ppid=6278 vsize=663440 CPUtime=0 cores=0,2,4,6
/proc/6369/task/6394/stat : 6394 (python) S 6278 6278 3621 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338421380 679362560 6443 33554432000 4194304 5683644 140734370940320 139783896793648 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=6425] ppid=6278 vsize=739232 CPUtime=0.08 cores=0,2,4,6
/proc/6425/stat : 6425 (python) S 6278 6278 3621 0 -1 4202560 1161 0 0 0 5 3 0 0 20 0 2 0 338438839 756973568 6440 33554432000 4194304 5683644 140734370940320 140734370931968 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/6425/statm: 184808 6440 384 364 0 120013 0
[pid=6427] ppid=6425 vsize=94476 CPUtime=47.4 cores=0,2,4,6
/proc/6427/stat : 6427 (bsolo_pb10) T 6425 6427 6427 0 -1 4202496 21654 0 0 0 4729 11 0 0 20 0 1 0 338438840 96743424 21186 33554432000 134512640 137138936 4288170016 4288168944 134542808 0 0 16781312 16384 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/6427/statm: 23619 21186 192 642 0 22975 0
[pid=6425/tid=6428] ppid=6278 vsize=739232 CPUtime=0 cores=0,2,4,6
/proc/6425/task/6428/stat : 6428 (python) S 6278 6278 3621 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338443583 756973568 6440 33554432000 4194304 5683644 140734370940320 139783886303792 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6429] ppid=6278 vsize=815024 CPUtime=0 cores=0,2,4,6
/proc/6429/stat : 6429 (python) S 6278 6278 3621 0 -1 4202560 852 0 0 0 0 0 0 0 20 0 1 0 338443585 834584576 6905 33554432000 4194304 5683644 140734370940320 140734370931408 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/6429/statm: 203756 6905 336 364 0 138958 0
[pid=6431] ppid=6429 vsize=13072 CPUtime=0.8 cores=0,2,4,6
/proc/6431/stat : 6431 (bsolo_pb10) R 6429 6431 6431 0 -1 4202496 1171 0 0 0 80 0 0 0 20 0 1 0 338443586 13385728 807 33554432000 134512640 137138936 4292817248 4292740472 136504080 0 0 16781312 16384 0 0 0 17 0 0 0 0 0 0
/proc/6431/statm: 3268 807 184 642 0 2624 0
[pid=6278/tid=6285] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6285/stat : 6285 (python) S 6276 6278 3621 0 -1 4202560 31 1009 0 0 0 0 0 0 20 0 9 0 338372282 912183296 8491 33554432000 4194304 5683644 140734370940320 139783938753072 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6289] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6289/stat : 6289 (python) S 6276 6278 3621 0 -1 4202560 30 1009 0 0 0 0 0 0 20 0 9 0 338377317 912183296 8491 33554432000 4194304 5683644 140734370940320 139783928263216 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6297] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6297/stat : 6297 (python) S 6276 6278 3621 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 9 0 338382032 912183296 8491 33554432000 4194304 5683644 140734370940320 139783917773360 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=6278/tid=6356] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6356/stat : 6356 (python) S 6276 6278 3621 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338392252 912183296 8491 33554432000 4194304 5683644 140734370940320 139783907283504 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6278/tid=6361] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6361/stat : 6361 (python) S 6276 6278 3621 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338397469 912183296 8491 33554432000 4194304 5683644 140734370940320 139783896793648 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6278/tid=6370] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6370/stat : 6370 (python) S 6276 6278 3621 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338410405 912183296 8491 33554432000 4194304 5683644 140734370940320 139783886303792 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=6278/tid=6426] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6426/stat : 6426 (python) S 6276 6278 3621 0 -1 4202560 22 1009 0 0 0 0 0 0 20 0 9 0 338438840 912183296 8491 33554432000 4194304 5683644 140734370940320 139783535845936 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=6278/tid=6430] ppid=6276 vsize=890804 CPUtime=0 cores=0,2,4,6
/proc/6278/task/6430/stat : 6430 (python) S 6276 6278 3621 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338443586 912183296 8491 33554432000 4194304 5683644 140734370940320 139783525356080 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 654.25
Current children cumulated vsize (KiB) 15951160

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 6278 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=362792
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=143978
# CPU time returned by wait4() is 1.50677
# while last known CPU time is 654.25
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 715.464
CPU time (s): 654.25
CPU user time (s): 650.68
CPU system time (s): 3.57
CPU usage (%): 91.4442
Max. virtual memory (cumulated for all children) (KiB): 15951160

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.36279
system time used= 0.143978
maximum resident set size= 33964
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19658
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1536
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2091
involuntary context switches= 23

runsolver used 2.12368 second user time and 5.16221 second system time

The end

Launcher Data 

Begin job on node140 at 2012-06-03 13:21:11
IDJOB=3741966
IDBENCH=90121
IDSOLVER=2327
FILE ID=node140/3741966-1338722471
RUNJOBID= node140-1338713222-3637
PBS_JOBID= 14636725
Free space on /tmp= 70524 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K92.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3741966-1338722471/watcher-3741966-1338722471 -o /tmp/evaluation-result-3741966-1338722471/solver-3741966-1338722471 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741966-1338722471.opb -seed 952373340 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 21cc97db69195cd3dcf69a6d797764c1
RANDOM SEED=952373340

node140.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.88
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        24884980 kB
Buffers:          477036 kB
Cached:          4124180 kB
SwapCached:         1620 kB
Active:          4094564 kB
Inactive:        2950988 kB
Active(anon):    2434544 kB
Inactive(anon):    12096 kB
Active(file):    1660020 kB
Inactive(file):  2938892 kB
Unevictable:        6608 kB
Mlocked:            6608 kB
SwapTotal:      67108856 kB
SwapFree:       67105264 kB
Dirty:            101972 kB
Writeback:             0 kB
AnonPages:       2449476 kB
Mapped:            34292 kB
Shmem:                92 kB
Slab:             793068 kB
SReclaimable:     233396 kB
SUnreclaim:       559672 kB
KernelStack:        1776 kB
PageTables:        13112 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    3258564 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1691648 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70424 MiB
End job on node140 at 2012-06-03 13:33:10