Trace number 3742109

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 611.57 609.357

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K86.opb
MD5SUMa7a3236cd0071452eec601ea09fe7bbe
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables5848
Total number of constraints6022
Number of constraints which are clauses5936
Number of constraints which are cardinality constraints (but not clauses)86
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint86
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 87
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.38/1.19	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.38/1.19	c INFO: solving HOME/instance-3742109-1338723511.opb
0.44/1.25	c INFO: parsing took 0.06 s
0.53/1.37	c INFO: features took 0.12 CPU seconds
0.62/1.40	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.69)
47.14/47.97	c INFO: running clasp-1.3.7@0 for 45 with 1753 remaining (b = 0.36)
94.39/95.13	c INFO: running scip-2.0.1-spx@0 for 45 with 1706 remaining (b = 0.06)
144.06/144.73	c INFO: running clasp-1.3.7@47 for 45 with 1657 remaining (b = 0.01)
189.83/190.45	c INFO: wrote minion input file to HOME/tmpWL7gJW.minion
189.83/190.45	c INFO: running minion-0.12@0 for 45 with 1612 remaining (b = 0.09)
237.27/237.81	c INFO: running wbo1.4a@0 for 45 with 1564 remaining (b = 0.02)
286.61/287.16	c INFO: running scip-2.0.1-clp@45 for 45 with 1515 remaining (b = 0.01)
332.39/332.84	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1469 remaining (b = 0.04)
382.27/380.36	c INFO: running clasp-1.3.7@92 for 136 with 1420 remaining (b = 0.02)
519.83/517.76	c INFO: running scip-2.0.1-spx@49 for 45 with 1283 remaining (b = 0.01)
562.51/560.34	c INFO: running bsolo_pb10-l3@0 for 45 with 1240 remaining (b = 0.02)
609.87/607.62	c INFO: running bsolo_pb10-l1@0 for 91 with 1193 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742109-1338723511/watcher-3742109-1338723511 -o /tmp/evaluation-result-3742109-1338723511/solver-3742109-1338723511 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742109-1338723511.opb -seed 934199701 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.06 2.06 2.17 3/195 2278
/proc/meminfo: memFree=26264844/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=15276 CPUtime=0 cores=0,2,4,6
/proc/2278/stat : 2278 (python) R 2276 2278 27000 0 -1 4202496 670 0 0 0 0 0 0 0 20 0 1 0 338478559 15642624 557 33554432000 4194304 5683644 140734090246896 140734090232808 4870469 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/2278/statm: 3819 559 324 364 0 261 0

[startup+0.105925 s]
/proc/loadavg: 2.06 2.06 2.17 3/195 2278
/proc/meminfo: memFree=26264844/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=23380 CPUtime=0.02 cores=0,2,4,6
/proc/2278/stat : 2278 (python) D 2276 2278 27000 0 -1 4202496 1438 0 0 0 2 0 0 0 20 0 1 0 338478559 23941120 1184 33554432000 4194304 5683644 140734090246896 140734090177400 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/2278/statm: 5845 1184 441 364 0 727 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 23380

[startup+0.200219 s]
/proc/loadavg: 2.06 2.06 2.17 3/195 2278
/proc/meminfo: memFree=26264844/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=83948 CPUtime=0.07 cores=0,2,4,6
/proc/2278/stat : 2278 (python) R 2276 2278 27000 0 -1 4202496 2917 0 0 0 5 2 0 0 20 0 1 0 338478559 85962752 2442 33554432000 4194304 5683644 140734090246896 140734090204856 4620062 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/2278/statm: 20987 2442 744 364 0 1615 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 83948

[startup+0.300275 s]
/proc/loadavg: 2.06 2.06 2.17 3/195 2278
/proc/meminfo: memFree=26264844/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=102048 CPUtime=0.11 cores=0,2,4,6
/proc/2278/stat : 2278 (python) D 2276 2278 27000 0 -1 4202496 4127 1011 1 0 9 2 0 0 20 0 1 0 338478559 104497152 2726 33554432000 4194304 5683644 140734090246896 140734090101592 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/2278/statm: 25512 2726 876 364 0 1722 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 102048

[startup+0.700274 s]
/proc/loadavg: 2.06 2.06 2.17 3/195 2278
/proc/meminfo: memFree=26264844/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=162440 CPUtime=0.23 cores=0,2,4,6
/proc/2278/stat : 2278 (python) D 2276 2278 27000 0 -1 4202496 6472 1011 1 0 17 6 0 0 20 0 1 0 338478559 166338560 4277 33554432000 4194304 5683644 140734090246896 140734090119416 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 2 0 0 0 0 0
/proc/2278/statm: 40610 4277 1192 364 0 3255 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 162440

[startup+1.50019 s]
/proc/loadavg: 2.06 2.06 2.17 2/196 2283
/proc/meminfo: memFree=26243244/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=359456 CPUtime=0.63 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 11246 1011 1 0 53 10 0 0 20 0 2 0 338478559 368082944 7740 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2278/statm: 89864 7740 1892 364 0 25087 0
Current children cumulated CPU time (s) 0.63
Current children cumulated vsize (KiB) 359456

[startup+3.10038 s]
/proc/loadavg: 2.06 2.06 2.17 3/199 2286
/proc/meminfo: memFree=26164108/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=359456 CPUtime=0.63 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 11246 1011 1 0 53 10 0 0 20 0 2 0 338478559 368082944 7740 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2278/statm: 89864 7740 1892 364 0 25087 0
[pid=2284] ppid=2278 vsize=283676 CPUtime=0 cores=0,2,4,6
/proc/2284/stat : 2284 (python) S 2278 2278 27000 0 -1 4202560 764 0 0 0 0 0 0 0 20 0 1 0 338478700 290484224 6183 33554432000 4194304 5683644 140734090246896 140734090237984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/2284/statm: 70919 6183 341 364 0 6142 0
[pid=2286] ppid=2284 vsize=106020 CPUtime=1.68 cores=0,2,4,6
/proc/2286/stat : 2286 (scip-2.0.1.linu) R 2284 2286 2286 0 -1 4202496 23001 0 0 0 164 4 0 0 20 0 1 0 338478700 108564480 22399 33554432000 4194304 15243407 140736331871536 140736331867848 4880954 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/2286/statm: 26505 22399 682 2698 0 23800 0
[pid=2278/tid=2285] ppid=2276 vsize=359456 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338478700 368082944 7740 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 2.31
Current children cumulated vsize (KiB) 749152
heavy processes:

[startup+6.3004 s]
/proc/loadavg: 2.05 2.06 2.17 3/199 2286
/proc/meminfo: memFree=26086268/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=359456 CPUtime=0.63 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 11246 1011 1 0 53 10 0 0 20 0 2 0 338478559 368082944 7740 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2278/statm: 89864 7740 1892 364 0 25087 0
[pid=2284] ppid=2278 vsize=283676 CPUtime=0 cores=0,2,4,6
/proc/2284/stat : 2284 (python) S 2278 2278 27000 0 -1 4202560 892 0 0 0 0 0 0 0 20 0 1 0 338478700 290484224 6205 33554432000 4194304 5683644 140734090246896 140734090237984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/2284/statm: 70919 6205 363 364 0 6142 0
[pid=2286] ppid=2284 vsize=156976 CPUtime=4.88 cores=0,2,4,6
/proc/2286/stat : 2286 (scip-2.0.1.linu) R 2284 2286 2286 0 -1 4202496 35801 0 0 0 477 11 0 0 20 0 1 0 338478700 160743424 35199 33554432000 4194304 15243407 140736331871536 140736331867848 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/2286/statm: 39244 35199 686 2698 0 36539 0
[pid=2278/tid=2285] ppid=2276 vsize=359456 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338478700 368082944 7740 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 5.51
Current children cumulated vsize (KiB) 800108

[startup+12.7004 s]
/proc/loadavg: 2.05 2.06 2.17 3/199 2286
/proc/meminfo: memFree=26031180/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=359456 CPUtime=0.63 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 11246 1011 1 0 53 10 0 0 20 0 2 0 338478559 368082944 7740 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2278/statm: 89864 7740 1892 364 0 25087 0
[pid=2284] ppid=2278 vsize=283676 CPUtime=0 cores=0,2,4,6
/proc/2284/stat : 2284 (python) S 2278 2278 27000 0 -1 4202560 897 0 0 0 0 0 0 0 20 0 1 0 338478700 290484224 6206 33554432000 4194304 5683644 140734090246896 140734090237984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/2284/statm: 70919 6206 364 364 0 6142 0
[pid=2286] ppid=2284 vsize=212784 CPUtime=11.28 cores=0,2,4,6
/proc/2286/stat : 2286 (scip-2.0.1.linu) R 2284 2286 2286 0 -1 4202496 49589 0 0 0 1108 20 0 0 20 0 1 0 338478700 217890816 48987 33554432000 4194304 15243407 140736331871536 140736331867848 4865612 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/2286/statm: 53196 48987 686 2698 0 50491 0
[pid=2278/tid=2285] ppid=2276 vsize=359456 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338478700 368082944 7740 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 11.91
Current children cumulated vsize (KiB) 855916

[startup+25.5004 s]
/proc/loadavg: 2.04 2.06 2.16 3/200 2287
/proc/meminfo: memFree=25966860/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=359456 CPUtime=0.63 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 11246 1011 1 0 53 10 0 0 20 0 2 0 338478559 368082944 7740 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2278/statm: 89864 7740 1892 364 0 25087 0
[pid=2284] ppid=2278 vsize=283676 CPUtime=0.02 cores=0,2,4,6
/proc/2284/stat : 2284 (python) S 2278 2278 27000 0 -1 4202560 1002 0 0 0 1 1 0 0 20 0 1 0 338478700 290484224 6211 33554432000 4194304 5683644 140734090246896 140734090237984 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/2284/statm: 70919 6211 369 364 0 6142 0
[pid=2286] ppid=2284 vsize=255256 CPUtime=24.07 cores=0,2,4,6
/proc/2286/stat : 2286 (scip-2.0.1.linu) R 2284 2286 2286 0 -1 4202496 60148 0 0 0 2359 48 0 0 20 0 1 0 338478700 261382144 59510 33554432000 4194304 15243407 140736331871536 140736331867624 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/2286/statm: 63814 59510 873 2698 0 61109 0
[pid=2278/tid=2285] ppid=2276 vsize=359456 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338478700 368082944 7740 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 24.72
Current children cumulated vsize (KiB) 898388

[startup+51.1004 s]
/proc/loadavg: 2.02 2.05 2.16 3/204 2291
/proc/meminfo: memFree=25934528/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=435248 CPUtime=0.65 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 12196 1011 1 0 55 10 0 0 20 0 3 0 338478559 445693952 7749 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2278/statm: 108812 7749 1895 364 0 44032 0
[pid=2284] ppid=2278 vsize=359456 CPUtime=0.07 cores=0,2,4,6
/proc/2284/stat : 2284 (python) S 2278 2278 27000 0 -1 4202560 1150 0 0 0 4 3 0 0 20 0 2 0 338478700 368082944 6246 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2284/statm: 89864 6246 397 364 0 25087 0
[pid=2286] ppid=2284 vsize=280692 CPUtime=46.51 cores=0,2,4,6
/proc/2286/stat : 2286 (scip-2.0.1.linu) T 2284 2286 2286 0 -1 4202496 65156 0 0 0 4579 72 0 0 20 0 1 0 338478700 287428608 65029 33554432000 4194304 15243407 140736331871536 140736331866248 8138851 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2286/statm: 70173 65029 898 2698 0 67468 0
[pid=2284/tid=2288] ppid=2278 vsize=359456 CPUtime=0 cores=0,2,4,6
/proc/2284/task/2288/stat : 2288 (python) S 2278 2278 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338483355 368082944 6246 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2289] ppid=2278 vsize=359468 CPUtime=0 cores=0,2,4,6
/proc/2289/stat : 2289 (python) S 2278 2278 27000 0 -1 4202560 897 0 0 0 0 0 0 0 20 0 1 0 338483357 368095232 6206 33554432000 4194304 5683644 140734090246896 140734090237984 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/2289/statm: 89867 6206 358 364 0 25087 0
[pid=2291] ppid=2289 vsize=5076 CPUtime=3.11 cores=0,2,4,6
/proc/2291/stat : 2291 (clasp-1.3.7-x86) R 2289 2291 2291 0 -1 4202496 1339 0 0 0 311 0 0 0 20 0 1 0 338483358 5197824 998 33554432000 134512640 136139019 4293163744 4293162608 134608547 0 0 16781312 16386 0 0 0 17 0 0 0 0 0 0
/proc/2291/statm: 1269 998 182 398 0 869 0
[pid=2278/tid=2285] ppid=2276 vsize=435248 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 3 0 338478700 445693952 7749 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2278/tid=2290] ppid=2276 vsize=435248 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2290/stat : 2290 (python) S 2276 2278 27000 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 3 0 338483357 445693952 7749 33554432000 4194304 5683644 140734090246896 139872782705200 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 50.34
Current children cumulated vsize (KiB) 1439940

[startup+102.301 s]
/proc/loadavg: 2.11 2.07 2.16 3/230 2319
/proc/meminfo: memFree=24675640/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=511636 CPUtime=0.68 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 13137 1011 1 0 57 11 0 0 20 0 4 0 338478559 523915264 7891 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2278/statm: 127909 7891 1905 364 0 63126 0
[pid=2284] ppid=2278 vsize=359456 CPUtime=0.07 cores=0,2,4,6
/proc/2284/stat : 2284 (python) S 2278 2278 27000 0 -1 4202560 1150 0 0 0 4 3 0 0 20 0 2 0 338478700 368082944 6246 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2284/statm: 89864 6246 397 364 0 25087 0
[pid=2286] ppid=2284 vsize=280692 CPUtime=46.51 cores=0,2,4,6
/proc/2286/stat : 2286 (scip-2.0.1.linu) T 2284 2286 2286 0 -1 4202496 65156 0 0 0 4579 72 0 0 20 0 1 0 338478700 287428608 65029 33554432000 4194304 15243407 140736331871536 140736331866248 8138851 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2286/statm: 70173 65029 898 2698 0 67468 0
[pid=2284/tid=2288] ppid=2278 vsize=359456 CPUtime=0 cores=0,2,4,6
/proc/2284/task/2288/stat : 2288 (python) S 2278 2278 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338483355 368082944 6246 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2289] ppid=2278 vsize=359468 CPUtime=0.07 cores=0,2,4,6
/proc/2289/stat : 2289 (python) S 2278 2278 27000 0 -1 4202560 1084 0 0 0 4 3 0 0 20 0 2 0 338483357 368095232 6233 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2289/statm: 89867 6233 384 364 0 25087 0
[pid=2291] ppid=2289 vsize=14184 CPUtime=47.1 cores=0,2,4,6
/proc/2291/stat : 2291 (clasp-1.3.7-x86) T 2289 2291 2291 0 -1 4202496 3634 0 0 0 4710 0 0 0 20 0 1 0 338483358 14524416 3293 33554432000 134512640 136139019 4293163744 4293162720 134883066 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2291/statm: 3546 3293 182 398 0 3146 0
[pid=2289/tid=2316] ppid=2278 vsize=359468 CPUtime=0 cores=0,2,4,6
/proc/2289/task/2316/stat : 2316 (python) S 2278 2278 27000 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338488070 368095232 6233 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2317] ppid=2278 vsize=435856 CPUtime=0 cores=0,2,4,6
/proc/2317/stat : 2317 (python) S 2278 2278 27000 0 -1 4202560 928 0 0 0 0 0 0 0 20 0 1 0 338488073 446316544 6339 33554432000 4194304 5683644 140734090246896 140734090237984 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/2317/statm: 108964 6339 359 364 0 44181 0
[pid=2319] ppid=2317 vsize=174124 CPUtime=7.15 cores=0,2,4,6
/proc/2319/stat : 2319 (scip-2.0.1.linu) R 2317 2319 2319 0 -1 4202496 40245 0 0 0 703 12 0 0 20 0 1 0 338488073 178302976 39629 33554432000 4194304 13684479 140734919542880 140734919539192 11876362 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/2319/statm: 43531 39629 639 2317 0 41209 0
[pid=2278/tid=2285] ppid=2276 vsize=511636 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 4 0 338478700 523915264 7891 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2278/tid=2290] ppid=2276 vsize=511636 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

[pid=2334/tid=2406] ppid=2278 vsize=663232 CPUtime=0 cores=0,2,4,6
/proc/2334/task/2406/stat : 2406 (python) S 2278 2278 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338516594 679149568 6383 33554432000 4194304 5683644 140734090246896 139872751235632 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2420] ppid=2278 vsize=739024 CPUtime=0.03 cores=0,2,4,6
/proc/2420/stat : 2420 (python) S 2278 2278 27000 0 -1 4202560 995 0 0 0 2 1 0 0 20 0 1 0 338534594 756760576 6363 33554432000 4194304 5683644 140734090246896 140734090237984 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/2420/statm: 184756 6363 359 364 0 119961 0
[pid=2422] ppid=2420 vsize=60336 CPUtime=21.94 cores=0,2,4,6
/proc/2422/stat : 2422 (bsolo_pb10) R 2420 2422 2422 0 -1 4202496 13098 0 0 0 2189 5 0 0 20 0 1 0 338534594 61784064 12650 33554432000 134512640 137138936 4287791936 4287790864 134542620 0 0 16781312 16384 0 0 0 17 2 0 0 0 0 0
/proc/2422/statm: 15084 12650 192 642 0 14440 0
[pid=2278/tid=2285] ppid=2276 vsize=814804 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 32 1011 0 0 0 0 0 0 20 0 8 0 338478700 834359296 8431 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2278/tid=2290] ppid=2276 vsize=814804 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2290/stat : 2290 (python) S 2276 2278 27000 0 -1 4202560 30 1011 0 0 0 0 0 0 20 0 8 0 338483357 834359296 8431 33554432000 4194304 5683644 140734090246896 139872782705200 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2278/tid=2318] ppid=2276 vsize=814804 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2318/stat : 2318 (python) S 2276 2278 27000 0 -1 4202560 25 1011 0 0 0 0 0 0 20 0 8 0 338488073 834359296 8431 33554432000 4194304 5683644 140734090246896 139872772215344 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2278/tid=2323] ppid=2276 vsize=814804 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2323/stat : 2323 (python) S 2276 2278 27000 0 -1 4202560 20 1011 0 0 0 0 0 0 20 0 8 0 338497605 834359296 8431 33554432000 4194304 5683644 140734090246896 139872761725488 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2278/tid=2328] ppid=2276 vsize=814804 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2328/stat : 2328 (python) S 2276 2278 27000 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 8 0 338502341 834359296 8431 33554432000 4194304 5683644 140734090246896 139872751235632 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2278/tid=2335] ppid=2276 vsize=814804 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2335/stat : 2335 (python) S 2276 2278 27000 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 8 0 338511844 834359296 8431 33554432000 4194304 5683644 140734090246896 139872740745776 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2278/tid=2421] ppid=2276 vsize=814804 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2421/stat : 2421 (python) S 2276 2278 27000 0 -1 4202560 20 1011 0 0 0 0 0 0 20 0 8 0 338534594 834359296 8431 33554432000 4194304 5683644 140734090246896 139872387981872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 584.53
Current children cumulated vsize (KiB) 14923772



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+609.302 s]
/proc/loadavg: 2.03 2.03 2.09 3/245 2430
/proc/meminfo: memFree=23606732/32873648 swapFree=7120/7120
[pid=2278] ppid=2276 vsize=890596 CPUtime=1.47 cores=0,2,4,6
/proc/2278/stat : 2278 (python) S 2276 2278 27000 0 -1 4202496 18144 1011 1 0 134 13 0 0 20 0 9 0 338478559 911970304 8439 33554432000 4194304 5683644 140734090246896 140734090241728 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/2278/statm: 222649 8439 1915 364 0 157851 0
[pid=2284] ppid=2278 vsize=359456 CPUtime=0.15 cores=0,2,4,6
/proc/2284/stat : 2284 (python) S 2278 2278 27000 0 -1 4202560 1185 0 0 0 9 6 0 0 20 0 2 0 338478700 368082944 6247 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2284/statm: 89864 6247 398 364 0 25087 0
[pid=2286] ppid=2284 vsize=288416 CPUtime=92.16 cores=0,2,4,6
/proc/2286/stat : 2286 (scip-2.0.1.linu) T 2284 2286 2286 0 -1 4202496 81534 0 0 0 9134 82 0 0 20 0 1 0 338478700 295337984 69308 33554432000 4194304 15243407 140736331871536 140736331867336 8113495 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2286/statm: 72104 69308 955 2698 0 69399 0
[pid=2284/tid=2288] ppid=2278 vsize=359456 CPUtime=0 cores=0,2,4,6
/proc/2284/task/2288/stat : 2288 (python) S 2278 2278 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338483355 368082944 6247 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2289] ppid=2278 vsize=359468 CPUtime=0.38 cores=0,2,4,6
/proc/2289/stat : 2289 (python) S 2278 2278 27000 0 -1 4202560 1177 0 0 0 24 14 0 0 20 0 2 0 338483357 368095232 6233 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2289/statm: 89867 6233 384 364 0 25087 0
[pid=2291] ppid=2289 vsize=23316 CPUtime=229.54 cores=0,2,4,6
/proc/2291/stat : 2291 (clasp-1.3.7-x86) T 2289 2291 2291 0 -1 4202496 5921 0 0 0 22954 0 0 0 20 0 1 0 338483358 23875584 5581 33554432000 134512640 136139019 4293163744 4293162720 134882959 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/2291/statm: 5829 5581 182 398 0 5429 0
[pid=2289/tid=2316] ppid=2278 vsize=359468 CPUtime=0 cores=0,2,4,6
/proc/2289/task/2316/stat : 2316 (python) S 2278 2278 27000 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338488070 368095232 6233 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2317] ppid=2278 vsize=435856 CPUtime=0.17 cores=0,2,4,6
/proc/2317/stat : 2317 (python) S 2278 2278 27000 0 -1 4202560 1152 0 0 0 11 6 0 0 20 0 2 0 338488073 446316544 6365 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2317/statm: 108964 6365 384 364 0 44181 0
[pid=2319] ppid=2317 vsize=278200 CPUtime=92.11 cores=0,2,4,6
/proc/2319/stat : 2319 (scip-2.0.1.linu) T 2317 2319 2319 0 -1 4202496 67674 0 0 0 9131 80 0 0 20 0 1 0 338488073 284876800 65140 33554432000 4194304 13684479 140734919542880 140734919536712 7388647 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2319/statm: 69550 65140 786 2317 0 67228 0
[pid=2317/tid=2321] ppid=2278 vsize=435856 CPUtime=0 cores=0,2,4,6
/proc/2317/task/2321/stat : 2321 (python) S 2278 2278 27000 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338493030 446316544 6365 33554432000 4194304 5683644 140734090246896 139872782705200 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2322] ppid=2278 vsize=511648 CPUtime=0.07 cores=0,2,4,6
/proc/2322/stat : 2322 (python) S 2278 2278 27000 0 -1 4202560 1124 0 0 0 4 3 0 0 20 0 2 0 338497605 523927552 6371 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2322/statm: 127912 6371 384 364 0 63126 0
[pid=2324] ppid=2322 vsize=83900 CPUtime=47.31 cores=0,2,4,6
/proc/2324/stat : 2324 (minion) T 2322 2324 2324 0 -1 4202496 7142 0 0 0 4730 1 0 0 20 0 1 0 338497605 85913600 13691 33554432000 134512640 165565024 4288061424 4288058672 142765000 0 0 16781312 8396802 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/2324/statm: 20975 13691 336 7582 0 13390 0
[pid=2322/tid=2326] ppid=2278 vsize=511648 CPUtime=0 cores=0,2,4,6
/proc/2322/task/2326/stat : 2326 (python) S 2278 2278 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338502339 523927552 6371 33554432000 4194304 5683644 140734090246896 139872772215344 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2327] ppid=2278 vsize=587440 CPUtime=0.08 cores=0,2,4,6
/proc/2327/stat : 2327 (python) S 2278 2278 27000 0 -1 4202560 1141 0 0 0 5 3 0 0 20 0 2 0 338502341 601538560 6377 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2327/statm: 146860 6377 384 364 0 82071 0
[pid=2329] ppid=2327 vsize=93704 CPUtime=49.28 cores=0,2,4,6
/proc/2329/stat : 2329 (wbo1.4a) T 2327 2329 2329 0 -1 4202496 23338 0 0 0 4887 41 0 0 20 0 1 0 338502342 95952896 22986 33554432000 134512640 135751879 4290095792 4290093408 134604786 0 0 16781312 3 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/2329/statm: 23426 22986 143 303 0 23121 0
[pid=2327/tid=2330] ppid=2278 vsize=587440 CPUtime=0 cores=0,2,4,6
/proc/2327/task/2330/stat : 2330 (python) S 2278 2278 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338507273 601538560 6377 33554432000 4194304 5683644 140734090246896 139872761725488 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2334] ppid=2278 vsize=663232 CPUtime=0.08 cores=0,2,4,6
/proc/2334/stat : 2334 (python) S 2278 2278 27000 0 -1 4202560 1150 0 0 0 4 4 0 0 20 0 2 0 338511844 679149568 6383 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2334/statm: 165808 6383 384 364 0 101016 0
[pid=2336] ppid=2334 vsize=9624972 CPUtime=49.79 cores=0,2,4,6
/proc/2336/stat : 2336 (java) T 2334 2336 2336 0 -1 4202496 22475 0 1 0 4937 42 0 0 20 0 19 0 338511844 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140734762168496 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2336/statm: 2406243 431170 2266 9 0 2396937 0
[pid=2336/tid=2337] ppid=2334 vsize=9624972 CPUtime=47.19 cores=0,2,4,6
/proc/2336/task/2337/stat : 2337 (java) T 2334 2336 2336 0 -1 4202560 4121 0 1 0 4692 27 0 0 20 0 19 0 338511845 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057210694784 140057083657582 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=2336/tid=2338] ppid=2334 vsize=9624972 CPUtime=0.11 cores=0,2,4,6
/proc/2336/task/2338/stat : 2338 (java) T 2334 2336 2336 0 -1 4202560 117 0 0 0 9 2 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057054575656 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=2336/tid=2339] ppid=2334 vsize=9624972 CPUtime=0.11 cores=0,2,4,6
/proc/2336/task/2339/stat : 2339 (java) T 2334 2336 2336 0 -1 4202560 110 0 0 0 9 2 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057053522856 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=2336/tid=2340] ppid=2334 vsize=9624972 CPUtime=0.08 cores=0,2,4,6
/proc/2336/task/2340/stat : 2340 (java) T 2334 2336 2336 0 -1 4202560 57 0 0 0 7 1 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057052470056 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=2336/tid=2341] ppid=2334 vsize=9624972 CPUtime=0.07 cores=0,2,4,6
/proc/2336/task/2341/stat : 2341 (java) T 2334 2336 2336 0 -1 4202560 68 0 0 0 7 0 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057051417256 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=2336/tid=2342] ppid=2334 vsize=9624972 CPUtime=0.09 cores=0,2,4,6
/proc/2336/task/2342/stat : 2342 (java) T 2334 2336 2336 0 -1 4202560 46 0 0 0 7 2 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057050364456 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=2336/tid=2343] ppid=2334 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/2336/task/2343/stat : 2343 (java) T 2334 2336 2336 0 -1 4202560 41 0 0 0 11 1 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057049311656 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=2336/tid=2344] ppid=2334 vsize=9624972 CPUtime=0.08 cores=0,2,4,6
/proc/2336/task/2344/stat : 2344 (java) T 2334 2336 2336 0 -1 4202560 40 0 0 0 8 0 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057048258856 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=2336/tid=2345] ppid=2334 vsize=9624972 CPUtime=0.1 cores=0,2,4,6
/proc/2336/task/2345/stat : 2345 (java) T 2334 2336 2336 0 -1 4202560 84 0 0 0 8 2 0 0 20 0 19 0 338511846 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057047206056 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=2336/tid=2346] ppid=2334 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/2336/task/2346/stat : 2346 (java) T 2334 2336 2336 0 -1 4202560 740 0 0 0 0 0 0 0 20 0 19 0 338511847 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057035009000 219517335465 0 0 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=2336/tid=2347] ppid=2334 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/2336/task/2347/stat : 2347 (java) T 2334 2336 2336 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338511848 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057033955064 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=2336/tid=2348] ppid=2334 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/2336/task/2348/stat : 2348 (java) T 2334 2336 2336 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338511848 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057032902152 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=2336/tid=2349] ppid=2334 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/2336/task/2349/stat : 2349 (java) T 2334 2336 2336 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338511850 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057031850640 219517343520 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=2336/tid=2350] ppid=2334 vsize=9624972 CPUtime=0.99 cores=0,2,4,6
/proc/2336/task/2350/stat : 2350 (java) T 2334 2336 2336 0 -1 4202560 6755 0 0 0 99 0 0 0 20 0 19 0 338511850 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057030797752 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=2336/tid=2351] ppid=2334 vsize=9624972 CPUtime=0.7 cores=0,2,4,6
/proc/2336/task/2351/stat : 2351 (java) T 2334 2336 2336 0 -1 4202560 8872 0 0 0 68 2 0 0 20 0 19 0 338511850 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057029744952 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=2336/tid=2352] ppid=2334 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/2336/task/2352/stat : 2352 (java) T 2334 2336 2336 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338511850 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057028692344 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=2336/tid=2353] ppid=2334 vsize=9624972 CPUtime=0.01 cores=0,2,4,6
/proc/2336/task/2353/stat : 2353 (java) T 2334 2336 2336 0 -1 4202560 3 0 0 0 1 0 0 0 20 0 19 0 338511850 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057027639752 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=2336/tid=2354] ppid=2334 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/2336/task/2354/stat : 2354 (java) T 2334 2336 2336 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338511882 9855971328 431170 33554432000 1073741824 1073778376 140734762177360 140057026557608 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=2334/tid=2406] ppid=2278 vsize=663232 CPUtime=0 cores=0,2,4,6
/proc/2334/task/2406/stat : 2406 (python) S 2278 2278 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338516594 679149568 6383 33554432000 4194304 5683644 140734090246896 139872751235632 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2420] ppid=2278 vsize=739024 CPUtime=0.08 cores=0,2,4,6
/proc/2420/stat : 2420 (python) S 2278 2278 27000 0 -1 4202560 1159 0 0 0 5 3 0 0 20 0 2 0 338534594 756760576 6389 33554432000 4194304 5683644 140734090246896 140734090238544 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2420/statm: 184756 6389 384 364 0 119961 0
[pid=2422] ppid=2420 vsize=103640 CPUtime=47.23 cores=0,2,4,6
/proc/2422/stat : 2422 (bsolo_pb10) T 2420 2422 2422 0 -1 4202496 23935 0 0 0 4714 9 0 0 20 0 1 0 338534594 106127360 23487 33554432000 134512640 137138936 4287791936 4287790864 134542480 0 0 16781312 16384 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/2422/statm: 25910 23487 192 642 0 25266 0
[pid=2420/tid=2427] ppid=2278 vsize=739024 CPUtime=0 cores=0,2,4,6
/proc/2420/task/2427/stat : 2427 (python) S 2278 2278 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338539319 756760576 6389 33554432000 4194304 5683644 140734090246896 139872740745776 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2428] ppid=2278 vsize=814816 CPUtime=0 cores=0,2,4,6
/proc/2428/stat : 2428 (python) S 2278 2278 27000 0 -1 4202560 862 0 0 0 0 0 0 0 20 0 1 0 338539322 834371584 6854 33554432000 4194304 5683644 140734090246896 140734090237984 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/2428/statm: 203704 6854 336 364 0 138906 0
[pid=2430] ppid=2428 vsize=14524 CPUtime=1.67 cores=0,2,4,6
/proc/2430/stat : 2430 (bsolo_pb10) R 2428 2430 2430 0 -1 4202496 1656 0 0 0 165 2 0 0 20 0 1 0 338539322 14872576 1202 33554432000 134512640 137138936 4288456640 4288455756 136570369 0 0 16781312 16384 0 0 0 17 2 0 0 0 0 0
/proc/2430/statm: 3631 1202 191 642 0 2987 0
[pid=2278/tid=2285] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2285/stat : 2285 (python) S 2276 2278 27000 0 -1 4202560 32 1011 0 0 0 0 0 0 20 0 9 0 338478700 911970304 8439 33554432000 4194304 5683644 140734090246896 139872793195056 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2278/tid=2290] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2290/stat : 2290 (python) S 2276 2278 27000 0 -1 4202560 30 1011 0 0 0 0 0 0 20 0 9 0 338483357 911970304 8439 33554432000 4194304 5683644 140734090246896 139872782705200 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2278/tid=2318] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2318/stat : 2318 (python) S 2276 2278 27000 0 -1 4202560 25 1011 0 0 0 0 0 0 20 0 9 0 338488073 911970304 8439 33554432000 4194304 5683644 140734090246896 139872772215344 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2278/tid=2323] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2323/stat : 2323 (python) S 2276 2278 27000 0 -1 4202560 20 1011 0 0 0 0 0 0 20 0 9 0 338497605 911970304 8439 33554432000 4194304 5683644 140734090246896 139872761725488 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2278/tid=2328] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2328/stat : 2328 (python) S 2276 2278 27000 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 9 0 338502341 911970304 8439 33554432000 4194304 5683644 140734090246896 139872751235632 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2278/tid=2335] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2335/stat : 2335 (python) S 2276 2278 27000 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 9 0 338511844 911970304 8439 33554432000 4194304 5683644 140734090246896 139872740745776 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2278/tid=2421] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2421/stat : 2421 (python) S 2276 2278 27000 0 -1 4202560 20 1011 0 0 0 0 0 0 20 0 9 0 338534594 911970304 8439 33554432000 4194304 5683644 140734090246896 139872387981872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2278/tid=2429] ppid=2276 vsize=890596 CPUtime=0 cores=0,2,4,6
/proc/2278/task/2429/stat : 2429 (python) S 2276 2278 27000 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 9 0 338539322 911970304 8439 33554432000 4194304 5683644 140734090246896 139872377492016 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 611.57
Current children cumulated vsize (KiB) 15872208

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2278 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=352794
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=131979
# CPU time returned by wait4() is 1.48477
# while last known CPU time is 611.57
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 609.357
CPU time (s): 611.57
CPU user time (s): 608.48
CPU system time (s): 3.09
CPU usage (%): 100.363
Max. virtual memory (cumulated for all children) (KiB): 15872208

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.35279
system time used= 0.131979
maximum resident set size= 33756
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19663
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1464
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2090
involuntary context switches= 15

runsolver used 2.77158 second user time and 5.97409 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 13:38:31
IDJOB=3742109
IDBENCH=90132
IDSOLVER=2327
FILE ID=node114/3742109-1338723511
RUNJOBID= node114-1338713908-27016
PBS_JOBID= 14636748
Free space on /tmp= 70040 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K86.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742109-1338723511/watcher-3742109-1338723511 -o /tmp/evaluation-result-3742109-1338723511/solver-3742109-1338723511 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742109-1338723511.opb -seed 934199701 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a7a3236cd0071452eec601ea09fe7bbe
RANDOM SEED=934199701

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        26265108 kB
Buffers:         1036508 kB
Cached:          3819508 kB
SwapCached:         2180 kB
Active:          3822548 kB
Inactive:        1773916 kB
Active(anon):     725200 kB
Inactive(anon):    10720 kB
Active(file):    3097348 kB
Inactive(file):  1763196 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:            102072 kB
Writeback:             8 kB
AnonPages:        738784 kB
Mapped:            26328 kB
Shmem:               172 kB
Slab:             869824 kB
SReclaimable:     312264 kB
SUnreclaim:       557560 kB
KernelStack:        1616 kB
PageTables:         9012 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    1149992 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:     30720 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 69940 MiB
End job on node114 at 2012-06-03 13:48:43