Trace number 3729025

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 1735.23 1736.07

General information on the benchmark

Namenormalized-PB06/SATUNSAT-SMALLINT/submitted-PB06/
prestwich/armies/normalized-army12.21bt.opb
MD5SUMcc11de80c07cf0d445d50327479f6689
Bench CategoryDEC-SMALLINT (no optimisation, small integers)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark1373.4
Has Objective FunctionNO
SatisfiableYES
(Un)Satisfiability was provedYES
Best value of the objective function
Optimality of the best value was proved NO
Number of variables428
Total number of constraints942
Number of constraints which are clauses645
Number of constraints which are cardinality constraints (but not clauses)2
Number of constraints which are nor clauses,nor cardinality constraints295
Minimum length of a constraint1
Maximum length of a constraint144
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 32
Number of bits of the biggest number in a constraint 6
Biggest sum of numbers in a constraint 189
Number of bits of the biggest sum of numbers8
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.52/4.49	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.52/4.49	c INFO: solving HOME/instance-3729025-1338713700.opb
0.56/4.50	c INFO: parsing took 0.01 s
0.56/4.52	c INFO: features took 0.02 CPU seconds
0.59/6.44	c INFO: running scip-2.0.1-spx@0 for 45 with 1799 remaining (b = 0.45)
49.90/55.79	c INFO: running clasp-1.3.7@0 for 45 with 1751 remaining (b = 0.07)
97.14/102.94	c INFO: running scip-2.0.1-clp@0 for 45 with 1703 remaining (b = 0.02)
144.78/150.59	c INFO: running scip-2.0.1-clp@47 for 45 with 1656 remaining (b = 0.01)
192.66/198.32	c INFO: running clasp-1.3.7@47 for 45 with 1609 remaining (b = 0.01)
237.82/243.47	c INFO: running pbct-0.1.2-linear@0 for 45 with 1564 remaining (b = 0.02)
289.06/294.65	c INFO: running clasp-1.3.7@92 for 136 with 1515 remaining (b = 0.02)
426.65/432.04	c INFO: running scip-2.0.1-spx@48 for 45 with 1377 remaining (b = 0.01)
473.05/478.39	c INFO: running bsolo_pb10-l3@0 for 45 with 1332 remaining (b = 0.02)
520.31/525.57	c INFO: running scip-2.0.1-spx@94 for 410 with 1285 remaining (b = 0.06)
935.35/939.88	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 877 remaining (b = 0.01)
985.20/987.30	c INFO: running bsolo_pb10-l1@0 for 91 with 827 remaining (b = 0.01)
1080.66/1082.62	c INFO: running scip-2.0.1-spx@502 for 547 with 732 remaining (b = 0.05)
1640.68/1641.66	c INFO: running wbo1.4a@0 for 45 with 181 remaining (b = 0.00)
1689.85/1690.76	c INFO: running scip-2.0.1-clp@94 for 45 with 133 remaining (b = 0.00)
1734.93/1735.74	c INFO: wrote minion input file to HOME/tmppBQUoD.minion
1734.93/1735.74	c INFO: running minion-0.12@0 for 88 with 88 remaining (b = 0.00)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3729025-1338713700/watcher-3729025-1338713700 -o /tmp/evaluation-result-3729025-1338713700/solver-3729025-1338713700 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3729025-1338713700.opb -seed 1404835575 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.10 1.10 1.43 5/175 2657
/proc/meminfo: memFree=31401172/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=1844 CPUtime=0 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4194304 48 0 0 0 0 0 0 0 20 0 1 0 337495209 1888256 1 33554432000 0 0 140735074711141 140736425221896 5057063 0 0 4096 0 18446744071579943789 0 0 17 2 0 0 0 0 0
/proc/2657/statm: 461 1 0 0 0 37 0

[startup+0.0974111 s]
/proc/loadavg: 1.10 1.10 1.43 5/175 2657
/proc/meminfo: memFree=31401172/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=8340 CPUtime=0 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4202496 101 0 2 0 0 0 0 0 20 0 1 0 337495209 8540160 54 33554432000 4194304 5683644 140735074707568 140735074704376 240556015248 0 0 4096 0 18446744071581216301 0 0 17 0 0 0 0 0 0
/proc/2657/statm: 2085 54 42 364 0 45 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8340

[startup+0.10028 s]
/proc/loadavg: 1.10 1.10 1.43 5/175 2657
/proc/meminfo: memFree=31401172/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=8340 CPUtime=0 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4202496 101 0 2 0 0 0 0 0 20 0 1 0 337495209 8540160 54 33554432000 4194304 5683644 140735074707568 140735074704376 240556015248 0 0 4096 0 18446744071581216301 0 0 17 0 0 0 0 0 0
/proc/2657/statm: 2085 54 42 364 0 45 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8340

[startup+0.30025 s]
/proc/loadavg: 1.10 1.10 1.43 5/175 2657
/proc/meminfo: memFree=31401172/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=8340 CPUtime=0 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4202496 101 0 2 0 0 0 0 0 20 0 1 0 337495209 8540160 54 33554432000 4194304 5683644 140735074707568 140735074704376 240556015248 0 0 4096 0 18446744071581216301 0 0 17 0 0 0 0 0 0
/proc/2657/statm: 2085 54 42 364 0 45 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8340

[startup+0.700245 s]
/proc/loadavg: 1.10 1.10 1.43 5/175 2657
/proc/meminfo: memFree=31401172/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=8340 CPUtime=0 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4202496 101 0 2 0 0 0 0 0 20 0 1 0 337495209 8540160 54 33554432000 4194304 5683644 140735074707568 140735074704376 240556015248 0 0 4096 0 18446744071579943789 0 0 17 0 0 0 48 0 0
/proc/2657/statm: 2085 54 42 364 0 45 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8340

[startup+1.5002 s]
/proc/loadavg: 1.10 1.10 1.43 2/177 2659
/proc/meminfo: memFree=31395288/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=59092 CPUtime=0.08 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4202496 2297 0 15 0 3 5 0 0 20 0 1 0 337495209 60510208 1880 33554432000 4194304 5683644 140735074707568 140735074638888 240556010887 0 0 16781312 2 18446744071579943789 0 0 17 0 0 0 101 0 0
/proc/2657/statm: 14773 1880 517 364 0 1313 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 59092

[startup+3.10021 s]
/proc/loadavg: 1.10 1.10 1.43 2/177 2659
/proc/meminfo: memFree=31386748/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=122212 CPUtime=0.26 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4202496 5878 998 34 10 13 13 0 0 20 0 1 0 337495209 125145088 3814 33554432000 4194304 5683644 140735074707568 140735074650232 240565175184 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 184 0 0
/proc/2657/statm: 30553 3814 1015 364 0 2796 0
Current children cumulated CPU time (s) 0.26
Current children cumulated vsize (KiB) 122212
heavy processes:

[startup+6.30025 s]
/proc/loadavg: 1.17 1.12 1.43 2/177 2667
/proc/meminfo: memFree=31312588/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=281468 CPUtime=0.59 cores=0,2,4,6
/proc/2657/stat : 2657 (python) D 2652 2657 2614 0 -1 4202496 10063 998 86 10 34 25 0 0 20 0 1 0 337495209 288223232 7157 33554432000 4194304 5683644 140735074707568 140735074697784 240572799796 0 0 16781312 33554434 18446744071579943789 0 0 17 0 0 0 264 0 0
/proc/2657/statm: 70367 7157 1877 364 0 5593 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 281468
heavy processes:

[startup+12.7003 s]
/proc/loadavg: 1.55 1.20 1.46 2/180 2716
/proc/meminfo: memFree=31278348/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=357564 CPUtime=0.61 cores=0,2,4,6
/proc/2657/stat : 2657 (python) S 2652 2657 2614 0 -1 4202496 10489 998 88 10 36 25 0 0 20 0 2 0 337495209 366145536 7259 33554432000 4194304 5683644 140735074707568 140735074702400 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 405 0 0
/proc/2657/statm: 89391 7259 1900 364 0 24614 0
[pid=2671] ppid=2657 vsize=281784 CPUtime=0 cores=0,2,4,6
/proc/2671/stat : 2671 (python) S 2657 2657 2614 0 -1 4202560 883 0 0 0 0 0 0 0 20 0 1 0 337495854 288546816 5717 33554432000 4194304 5683644 140735074707568 140735074698656 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/2671/statm: 70446 5717 364 364 0 5669 0
[pid=2673] ppid=2671 vsize=48936 CPUtime=6.23 cores=0,2,4,6
/proc/2673/stat : 2673 (scip-2.0.1.linu) R 2671 2673 2673 0 -1 4202496 11400 0 0 0 617 6 0 0 20 0 1 0 337495854 50110464 9981 33554432000 4194304 13684479 140735563323920 140735563317720 11876362 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/2673/statm: 12234 9981 817 2317 0 9912 0
[pid=2657/tid=2672] ppid=2652 vsize=357564 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2672/stat : 2672 (python) S 2652 2657 2614 0 -1 4202560 25 998 0 10 0 0 0 0 20 0 2 0 337495854 366145536 7259 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 6.84
Current children cumulated vsize (KiB) 688284
heavy processes:

[startup+25.5003 s]
/proc/loadavg: 1.80 1.27 1.48 3/183 2719
/proc/meminfo: memFree=31241684/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=357564 CPUtime=0.61 cores=0,2,4,6
/proc/2657/stat : 2657 (python) S 2652 2657 2614 0 -1 4202496 10489 998 88 10 36 25 0 0 20 0 2 0 337495209 366145536 7259 33554432000 4194304 5683644 140735074707568 140735074702400 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 405 0 0
/proc/2657/statm: 89391 7259 1900 364 0 24614 0
[pid=2671] ppid=2657 vsize=281784 CPUtime=0.02 cores=0,2,4,6
/proc/2671/stat : 2671 (python) S 2657 2657 2614 0 -1 4202560 986 0 0 0 1 1 0 0 20 0 1 0 337495854 288546816 5722 33554432000 4194304 5683644 140735074707568 140735074698656 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/2671/statm: 70446 5722 369 364 0 5669 0
[pid=2673] ppid=2671 vsize=48936 CPUtime=19.04 cores=0,2,4,6
/proc/2673/stat : 2673 (scip-2.0.1.linu) R 2671 2673 2673 0 -1 4202496 11404 0 0 0 1881 23 0 0 20 0 1 0 337495854 50110464 9985 33554432000 4194304 13684479 140735563323920 140735563317720 11876362 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/2673/statm: 12234 9985 821 2317 0 9912 0
[pid=2657/tid=2672] ppid=2652 vsize=357564 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2672/stat : 2672 (python) S 2652 2657 2614 0 -1 4202560 25 998 0 10 0 0 0 0 20 0 2 0 337495854 366145536 7259 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 19.67
Current children cumulated vsize (KiB) 688284
heavy processes:

[startup+51.1003 s]
/proc/loadavg: 1.94 1.34 1.49 3/183 2719
/proc/meminfo: memFree=31241156/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=357564 CPUtime=0.61 cores=0,2,4,6
/proc/2657/stat : 2657 (python) S 2652 2657 2614 0 -1 4202496 10489 998 88 10 36 25 0 0 20 0 2 0 337495209 366145536 7259 33554432000 4194304 5683644 140735074707568 140735074702400 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 405 0 0
/proc/2657/statm: 89391 7259 1900 364 0 24614 0
[pid=2671] ppid=2657 vsize=281784 CPUtime=0.07 cores=0,2,4,6
/proc/2671/stat : 2671 (python) S 2657 2657 2614 0 -1 4202560 1016 0 0 0 4 3 0 0 20 0 1 0 337495854 288546816 5722 33554432000 4194304 5683644 140735074707568 140735074698656 240565191080 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/2671/statm: 70446 5722 369 364 0 5669 0
[pid=2673] ppid=2671 vsize=48936 CPUtime=44.63 cores=0,2,4,6
/proc/2673/stat : 2673 (scip-2.0.1.linu) R 2671 2673 2673 0 -1 4202496 11416 0 0 0 4399 64 0 0 20 0 1 0 337495854 50110464 10246 33554432000 4194304 13684479 140735563323920 140735563317768 11876362 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/2673/statm: 12234 10246 821 2317 0 9912 0
[pid=2657/tid=2672] ppid=2652 vsize=357564 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2672/stat : 2672 (python) S 2652 2657 2614 0 -1 4202560 25 998 0 10 0 0 0 0 20 0 2 0 337495854 366145536 7259 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 45.31
Current children cumulated vsize (KiB) 688284

[startup+102.3 s]
/proc/loadavg: 2.19 1.50 1.54 3/187 2819
/proc/meminfo: memFree=31222088/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=433784 CPUtime=0.64 cores=0,2,4,6
/proc/2657/stat : 2657 (python) S 2652 2657 2614 0 -1 4202496 11377 998 88 10 38 26 0 0 20 0 3 0 337495209 444194816 7384 33554432000 4194304 5683644 140735074707568 140735074702400 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 405 0 0
/proc/2657/statm: 108446 7384 1903 364 0 43666 0
[pid=2671] ppid=2657 vsize=357564 CPUtime=0.07 cores=0,2,4,6
/proc/2671/stat : 2671 (python) S 2657 2657 2614 0 -1 4202560 1157 0 0 0 4 3 0 0 20 0 2 0 337495854 366145536 5757 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2671/statm: 89391 5757 397 364 0 24614 0
[pid=2673] ppid=2671 vsize=48936 CPUtime=49.29 cores=0,2,4,6
/proc/2673/stat : 2673 (scip-2.0.1.linu) T 2671 2673 2673 0 -1 4202496 11416 0 0 0 4859 70 0 0 20 0 1 0 337495854 50110464 10246 33554432000 4194304 13684479 140735563323920 140735563317752 7390793 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2673/statm: 12234 10246 821 2317 0 9912 0
[pid=2671/tid=2720] ppid=2657 vsize=357564 CPUtime=0 cores=0,2,4,6
/proc/2671/task/2720/stat : 2720 (python) S 2657 2657 2614 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 337500787 366145536 5757 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2721] ppid=2657 vsize=358004 CPUtime=0.06 cores=0,2,4,6
/proc/2721/stat : 2721 (python) S 2657 2657 2614 0 -1 4202560 913 0 0 0 4 2 0 0 20 0 1 0 337500789 366596096 5834 33554432000 4194304 5683644 140735074707568 140735074698656 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/2721/statm: 89501 5834 359 364 0 24721 0
[pid=2723] ppid=2721 vsize=3316 CPUtime=46.48 cores=0,2,4,6
/proc/2723/stat : 2723 (clasp-1.3.7-x86) R 2721 2723 2723 0 -1 4202496 841 0 0 0 4648 0 0 0 20 0 1 0 337500789 3395584 588 33554432000 134512640 136139019 4291282752 4291281676 134828711 0 0 16781312 16386 0 0 0 17 0 0 0 0 0 0
/proc/2723/statm: 829 588 183 398 0 429 0
[pid=2657/tid=2672] ppid=2652 vsize=433784 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2672/stat : 2672 (python) S 2652 2657 2614 0 -1 4202560 25 998 0 10 0 0 0 0 20 0 3 0 337495854 444194816 7384 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2657/tid=2722] ppid=2652 vsize=433784 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2722/stat : 2722 (python) S 2652 2657 2614 0 -1 4202560 21 998 0 10 0 0 0 0 20 0 3 0 337500789 444194816 7384 33554432000 4194304 5683644 140735074707568 139842678354480 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 96.54
Current children cumulated vsize (KiB) 1201604

[startup+162.301 s]
/proc/loadavg: 2.13 1.61 1.57 3/196 2828
/proc/meminfo: memFree=31125044/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=509784 CPUtime=0.69 cores=0,2,4,6
/proc/2657/stat : 2657 (python) S 2652 2657 2614 0 -1 4202496 12919 998 88 10 43 26 0 0 20 0 4 0 337495209 522018816 7936 33554432000 4194304 5683644 140735074707568 140735074702400 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 405 0 0
/proc/2657/statm: 127446 7936 1905 364 0 62663 0
[pid=2671] ppid=2657 vsize=357564 CPUtime=0.07 cores=0,2,4,6
/proc/2671/stat : 2671 (python) S 2657 2657 2614 0 -1 4202560 1157 0 0 0 4 3 0 0 20 0 2 0 337495854 366145536 5757 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2671/statm: 89391 5757 397 364 0 24614 0
[pid=2673] ppid=2671 vsize=48936 CPUtime=49.29 cores=0,2,4,6
/proc/2673/stat : 2673 (scip-2.0.1.linu) T 2671 2673 2673 0 -1 4202496 11416 0 0 0 4859 70 0 0 20 0 1 0 337495854 50110464 10246 33554432000 4194304 13684479 140735563323920 140735563317752 7390793 0 0 16781312 2 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2673/statm: 12234 10246 821 2317 0 9912 0
[pid=2671/tid=2720] ppid=2657 vsize=357564 CPUtime=0 cores=0,2,4,6
/proc/2671/task/2720/stat : 2720 (python) S 2657 2657 2614 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 337500787 366145536 5757 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2721] ppid=2657 vsize=358004 CPUtime=0.07 cores=0,2,4,6
/proc/2721/stat : 2721 (python) S 2657 2657 2614 0 -1 4202560 1074 0 0 0 4 3 0 0 20 0 2 0 337500789 366596096 5860 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/2721/statm: 89501 5860 384 364 0 24721 0
[pid=2723] ppid=2721 vsize=3316 CPUtime=47.1 cores=0,2,4,6
/proc/2723/stat : 2723 (clasp-1.3.7-x86) T 2721 2723 2723 0 -1 4202496 841 0 0 0 4710 0 0 0 20 0 1 0 337500789 3395584 588 33554432000 134512640 136139019 4291282752 4291281684 134632451 0 0 16781312 16386 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/2723/statm: 829 588 183 398 0 429 0
[pid=2721/tid=2820] ppid=2657 vsize=358004 CPUtime=0 cores=0,2,4,6
/proc/2721/task/2820/stat : 2820 (python) S 2657 2657 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 337505502 366596096 5860 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2821] ppid=2657 vsize=433500 CPUtime=0.09 cores=0,2,4,6
/proc/2821/stat : 2821 (python) S 2657 2657 2614 0 -1 4202560 1101 0 0 0 5 4 0 0 20 0 2 0 337505504 443904000 5808 33554432000 4194304 5683644 140735074707568 140735074698656 240565191139 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2821/statm: 108375 5808 384 364 0 43592 0
[pid=2823] ppid=2821 vsize=54680 CPUtime=59.29 cores=0,2,4,6
/proc/2823/stat : 2823 (scip-2.0.1.linu) R 2821 2823 2823 0 -1 4202496 16991 0 0 0 5892 37 0 0 20 0 1 0 337505504 55992320 11261 33554432000 4194304 15243407 140736559293712 140736559288104 13354455 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0

################
# More data... #
################

[pid=2657/tid=2822] ppid=2652 vsize=888956 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2822/stat : 2822 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 9 0 337505504 910290944 8029 33554432000 4194304 5683644 140735074707568 139842591646256 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2657/tid=3156] ppid=2652 vsize=888956 CPUtime=0 cores=0,2,4,6
/proc/2657/task/3156/stat : 3156 (python) S 2652 2657 2614 0 -1 4202560 21 998 0 10 0 0 0 0 20 0 9 0 337519557 910290944 8029 33554432000 4194304 5683644 140735074707568 139842581156400 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2657/tid=4157] ppid=2652 vsize=888956 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4157/stat : 4157 (python) S 2652 2657 2614 0 -1 4202560 21 998 0 10 0 0 0 0 20 0 9 0 337543049 910290944 8029 33554432000 4194304 5683644 140735074707568 139842570666544 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2657/tid=4175] ppid=2652 vsize=888956 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4175/stat : 4175 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 9 0 337589198 910290944 8029 33554432000 4194304 5683644 140735074707568 139842560176688 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2657/tid=4197] ppid=2652 vsize=888956 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4197/stat : 4197 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 9 0 337593940 910290944 8029 33554432000 4194304 5683644 140735074707568 139842549686832 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2657/tid=4214] ppid=2652 vsize=888956 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4214/stat : 4214 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 9 0 337659376 910290944 8029 33554432000 4194304 5683644 140735074707568 139842539196976 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 1721.49
Current children cumulated vsize (KiB) 15295244



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+1736 s]
/proc/loadavg: 2.04 2.02 1.92 3/254 4219
/proc/meminfo: memFree=29175036/32873844 swapFree=7116/7116
[pid=2657] ppid=2652 vsize=964748 CPUtime=0.98 cores=0,2,4,6
/proc/2657/stat : 2657 (python) S 2652 2657 2614 0 -1 4202496 18207 998 88 10 70 28 0 0 20 0 10 0 337495209 987901952 8545 33554432000 4194304 5683644 140735074707568 140735074702400 240568558861 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 405 0 0
/proc/2657/statm: 241187 8545 1917 364 0 176386 0
[pid=2671] ppid=2657 vsize=358252 CPUtime=2.09 cores=0,2,4,6
/proc/2671/stat : 2671 (python) S 2657 2657 2614 0 -1 4202560 1404 0 0 0 123 86 0 0 20 0 2 0 337495854 366850048 5918 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2671/statm: 89563 5918 400 364 0 24786 0
[pid=2673] ppid=2671 vsize=53640 CPUtime=1068.81 cores=0,2,4,6
/proc/2673/stat : 2673 (scip-2.0.1.linu) T 2671 2673 2673 0 -1 4202496 12596 0 0 0 105278 1603 0 0 20 0 1 0 337495854 54927360 11426 33554432000 4194304 13684479 140735563323920 140735563317752 11876362 0 0 16781312 2 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/2673/statm: 13410 11426 821 2317 0 11088 0
[pid=2671/tid=2720] ppid=2657 vsize=358252 CPUtime=0 cores=0,2,4,6
/proc/2671/task/2720/stat : 2720 (python) S 2657 2657 2614 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 337500787 366850048 5918 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2721] ppid=2657 vsize=358004 CPUtime=0.37 cores=0,2,4,6
/proc/2721/stat : 2721 (python) S 2657 2657 2614 0 -1 4202560 1165 0 0 0 23 14 0 0 20 0 2 0 337500789 366596096 5860 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2721/statm: 89501 5860 384 364 0 24721 0
[pid=2723] ppid=2721 vsize=4120 CPUtime=229.55 cores=0,2,4,6
/proc/2723/stat : 2723 (clasp-1.3.7-x86) T 2721 2723 2723 0 -1 4202496 1030 0 0 0 22955 0 0 0 20 0 1 0 337500789 4218880 777 33554432000 134512640 136139019 4291282752 4291281616 134608488 0 0 16781312 16386 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/2723/statm: 1030 777 183 398 0 630 0
[pid=2721/tid=2820] ppid=2657 vsize=358004 CPUtime=0 cores=0,2,4,6
/proc/2721/task/2820/stat : 2820 (python) S 2657 2657 2614 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 337505502 366596096 5860 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2821] ppid=2657 vsize=433500 CPUtime=0.24 cores=0,2,4,6
/proc/2821/stat : 2821 (python) S 2657 2657 2614 0 -1 4202560 1134 0 0 0 14 10 0 0 20 0 2 0 337505504 443904000 5808 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2821/statm: 108375 5808 384 364 0 43592 0
[pid=2823] ppid=2821 vsize=54680 CPUtime=140.17 cores=0,2,4,6
/proc/2823/stat : 2823 (scip-2.0.1.linu) T 2821 2823 2823 0 -1 4202496 17087 0 0 0 13927 90 0 0 20 0 1 0 337505504 55992320 11455 33554432000 4194304 15243407 140736559293712 140736559288008 7917560 0 0 16781312 2 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/2823/statm: 13670 11455 946 2698 0 10965 0
[pid=2821/tid=2828] ppid=2657 vsize=433500 CPUtime=0 cores=0,2,4,6
/proc/2821/task/2828/stat : 2828 (python) S 2657 2657 2614 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 337510266 443904000 5808 33554432000 4194304 5683644 140735074707568 139842678354480 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=3155] ppid=2657 vsize=509796 CPUtime=0.07 cores=0,2,4,6
/proc/3155/stat : 3155 (python) S 2657 2657 2614 0 -1 4202560 1124 0 0 0 4 3 0 0 20 0 2 0 337519557 522031104 6416 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/3155/statm: 127449 6416 384 364 0 62663 0
[pid=3157] ppid=3155 vsize=20216 CPUtime=51.12 cores=0,2,4,6
/proc/3157/stat : 3157 (pbct-0.1.2-linu) T 3155 3157 3157 0 -1 4202496 4922 0 0 0 4889 223 0 0 20 0 1 0 337519558 20701184 4654 33554432000 134512640 136422188 4289160896 4289158620 135874785 0 0 16781312 16387 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/3157/statm: 5054 4654 214 467 0 4585 0
[pid=3155/tid=3389] ppid=2657 vsize=509796 CPUtime=0 cores=0,2,4,6
/proc/3155/task/3389/stat : 3389 (python) S 2657 2657 2614 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 337524672 522031104 6416 33554432000 4194304 5683644 140735074707568 139842591646256 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=4156] ppid=2657 vsize=585760 CPUtime=0.08 cores=0,2,4,6
/proc/4156/stat : 4156 (python) S 2657 2657 2614 0 -1 4202560 1124 0 0 0 4 4 0 0 20 0 2 0 337543049 599818240 6465 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/4156/statm: 146440 6465 384 364 0 81651 0
[pid=4158] ppid=4156 vsize=40056 CPUtime=47.13 cores=0,2,4,6
/proc/4158/stat : 4158 (bsolo_pb10) T 4156 4158 4158 0 -1 4202496 7854 0 0 0 4711 2 0 0 20 0 1 0 337543050 41017344 7582 33554432000 134512640 137138936 4292989760 4292988472 136506283 0 0 16781312 16384 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/4158/statm: 10014 7582 192 642 0 9370 0
[pid=4156/tid=4163] ppid=2657 vsize=585760 CPUtime=0 cores=0,2,4,6
/proc/4156/task/4163/stat : 4163 (python) S 2657 2657 2614 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 337547765 599818240 6465 33554432000 4194304 5683644 140735074707568 139842581156400 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=4174] ppid=2657 vsize=661552 CPUtime=0.09 cores=0,2,4,6
/proc/4174/stat : 4174 (python) S 2657 2657 2614 0 -1 4202560 1132 0 0 0 5 4 0 0 20 0 2 0 337589198 677429248 6473 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/4174/statm: 165388 6473 384 364 0 100596 0
[pid=4176] ppid=4174 vsize=9624972 CPUtime=49.68 cores=0,2,4,6
/proc/4176/stat : 4176 (java) T 4174 4176 4176 0 -1 4202496 21049 0 1 0 4938 30 0 0 20 0 19 0 337589198 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140736650124416 240568533053 0 0 0 16800975 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/4176/statm: 2406243 232335 2256 9 0 2396937 0
[pid=4176/tid=4177] ppid=4174 vsize=9624972 CPUtime=46.86 cores=0,2,4,6
/proc/4176/task/4177/stat : 4177 (java) T 4174 4176 4176 0 -1 4202560 3349 0 1 0 4674 12 0 0 20 0 19 0 337589199 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132822543616 140132653500514 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4178] ppid=4174 vsize=9624972 CPUtime=0.14 cores=0,2,4,6
/proc/4176/task/4178/stat : 4178 (java) T 4174 4176 4176 0 -1 4202560 41 0 0 0 12 2 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132784036904 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4179] ppid=4174 vsize=9624972 CPUtime=0.15 cores=0,2,4,6
/proc/4176/task/4179/stat : 4179 (java) T 4174 4176 4176 0 -1 4202560 53 0 0 0 13 2 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132782984104 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4180] ppid=4174 vsize=9624972 CPUtime=0.11 cores=0,2,4,6
/proc/4176/task/4180/stat : 4180 (java) T 4174 4176 4176 0 -1 4202560 24 0 0 0 11 0 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132781931304 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4181] ppid=4174 vsize=9624972 CPUtime=0.14 cores=0,2,4,6
/proc/4176/task/4181/stat : 4181 (java) T 4174 4176 4176 0 -1 4202560 49 0 0 0 12 2 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132780878504 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=4176/tid=4182] ppid=4174 vsize=9624972 CPUtime=0.14 cores=0,2,4,6
/proc/4176/task/4182/stat : 4182 (java) T 4174 4176 4176 0 -1 4202560 38 0 0 0 12 2 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132779825704 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4183] ppid=4174 vsize=9624972 CPUtime=0.14 cores=0,2,4,6
/proc/4176/task/4183/stat : 4183 (java) T 4174 4176 4176 0 -1 4202560 29 0 0 0 12 2 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132778772904 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=4176/tid=4184] ppid=4174 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/4176/task/4184/stat : 4184 (java) T 4174 4176 4176 0 -1 4202560 34 0 0 0 10 2 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132777720104 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=4176/tid=4185] ppid=4174 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/4176/task/4185/stat : 4185 (java) T 4174 4176 4176 0 -1 4202560 43 0 0 0 10 2 0 0 20 0 19 0 337589201 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132776667304 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=4176/tid=4186] ppid=4174 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/4176/task/4186/stat : 4186 (java) T 4174 4176 4176 0 -1 4202560 592 0 0 0 0 0 0 0 20 0 19 0 337589202 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132646620136 240568547241 0 0 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4187] ppid=4174 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/4176/task/4187/stat : 4187 (java) T 4174 4176 4176 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 337589202 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132645566200 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4188] ppid=4174 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/4176/task/4188/stat : 4188 (java) T 4174 4176 4176 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 337589202 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132644513288 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4189] ppid=4174 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/4176/task/4189/stat : 4189 (java) T 4174 4176 4176 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 337589205 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132643461776 240568555296 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4190] ppid=4174 vsize=9624972 CPUtime=0.74 cores=0,2,4,6
/proc/4176/task/4190/stat : 4190 (java) T 4174 4176 4176 0 -1 4202560 6493 0 0 0 74 0 0 0 20 0 19 0 337589205 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132642408888 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4191] ppid=4174 vsize=9624972 CPUtime=0.74 cores=0,2,4,6
/proc/4176/task/4191/stat : 4191 (java) T 4174 4176 4176 0 -1 4202560 8880 0 0 0 72 2 0 0 20 0 19 0 337589205 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132641356088 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4192] ppid=4174 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/4176/task/4192/stat : 4192 (java) T 4174 4176 4176 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 337589205 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132640303480 240568546364 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=4176/tid=4193] ppid=4174 vsize=9624972 CPUtime=0.01 cores=0,2,4,6
/proc/4176/task/4193/stat : 4193 (java) T 4174 4176 4176 0 -1 4202560 5 0 0 0 0 1 0 0 20 0 19 0 337589205 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132639250888 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=4176/tid=4194] ppid=4174 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/4176/task/4194/stat : 4194 (java) T 4174 4176 4176 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 337589225 9855971328 232335 33554432000 1073741824 1073778376 140736650133280 140132638197416 240568547241 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=4174/tid=4195] ppid=2657 vsize=661552 CPUtime=0 cores=0,2,4,6
/proc/4174/task/4195/stat : 4195 (python) S 2657 2657 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 337593938 677429248 6473 33554432000 4194304 5683644 140735074707568 139842570666544 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=4196] ppid=2657 vsize=737344 CPUtime=0.18 cores=0,2,4,6
/proc/4196/stat : 4196 (python) S 2657 2657 2614 0 -1 4202560 1170 0 0 0 11 7 0 0 20 0 2 0 337593940 755040256 6479 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/4196/statm: 184336 6479 384 364 0 119541 0
[pid=4198] ppid=4196 vsize=45280 CPUtime=95.27 cores=0,2,4,6
/proc/4198/stat : 4198 (bsolo_pb10) T 4196 4198 4198 0 -1 4202496 9171 0 0 0 9513 14 0 0 20 0 1 0 337593940 46366720 8890 33554432000 134512640 137138936 4292574592 4292573504 136296988 0 0 16781312 16384 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/4198/statm: 11320 8890 191 642 0 10676 0
[pid=4196/tid=4199] ppid=2657 vsize=737344 CPUtime=0 cores=0,2,4,6
/proc/4196/task/4199/stat : 4199 (python) S 2657 2657 2614 0 -1 4202560 48 0 0 0 0 0 0 0 20 0 2 0 337603470 755040256 6479 33554432000 4194304 5683644 140735074707568 139842560176688 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=4213] ppid=2657 vsize=813176 CPUtime=0.08 cores=0,2,4,6
/proc/4213/stat : 4213 (python) S 2657 2657 2614 0 -1 4202560 1213 0 0 0 5 3 0 0 20 0 2 0 337659376 832692224 6499 33554432000 4194304 5683644 140735074707568 140735074699216 240568558861 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/4213/statm: 203294 6499 390 364 0 138496 0
[pid=4215] ppid=4213 vsize=105940 CPUtime=49.07 cores=0,2,4,6
/proc/4215/stat : 4215 (wbo1.4a) T 4213 4215 4215 0 -1 4202496 26240 0 0 0 4884 23 0 0 20 0 1 0 337659377 108482560 25967 33554432000 134512640 135751879 4288003664 4288001840 134616516 0 0 16781312 3 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/4215/statm: 26485 25967 143 303 0 26180 0
[pid=4213/tid=4216] ppid=2657 vsize=813176 CPUtime=0 cores=0,2,4,6
/proc/4213/task/4216/stat : 4216 (python) S 2657 2657 2614 0 -1 4202560 42 0 0 0 0 0 0 0 20 0 2 0 337664285 832692224 6499 33554432000 4194304 5683644 140735074707568 139842549686832 240568555296 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=4217] ppid=2657 vsize=888968 CPUtime=0 cores=0,2,4,6
/proc/4217/stat : 4217 (python) S 2657 2657 2614 0 -1 4202560 893 0 0 0 0 0 0 0 20 0 1 0 337668784 910303232 6457 33554432000 4194304 5683644 140735074707568 140735074698656 240565191139 0 0 16781312 33554946 18446744071580425449 0 0 17 6 0 0 0 0 0
/proc/4217/statm: 222242 6457 343 364 0 157441 0
[pid=4219] ppid=4217 vsize=64408 CPUtime=0.25 cores=0,2,4,6
/proc/4219/stat : 4219 (minion) R 4217 4219 4219 0 -1 4202496 1963 0 0 0 25 0 0 0 20 0 1 0 337668784 65953792 8842 33554432000 134512640 165565024 4288388432 4288385664 159737071 0 0 16781312 8396802 0 0 0 17 2 0 0 0 0 0
/proc/4219/statm: 16102 8842 345 7582 0 8517 0
[pid=2657/tid=2672] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2672/stat : 2672 (python) S 2652 2657 2614 0 -1 4202560 39 998 0 10 0 0 0 0 20 0 10 0 337495854 987901952 8545 33554432000 4194304 5683644 140735074707568 139842688844336 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2657/tid=2722] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2722/stat : 2722 (python) S 2652 2657 2614 0 -1 4202560 30 998 0 10 0 0 0 0 20 0 10 0 337500789 987901952 8545 33554432000 4194304 5683644 140735074707568 139842678354480 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2657/tid=2822] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/2822/stat : 2822 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 10 0 337505504 987901952 8545 33554432000 4194304 5683644 140735074707568 139842591646256 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2657/tid=3156] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/3156/stat : 3156 (python) S 2652 2657 2614 0 -1 4202560 21 998 0 10 0 0 0 0 20 0 10 0 337519557 987901952 8545 33554432000 4194304 5683644 140735074707568 139842581156400 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2657/tid=4157] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4157/stat : 4157 (python) S 2652 2657 2614 0 -1 4202560 21 998 0 10 0 0 0 0 20 0 10 0 337543049 987901952 8545 33554432000 4194304 5683644 140735074707568 139842570666544 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2657/tid=4175] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4175/stat : 4175 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 10 0 337589198 987901952 8545 33554432000 4194304 5683644 140735074707568 139842560176688 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2657/tid=4197] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4197/stat : 4197 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 10 0 337593940 987901952 8545 33554432000 4194304 5683644 140735074707568 139842549686832 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2657/tid=4214] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4214/stat : 4214 (python) S 2652 2657 2614 0 -1 4202560 22 998 0 10 0 0 0 0 20 0 10 0 337659376 987901952 8545 33554432000 4194304 5683644 140735074707568 139842539196976 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2657/tid=4218] ppid=2652 vsize=964748 CPUtime=0 cores=0,2,4,6
/proc/2657/task/4218/stat : 4218 (python) S 2652 2657 2614 0 -1 4202560 20 998 0 10 0 0 0 0 20 0 10 0 337668784 987901952 8545 33554432000 4194304 5683644 140735074707568 139842121884208 240568555296 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 1735.23
Current children cumulated vsize (KiB) 16324412

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2657 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=706892
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=285956
# CPU time returned by wait4() is 0.992848
# while last known CPU time is 1735.23
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1736.07
CPU time (s): 1735.23
CPU user time (s): 1713.79
CPU system time (s): 21.44
CPU usage (%): 99.9517
Max. virtual memory (cumulated for all children) (KiB): 16324412

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.706892
system time used= 0.285956
maximum resident set size= 34180
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19688
page faults= 98
swaps= 0
block input operations= 41552
block output operations= 592
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7294
involuntary context switches= 57

runsolver used 7.54885 second user time and 18.7342 second system time

The end

Launcher Data 

Begin job on node146 at 2012-06-03 10:55:00
IDJOB=3729025
IDBENCH=1161
IDSOLVER=2327
FILE ID=node146/3729025-1338713700
RUNJOBID= node146-1338713700-2630
PBS_JOBID= 14636739
Free space on /tmp= 70912 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB06//final/normalized-PB06/SATUNSAT-SMALLINT/submitted-PB06/prestwich/armies/normalized-army12.21bt.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3729025-1338713700/watcher-3729025-1338713700 -o /tmp/evaluation-result-3729025-1338713700/solver-3729025-1338713700 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3729025-1338713700.opb -seed 1404835575 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= cc11de80c07cf0d445d50327479f6689
RANDOM SEED=1404835575

node146.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.034
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        31401816 kB
Buffers:          154000 kB
Cached:           546720 kB
SwapCached:         1792 kB
Active:           450880 kB
Inactive:         253312 kB
Active(anon):       4240 kB
Inactive(anon):     1664 kB
Active(file):     446640 kB
Inactive(file):   251648 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67099888 kB
Dirty:            203724 kB
Writeback:             0 kB
AnonPages:          9732 kB
Mapped:             8212 kB
Shmem:                 0 kB
Slab:             629504 kB
SReclaimable:      66112 kB
SUnreclaim:       563392 kB
KernelStack:        1400 kB
PageTables:         3836 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:     103564 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70704 MiB
End job on node146 at 2012-06-03 11:24:01