Trace number 3482303

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg pb-opt-11.04.03? (MO) 242 238.702

General information on the benchmark

Name/OPT-SMALLINT-LIN/heinz/
normalized-neos-631710.opb
MD5SUMba0215d19182fa8cb27dee737825dd09
Bench CategoryOPT-SMALLINT-LIN (optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark205
Best CPU time to get the best result obtained on this benchmark1798.07
Has Objective FunctionYES
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables167056
Total number of constraints169576
Number of constraints which are clauses167055
Number of constraints which are cardinality constraints (but not clauses)1965
Number of constraints which are nor clauses,nor cardinality constraints556
Minimum length of a constraint2
Maximum length of a constraint556
Number of terms in the objective function 556
Biggest coefficient in the objective function 1
Number of bits for the biggest coefficient in the objective function 1
Sum of the numbers in the objective function 556
Number of bits of the sum of numbers in the objective function 10
Biggest number in a constraint 4
Number of bits of the biggest number in a constraint 3
Biggest sum of numbers in a constraint 757
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.32/0.47	c INFO: loaded portfolio from HOME/etc/borg-mix+class.opt.pickle
0.32/0.47	c INFO: solving HOME/instance-3482303-1307578897.opb
1.81/1.92	c INFO: parsing took 1.42 s
4.51/4.68	c INFO: features took 2.76 CPU seconds
4.61/4.71	c INFO: running wbo1.4a@0 for 45 with 1795 remaining (b = 0.02)
52.81/52.79	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1747 remaining (b = 0.00)
99.27/97.03	c INFO: running bsolo_pb10-l3@0 for 136 with 1701 remaining (b = 0.01)
240.81/237.88	c INFO: running sat4j-pb-v20101225@0 for 91 with 1560 remaining (b = 0.00)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3482303-1307578897/watcher-3482303-1307578897 -o /tmp/evaluation-result-3482303-1307578897/solver-3482303-1307578897 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.opt.pickle HOME/instance-3482303-1307578897.opb -seed 549437552 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.23 2.11 1.93 3/952 393
/proc/meminfo: memFree=10772248/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=14820 CPUtime=0 cores=0,2,4,6
/proc/393/stat : 393 (python) R 390 393 30488 0 -1 4202496 512 0 0 0 0 0 0 0 20 0 1 0 869488602 15175680 419 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272684901 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/393/statm: 3705 419 259 364 0 192 0

[startup+0.0410409 s]
/proc/loadavg: 2.23 2.11 1.93 3/952 393
/proc/meminfo: memFree=10772248/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=20976 CPUtime=0.02 cores=0,2,4,6
/proc/393/stat : 393 (python) S 390 393 30488 0 -1 4202496 1301 0 0 0 2 0 0 0 21 0 1 0 869488602 21479424 1152 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272687104 0 2147483391 16781312 2 18446744071699199166 0 0 17 2 0 0 0
/proc/393/statm: 5244 1152 428 364 0 691 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 20976

[startup+0.101071 s]
/proc/loadavg: 2.23 2.11 1.93 3/952 393
/proc/meminfo: memFree=10772248/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=96128 CPUtime=0.07 cores=0,2,4,6
/proc/393/stat : 393 (python) R 390 393 30488 0 -1 4202496 3170 933 1 0 6 1 0 0 18 0 1 0 869488602 98435072 2483 33554432000 4194304 5683644 140736702196192 18446744073709551615 5178128 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/393/statm: 24032 2483 792 364 0 1627 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 96128

[startup+0.301056 s]
/proc/loadavg: 2.23 2.11 1.93 3/952 393
/proc/meminfo: memFree=10772248/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=203284 CPUtime=0.23 cores=0,2,4,6
/proc/393/stat : 393 (python) R 390 393 30488 0 -1 4202496 6732 933 1 0 17 6 0 0 17 0 1 0 869488602 208162816 5008 33554432000 4194304 5683644 140736702196192 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/393/statm: 50821 5009 1472 364 0 3783 0
Current children cumulated CPU time (s) 0.23
Current children cumulated vsize (KiB) 203284

[startup+0.700986 s]
/proc/loadavg: 2.23 2.11 1.93 3/952 393
/proc/meminfo: memFree=10772248/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=298920 CPUtime=0.62 cores=0,2,4,6
/proc/393/stat : 393 (python) R 390 393 30488 0 -1 4202496 13135 933 1 0 52 10 0 0 17 0 1 0 869488602 306094080 10996 33554432000 4194304 5683644 140736702196192 18446744073709551615 47094907940315 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/393/statm: 74730 10997 1781 364 0 9623 0
Current children cumulated CPU time (s) 0.62
Current children cumulated vsize (KiB) 298920

[startup+1.50085 s]
/proc/loadavg: 2.23 2.11 1.93 3/953 400
/proc/meminfo: memFree=10723384/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=326460 CPUtime=1.41 cores=0,2,4,6
/proc/393/stat : 393 (python) R 390 393 30488 0 -1 4202496 19637 933 1 0 131 10 0 0 18 0 1 0 869488602 334295040 17498 33554432000 4194304 5683644 140736702196192 18446744073709551615 47094907926960 0 0 16781312 2 0 0 0 17 2 0 0 0
/proc/393/statm: 81615 17499 1781 364 0 16508 0
Current children cumulated CPU time (s) 1.41
Current children cumulated vsize (KiB) 326460

[startup+3.10045 s]
/proc/loadavg: 2.23 2.11 1.93 3/953 400
/proc/meminfo: memFree=10690640/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=340016 CPUtime=3.01 cores=0,2,4,6
/proc/393/stat : 393 (python) R 390 393 30488 0 -1 4202496 33632 933 1 0 289 12 0 0 22 0 1 0 869488602 348176384 21350 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272356229 0 0 16781312 33554434 0 0 0 17 2 0 0 0
/proc/393/statm: 85004 21350 1815 364 0 19897 0
Current children cumulated CPU time (s) 3.01
Current children cumulated vsize (KiB) 340016

[startup+6.30099 s]
/proc/loadavg: 2.29 2.13 1.93 3/956 403
/proc/meminfo: memFree=10651644/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=354260 CPUtime=4.62 cores=0,2,4,6
/proc/393/stat : 393 (python) S 390 393 30488 0 -1 4202496 34913 933 1 0 450 12 0 0 25 0 2 0 869488602 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0
/proc/393/statm: 88565 22403 1870 364 0 23452 0
[pid=401] ppid=393 vsize=344016 CPUtime=0 cores=0,2,4,6
/proc/401/stat : 401 (python) S 393 393 30488 0 -1 4202560 749 0 0 0 0 0 0 0 15 0 1 0 869489074 352272384 20880 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272709135 0 0 16781312 33554946 12627634828733284611 0 0 17 2 0 0 0
/proc/401/statm: 86004 20880 351 364 0 20891 0
[pid=403] ppid=401 vsize=87448 CPUtime=1.57 cores=0,2,4,6
/proc/403/stat : 403 (wbo1.4a) R 401 403 403 0 -1 4202496 30263 0 0 0 154 3 0 0 25 0 1 0 869489074 89546752 20057 33554432000 134512640 135751879 4291161840 18446744073709551615 134616514 0 0 16781312 3 0 0 0 17 4 0 0 0
/proc/403/statm: 21862 20057 145 303 0 21557 0
[pid=393/tid=402] ppid=390 vsize=354260 CPUtime=0 cores=0,2,4,6
/proc/393/task/402/stat : 402 (python) S 390 393 30488 0 -1 4202560 25 933 0 0 0 0 0 0 25 0 2 0 869489074 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 6.19
Current children cumulated vsize (KiB) 785724

[startup+12.7009 s]
/proc/loadavg: 2.34 2.14 1.94 3/956 403
/proc/meminfo: memFree=10592908/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=354260 CPUtime=4.62 cores=0,2,4,6
/proc/393/stat : 393 (python) S 390 393 30488 0 -1 4202496 34913 933 1 0 450 12 0 0 25 0 2 0 869488602 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0
/proc/393/statm: 88565 22403 1870 364 0 23452 0
[pid=401] ppid=393 vsize=344016 CPUtime=0.02 cores=0,2,4,6
/proc/401/stat : 401 (python) S 393 393 30488 0 -1 4202560 899 0 0 0 1 1 0 0 15 0 1 0 869489074 352272384 20904 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272709135 0 0 16781312 33554946 349382159166832899 0 0 17 6 0 0 0
/proc/401/statm: 86004 20904 375 364 0 20891 0
[pid=403] ppid=401 vsize=97076 CPUtime=7.98 cores=0,2,4,6
/proc/403/stat : 403 (wbo1.4a) R 401 403 403 0 -1 4202496 38178 0 0 0 794 4 0 0 25 0 1 0 869489074 99405824 22722 33554432000 134512640 135751879 4291161840 18446744073709551615 134636289 0 0 16781312 3 0 0 0 17 4 0 0 0
/proc/403/statm: 24269 22722 145 303 0 23964 0
[pid=393/tid=402] ppid=390 vsize=354260 CPUtime=0 cores=0,2,4,6
/proc/393/task/402/stat : 402 (python) S 390 393 30488 0 -1 4202560 25 933 0 0 0 0 0 0 25 0 2 0 869489074 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 12.62
Current children cumulated vsize (KiB) 795352

[startup+25.5006 s]
/proc/loadavg: 2.27 2.14 1.94 3/960 409
/proc/meminfo: memFree=10561132/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=354260 CPUtime=4.62 cores=0,2,4,6
/proc/393/stat : 393 (python) S 390 393 30488 0 -1 4202496 34913 933 1 0 450 12 0 0 25 0 2 0 869488602 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0
/proc/393/statm: 88565 22403 1870 364 0 23452 0
[pid=401] ppid=393 vsize=344016 CPUtime=0.09 cores=0,2,4,6
/proc/401/stat : 401 (python) S 393 393 30488 0 -1 4202560 905 0 0 0 5 4 0 0 15 0 1 0 869489074 352272384 20904 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272709135 0 0 16781312 33554946 72556772316949205 0 0 17 6 0 0 0
/proc/401/statm: 86004 20904 375 364 0 20891 0
[pid=403] ppid=401 vsize=97076 CPUtime=20.78 cores=0,2,4,6
/proc/403/stat : 403 (wbo1.4a) R 401 403 403 0 -1 4202496 38178 0 0 0 2074 4 0 0 25 0 1 0 869489074 99405824 22722 33554432000 134512640 135751879 4291161840 18446744073709551615 134636949 0 0 16781312 3 0 0 0 17 4 0 0 0
/proc/403/statm: 24269 22722 145 303 0 23964 0
[pid=393/tid=402] ppid=390 vsize=354260 CPUtime=0 cores=0,2,4,6
/proc/393/task/402/stat : 402 (python) S 390 393 30488 0 -1 4202560 25 933 0 0 0 0 0 0 25 0 2 0 869489074 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 25.49
Current children cumulated vsize (KiB) 795352

[startup+51.1009 s]
/proc/loadavg: 2.17 2.12 1.94 3/960 409
/proc/meminfo: memFree=10548224/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=354260 CPUtime=4.62 cores=0,2,4,6
/proc/393/stat : 393 (python) S 390 393 30488 0 -1 4202496 34913 933 1 0 450 12 0 0 25 0 2 0 869488602 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0
/proc/393/statm: 88565 22403 1870 364 0 23452 0
[pid=401] ppid=393 vsize=344016 CPUtime=0.23 cores=0,2,4,6
/proc/401/stat : 401 (python) S 393 393 30488 0 -1 4202560 921 0 0 0 12 11 0 0 15 0 1 0 869489074 352272384 20904 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272709135 0 0 16781312 33554946 72262103200705237 0 0 17 2 0 0 0
/proc/401/statm: 86004 20904 375 364 0 20891 0
[pid=403] ppid=401 vsize=102680 CPUtime=46.36 cores=0,2,4,6
/proc/403/stat : 403 (wbo1.4a) R 401 403 403 0 -1 4202496 43567 0 0 0 4631 5 0 0 25 0 1 0 869489074 105144320 24012 33554432000 134512640 135751879 4291161840 18446744073709551615 134636949 0 0 16781312 3 0 0 0 17 2 0 0 0
/proc/403/statm: 25670 24012 146 303 0 25365 0
[pid=393/tid=402] ppid=390 vsize=354260 CPUtime=0 cores=0,2,4,6
/proc/393/task/402/stat : 402 (python) S 390 393 30488 0 -1 4202560 25 933 0 0 0 0 0 0 25 0 2 0 869489074 362762240 22403 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 51.21
Current children cumulated vsize (KiB) 800956

[startup+102.326 s]
/proc/loadavg: 2.15 2.12 1.94 3/986 439
/proc/meminfo: memFree=9752980/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=374772 CPUtime=4.67 cores=0,2,4,6
/proc/393/stat : 393 (python) S 390 393 30488 0 -1 4202496 36657 933 1 0 455 12 0 0 19 0 4 0 869488602 383766528 22416 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0
/proc/393/statm: 93693 22416 1875 364 0 28574 0
[pid=401] ppid=393 vsize=354260 CPUtime=0.25 cores=0,2,4,6
/proc/401/stat : 401 (python) S 393 393 30488 0 -1 4202560 1091 0 0 0 14 11 0 0 17 0 2 0 869489074 362762240 20940 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0
/proc/401/statm: 88565 20940 406 364 0 23452 0
[pid=403] ppid=401 vsize=102680 CPUtime=48.02 cores=0,2,4,6
/proc/403/stat : 403 (wbo1.4a) T 401 403 403 0 -1 4202496 43567 0 0 0 4797 5 0 0 25 0 1 0 869489074 105144320 24012 33554432000 134512640 135751879 4291161840 18446744073709551615 134636949 0 0 16781312 3 18446744071562691360 0 0 17 6 0 0 0
/proc/403/statm: 25670 24012 146 303 0 25365 0
[pid=401/tid=411] ppid=393 vsize=354260 CPUtime=0 cores=0,2,4,6
/proc/401/task/411/stat : 411 (python) S 393 393 30488 0 -1 4202560 50 0 0 0 0 0 0 0 17 0 2 0 869493880 362762240 20940 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554946 18446604451953544192 0 0 -1 2 0 0 0
[pid=412] ppid=393 vsize=354272 CPUtime=0.24 cores=0,2,4,6
/proc/412/stat : 412 (python) S 393 393 30488 0 -1 4202560 1199 0 0 0 13 11 0 0 19 0 2 0 869493882 362774528 20936 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0
/proc/412/statm: 88568 20936 402 364 0 23452 0
[pid=415] ppid=412 vsize=8539444 CPUtime=46.14 cores=0,2,4,6
/proc/415/stat : 415 (java) T 412 415 415 0 -1 4202496 181972 0 1 0 4594 20 0 0 18 0 19 0 869493883 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284491749 0 0 0 16800975 18446744073709551615 0 0 17 2 0 0 0
/proc/415/statm: 2134861 176698 2242 9 0 2125602 0
[pid=415/tid=416] ppid=412 vsize=8539444 CPUtime=43.84 cores=0,2,4,6
/proc/415/task/416/stat : 416 (java) T 412 415 415 0 -1 4202560 115689 0 1 0 4373 11 0 0 25 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 46912506000703 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=417] ppid=412 vsize=8539444 CPUtime=0.17 cores=0,2,4,6
/proc/415/task/417/stat : 417 (java) T 412 415 415 0 -1 4202560 13761 0 0 0 16 1 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=418] ppid=412 vsize=8539444 CPUtime=0.11 cores=0,2,4,6
/proc/415/task/418/stat : 418 (java) T 412 415 415 0 -1 4202560 8149 0 0 0 10 1 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=419] ppid=412 vsize=8539444 CPUtime=0.09 cores=0,2,4,6
/proc/415/task/419/stat : 419 (java) T 412 415 415 0 -1 4202560 6870 0 0 0 9 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=420] ppid=412 vsize=8539444 CPUtime=0.04 cores=0,2,4,6
/proc/415/task/420/stat : 420 (java) T 412 415 415 0 -1 4202560 2958 0 0 0 4 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=421] ppid=412 vsize=8539444 CPUtime=0.03 cores=0,2,4,6
/proc/415/task/421/stat : 421 (java) T 412 415 415 0 -1 4202560 2646 0 0 0 3 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=422] ppid=412 vsize=8539444 CPUtime=0.02 cores=0,2,4,6
/proc/415/task/422/stat : 422 (java) T 412 415 415 0 -1 4202560 2437 0 0 0 2 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=423] ppid=412 vsize=8539444 CPUtime=0.1 cores=0,2,4,6
/proc/415/task/423/stat : 423 (java) T 412 415 415 0 -1 4202560 9551 0 0 0 10 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=424] ppid=412 vsize=8539444 CPUtime=0.1 cores=0,2,4,6
/proc/415/task/424/stat : 424 (java) T 412 415 415 0 -1 4202560 7186 0 0 0 10 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=425] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/425/stat : 425 (java) T 412 415 415 0 -1 4202560 500 0 0 0 0 0 0 0 15 0 19 0 869493885 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284505600 0 0 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=426] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/426/stat : 426 (java) T 412 415 415 0 -1 4202560 17 0 0 0 0 0 0 0 18 0 19 0 869493886 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=427] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/427/stat : 427 (java) T 412 415 415 0 -1 4202560 9 0 0 0 0 0 0 0 18 0 19 0 869493886 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=428] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/428/stat : 428 (java) T 412 415 415 0 -1 4202560 1 0 0 0 0 0 0 0 18 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284512689 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=429] ppid=412 vsize=8539444 CPUtime=0.66 cores=0,2,4,6
/proc/415/task/429/stat : 429 (java) T 412 415 415 0 -1 4202560 2284 0 0 0 66 0 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=430] ppid=412 vsize=8539444 CPUtime=0.86 cores=0,2,4,6
/proc/415/task/430/stat : 430 (java) T 412 415 415 0 -1 4202560 8546 0 0 0 85 1 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0

################
# More data... #
################

[pid=415/tid=427] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/427/stat : 427 (java) T 412 415 415 0 -1 4202560 9 0 0 0 0 0 0 0 18 0 19 0 869493886 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=428] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/428/stat : 428 (java) T 412 415 415 0 -1 4202560 1 0 0 0 0 0 0 0 18 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284512689 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=429] ppid=412 vsize=8539444 CPUtime=0.66 cores=0,2,4,6
/proc/415/task/429/stat : 429 (java) T 412 415 415 0 -1 4202560 2284 0 0 0 66 0 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=430] ppid=412 vsize=8539444 CPUtime=0.86 cores=0,2,4,6
/proc/415/task/430/stat : 430 (java) T 412 415 415 0 -1 4202560 8546 0 0 0 85 1 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=431] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/431/stat : 431 (java) T 412 415 415 0 -1 4202560 1 0 0 0 0 0 0 0 18 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=432] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/432/stat : 432 (java) T 412 415 415 0 -1 4202560 3 0 0 0 0 0 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284505600 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=433] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/433/stat : 433 (java) T 412 415 415 0 -1 4202560 7 0 0 0 0 0 0 0 18 0 19 0 869494082 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284505600 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=412/tid=436] ppid=393 vsize=354272 CPUtime=0 cores=0,2,4,6
/proc/412/task/436/stat : 436 (python) S 393 393 30488 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 869498304 362774528 20936 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554946 18446604451953541504 0 0 -1 2 0 0 0
[pid=437] ppid=393 vsize=364528 CPUtime=0.67 cores=0,2,4,6
/proc/437/stat : 437 (python) S 393 393 30488 0 -1 4202560 1072 0 0 0 33 34 0 0 15 0 1 0 869498306 373276672 20915 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272709190 0 0 16781312 33554946 72556772316949205 0 0 17 6 0 0 0
/proc/437/statm: 91132 20915 378 364 0 26013 0
[pid=439] ppid=437 vsize=87404 CPUtime=125.25 cores=0,2,4,6
/proc/439/stat : 439 (bsolo_pb10) R 437 439 439 0 -1 4202496 35921 0 0 0 12511 14 0 0 25 0 1 0 869498307 89501696 18491 33554432000 134512640 137138936 4294375808 18446744073709551615 136570369 0 0 16781312 16384 0 0 0 17 0 0 0 0
/proc/439/statm: 21851 18491 184 642 0 21207 0
[pid=393/tid=402] ppid=390 vsize=374772 CPUtime=0 cores=0,2,4,6
/proc/393/task/402/stat : 402 (python) S 390 393 30488 0 -1 4202560 25 933 0 0 0 0 0 0 25 0 4 0 869489074 383766528 22416 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
[pid=393/tid=413] ppid=390 vsize=374772 CPUtime=0 cores=0,2,4,6
/proc/393/task/413/stat : 413 (python) S 390 393 30488 0 -1 4202560 32 933 0 0 0 0 0 0 20 0 4 0 869493882 383766528 22416 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 6 0 0 0
[pid=393/tid=438] ppid=390 vsize=374772 CPUtime=0 cores=0,2,4,6
/proc/393/task/438/stat : 438 (python) S 390 393 30488 0 -1 4202560 32 933 0 0 0 0 0 0 20 0 4 0 869498307 383766528 22416 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 225.24
Current children cumulated vsize (KiB) 10177360



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+238.601 s]
/proc/loadavg: 2.02 2.07 1.94 5/1025 561
/proc/meminfo: memFree=8147656/32950928 swapFree=43925852/67111528
[pid=393] ppid=390 vsize=385028 CPUtime=4.7 cores=0,2,4,6
/proc/393/stat : 393 (python) S 390 393 30488 0 -1 4202496 37544 933 1 0 457 13 0 0 20 0 5 0 869488602 394268672 22422 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0
/proc/393/statm: 96257 22422 1877 364 0 31135 0
[pid=401] ppid=393 vsize=354260 CPUtime=0.25 cores=0,2,4,6
/proc/401/stat : 401 (python) S 393 393 30488 0 -1 4202560 1091 0 0 0 14 11 0 0 17 0 2 0 869489074 362762240 20940 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0
/proc/401/statm: 88565 20940 406 364 0 23452 0
[pid=403] ppid=401 vsize=102680 CPUtime=48.02 cores=0,2,4,6
/proc/403/stat : 403 (wbo1.4a) T 401 403 403 0 -1 4202496 43567 0 0 0 4797 5 0 0 25 0 1 0 869489074 105144320 24012 33554432000 134512640 135751879 4291161840 18446744073709551615 134636949 0 0 16781312 3 18446744071562691360 0 0 17 6 0 0 0
/proc/403/statm: 25670 24012 146 303 0 25365 0
[pid=401/tid=411] ppid=393 vsize=354260 CPUtime=0 cores=0,2,4,6
/proc/401/task/411/stat : 411 (python) S 393 393 30488 0 -1 4202560 50 0 0 0 0 0 0 0 17 0 2 0 869493880 362762240 20940 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554946 18446604451953544192 0 0 -1 2 0 0 0
[pid=412] ppid=393 vsize=354272 CPUtime=0.24 cores=0,2,4,6
/proc/412/stat : 412 (python) S 393 393 30488 0 -1 4202560 1199 0 0 0 13 11 0 0 19 0 2 0 869493882 362774528 20936 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0
/proc/412/statm: 88568 20936 402 364 0 23452 0
[pid=415] ppid=412 vsize=8539444 CPUtime=46.14 cores=0,2,4,6
/proc/415/stat : 415 (java) T 412 415 415 0 -1 4202496 181972 0 1 0 4594 20 0 0 18 0 19 0 869493883 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284491749 0 0 0 16800975 18446744073709551615 0 0 17 2 0 0 0
/proc/415/statm: 2134861 176698 2242 9 0 2125602 0
[pid=415/tid=416] ppid=412 vsize=8539444 CPUtime=43.84 cores=0,2,4,6
/proc/415/task/416/stat : 416 (java) T 412 415 415 0 -1 4202560 115689 0 1 0 4373 11 0 0 25 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 46912506000703 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=417] ppid=412 vsize=8539444 CPUtime=0.17 cores=0,2,4,6
/proc/415/task/417/stat : 417 (java) T 412 415 415 0 -1 4202560 13761 0 0 0 16 1 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=418] ppid=412 vsize=8539444 CPUtime=0.11 cores=0,2,4,6
/proc/415/task/418/stat : 418 (java) T 412 415 415 0 -1 4202560 8149 0 0 0 10 1 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=419] ppid=412 vsize=8539444 CPUtime=0.09 cores=0,2,4,6
/proc/415/task/419/stat : 419 (java) T 412 415 415 0 -1 4202560 6870 0 0 0 9 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=420] ppid=412 vsize=8539444 CPUtime=0.04 cores=0,2,4,6
/proc/415/task/420/stat : 420 (java) T 412 415 415 0 -1 4202560 2958 0 0 0 4 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=421] ppid=412 vsize=8539444 CPUtime=0.03 cores=0,2,4,6
/proc/415/task/421/stat : 421 (java) T 412 415 415 0 -1 4202560 2646 0 0 0 3 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=422] ppid=412 vsize=8539444 CPUtime=0.02 cores=0,2,4,6
/proc/415/task/422/stat : 422 (java) T 412 415 415 0 -1 4202560 2437 0 0 0 2 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=423] ppid=412 vsize=8539444 CPUtime=0.1 cores=0,2,4,6
/proc/415/task/423/stat : 423 (java) T 412 415 415 0 -1 4202560 9551 0 0 0 10 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=424] ppid=412 vsize=8539444 CPUtime=0.1 cores=0,2,4,6
/proc/415/task/424/stat : 424 (java) T 412 415 415 0 -1 4202560 7186 0 0 0 10 0 0 0 15 0 19 0 869493884 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=425] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/425/stat : 425 (java) T 412 415 415 0 -1 4202560 500 0 0 0 0 0 0 0 15 0 19 0 869493885 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284505600 0 0 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=426] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/426/stat : 426 (java) T 412 415 415 0 -1 4202560 17 0 0 0 0 0 0 0 18 0 19 0 869493886 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=427] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/427/stat : 427 (java) T 412 415 415 0 -1 4202560 9 0 0 0 0 0 0 0 18 0 19 0 869493886 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 6 0 0 0
[pid=415/tid=428] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/428/stat : 428 (java) T 412 415 415 0 -1 4202560 1 0 0 0 0 0 0 0 18 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284512689 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=429] ppid=412 vsize=8539444 CPUtime=0.66 cores=0,2,4,6
/proc/415/task/429/stat : 429 (java) T 412 415 415 0 -1 4202560 2284 0 0 0 66 0 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 0 0 0 0
[pid=415/tid=430] ppid=412 vsize=8539444 CPUtime=0.86 cores=0,2,4,6
/proc/415/task/430/stat : 430 (java) T 412 415 415 0 -1 4202560 8546 0 0 0 85 1 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=431] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/431/stat : 431 (java) T 412 415 415 0 -1 4202560 1 0 0 0 0 0 0 0 18 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284504985 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=432] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/432/stat : 432 (java) T 412 415 415 0 -1 4202560 3 0 0 0 0 0 0 0 15 0 19 0 869493888 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284505600 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=415/tid=433] ppid=412 vsize=8539444 CPUtime=0 cores=0,2,4,6
/proc/415/task/433/stat : 433 (java) T 412 415 415 0 -1 4202560 7 0 0 0 0 0 0 0 18 0 19 0 869494082 8744390656 176698 33554432000 1073741824 1073778376 140736536594256 18446744073709551615 268284505600 0 4 0 16800975 18446744071562691360 0 0 -1 2 0 0 0
[pid=412/tid=436] ppid=393 vsize=354272 CPUtime=0 cores=0,2,4,6
/proc/412/task/436/stat : 436 (python) S 393 393 30488 0 -1 4202560 51 0 0 0 0 0 0 0 19 0 2 0 869498304 362774528 20936 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554946 18446604451953541504 0 0 -1 2 0 0 0
[pid=437] ppid=393 vsize=364528 CPUtime=0.78 cores=0,2,4,6
/proc/437/stat : 437 (python) S 393 393 30488 0 -1 4202560 1245 0 0 0 39 39 0 0 16 0 2 0 869498306 373276672 20940 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284515787 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/437/statm: 91132 20940 402 364 0 26013 0
[pid=439] ppid=437 vsize=87404 CPUtime=140.81 cores=0,2,4,6
/proc/439/stat : 439 (bsolo_pb10) T 437 439 439 0 -1 4202496 35921 0 0 0 14066 15 0 0 25 0 1 0 869498307 89501696 18491 33554432000 134512640 137138936 4294375808 18446744073709551615 134542516 0 0 16781312 16384 18446744071562691360 0 0 17 0 0 0 0
/proc/439/statm: 21851 18491 184 642 0 21207 0
[pid=437/tid=538] ppid=393 vsize=364528 CPUtime=0 cores=0,2,4,6
/proc/437/task/538/stat : 538 (python) S 393 393 30488 0 -1 4202560 53 0 0 0 0 0 0 0 16 0 2 0 869512389 373276672 20940 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554946 18446604447121024128 0 0 -1 6 0 0 0
[pid=539] ppid=393 vsize=374784 CPUtime=0 cores=0,2,4,6
/proc/539/stat : 539 (python) S 393 393 30488 0 -1 4202560 805 0 0 0 0 0 0 0 19 0 1 0 869512391 383778816 20889 33554432000 4194304 5683644 140736702196192 18446744073709551615 268272709190 0 0 16781312 33554946 1270850831129739522 0 0 17 0 0 0 0
/proc/539/statm: 93696 20889 348 364 0 28574 0
[pid=541] ppid=539 vsize=8534364 CPUtime=1.06 cores=0,2,4,6
/proc/541/stat : 541 (java) S 539 541 541 0 -1 4202496 130842 0 1 0 91 15 0 0 20 0 18 0 869512391 8739188736 129748 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284491749 0 0 0 16800975 18446744073709551615 0 0 17 6 0 0 0
/proc/541/statm: 2133591 129762 2212 9 0 2124332 0
[pid=541/tid=542] ppid=539 vsize=8534364 CPUtime=0.69 cores=0,2,4,6
/proc/541/task/542/stat : 542 (java) S 539 541 541 0 -1 4202560 125313 0 1 0 55 14 0 0 17 0 18 0 869512392 8739188736 139204 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 0 0 0 0
[pid=541/tid=544] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/544/stat : 544 (java) R 539 541 541 0 -1 4202560 520 0 0 0 0 0 0 0 15 0 18 0 869512392 8739188736 139209 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 0 0 0 -1 6 0 0 0
[pid=541/tid=545] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/545/stat : 545 (java) R 539 541 541 0 -1 4202560 1072 0 0 0 0 0 0 0 16 0 18 0 869512392 8739188736 139213 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 47714456598201 0 4 0 16800975 0 0 0 -1 0 0 0 0
[pid=541/tid=546] ppid=539 vsize=8534364 CPUtime=0.02 cores=0,2,4,6
/proc/541/task/546/stat : 546 (java) S 539 541 541 0 -1 4202560 3499 0 0 0 2 0 0 0 16 0 18 0 869512392 8739188736 139219 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 0 0 0 0
[pid=541/tid=548] ppid=539 vsize=8534364 CPUtime=0.01 cores=0,2,4,6
/proc/541/task/548/stat : 548 (java) S 539 541 541 0 -1 4202560 1624 0 0 0 1 0 0 0 15 0 18 0 869512392 8739188736 139223 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 6 0 0 0
[pid=541/tid=549] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/549/stat : 549 (java) R 539 541 541 0 -1 4202560 640 0 0 0 0 0 0 0 18 0 18 0 869512392 8739188736 139227 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284514997 0 4 0 16800975 0 0 0 -1 0 0 0 0
[pid=541/tid=550] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/550/stat : 550 (java) S 539 541 541 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 869512392 8739188736 139233 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 6 0 0 0
[pid=541/tid=551] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/551/stat : 551 (java) S 539 541 541 0 -1 4202560 711 0 0 0 0 0 0 0 20 0 18 0 869512392 8739188736 139237 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 0 0 0 0
[pid=541/tid=552] ppid=539 vsize=8534364 CPUtime=0.01 cores=0,2,4,6
/proc/541/task/552/stat : 552 (java) R 539 541 541 0 -1 4202560 3090 0 0 0 1 0 0 0 16 0 18 0 869512392 8739188736 139241 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 47714456598201 0 4 0 16800975 0 0 0 -1 2 0 0 0
[pid=541/tid=553] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/553/stat : 553 (java) S 539 541 541 0 -1 4202560 197 0 0 0 0 0 0 0 19 0 18 0 869512393 8739188736 139246 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 0 0 16800975 18446604448351085504 0 0 -1 2 0 0 0
[pid=541/tid=554] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/554/stat : 554 (java) S 539 541 541 0 -1 4202560 17 0 0 0 0 0 0 0 23 0 18 0 869512394 8739188736 139250 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 2 0 0 0
[pid=541/tid=555] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/555/stat : 555 (java) S 539 541 541 0 -1 4202560 10 0 0 0 0 0 0 0 23 0 18 0 869512394 8739188736 139254 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 0 0 0 0
[pid=541/tid=556] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/556/stat : 556 (java) S 539 541 541 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 18 0 869512396 8739188736 139257 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284512689 0 4 0 16800975 18446604448351085504 0 0 -1 6 0 0 0
[pid=541/tid=558] ppid=539 vsize=8534364 CPUtime=0.14 cores=0,2,4,6
/proc/541/task/558/stat : 558 (java) S 539 541 541 0 -1 4202560 925 0 0 0 14 0 0 0 16 0 18 0 869512396 8739188736 139261 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 2 0 0 0
[pid=541/tid=559] ppid=539 vsize=8534364 CPUtime=0.13 cores=0,2,4,6
/proc/541/task/559/stat : 559 (java) S 539 541 541 0 -1 4202560 1345 0 0 0 13 0 0 0 18 0 18 0 869512396 8739188736 139266 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 0 0 0 0
[pid=541/tid=560] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/560/stat : 560 (java) S 539 541 541 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 18 0 869512396 8739188736 139270 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284504985 0 4 0 16800975 18446604448351085504 0 0 -1 6 0 0 0
[pid=541/tid=561] ppid=539 vsize=8534364 CPUtime=0 cores=0,2,4,6
/proc/541/task/561/stat : 561 (java) S 539 541 541 0 -1 4202560 2 0 0 0 0 0 0 0 18 0 18 0 869512396 8739188736 139274 33554432000 1073741824 1073778376 140735614878288 18446744073709551615 268284505600 0 4 0 16800975 18446604448351085504 0 0 -1 0 0 0 0
[pid=393/tid=402] ppid=390 vsize=385028 CPUtime=0 cores=0,2,4,6
/proc/393/task/402/stat : 402 (python) S 390 393 30488 0 -1 4202560 25 933 0 0 0 0 0 0 25 0 5 0 869489074 394268672 22422 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
[pid=393/tid=413] ppid=390 vsize=385028 CPUtime=0 cores=0,2,4,6
/proc/393/task/413/stat : 413 (python) S 390 393 30488 0 -1 4202560 32 933 0 0 0 0 0 0 20 0 5 0 869493882 394268672 22422 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 6 0 0 0
[pid=393/tid=438] ppid=390 vsize=385028 CPUtime=0 cores=0,2,4,6
/proc/393/task/438/stat : 438 (python) S 390 393 30488 0 -1 4202560 32 933 0 0 0 0 0 0 20 0 5 0 869498307 394268672 22422 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 2 0 0 0
[pid=393/tid=540] ppid=390 vsize=385028 CPUtime=0 cores=0,2,4,6
/proc/393/task/540/stat : 540 (python) S 390 393 30488 0 -1 4202560 33 933 0 0 0 0 0 0 20 0 5 0 869512391 394268672 22422 33554432000 4194304 5683644 140736702196192 18446744073709551615 268284512689 0 0 16781312 33554434 18446604448564934464 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 242
Current children cumulated vsize (KiB) 19096764

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 393 and gives
#  childrusage.ru_utime.tv_sec=4
#  childrusage.ru_utime.tv_usec=580303
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=135979
# CPU time returned by wait4() is 4.71628
# while last known CPU time is 242
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 238.702
CPU time (s): 242
CPU user time (s): 240.71
CPU system time (s): 1.29
CPU usage (%): 101.381
Max. virtual memory (cumulated for all children) (KiB): 19096764

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4.5803
system time used= 0.135979
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 38478
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1008
involuntary context switches= 21

runsolver used 2.13468 second user time and 3.60245 second system time

The end

Launcher Data 

Begin job on node122 at 2011-06-09 02:21:37
IDJOB=3482303
IDBENCH=89840
IDSOLVER=1970
FILE ID=node122/3482303-1307578897
RUNJOBID= node122-1307576358-30506
PBS_JOBID= 13547677
Free space on /tmp= 73228 MiB

SOLVER NAME= borg pb-opt-11.04.03
BENCH NAME= PB11/normalized-PB11/OPT-SMALLINT-LIN/heinz/normalized-neos-631710.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.opt.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3482303-1307578897/watcher-3482303-1307578897 -o /tmp/evaluation-result-3482303-1307578897/solver-3482303-1307578897 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.opt.pickle HOME/instance-3482303-1307578897.opb -seed 549437552 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ba0215d19182fa8cb27dee737825dd09
RANDOM SEED=549437552

node122.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      10772280 kB
Buffers:        167336 kB
Cached:        1006028 kB
SwapCached:      16764 kB
Active:       21642008 kB
Inactive:       166408 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      10772280 kB
SwapTotal:    67111528 kB
SwapFree:     43925852 kB
Dirty:          115744 kB
Writeback:           0 kB
AnonPages:    20632584 kB
Mapped:         123460 kB
Slab:           150844 kB
PageTables:     142112 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 210631448 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73076 MiB
End job on node122 at 2011-06-09 02:25:39