Trace number 3482898

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg pb-dec-11.04.03? (MO) 304.2 303.386

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-40.opb
MD5SUM96b10f202af149572cd99ac873608650
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables9342
Total number of constraints199352
Number of constraints which are clauses178530
Number of constraints which are cardinality constraints (but not clauses)720
Number of constraints which are nor clauses,nor cardinality constraints20102
Minimum length of a constraint2
Maximum length of a constraint231
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 232
Number of bits of the biggest sum of numbers8
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.28/0.49	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.28/0.49	c INFO: solving HOME/instance-3482898-1307578362.opb
1.76/1.90	c INFO: parsing took 1.38 s
4.16/4.35	c INFO: features took 2.45 CPU seconds
4.16/4.38	c INFO: running wbo1.4b-fixed@0 for 45 with 1795 remaining (b = 0.96)
51.29/51.35	c INFO: running pbct-0.1.2-linear@0 for 45 with 1749 remaining (b = 0.96)
100.36/100.24	c INFO: running wbo1.4a@0 for 45 with 1701 remaining (b = 0.94)
161.25/160.91	c INFO: wrote minion input file to HOME/tmpOvw4Z9.minion
161.25/160.91	c INFO: running minion-0.12@0 for 45 with 1655 remaining (b = 0.71)
209.41/208.90	c INFO: running bsolo_pb10-l2@0 for 45 with 1594 remaining (b = 0.66)
256.57/255.96	c INFO: running clasp-1.3.7@0 for 45 with 1547 remaining (b = 0.62)
303.93/303.19	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1500 remaining (b = 0.23)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 977)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3482898-1307578362/watcher-3482898-1307578362 -o /tmp/evaluation-result-3482898-1307578362/solver-3482898-1307578362 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3482898-1307578362.opb -seed 1283641863 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.42 2.12 1.80 2/482 16246
/proc/meminfo: memFree=21294228/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=16244 CPUtime=0 cores=0,2,4,6
/proc/16246/stat : 16246 (python) R 16244 16246 15040 0 -1 4202496 1015 0 0 0 0 0 0 0 18 0 1 0 850251909 16633856 887 33554432000 4194304 5683644 140733497739568 18446744073709551615 4540219 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/16246/statm: 4061 888 372 364 0 548 0

[startup+0.074619 s]
/proc/loadavg: 2.42 2.12 1.80 2/482 16246
/proc/meminfo: memFree=21294228/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=84436 CPUtime=0.05 cores=0,2,4,6
/proc/16246/stat : 16246 (python) R 16244 16246 15040 0 -1 4202496 2386 0 0 0 4 1 0 0 17 0 1 0 850251909 86462464 2114 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338044613 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/16246/statm: 21109 2114 706 364 0 1326 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 84436

[startup+0.100613 s]
/proc/loadavg: 2.42 2.12 1.80 2/482 16246
/proc/meminfo: memFree=21294228/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=96124 CPUtime=0.06 cores=0,2,4,6
/proc/16246/stat : 16246 (python) R 16244 16246 15040 0 -1 4202496 3170 926 1 0 5 1 0 0 18 0 1 0 850251909 98430976 2483 33554432000 4194304 5683644 140733497739568 18446744073709551615 5178064 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/16246/statm: 24031 2483 792 364 0 1626 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 96124

[startup+0.300548 s]
/proc/loadavg: 2.42 2.12 1.80 2/482 16246
/proc/meminfo: memFree=21294228/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=190664 CPUtime=0.2 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 6544 926 1 0 16 4 0 0 17 0 1 0 850251909 195239936 4837 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338381824 0 2147483391 16781312 2 18446744071699199166 0 0 17 4 0 0 0
/proc/16246/statm: 47666 4837 1400 364 0 3676 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 190664

[startup+0.70053 s]
/proc/loadavg: 2.42 2.12 1.80 2/482 16246
/proc/meminfo: memFree=21294228/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=297760 CPUtime=0.56 cores=0,2,4,6
/proc/16246/stat : 16246 (python) R 16244 16246 15040 0 -1 4202496 12838 926 1 0 50 6 0 0 17 0 1 0 850251909 304906240 10699 33554432000 4194304 5683644 140733497739568 18446744073709551615 4960460 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/16246/statm: 74440 10699 1781 364 0 9333 0
Current children cumulated CPU time (s) 0.56
Current children cumulated vsize (KiB) 297760

[startup+1.50038 s]
/proc/loadavg: 2.42 2.12 1.80 3/483 16251
/proc/meminfo: memFree=21240528/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=324808 CPUtime=1.37 cores=0,2,4,6
/proc/16246/stat : 16246 (python) R 16244 16246 15040 0 -1 4202496 19506 926 1 0 129 8 0 0 18 0 1 0 850251909 332603392 17367 33554432000 4194304 5683644 140733497739568 18446744073709551615 4445115 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/16246/statm: 81202 17367 1781 364 0 16095 0
Current children cumulated CPU time (s) 1.37
Current children cumulated vsize (KiB) 324808

[startup+3.10102 s]
/proc/loadavg: 2.42 2.12 1.80 3/483 16251
/proc/meminfo: memFree=21204816/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=338132 CPUtime=2.96 cores=0,2,4,6
/proc/16246/stat : 16246 (python) R 16244 16246 15040 0 -1 4202496 32716 926 1 0 287 9 0 0 21 0 1 0 850251909 346247168 20881 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338038768 0 0 16781312 33554434 0 0 0 17 4 0 0 0
/proc/16246/statm: 84533 20881 1816 364 0 19426 0
Current children cumulated CPU time (s) 2.96
Current children cumulated vsize (KiB) 338132

[startup+6.30141 s]
/proc/loadavg: 2.38 2.12 1.81 3/486 16254
/proc/meminfo: memFree=21165676/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=351516 CPUtime=4.24 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 33780 926 1 0 415 9 0 0 20 0 2 0 850251909 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 0
/proc/16246/statm: 87879 21720 1872 364 0 22766 0
[pid=16252] ppid=16246 vsize=341272 CPUtime=0 cores=0,2,4,6
/proc/16252/stat : 16252 (python) S 16246 16246 15040 0 -1 4202560 744 0 0 0 0 0 0 0 15 0 1 0 850252347 349462528 20195 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338403855 0 0 16781312 33554946 7737168667875442945 0 0 17 4 0 0 0
/proc/16252/statm: 85318 20195 351 364 0 20205 0
[pid=16254] ppid=16252 vsize=24572 CPUtime=1.91 cores=0,2,4,6
/proc/16254/stat : 16254 (wbo1.4b-fixed) R 16252 16254 16254 0 -1 4202496 8127 0 0 0 185 6 0 0 25 0 1 0 850252348 25161728 5580 33554432000 134512640 135752315 4288473824 18446744073709551615 135270849 0 0 16781312 3 0 0 0 17 4 0 0 0
/proc/16254/statm: 6143 5580 146 303 0 5838 0
[pid=16246/tid=16253] ppid=16244 vsize=351516 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16253/stat : 16253 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 2 0 850252348 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 6.15
Current children cumulated vsize (KiB) 717360

[startup+12.7011 s]
/proc/loadavg: 2.35 2.12 1.81 3/486 16255
/proc/meminfo: memFree=21149868/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=351516 CPUtime=4.24 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 33780 926 1 0 415 9 0 0 20 0 2 0 850251909 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 0
/proc/16246/statm: 87879 21720 1872 364 0 22766 0
[pid=16252] ppid=16246 vsize=341272 CPUtime=0.03 cores=0,2,4,6
/proc/16252/stat : 16252 (python) S 16246 16246 15040 0 -1 4202560 894 0 0 0 1 2 0 0 15 0 1 0 850252347 349462528 20219 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338403855 0 0 16781312 33554946 4294934786 0 0 17 4 0 0 0
/proc/16252/statm: 85318 20219 375 364 0 20205 0
[pid=16254] ppid=16252 vsize=30596 CPUtime=8.3 cores=0,2,4,6
/proc/16254/stat : 16254 (wbo1.4b-fixed) R 16252 16254 16254 0 -1 4202496 9634 0 0 0 808 22 0 0 25 0 1 0 850252348 31330304 7087 33554432000 134512640 135752315 4288473824 18446744073709551615 134641602 0 0 16781312 3 0 0 0 17 6 0 0 0
/proc/16254/statm: 7649 7087 146 303 0 7344 0
[pid=16246/tid=16253] ppid=16244 vsize=351516 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16253/stat : 16253 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 2 0 850252348 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 12.57
Current children cumulated vsize (KiB) 723384

[startup+25.5006 s]
/proc/loadavg: 2.42 2.14 1.82 3/486 16255
/proc/meminfo: memFree=21110716/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=351516 CPUtime=4.24 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 33780 926 1 0 415 9 0 0 20 0 2 0 850251909 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 0
/proc/16246/statm: 87879 21720 1872 364 0 22766 0
[pid=16252] ppid=16246 vsize=341272 CPUtime=0.07 cores=0,2,4,6
/proc/16252/stat : 16252 (python) S 16246 16246 15040 0 -1 4202560 992 0 0 0 3 4 0 0 15 0 1 0 850252347 349462528 20224 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338403855 0 0 16781312 33554946 4294934786 0 0 17 6 0 0 0
/proc/16252/statm: 85318 20224 380 364 0 20205 0
[pid=16254] ppid=16252 vsize=39348 CPUtime=21.09 cores=0,2,4,6
/proc/16254/stat : 16254 (wbo1.4b-fixed) R 16252 16254 16254 0 -1 4202496 11784 0 0 0 2076 33 0 0 25 0 1 0 850252348 40292352 9237 33554432000 134512640 135752315 4288473824 18446744073709551615 134611194 0 0 16781312 3 0 0 0 17 0 0 0 0
/proc/16254/statm: 9837 9237 146 303 0 9532 0
[pid=16246/tid=16253] ppid=16244 vsize=351516 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16253/stat : 16253 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 2 0 850252348 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 25.4
Current children cumulated vsize (KiB) 732136

[startup+51.1005 s]
/proc/loadavg: 2.28 2.13 1.82 3/487 16257
/proc/meminfo: memFree=21080076/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=351516 CPUtime=4.24 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 33780 926 1 0 415 9 0 0 20 0 2 0 850251909 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 0
/proc/16246/statm: 87879 21720 1872 364 0 22766 0
[pid=16252] ppid=16246 vsize=341272 CPUtime=0.16 cores=0,2,4,6
/proc/16252/stat : 16252 (python) S 16246 16246 15040 0 -1 4202560 1009 0 0 0 8 8 0 0 15 0 1 0 850252347 349462528 20224 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338403855 0 0 16781312 33554946 4294934786 0 0 17 4 0 0 0
/proc/16252/statm: 85318 20224 380 364 0 20205 0
[pid=16254] ppid=16252 vsize=49412 CPUtime=46.69 cores=0,2,4,6
/proc/16254/stat : 16254 (wbo1.4b-fixed) R 16252 16254 16254 0 -1 4202496 14216 0 0 0 4622 47 0 0 25 0 1 0 850252348 50597888 11669 33554432000 134512640 135752315 4288473824 18446744073709551615 135270849 0 0 16781312 3 0 0 0 17 0 0 0 0
/proc/16254/statm: 12353 11669 146 303 0 12048 0
[pid=16246/tid=16253] ppid=16244 vsize=351516 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16253/stat : 16253 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 2 0 850252348 359952384 21720 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 51.09
Current children cumulated vsize (KiB) 742200

[startup+102.301 s]
/proc/loadavg: 2.12 2.11 1.83 3/498 16269
/proc/meminfo: memFree=20225736/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=372028 CPUtime=4.3 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 35540 926 1 0 420 10 0 0 20 0 4 0 850251909 380956672 21733 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0
/proc/16246/statm: 93007 21733 1877 364 0 27888 0
[pid=16252] ppid=16246 vsize=351516 CPUtime=0.18 cores=0,2,4,6
/proc/16252/stat : 16252 (python) S 16246 16246 15040 0 -1 4202560 1142 0 0 0 9 9 0 0 17 0 2 0 850252347 359952384 20260 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16252/statm: 87879 20260 411 364 0 22766 0
[pid=16254] ppid=16252 vsize=49544 CPUtime=46.92 cores=0,2,4,6
/proc/16254/stat : 16254 (wbo1.4b-fixed) T 16252 16254 16254 0 -1 4202496 14267 0 0 0 4645 47 0 0 25 0 1 0 850252348 50733056 11720 33554432000 134512640 135752315 4288473824 18446744073709551615 134611253 0 0 16781312 3 18446744071562691360 0 0 17 0 0 0 0
/proc/16254/statm: 12386 11720 146 303 0 12081 0
[pid=16252/tid=16258] ppid=16246 vsize=351516 CPUtime=0 cores=0,2,4,6
/proc/16252/task/16258/stat : 16258 (python) S 16246 16246 15040 0 -1 4202560 41 0 0 0 0 0 0 0 17 0 2 0 850257042 359952384 20260 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604442885226624 0 0 -1 6 0 0 0
[pid=16259] ppid=16246 vsize=351528 CPUtime=0.18 cores=0,2,4,6
/proc/16259/stat : 16259 (python) S 16246 16246 15040 0 -1 4202560 1104 0 0 0 9 9 0 0 16 0 2 0 850257045 359964672 20245 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16259/statm: 87882 20245 396 364 0 22766 0
[pid=16261] ppid=16259 vsize=808368 CPUtime=48.85 cores=0,2,4,6
/proc/16261/stat : 16261 (pbct-0.1.2-linu) T 16259 16261 16261 0 -1 4202496 212184 0 0 0 4828 57 0 0 25 0 1 0 850257045 827768832 195449 33554432000 134512640 136422188 4287019552 18446744073709551615 134862323 0 0 16781312 16387 18446744071562691360 0 0 17 0 0 0 0
/proc/16261/statm: 202092 195449 215 467 0 201623 0
[pid=16259/tid=16266] ppid=16246 vsize=351528 CPUtime=0 cores=0,2,4,6
/proc/16259/task/16266/stat : 16266 (python) S 16246 16246 15040 0 -1 4202560 50 0 0 0 0 0 0 0 16 0 2 0 850261931 359964672 20245 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604444015478656 0 0 -1 6 0 0 0
[pid=16267] ppid=16246 vsize=361784 CPUtime=0 cores=0,2,4,6
/proc/16267/stat : 16267 (python) S 16246 16246 15040 0 -1 4202560 792 0 0 0 0 0 0 0 15 0 1 0 850261934 370466816 20200 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338403910 0 0 16781312 33554946 4294934787 0 0 17 4 0 0 0
/proc/16267/statm: 90446 20200 348 364 0 25327 0
[pid=16269] ppid=16267 vsize=20932 CPUtime=2.04 cores=0,2,4,6
/proc/16269/stat : 16269 (wbo1.4a) R 16267 16269 16269 0 -1 4202496 7205 0 0 0 196 8 0 0 25 0 1 0 850261934 21434368 4649 33554432000 134512640 135751879 4292422752 18446744073709551615 135270609 0 0 16781312 3 0 0 0 17 0 0 0 0
/proc/16269/statm: 5233 4649 143 303 0 4928 0
[pid=16246/tid=16253] ppid=16244 vsize=372028 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16253/stat : 16253 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 4 0 850252348 380956672 21733 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
[pid=16246/tid=16260] ppid=16244 vsize=372028 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16260/stat : 16260 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 4 0 850257045 380956672 21733 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 4 0 0 0
[pid=16246/tid=16268] ppid=16244 vsize=372028 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16268/stat : 16268 (python) S 16244 16246 15040 0 -1 4202560 28 926 0 0 0 0 0 0 20 0 4 0 850261934 380956672 21733 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 102.47
Current children cumulated vsize (KiB) 2315700

[startup+162.301 s]
/proc/loadavg: 2.04 2.08 1.83 3/503 16276
/proc/meminfo: memFree=20066664/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=382284 CPUtime=18.49 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 36495 926 1 0 1835 14 0 0 25 0 5 0 850251909 391458816 21739 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0
/proc/16246/statm: 95571 21739 1879 364 0 30449 0
[pid=16252] ppid=16246 vsize=351516 CPUtime=0.18 cores=0,2,4,6
/proc/16252/stat : 16252 (python) S 16246 16246 15040 0 -1 4202560 1142 0 0 0 9 9 0 0 17 0 2 0 850252347 359952384 20260 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16252/statm: 87879 20260 411 364 0 22766 0
[pid=16254] ppid=16252 vsize=49544 CPUtime=46.92 cores=0,2,4,6
/proc/16254/stat : 16254 (wbo1.4b-fixed) T 16252 16254 16254 0 -1 4202496 14267 0 0 0 4645 47 0 0 25 0 1 0 850252348 50733056 11720 33554432000 134512640 135752315 4288473824 18446744073709551615 134611253 0 0 16781312 3 18446744071562691360 0 0 17 0 0 0 0
/proc/16254/statm: 12386 11720 146 303 0 12081 0
[pid=16252/tid=16258] ppid=16246 vsize=351516 CPUtime=0 cores=0,2,4,6

################
# More data... #
################

[pid=16274] ppid=16246 vsize=372040 CPUtime=0.18 cores=0,2,4,6
/proc/16274/stat : 16274 (python) S 16246 16246 15040 0 -1 4202560 1149 0 0 0 10 8 0 0 16 0 2 0 850268000 380968960 20253 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16274/statm: 93010 20253 396 364 0 27888 0
[pid=16276] ppid=16274 vsize=3691996 CPUtime=47.95 cores=0,2,4,6
/proc/16276/stat : 16276 (minion) T 16274 16276 16276 0 -1 4202496 928605 0 0 0 4692 103 0 0 25 0 1 0 850268001 3780603904 915599 33554432000 134512640 165565024 4288850464 18446744073709551615 142764899 0 0 16781312 8396802 18446744071562691360 0 0 17 0 0 0 0
/proc/16276/statm: 922999 915599 359 7582 0 915414 0
[pid=16274/tid=16278] ppid=16246 vsize=372040 CPUtime=0 cores=0,2,4,6
/proc/16274/task/16278/stat : 16278 (python) S 16246 16246 15040 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 850272797 380968960 20253 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604447616456640 0 0 -1 6 0 0 0
[pid=16279] ppid=16246 vsize=382296 CPUtime=0.18 cores=0,2,4,6
/proc/16279/stat : 16279 (python) S 16246 16246 15040 0 -1 4202560 1122 0 0 0 9 9 0 0 16 0 2 0 850272800 391471104 20257 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16279/statm: 95574 20257 396 364 0 30449 0
[pid=16281] ppid=16279 vsize=90372 CPUtime=47.02 cores=0,2,4,6
/proc/16281/stat : 16281 (bsolo_pb10) T 16279 16281 16281 0 -1 4202496 25583 0 0 0 4659 43 0 0 25 0 1 0 850272800 92540928 20147 33554432000 134512640 137138936 4287045744 18446744073709551615 134542233 0 0 16781312 16384 18446744071562691360 0 0 17 2 0 0 0
/proc/16281/statm: 22593 20147 192 642 0 21949 0
[pid=16279/tid=16283] ppid=16246 vsize=382296 CPUtime=0 cores=0,2,4,6
/proc/16279/task/16283/stat : 16283 (python) S 16246 16246 15040 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 850277503 391471104 20257 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604453904987200 0 0 -1 6 0 0 0
[pid=16284] ppid=16246 vsize=392552 CPUtime=0.09 cores=0,2,4,6
/proc/16284/stat : 16284 (python) S 16246 16246 15040 0 -1 4202560 976 0 0 0 5 4 0 0 18 0 1 0 850277505 401973248 20236 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338403910 0 0 16781312 33554946 72556772316949191 0 0 17 6 0 0 0
/proc/16284/statm: 98138 20236 372 364 0 33010 0
[pid=16286] ppid=16284 vsize=36880 CPUtime=26.32 cores=0,2,4,6
/proc/16286/stat : 16286 (clasp-1.3.7-x86) R 16284 16286 16286 0 -1 4202496 9964 0 0 0 2631 1 0 0 25 0 1 0 850277506 37765120 8805 33554432000 134512640 136139019 4286721232 18446744073709551615 134831864 0 0 16781312 16386 0 0 0 17 0 0 0 0
/proc/16286/statm: 9220 8805 184 398 0 8820 0
[pid=16246/tid=16253] ppid=16244 vsize=402796 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16253/stat : 16253 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 7 0 850252348 412463104 21751 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
[pid=16246/tid=16260] ppid=16244 vsize=402796 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16260/stat : 16260 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 7 0 850257045 412463104 21751 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 4 0 0 0
[pid=16246/tid=16268] ppid=16244 vsize=402796 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16268/stat : 16268 (python) S 16244 16246 15040 0 -1 4202560 28 926 0 0 0 0 0 0 20 0 7 0 850261934 412463104 21751 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 0 0 0 0
[pid=16246/tid=16275] ppid=16244 vsize=402796 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16275/stat : 16275 (python) S 16244 16246 15040 0 -1 4202560 32 926 0 0 0 0 0 0 25 0 7 0 850268001 412463104 21751 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 0 0 0 0
[pid=16246/tid=16280] ppid=16244 vsize=402796 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16280/stat : 16280 (python) S 16244 16246 15040 0 -1 4202560 26 926 0 0 0 0 0 0 20 0 7 0 850272800 412463104 21751 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
[pid=16246/tid=16285] ppid=16244 vsize=402796 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16285/stat : 16285 (python) S 16244 16246 15040 0 -1 4202560 29 926 0 0 0 0 0 0 20 0 7 0 850277505 412463104 21751 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 283.06
Current children cumulated vsize (KiB) 7336972



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+303.301 s]
/proc/loadavg: 2.01 2.06 1.85 4/532 16308
/proc/meminfo: memFree=16254920/32951124 swapFree=50923800/67111528
[pid=16246] ppid=16244 vsize=413052 CPUtime=18.58 cores=0,2,4,6
/proc/16246/stat : 16246 (python) S 16244 16246 15040 0 -1 4202496 39204 926 1 0 1843 15 0 0 20 0 8 0 850251909 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0
/proc/16246/statm: 103263 21757 1885 364 0 38132 0
[pid=16252] ppid=16246 vsize=351516 CPUtime=0.18 cores=0,2,4,6
/proc/16252/stat : 16252 (python) S 16246 16246 15040 0 -1 4202560 1142 0 0 0 9 9 0 0 17 0 2 0 850252347 359952384 20260 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16252/statm: 87879 20260 411 364 0 22766 0
[pid=16254] ppid=16252 vsize=49544 CPUtime=46.92 cores=0,2,4,6
/proc/16254/stat : 16254 (wbo1.4b-fixed) T 16252 16254 16254 0 -1 4202496 14267 0 0 0 4645 47 0 0 25 0 1 0 850252348 50733056 11720 33554432000 134512640 135752315 4288473824 18446744073709551615 134611253 0 0 16781312 3 18446744071562691360 0 0 17 0 0 0 0
/proc/16254/statm: 12386 11720 146 303 0 12081 0
[pid=16252/tid=16258] ppid=16246 vsize=351516 CPUtime=0 cores=0,2,4,6
/proc/16252/task/16258/stat : 16258 (python) S 16246 16246 15040 0 -1 4202560 41 0 0 0 0 0 0 0 17 0 2 0 850257042 359952384 20260 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604442885226624 0 0 -1 6 0 0 0
[pid=16259] ppid=16246 vsize=351528 CPUtime=0.18 cores=0,2,4,6
/proc/16259/stat : 16259 (python) S 16246 16246 15040 0 -1 4202560 1104 0 0 0 9 9 0 0 16 0 2 0 850257045 359964672 20245 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16259/statm: 87882 20245 396 364 0 22766 0
[pid=16261] ppid=16259 vsize=808368 CPUtime=48.85 cores=0,2,4,6
/proc/16261/stat : 16261 (pbct-0.1.2-linu) T 16259 16261 16261 0 -1 4202496 212184 0 0 0 4828 57 0 0 25 0 1 0 850257045 827768832 195449 33554432000 134512640 136422188 4287019552 18446744073709551615 134862323 0 0 16781312 16387 18446744071562691360 0 0 17 0 0 0 0
/proc/16261/statm: 202092 195449 215 467 0 201623 0
[pid=16259/tid=16266] ppid=16246 vsize=351528 CPUtime=0 cores=0,2,4,6
/proc/16259/task/16266/stat : 16266 (python) S 16246 16246 15040 0 -1 4202560 50 0 0 0 0 0 0 0 16 0 2 0 850261931 359964672 20245 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604444015478656 0 0 -1 6 0 0 0
[pid=16267] ppid=16246 vsize=361784 CPUtime=0.18 cores=0,2,4,6
/proc/16267/stat : 16267 (python) S 16246 16246 15040 0 -1 4202560 1114 0 0 0 10 8 0 0 16 0 2 0 850261934 370466816 20249 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16267/statm: 90446 20249 396 364 0 25327 0
[pid=16269] ppid=16267 vsize=45300 CPUtime=46.46 cores=0,2,4,6
/proc/16269/stat : 16269 (wbo1.4a) T 16267 16269 16269 0 -1 4202496 13195 0 0 0 4593 53 0 0 25 0 1 0 850261934 46387200 10639 33554432000 134512640 135751879 4292422752 18446744073709551615 134616371 0 0 16781312 3 18446744071562691360 0 0 17 0 0 0 0
/proc/16269/statm: 11325 10639 143 303 0 11020 0
[pid=16267/tid=16272] ppid=16246 vsize=361784 CPUtime=0 cores=0,2,4,6
/proc/16267/task/16272/stat : 16272 (python) S 16246 16246 15040 0 -1 4202560 44 0 0 0 0 0 0 0 16 0 2 0 850266581 370466816 20249 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604440278879104 0 0 -1 6 0 0 0
[pid=16274] ppid=16246 vsize=372040 CPUtime=0.18 cores=0,2,4,6
/proc/16274/stat : 16274 (python) S 16246 16246 15040 0 -1 4202560 1149 0 0 0 10 8 0 0 16 0 2 0 850268000 380968960 20253 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16274/statm: 93010 20253 396 364 0 27888 0
[pid=16276] ppid=16274 vsize=3691996 CPUtime=47.95 cores=0,2,4,6
/proc/16276/stat : 16276 (minion) T 16274 16276 16276 0 -1 4202496 928605 0 0 0 4692 103 0 0 25 0 1 0 850268001 3780603904 915599 33554432000 134512640 165565024 4288850464 18446744073709551615 142764899 0 0 16781312 8396802 18446744071562691360 0 0 17 0 0 0 0
/proc/16276/statm: 922999 915599 359 7582 0 915414 0
[pid=16274/tid=16278] ppid=16246 vsize=372040 CPUtime=0 cores=0,2,4,6
/proc/16274/task/16278/stat : 16278 (python) S 16246 16246 15040 0 -1 4202560 51 0 0 0 0 0 0 0 16 0 2 0 850272797 380968960 20253 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604447616456640 0 0 -1 6 0 0 0
[pid=16279] ppid=16246 vsize=382296 CPUtime=0.18 cores=0,2,4,6
/proc/16279/stat : 16279 (python) S 16246 16246 15040 0 -1 4202560 1122 0 0 0 9 9 0 0 16 0 2 0 850272800 391471104 20257 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16279/statm: 95574 20257 396 364 0 30449 0
[pid=16281] ppid=16279 vsize=90372 CPUtime=47.02 cores=0,2,4,6
/proc/16281/stat : 16281 (bsolo_pb10) T 16279 16281 16281 0 -1 4202496 25583 0 0 0 4659 43 0 0 25 0 1 0 850272800 92540928 20147 33554432000 134512640 137138936 4287045744 18446744073709551615 134542233 0 0 16781312 16384 18446744071562691360 0 0 17 2 0 0 0
/proc/16281/statm: 22593 20147 192 642 0 21949 0
[pid=16279/tid=16283] ppid=16246 vsize=382296 CPUtime=0 cores=0,2,4,6
/proc/16279/task/16283/stat : 16283 (python) S 16246 16246 15040 0 -1 4202560 46 0 0 0 0 0 0 0 16 0 2 0 850277503 391471104 20257 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604453904987200 0 0 -1 6 0 0 0
[pid=16284] ppid=16246 vsize=392552 CPUtime=0.18 cores=0,2,4,6
/proc/16284/stat : 16284 (python) S 16246 16246 15040 0 -1 4202560 1155 0 0 0 9 9 0 0 19 0 2 0 850277505 401973248 20261 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350210507 0 0 16781312 33554946 18446744073709551615 0 0 17 6 0 0 0
/proc/16284/statm: 98138 20261 396 364 0 33010 0
[pid=16286] ppid=16284 vsize=41476 CPUtime=47.19 cores=0,2,4,6
/proc/16286/stat : 16286 (clasp-1.3.7-x86) T 16284 16286 16286 0 -1 4202496 11030 0 0 0 4718 1 0 0 25 0 1 0 850277506 42471424 9871 33554432000 134512640 136139019 4286721232 18446744073709551615 134883867 0 0 16781312 16386 18446744071562691360 0 0 17 0 0 0 0
/proc/16286/statm: 10369 9871 184 398 0 9969 0
[pid=16284/tid=16288] ppid=16246 vsize=392552 CPUtime=0 cores=0,2,4,6
/proc/16284/task/16288/stat : 16288 (python) S 16246 16246 15040 0 -1 4202560 48 0 0 0 0 0 0 0 19 0 2 0 850282225 401973248 20261 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554946 18446604453948810560 0 0 -1 0 0 0 0
[pid=16289] ppid=16246 vsize=402808 CPUtime=0 cores=0,2,4,6
/proc/16289/stat : 16289 (python) S 16246 16246 15040 0 -1 4202560 823 0 0 0 0 0 0 0 18 0 1 0 850282228 412475392 20216 33554432000 4194304 5683644 140733497739568 18446744073709551615 270338403910 0 0 16781312 33554946 3262062287338340352 0 0 17 0 0 0 0
/proc/16289/statm: 100702 20216 348 364 0 35571 0
[pid=16291] ppid=16289 vsize=8532520 CPUtime=0.15 cores=0,2,4,6
/proc/16291/stat : 16291 (java) S 16289 16291 16291 0 -1 4202496 5847 0 1 0 14 1 0 0 17 0 18 0 850282229 8737300480 5284 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350186469 0 0 0 16800975 18446744073709551615 0 0 17 6 0 0 0
/proc/16291/statm: 2133130 5284 2129 9 0 2123871 0
[pid=16291/tid=16292] ppid=16289 vsize=8532520 CPUtime=0.12 cores=0,2,4,6
/proc/16291/task/16292/stat : 16292 (java) R 16289 16291 16291 0 -1 4202560 3589 0 1 0 11 1 0 0 18 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 46912505219208 0 4 0 16800975 0 0 0 -1 2 0 0 0
[pid=16291/tid=16293] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16293/stat : 16293 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 19 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 6 0 0 0
[pid=16291/tid=16294] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16294/stat : 16294 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16295] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16295/stat : 16295 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16296] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16296/stat : 16296 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 21 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16297] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16297/stat : 16297 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 21 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16298] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16298/stat : 16298 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 21 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16299] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16299/stat : 16299 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 21 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16300] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16300/stat : 16300 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 21 0 18 0 850282230 8737300480 5329 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16301] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16301/stat : 16301 (java) S 16289 16291 16291 0 -1 4202560 2 0 0 0 0 0 0 0 23 0 18 0 850282231 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350200320 0 0 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16302] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16302/stat : 16302 (java) S 16289 16291 16291 0 -1 4202560 16 0 0 0 0 0 0 0 23 0 18 0 850282232 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 6 0 0 0
[pid=16291/tid=16303] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16303/stat : 16303 (java) S 16289 16291 16291 0 -1 4202560 7 0 0 0 0 0 0 0 24 0 18 0 850282232 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 6 0 0 0
[pid=16291/tid=16304] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16304/stat : 16304 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 18 0 850282234 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350207409 0 4 0 16800975 18446604442885227520 0 0 -1 6 0 0 0
[pid=16291/tid=16305] ppid=16289 vsize=8532520 CPUtime=0.02 cores=0,2,4,6
/proc/16291/task/16305/stat : 16305 (java) S 16289 16291 16291 0 -1 4202560 368 0 0 0 2 0 0 0 17 0 18 0 850282234 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16306] ppid=16289 vsize=8532520 CPUtime=0.01 cores=0,2,4,6
/proc/16291/task/16306/stat : 16306 (java) S 16289 16291 16291 0 -1 4202560 542 0 0 0 1 0 0 0 25 0 18 0 850282234 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16291/tid=16307] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16307/stat : 16307 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 18 0 850282234 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350199705 0 4 0 16800975 18446604442885227520 0 0 -1 6 0 0 0
[pid=16291/tid=16308] ppid=16289 vsize=8532520 CPUtime=0 cores=0,2,4,6
/proc/16291/task/16308/stat : 16308 (java) S 16289 16291 16291 0 -1 4202560 1 0 0 0 0 0 0 0 25 0 18 0 850282234 8737300480 5330 33554432000 1073741824 1073778376 140734564355872 18446744073709551615 270350200320 0 4 0 16800975 18446604442885227520 0 0 -1 0 0 0 0
[pid=16246/tid=16253] ppid=16244 vsize=413052 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16253/stat : 16253 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 8 0 850252348 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
[pid=16246/tid=16260] ppid=16244 vsize=413052 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16260/stat : 16260 (python) S 16244 16246 15040 0 -1 4202560 30 926 0 0 0 0 0 0 20 0 8 0 850257045 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 4 0 0 0
[pid=16246/tid=16268] ppid=16244 vsize=413052 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16268/stat : 16268 (python) S 16244 16246 15040 0 -1 4202560 28 926 0 0 0 0 0 0 20 0 8 0 850261934 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 0 0 0 0
[pid=16246/tid=16275] ppid=16244 vsize=413052 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16275/stat : 16275 (python) S 16244 16246 15040 0 -1 4202560 32 926 0 0 0 0 0 0 25 0 8 0 850268001 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 0 0 0 0
[pid=16246/tid=16280] ppid=16244 vsize=413052 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16280/stat : 16280 (python) S 16244 16246 15040 0 -1 4202560 26 926 0 0 0 0 0 0 20 0 8 0 850272800 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
[pid=16246/tid=16285] ppid=16244 vsize=413052 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16285/stat : 16285 (python) S 16244 16246 15040 0 -1 4202560 29 926 0 0 0 0 0 0 20 0 8 0 850277505 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 6 0 0 0
[pid=16246/tid=16290] ppid=16244 vsize=413052 CPUtime=0 cores=0,2,4,6
/proc/16246/task/16290/stat : 16290 (python) S 16244 16246 15040 0 -1 4202560 28 926 0 0 0 0 0 0 20 0 8 0 850282229 422965248 21757 33554432000 4194304 5683644 140733497739568 18446744073709551615 270350207409 0 0 16781312 33554434 18446604442946440064 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 304.2
Current children cumulated vsize (KiB) 16287152

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 16246 and gives
#  childrusage.ru_utime.tv_sec=18
#  childrusage.ru_utime.tv_usec=435197
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=160975
# CPU time returned by wait4() is 18.5962
# while last known CPU time is 304.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 303.386
CPU time (s): 304.2
CPU user time (s): 300.48
CPU system time (s): 3.72
CPU usage (%): 100.268
Max. virtual memory (cumulated for all children) (KiB): 16287152

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 18.4352
system time used= 0.160975
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 40135
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1022
involuntary context switches= 56

runsolver used 1.55576 second user time and 3.14852 second system time

The end

Launcher Data 

Begin job on node143 at 2011-06-09 02:12:42
IDJOB=3482898
IDBENCH=89687
IDSOLVER=1971
FILE ID=node143/3482898-1307578362
RUNJOBID= node143-1307576354-15058
PBS_JOBID= 13547656
Free space on /tmp= 73784 MiB

SOLVER NAME= borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-40.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3482898-1307578362/watcher-3482898-1307578362 -o /tmp/evaluation-result-3482898-1307578362/solver-3482898-1307578362 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3482898-1307578362.opb -seed 1283641863 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 96b10f202af149572cd99ac873608650
RANDOM SEED=1283641863

node143.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      21294384 kB
Buffers:        176016 kB
Cached:         659268 kB
SwapCached:       8640 kB
Active:       11121080 kB
Inactive:       231496 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      21294384 kB
SwapTotal:    67111528 kB
SwapFree:     50923800 kB
Dirty:          123612 kB
Writeback:           0 kB
AnonPages:    10515104 kB
Mapped:          56920 kB
Slab:           145188 kB
PageTables:      90088 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 119568304 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73672 MiB
End job on node143 at 2011-06-09 02:17:48