Trace number 3741085

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 487.6 762.968

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-129.opb
MD5SUM67d021e5b92124566c71fb431934058c
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables48368
Total number of constraints1121286
Number of constraints which are clauses1007481
Number of constraints which are cardinality constraints (but not clauses)3534
Number of constraints which are nor clauses,nor cardinality constraints110271
Minimum length of a constraint1
Maximum length of a constraint690
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 691
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.36/1.17	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.36/1.17	c INFO: solving HOME/instance-3741085-1338721014.opb
8.22/9.09	c INFO: parsing took 7.78 s
22.81/23.67	c INFO: features took 14.54 CPU seconds
22.81/23.69	c INFO: running wbo1.4b-fixed@0 for 45 with 1777 remaining (b = 0.96)
72.96/73.79	c INFO: running pbct-0.1.2-linear@0 for 45 with 1728 remaining (b = 0.96)
104.72/227.81	c INFO: running wbo1.4a@0 for 45 with 1701 remaining (b = 0.95)
155.77/278.82	c INFO: running scip-2.0.1-clp@0 for 45 with 1651 remaining (b = 0.83)
203.53/326.53	c INFO: running scip-2.0.1-spx@0 for 45 with 1605 remaining (b = 0.68)
320.52/455.59	c INFO: wrote minion input file to HOME/tmpSXo7tx.minion
320.52/455.60	c INFO: running minion-0.12@0 for 45 with 1559 remaining (b = 0.55)
344.12/619.67	c INFO: running bsolo_pb10-l2@0 for 45 with 1471 remaining (b = 0.29)
391.95/667.44	c INFO: running clasp-1.3.7@0 for 45 with 1424 remaining (b = 0.17)
439.22/714.61	c INFO: running clasp-1.3.7@47 for 45 with 1377 remaining (b = 0.02)
487.38/762.76	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1329 remaining (b = 0.05)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3741085-1338721014/watcher-3741085-1338721014 -o /tmp/evaluation-result-3741085-1338721014/solver-3741085-1338721014 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741085-1338721014.opb -seed 375858065 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.53 3.42 2.71 2/184 8109
/proc/meminfo: memFree=17040900/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/8109/stat : 8109 (python) D 8107 8109 3958 0 -1 4202496 634 0 0 0 0 0 0 0 20 0 1 0 338227721 15642624 527 33554432000 4194304 5683644 140734936071968 140734936057880 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 3819 527 312 364 0 261 0

[startup+0.0615871 s]
/proc/loadavg: 4.53 3.42 2.71 2/184 8109
/proc/meminfo: memFree=17040900/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=16592 CPUtime=0.01 cores=1,3,5,7
/proc/8109/stat : 8109 (python) D 8107 8109 3958 0 -1 4202496 1202 0 0 0 1 0 0 0 20 0 1 0 338227721 16990208 1001 33554432000 4194304 5683644 140734936071968 140734936049736 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 4148 1001 422 364 0 590 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16592

[startup+0.100226 s]
/proc/loadavg: 4.53 3.42 2.71 2/184 8109
/proc/meminfo: memFree=17040900/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=21172 CPUtime=0.02 cores=1,3,5,7
/proc/8109/stat : 8109 (python) R 8107 8109 3958 0 -1 4202496 1402 0 0 0 2 0 0 0 20 0 1 0 338227721 21680128 1155 33554432000 4194304 5683644 140734936071968 140734936017688 217401355607 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 5293 1159 434 364 0 695 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 21172

[startup+0.300259 s]
/proc/loadavg: 4.53 3.42 2.71 2/184 8109
/proc/meminfo: memFree=17040900/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=102080 CPUtime=0.12 cores=1,3,5,7
/proc/8109/stat : 8109 (python) D 8107 8109 3958 0 -1 4202496 4203 1008 1 0 9 3 0 0 20 0 1 0 338227721 104529920 2755 33554432000 4194304 5683644 140734936071968 140734935946056 217401355559 0 0 16781312 2 18446744072100588836 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 25520 2755 890 364 0 1730 0
Current children cumulated CPU time (s) 0.12
Current children cumulated vsize (KiB) 102080

[startup+0.700403 s]
/proc/loadavg: 4.53 3.42 2.71 2/184 8109
/proc/meminfo: memFree=17040900/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=166948 CPUtime=0.22 cores=1,3,5,7
/proc/8109/stat : 8109 (python) R 8107 8109 3958 0 -1 4202496 6562 1008 1 0 15 7 0 0 20 0 1 0 338227721 170954752 4352 33554432000 4194304 5683644 140734936071968 140734935976200 139794463332801 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 41737 4352 1227 364 0 3290 0
Current children cumulated CPU time (s) 0.22
Current children cumulated vsize (KiB) 166948

[startup+1.50012 s]
/proc/loadavg: 4.53 3.42 2.71 1/185 8114
/proc/meminfo: memFree=17022772/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=340632 CPUtime=0.72 cores=1,3,5,7
/proc/8109/stat : 8109 (python) R 8107 8109 3958 0 -1 4202496 13035 1008 1 0 60 12 0 0 20 0 1 0 338227721 348807168 21822 33554432000 4194304 5683644 140734936071968 140734936065032 217410146196 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 85158 21822 1811 364 0 20387 0
Current children cumulated CPU time (s) 0.72
Current children cumulated vsize (KiB) 340632

[startup+3.10013 s]
/proc/loadavg: 4.53 3.42 2.71 2/185 8114
/proc/meminfo: memFree=16938228/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=396748 CPUtime=2.32 cores=1,3,5,7
/proc/8109/stat : 8109 (python) R 8107 8109 3958 0 -1 4202496 26553 1008 1 0 218 14 0 0 20 0 1 0 338227721 406269952 35340 33554432000 4194304 5683644 140734936071968 140734936064584 139794296620012 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 99187 35341 1811 364 0 34416 0
Current children cumulated CPU time (s) 2.32
Current children cumulated vsize (KiB) 396748
heavy processes:
  %CPU=18 pid=8024 uid=0 cmd=

[startup+6.30014 s]
/proc/loadavg: 4.57 3.44 2.72 2/185 8114
/proc/meminfo: memFree=16826436/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=507424 CPUtime=5.53 cores=1,3,5,7
/proc/8109/stat : 8109 (python) R 8107 8109 3958 0 -1 4202496 52900 1008 1 0 534 19 0 0 20 0 1 0 338227721 519602176 62709 33554432000 4194304 5683644 140734936071968 140734936064584 217410146196 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 126856 62709 1811 364 0 62085 0
Current children cumulated CPU time (s) 5.53
Current children cumulated vsize (KiB) 507424

[startup+12.7001 s]
/proc/loadavg: 4.60 3.47 2.73 2/185 8114
/proc/meminfo: memFree=16710556/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=585336 CPUtime=11.92 cores=1,3,5,7
/proc/8109/stat : 8109 (python) R 8107 8109 3958 0 -1 4202496 71195 1008 1 0 1166 26 0 0 20 0 1 0 338227721 599384064 83086 33554432000 4194304 5683644 140734936071968 140734936058792 139794463155120 0 0 16781312 33554434 0 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 146334 83086 1846 364 0 81563 0
Current children cumulated CPU time (s) 11.92
Current children cumulated vsize (KiB) 585336

[startup+25.5002 s]
/proc/loadavg: 4.69 3.54 2.77 2/188 8117
/proc/meminfo: memFree=16719084/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=611848 CPUtime=22.91 cores=1,3,5,7
/proc/8109/stat : 8109 (python) S 8107 8109 3958 0 -1 4202496 73598 1008 1 0 2264 27 0 0 20 0 2 0 338227721 626532352 70825 33554432000 4194304 5683644 140734936071968 140734936066800 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 152962 70825 1902 364 0 88185 0
[pid=8115] ppid=8109 vsize=536068 CPUtime=0 cores=1,3,5,7
/proc/8115/stat : 8115 (python) S 8109 8109 3958 0 -1 4202560 746 0 0 0 0 0 0 0 20 0 1 0 338230091 548933632 69257 33554432000 4194304 5683644 140734936071968 140734936063056 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8115/statm: 134017 69257 340 364 0 69240 0
[pid=8117] ppid=8115 vsize=97092 CPUtime=1.78 cores=1,3,5,7
/proc/8117/stat : 8117 (wbo1.4b-fixed) R 8115 8117 8117 0 -1 4202496 17834 0 0 0 176 2 0 0 20 0 1 0 338230091 99422208 17587 33554432000 134512640 135752315 4294883184 4293832576 134564550 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/8117/statm: 24273 17587 6519 303 0 11562 0
[pid=8109/tid=8116] ppid=8107 vsize=611848 CPUtime=0 cores=1,3,5,7
/proc/8109/task/8116/stat : 8116 (python) S 8107 8109 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338230091 626532352 70825 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 24.69
Current children cumulated vsize (KiB) 1245008

[startup+51.1003 s]
/proc/loadavg: 4.80 3.66 2.83 4/188 8117
/proc/meminfo: memFree=16635924/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=611848 CPUtime=22.91 cores=1,3,5,7
/proc/8109/stat : 8109 (python) S 8107 8109 3958 0 -1 4202496 73598 1008 1 0 2264 27 0 0 20 0 2 0 338227721 626532352 70825 33554432000 4194304 5683644 140734936071968 140734936066800 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 152962 70825 1902 364 0 88185 0
[pid=8115] ppid=8109 vsize=536068 CPUtime=0.03 cores=1,3,5,7
/proc/8115/stat : 8115 (python) S 8109 8109 3958 0 -1 4202560 902 0 0 0 1 2 0 0 20 0 1 0 338230091 548933632 69280 33554432000 4194304 5683644 140734936071968 140734936063056 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8115/statm: 134017 69280 363 364 0 69240 0
[pid=8117] ppid=8115 vsize=131744 CPUtime=27.38 cores=1,3,5,7
/proc/8117/stat : 8117 (wbo1.4b-fixed) R 8115 8117 8117 0 -1 4202496 43907 0 0 0 2682 56 0 0 20 0 1 0 338230091 134905856 30693 33554432000 134512640 135752315 4294883184 4294881260 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/8117/statm: 32936 30693 146 303 0 32631 0
[pid=8109/tid=8116] ppid=8107 vsize=611848 CPUtime=0 cores=1,3,5,7
/proc/8109/task/8116/stat : 8116 (python) S 8107 8109 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338230091 626532352 70825 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 50.32
Current children cumulated vsize (KiB) 1279660

[startup+102.307 s]
/proc/loadavg: 3.73 3.58 2.84 4/194 8127
/proc/meminfo: memFree=20504008/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=687640 CPUtime=22.94 cores=1,3,5,7
/proc/8109/stat : 8109 (python) S 8107 8109 3958 0 -1 4202496 74390 1008 1 0 2267 27 0 0 20 0 3 0 338227721 704143360 71342 33554432000 4194304 5683644 140734936071968 140734936066800 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 171910 71342 1905 364 0 107130 0
[pid=8115] ppid=8109 vsize=611848 CPUtime=0.07 cores=1,3,5,7
/proc/8115/stat : 8115 (python) S 8109 8109 3958 0 -1 4202560 1140 0 0 0 3 4 0 0 20 0 2 0 338230091 626532352 69320 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8115/statm: 152962 69320 396 364 0 88185 0
[pid=8117] ppid=8115 vsize=141116 CPUtime=50.04 cores=1,3,5,7
/proc/8117/stat : 8117 (wbo1.4b-fixed) T 8115 8117 8117 0 -1 4202496 46229 0 0 0 4911 93 0 0 20 0 1 0 338230091 144502784 33015 33554432000 134512640 135752315 4294883184 4294881260 135270849 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8117/statm: 35279 33015 146 303 0 34974 0
[pid=8115/tid=8123] ppid=8109 vsize=611848 CPUtime=0 cores=1,3,5,7
/proc/8115/task/8123/stat : 8123 (python) S 8109 8109 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338235098 626532352 69320 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8124] ppid=8109 vsize=611860 CPUtime=0.04 cores=1,3,5,7
/proc/8124/stat : 8124 (python) S 8109 8109 3958 0 -1 4202560 911 0 0 0 2 2 0 0 20 0 1 0 338235101 626544640 69281 33554432000 4194304 5683644 140734936071968 140734936063056 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/8124/statm: 152965 69281 358 364 0 88185 0
[pid=8126] ppid=8124 vsize=4175680 CPUtime=28.32 cores=1,3,5,7
/proc/8126/stat : 8126 (pbct-0.1.2-linu) R 8124 8126 8126 0 -1 4204032 924725 0 0 0 2678 154 0 0 20 0 1 0 338235101 4275896320 987019 33554432000 134512640 136422188 4290921904 4290919916 135736970 0 0 16781312 16387 0 0 0 17 1 0 0 0 0 0
/proc/8126/statm: 1043920 987019 249 467 0 1043451 0
[pid=8109/tid=8116] ppid=8107 vsize=687640 CPUtime=0 cores=1,3,5,7
/proc/8109/task/8116/stat : 8116 (python) S 8107 8109 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 3 0 338230091 704143360 71342 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8125] ppid=8107 vsize=687640 CPUtime=0 cores=1,3,5,7
/proc/8109/task/8125/stat : 8125 (python) S 8107 8109 3958 0 -1 4202560 19 1008 0 0 0 0 0 0 20 0 3 0 338235101 704143360 71342 33554432000 4194304 5683644 140734936071968 139794206155312 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 101.41
Current children cumulated vsize (KiB) 6228144

[startup+162.3 s]
/proc/loadavg: 4.47 3.81 2.97 2/198 8131
/proc/meminfo: memFree=10360020/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=687640 CPUtime=22.94 cores=1,3,5,7
/proc/8109/stat : 8109 (python) S 8107 8109 3958 0 -1 4202496 74390 1008 1 0 2267 27 0 0 20 0 3 0 338227721 704143360 71342 33554432000 4194304 5683644 140734936071968 140734936066800 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 171910 71342 1905 364 0 107130 0
[pid=8115] ppid=8109 vsize=611848 CPUtime=0.07 cores=1,3,5,7
/proc/8115/stat : 8115 (python) S 8109 8109 3958 0 -1 4202560 1140 0 0 0 3 4 0 0 20 0 2 0 338230091 626532352 69320 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8115/statm: 152962 69320 396 364 0 88185 0
[pid=8117] ppid=8115 vsize=141116 CPUtime=50.04 cores=1,3,5,7
/proc/8117/stat : 8117 (wbo1.4b-fixed) T 8115 8117 8117 0 -1 4202496 46229 0 0 0 4911 93 0 0 20 0 1 0 338230091 144502784 33015 33554432000 134512640 135752315 4294883184 4294881260 135270849 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8117/statm: 35279 33015 146 303 0 34974 0
[pid=8115/tid=8123] ppid=8109 vsize=611848 CPUtime=0 cores=1,3,5,7
/proc/8115/task/8123/stat : 8123 (python) S 8109 8109 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338235098 626532352 69320 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8124] ppid=8109 vsize=611860 CPUtime=0.13 cores=1,3,5,7
/proc/8124/stat : 8124 (python) S 8109 8109 3958 0 -1 4202560 947 0 0 0 7 6 0 0 20 0 1 0 338235101 626544640 69281 33554432000 4194304 5683644 140734936071968 140734936063056 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/8124/statm: 152965 69281 358 364 0 88185 0
[pid=8126] ppid=8124 vsize=4175680 CPUtime=31.22 cores=1,3,5,7
/proc/8126/stat : 8126 (pbct-0.1.2-linu) S 8124 8126 8126 0 -1 4204032 924725 0 0 0 2678 444 0 0 20 0 1 0 338235101 4275896320 987019 33554432000 134512640 136422188 4290921904 4290919916 135736970 0 0 16781312 16387 18446744071580380219 0 0 17 5 0 0 0 0 0
/proc/8126/statm: 1043920 987019 249 467 0 1043451 0
[pid=8109/tid=8116] ppid=8107 vsize=687640 CPUtime=0 cores=1,3,5,7
/proc/8109/task/8116/stat : 8116 (python) S 8107 8109 3958 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 3 0 338230091 704143360 71342 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8125] ppid=8107 vsize=687640 CPUtime=0 cores=1,3,5,7
/proc/8109/task/8125/stat : 8125 (python) S 8107 8109 3958 0 -1 4202560 19 1008 0 0 0 0 0 0 20 0 3 0 338235101 704143360 71342 33554432000 4194304 5683644 140734936071968 139794206155312 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 104.4
Current children cumulated vsize (KiB) 6228144
heavy processes:
  %CPU=23 pid=8127 uid=0 cmd=/usr/libexec/abrt-hook-ccpp /var/spool/abrt 8126 11 506 18446744073709551615 

[startup+222.3 s]
/proc/loadavg: 5.42 4.20 3.15 2/205 8139
/proc/meminfo: memFree=7053304/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=687640 CPUtime=22.94 cores=1,3,5,7
/proc/8109/stat : 8109 (python) S 8107 8109 3958 0 -1 4202496 74390 1008 1 0 2267 27 0 0 20 0 3 0 338227721 704143360 71342 33554432000 4194304 5683644 140734936071968 140734936066800 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8109/statm: 171910 71342 1905 364 0 107130 0
[pid=8115] ppid=8109 vsize=611848 CPUtime=0.07 cores=1,3,5,7
/proc/8115/stat : 8115 (python) S 8109 8109 3958 0 -1 4202560 1140 0 0 0 3 4 0 0 20 0 2 0 338230091 626532352 69320 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0

################
# More data... #
################

/proc/8324/task/8334/stat : 8334 (java) S 8322 8324 8324 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338304002 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449850284520 217413892009 0 0 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8335] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8335/stat : 8335 (java) S 8322 8324 8324 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338304003 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449849230584 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8336] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8336/stat : 8336 (java) S 8322 8324 8324 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338304003 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449848177672 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8337] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8337/stat : 8337 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449847126160 217413900064 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8338] ppid=8322 vsize=9558408 CPUtime=0.01 cores=1,3,5,7
/proc/8324/task/8338/stat : 8338 (java) S 8322 8324 8324 0 -1 4202560 381 0 0 0 1 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449846074296 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8339] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8339/stat : 8339 (java) S 8322 8324 8324 0 -1 4202560 507 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449845021496 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8324/tid=8340] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8340/stat : 8340 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449843968888 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8324/tid=8341] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8341/stat : 8341 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449842916296 217413892009 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8109/tid=8116] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8116/stat : 8116 (python) S 8107 8109 3958 0 -1 4202560 27 1327328 0 0 0 0 4684 839 20 0 10 0 338230091 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8125] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8125/stat : 8125 (python) S 8107 8109 3958 0 -1 4202560 19 1327328 0 0 0 0 4684 839 20 0 10 0 338235101 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794206155312 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8142] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8142/stat : 8142 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338250503 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794195665456 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8168] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8168/stat : 8168 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338255604 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794185175600 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8178] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8178/stat : 8178 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338260375 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794174685744 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8182] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8182/stat : 8182 (python) S 8107 8109 3958 0 -1 4202560 22 1327328 0 0 0 0 4684 839 20 0 10 0 338273282 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794164195888 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8109/tid=8216] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8216/stat : 8216 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338289689 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794153706032 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8109/tid=8265] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8265/stat : 8265 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338294466 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794143216176 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8109/tid=8323] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8323/stat : 8323 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338303998 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794132726320 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 487.6
Current children cumulated vsize (KiB) 20837384
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+762.901 s]
/proc/loadavg: 2.32 3.59 3.39 3/230 8341
/proc/meminfo: memFree=18862004/32873648 swapFree=7628/7628
[pid=8109] ppid=8107 vsize=1218184 CPUtime=146.66 cores=1,3,5,7
/proc/8109/stat : 8109 (python) S 8107 8109 3958 0 -1 4202496 79903 1327328 1 0 9091 52 4684 839 20 0 10 0 338227721 1247420416 71398 33554432000 4194304 5683644 140734936071968 140734936066800 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8109/statm: 304546 71398 1919 364 0 239745 0
[pid=8115] ppid=8109 vsize=611848 CPUtime=0.07 cores=1,3,5,7
/proc/8115/stat : 8115 (python) S 8109 8109 3958 0 -1 4202560 1140 0 0 0 3 4 0 0 20 0 2 0 338230091 626532352 69320 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8115/statm: 152962 69320 396 364 0 88185 0
[pid=8117] ppid=8115 vsize=141116 CPUtime=50.04 cores=1,3,5,7
/proc/8117/stat : 8117 (wbo1.4b-fixed) T 8115 8117 8117 0 -1 4202496 46229 0 0 0 4911 93 0 0 20 0 1 0 338230091 144502784 33015 33554432000 134512640 135752315 4294883184 4294881260 135270849 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8117/statm: 35279 33015 146 303 0 34974 0
[pid=8115/tid=8123] ppid=8109 vsize=611848 CPUtime=0 cores=1,3,5,7
/proc/8115/task/8123/stat : 8123 (python) S 8109 8109 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338235098 626532352 69320 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8141] ppid=8109 vsize=687652 CPUtime=0.09 cores=1,3,5,7
/proc/8141/stat : 8141 (python) S 8109 8109 3958 0 -1 4202560 1100 0 0 0 4 5 0 0 20 0 2 0 338250502 704155648 69820 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8141/statm: 171913 69820 383 364 0 107130 0
[pid=8143] ppid=8141 vsize=121704 CPUtime=50.96 cores=1,3,5,7
/proc/8143/stat : 8143 (wbo1.4a) T 8141 8143 8143 0 -1 4202496 41362 0 0 0 4996 100 0 0 20 0 1 0 338250503 124624896 28656 33554432000 134512640 135751879 4289868512 4289866688 134616520 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8143/statm: 30426 28656 143 303 0 30121 0
[pid=8141/tid=8166] ppid=8109 vsize=687652 CPUtime=0 cores=1,3,5,7
/proc/8141/task/8166/stat : 8166 (python) S 8109 8109 3958 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338255601 704155648 69820 33554432000 4194304 5683644 140734936071968 139794206155312 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8167] ppid=8109 vsize=763444 CPUtime=0.08 cores=1,3,5,7
/proc/8167/stat : 8167 (python) S 8109 8109 3958 0 -1 4202560 1092 0 0 0 4 4 0 0 20 0 2 0 338255604 781766656 69827 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/8167/statm: 190861 69827 383 364 0 126075 0
[pid=8169] ppid=8167 vsize=1656468 CPUtime=47.65 cores=1,3,5,7
/proc/8169/stat : 8169 (scip-2.0.1.linu) T 8167 8169 8169 0 -1 4202496 406292 0 0 0 4583 182 0 0 20 0 1 0 338255604 1696223232 406044 33554432000 4194304 15243407 140734361412160 140734361408456 5688624 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/8169/statm: 414117 406044 701 2698 0 411412 0
[pid=8167/tid=8176] ppid=8109 vsize=763444 CPUtime=0 cores=1,3,5,7
/proc/8167/task/8176/stat : 8176 (python) S 8109 8109 3958 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338260372 781766656 69827 33554432000 4194304 5683644 140734936071968 139794195665456 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8177] ppid=8109 vsize=839236 CPUtime=0.11 cores=1,3,5,7
/proc/8177/stat : 8177 (python) S 8109 8109 3958 0 -1 4202560 1121 0 0 0 5 6 0 0 20 0 2 0 338260375 859377664 69833 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8177/statm: 209809 69833 383 364 0 145020 0
[pid=8179] ppid=8177 vsize=1655100 CPUtime=48.63 cores=1,3,5,7
/proc/8179/stat : 8179 (scip-2.0.1.linu) T 8177 8179 8179 0 -1 4202496 407300 0 0 0 4616 247 0 0 20 0 1 0 338260375 1694822400 406038 33554432000 4194304 13684479 140736778663760 140736778660056 5688624 0 0 16781312 2 18446744071579387992 0 0 17 5 0 0 1189 0 0
/proc/8179/statm: 413775 406038 653 2317 0 411453 0
[pid=8177/tid=8180] ppid=8109 vsize=839236 CPUtime=0 cores=1,3,5,7
/proc/8177/task/8180/stat : 8180 (python) S 8109 8109 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338266449 859377664 69833 33554432000 4194304 5683644 140734936071968 139794185175600 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8215] ppid=8109 vsize=990820 CPUtime=0.08 cores=1,3,5,7
/proc/8215/stat : 8215 (python) S 8109 8109 3958 0 -1 4202560 1139 0 0 0 4 4 0 0 20 0 2 0 338289689 1014599680 69845 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8215/statm: 247705 69845 383 364 0 182910 0
[pid=8217] ppid=8215 vsize=280252 CPUtime=47.7 cores=1,3,5,7
/proc/8217/stat : 8217 (bsolo_pb10) T 8215 8217 8217 0 -1 4202496 80259 0 0 0 4731 39 0 0 20 0 1 0 338289689 286978048 67375 33554432000 134512640 137138936 4290485376 4289903424 134540085 0 0 16781312 16384 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/8217/statm: 70063 67375 185 642 0 69419 0
[pid=8215/tid=8263] ppid=8109 vsize=990820 CPUtime=0 cores=1,3,5,7
/proc/8215/task/8263/stat : 8263 (python) S 8109 8109 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338294464 1014599680 69845 33554432000 4194304 5683644 140734936071968 139794164195888 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8264] ppid=8109 vsize=1066612 CPUtime=0.15 cores=1,3,5,7
/proc/8264/stat : 8264 (python) S 8109 8109 3958 0 -1 4202560 1230 0 0 0 8 7 0 0 20 0 2 0 338294466 1092210688 69857 33554432000 4194304 5683644 140734936071968 140734936063616 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/8264/statm: 266653 69857 389 364 0 201855 0
[pid=8266] ppid=8264 vsize=104136 CPUtime=95.23 cores=1,3,5,7
/proc/8266/stat : 8266 (clasp-1.3.7-x86) T 8264 8266 8266 0 -1 4202496 31224 0 0 0 9516 7 0 0 20 0 1 0 338294466 106635264 25431 33554432000 134512640 136139019 4286906528 4286905360 134832139 0 0 16781312 16386 18446744071579387992 0 0 17 3 0 0 1 0 0
/proc/8266/statm: 26034 25431 183 398 0 25634 0
[pid=8264/tid=8317] ppid=8109 vsize=1066612 CPUtime=0 cores=1,3,5,7
/proc/8264/task/8317/stat : 8317 (python) S 8109 8109 3958 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338299181 1092210688 69857 33554432000 4194304 5683644 140734936071968 139794153706032 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8322] ppid=8109 vsize=1142404 CPUtime=0 cores=1,3,5,7
/proc/8322/stat : 8322 (python) S 8109 8109 3958 0 -1 4202560 890 0 0 0 0 0 0 0 20 0 1 0 338303998 1169821696 69815 33554432000 4194304 5683644 140734936071968 140734936063056 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/8322/statm: 285601 69815 342 364 0 220800 0
[pid=8324] ppid=8322 vsize=9558408 CPUtime=0.15 cores=1,3,5,7
/proc/8324/stat : 8324 (java) S 8322 8324 8324 0 -1 4202496 4721 0 1 0 14 1 0 0 20 0 18 0 338303998 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140734596886992 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/8324/statm: 2389602 6146 2152 9 0 2380296 0
[pid=8324/tid=8325] ppid=8322 vsize=9558408 CPUtime=0.1 cores=1,3,5,7
/proc/8324/task/8325/stat : 8325 (java) R 8322 8324 8324 0 -1 4202560 2411 0 1 0 10 0 0 0 20 0 18 0 338304000 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450497485392 140450346378415 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=8324/tid=8326] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8326/stat : 8326 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450469968424 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8327] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8327/stat : 8327 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450468915624 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8324/tid=8328] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8328/stat : 8328 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450467862824 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8324/tid=8329] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8329/stat : 8329 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450466810024 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8324/tid=8330] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8330/stat : 8330 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450465758248 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8331] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8331/stat : 8331 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450335337384 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8324/tid=8332] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8332/stat : 8332 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450334284584 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8324/tid=8333] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8333/stat : 8333 (java) S 8322 8324 8324 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338304001 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140450333231784 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8334] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8334/stat : 8334 (java) S 8322 8324 8324 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338304002 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449850284520 217413892009 0 0 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8335] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8335/stat : 8335 (java) S 8322 8324 8324 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338304003 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449849230584 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8336] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8336/stat : 8336 (java) S 8322 8324 8324 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338304003 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449848177672 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8337] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8337/stat : 8337 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449847126160 217413900064 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8338] ppid=8322 vsize=9558408 CPUtime=0.01 cores=1,3,5,7
/proc/8324/task/8338/stat : 8338 (java) S 8322 8324 8324 0 -1 4202560 381 0 0 0 1 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449846074296 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=8324/tid=8339] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8339/stat : 8339 (java) S 8322 8324 8324 0 -1 4202560 507 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449845021496 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8324/tid=8340] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8340/stat : 8340 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449843968888 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8324/tid=8341] ppid=8322 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/8324/task/8341/stat : 8341 (java) S 8322 8324 8324 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338304005 9787809792 6146 33554432000 1073741824 1073778376 140734596895856 140449842916296 217413892009 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8109/tid=8116] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8116/stat : 8116 (python) S 8107 8109 3958 0 -1 4202560 27 1327328 0 0 0 0 4684 839 20 0 10 0 338230091 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794289984048 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8125] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8125/stat : 8125 (python) S 8107 8109 3958 0 -1 4202560 19 1327328 0 0 0 0 4684 839 20 0 10 0 338235101 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794206155312 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8142] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8142/stat : 8142 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338250503 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794195665456 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8168] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8168/stat : 8168 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338255604 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794185175600 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8178] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8178/stat : 8178 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338260375 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794174685744 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=8109/tid=8182] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8182/stat : 8182 (python) S 8107 8109 3958 0 -1 4202560 22 1327328 0 0 0 0 4684 839 20 0 10 0 338273282 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794164195888 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8109/tid=8216] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8216/stat : 8216 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338289689 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794153706032 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=8109/tid=8265] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8265/stat : 8265 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338294466 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794143216176 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=8109/tid=8323] ppid=8107 vsize=1218184 CPUtime=55.23 cores=1,3,5,7
/proc/8109/task/8323/stat : 8323 (python) S 8107 8109 3958 0 -1 4202560 21 1327328 0 0 0 0 4684 839 20 0 10 0 338303998 1247420416 71398 33554432000 4194304 5683644 140734936071968 139794132726320 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 487.6
Current children cumulated vsize (KiB) 20837384

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 8109 and gives
#  childrusage.ru_utime.tv_sec=137
#  childrusage.ru_utime.tv_usec=754058
#  childrusage.ru_stime.tv_sec=8
#  childrusage.ru_stime.tv_usec=933641
# CPU time returned by wait4() is 146.688
# while last known CPU time is 487.6
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 762.968
CPU time (s): 487.6
CPU user time (s): 471.7
CPU system time (s): 15.9
CPU usage (%): 63.9083
Max. virtual memory (cumulated for all children) (KiB): 20837384

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 137.754
system time used= 8.93364
maximum resident set size= 4134132
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1407231
page faults= 1
swaps= 0
block input operations= 200
block output operations= 147096
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2027873
involuntary context switches= 311

runsolver used 2.42363 second user time and 6.48201 second system time

The end

Launcher Data 

Begin job on node122 at 2012-06-03 12:56:55
IDJOB=3741085
IDBENCH=89756
IDSOLVER=2327
FILE ID=node122/3741085-1338721014
RUNJOBID= node122-1338713392-3975
PBS_JOBID= 14636732
Free space on /tmp= 67180 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-129.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3741085-1338721014/watcher-3741085-1338721014 -o /tmp/evaluation-result-3741085-1338721014/solver-3741085-1338721014 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741085-1338721014.opb -seed 375858065 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 67d021e5b92124566c71fb431934058c
RANDOM SEED=375858065

node122.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5438.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        17041056 kB
Buffers:          299624 kB
Cached:          9841288 kB
SwapCached:         1804 kB
Active:          5313080 kB
Inactive:        9219864 kB
Active(anon):    4392436 kB
Inactive(anon):     2024 kB
Active(file):     920644 kB
Inactive(file):  9217840 kB
Unevictable:        7628 kB
Mlocked:            7628 kB
SwapTotal:      67108856 kB
SwapFree:       67100164 kB
Dirty:           4718240 kB
Writeback:           548 kB
AnonPages:       4398176 kB
Mapped:            16884 kB
Shmem:                36 kB
Slab:            1149708 kB
SReclaimable:     588736 kB
SUnreclaim:       560972 kB
KernelStack:        1504 kB
PageTables:        15480 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    5319780 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:    481280 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 63036 MiB
End job on node122 at 2012-06-03 13:09:42