Trace number 3728789

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pwbo 2.02MSAT (TO) 1800.41 900.215

General information on the benchmark

Name/SOFT-SMALLINT-LIN/PB06/submitted-PB05/aloul/FPGA_SAT05/
normalized-chnl10_11_pb.cnf.cr--soft-0-100-0.wbo
MD5SUM9e134c91eb84e264e2a3479232b34054
Bench CategorySOFT-SMALLINT-LIN (only soft constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark9
Best CPU time to get the best result obtained on this benchmark0.05699
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables220
Total number of constraints42
Number of soft constraints42
Number of constraints which are clauses22
Number of constraints which are cardinality constraints (but not clauses)20
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint10
Maximum length of a constraint11
Top cost 2167
Min constraint cost 1
Max constraint cost 97
Sum of constraints costs 2166
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 12
Number of bits of the biggest sum of numbers4
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Quality of the solution as a function of time


Download the above graph as a PDF file / as an EPS file

Solver Data 

0.00/0.00	c ==============================[ Solver Version ]===============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  pwbo:                     2.02 beta - May, 2012                            |
0.00/0.00	c |                                                                             |
0.00/0.00	c Parsing wbo file format. File HOME/instance-3728789-1338673243.wbo.
0.00/0.00	c Instance file HOME/instance-3728789-1338673243.wbo
0.00/0.00	c File size is 4084 bytes.
0.00/0.00	c Highest Coefficient sum: 12
0.00/0.00	c ============================[ Problem Statistics ]=============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Parsing time:          0.00         s                                      |
0.00/0.00	c |  Number Variables:      262                                                 |
0.00/0.00	c |  Number Clauses:        22                                                  |
0.00/0.00	c |  Number Cardinality:    0                                                   |
0.00/0.00	c |  Number PB Constraints: 20                                                  |
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Problem Type:          Optimization                                        |
0.00/0.00	c |  Objective Function:    Cardinality Constraint                              |
0.00/0.00	c |                                                                             |
0.00/0.00	c ===============================================================================
1800.06/900.21	o 117
1800.06/900.21	o 105
1800.06/900.21	o 98
1800.06/900.21	o 78
1800.06/900.21	o 25
1800.06/900.21	o 22
1800.06/900.21	o 14
1800.06/900.21	o 11
1800.06/900.21	o 9
1800.06/900.21	c ===============================================================================
1800.06/900.21	c Wall Clock Time: 900.21 s
1800.06/900.21	c Best Solution: 9
1800.06/900.21	s SATISFIABLE
1800.06/900.21	v -x1 -x2 x3 -x4 -x5 -x6 -x7 -x8 -x9 -x10 -x11 -x12 x13 -x14 -x15 -x16 -x17 -x18 -x19 -x20 -x21 -x22 -x23 -x24 -x25 x26 -x27 -x28 -x29 -x30 -x31 -x32 x33 -x34 -x35 -x36 -x37 -x38 -x39 -x40 -x41 -x42 -x43 x44 -x45 -x46 -x47 -x48 -x49 -x50 -x51 -x52 x53 -x54 -x55 -x56 -x57 -x58 -x59 -x60 x61 -x62 -x63 -x64 -x65 -x66 -x67 -x68 -x69 -x70 -x71 -x72 -x73 -x74 -x75 -x76 -x77 -x78 -x79 x80 -x81 -x82 -x83 -x84 -x85 -x86 -x87 -x88 x89 -x90 -x91 -x92 x93 -x94 -x95 -x96 -x97 -x98 -x99 -x100 -x101 -x102 -x103 -x104 -x105 -x106 -x107 x108 -x109 -x110 -x111 -x112 x113 -x114 -x115 -x116 -x117 -x118 -x119 -x120 -x121 -x122 x123 -x124 -x125 -x126 -x127 -x128 -x129 -x130 -x131 -x132 -x133 -x134 x135 -x136 -x137 -x138 -x139 -x140 -x141 -x142 x143 -x144 -x145 -x146 -x147 -x148 -x149 -x150 -x151 x152 -x153 -x154 -x155 -x156 -x157 -x158 -x159 -x160 -x161 -x162 x163 -x164 -x165 -x166 -x167 -x168 -x169 -x170 -x171 -x172 x173 -x174 -x175 -x176 -x177 -x178 -x179 -x180 -x181 -x182 -x183 -x184 -x185 -x186 -x187 -x188 x189 -x190 -x191 -x192 -x193 -x194 x195 -x196 -x197 -x198 -x199 -x200 -x201 -x202 -x203 -x204 -x205 -x206 -x207 -x208 x209 -x210 -x211 -x212 x213 -x214 -x215 -x216 -x217 -x218 -x219 -x220

Verifier Data 

OK	9

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3728789-1338673243/watcher-3728789-1338673243 -o /tmp/evaluation-result-3728789-1338673243/solver-3728789-1338673243 -C 1800 -W 1900 -M 15500 pwbo2.02 -file-format=wbo -verbosity=0 HOME/instance-3728789-1338673243.wbo 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.85 4.00 3.93 4/179 916
/proc/meminfo: memFree=29492752/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=162860 CPUtime=0 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 1012 0 0 0 0 0 0 0 20 0 3 0 333449724 166768640 964 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 40715 964 198 433 0 40279 0
[pid=916/tid=917] ppid=914 vsize=163392 CPUtime=0 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 59 0 0 0 0 0 0 0 20 0 3 0 333449724 167313408 1208 33554432000 4194304 5964478 140735177107904 140328940210168 4382008 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=163392 CPUtime=0 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 432 0 0 0 0 0 0 0 20 0 3 0 333449724 167313408 1209 33554432000 4194304 5964478 140735177107904 140328929720104 5244295 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0

[startup+0.046962 s]
/proc/loadavg: 3.85 4.00 3.93 4/179 916
/proc/meminfo: memFree=29492752/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=174796 CPUtime=0.08 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 5541 0 0 0 4 4 0 0 20 0 3 0 333449724 178991104 5396 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 43699 5408 206 433 0 43263 0
[pid=916/tid=917] ppid=914 vsize=174796 CPUtime=0.03 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 167 0 0 0 3 0 0 0 20 0 3 0 333449724 178991104 5410 33554432000 4194304 5964478 140735177107904 140328940210168 5136420 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=174796 CPUtime=0.03 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 4624 0 0 0 0 3 0 0 20 0 3 0 333449724 178991104 5412 33554432000 4194304 5964478 140735177107904 140328929720008 5136420 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 174796

[startup+0.100346 s]
/proc/loadavg: 3.85 4.00 3.93 4/179 916
/proc/meminfo: memFree=29492752/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=184384 CPUtime=0.19 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 10084 0 0 0 11 8 0 0 20 0 3 0 333449724 188809216 9925 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 46096 9925 207 433 0 45660 0
[pid=916/tid=917] ppid=914 vsize=184384 CPUtime=0.08 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 263 0 0 0 8 0 0 0 20 0 3 0 333449724 188809216 9925 33554432000 4194304 5964478 140735177107904 140328940209688 4400067 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=184384 CPUtime=0.08 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 9055 0 0 0 0 8 0 0 20 0 3 0 333449724 188809216 9925 33554432000 4194304 5964478 140735177107904 140328929720104 4422225 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 184384

[startup+0.300349 s]
/proc/loadavg: 3.85 4.00 3.93 4/179 916
/proc/meminfo: memFree=29492752/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=185124 CPUtime=0.59 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 10515 0 0 0 51 8 0 0 20 0 3 0 333449724 189566976 10356 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 46281 10356 207 433 0 45845 0
[pid=916/tid=917] ppid=914 vsize=185124 CPUtime=0.28 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 459 0 0 0 28 0 0 0 20 0 3 0 333449724 189566976 10356 33554432000 4194304 5964478 140735177107904 140328940209976 4400067 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=185124 CPUtime=0.28 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 9290 0 0 0 20 8 0 0 20 0 3 0 333449724 189566976 10356 33554432000 4194304 5964478 140735177107904 140328929720552 4403343 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 0.59
Current children cumulated vsize (KiB) 185124

[startup+0.700322 s]
/proc/loadavg: 3.85 4.00 3.93 4/179 916
/proc/meminfo: memFree=29492752/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=185124 CPUtime=1.39 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 11051 0 0 0 130 9 0 0 20 0 3 0 333449724 189566976 10892 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 46281 10892 207 433 0 45845 0
[pid=916/tid=917] ppid=914 vsize=185124 CPUtime=0.68 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 673 0 0 0 68 0 0 0 20 0 3 0 333449724 189566976 10892 33554432000 4194304 5964478 140735177107904 140328940210216 4400422 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=185124 CPUtime=0.68 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 9612 0 0 0 59 9 0 0 20 0 3 0 333449724 189566976 10892 33554432000 4194304 5964478 140735177107904 140328929720504 4400087 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1.39
Current children cumulated vsize (KiB) 185124

[startup+1.50016 s]
/proc/loadavg: 3.85 4.00 3.93 5/182 919
/proc/meminfo: memFree=29449428/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=185124 CPUtime=2.99 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 11741 0 0 0 290 9 0 0 20 0 3 0 333449724 189566976 11582 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 46281 11582 207 433 0 45845 0
[pid=916/tid=917] ppid=914 vsize=185124 CPUtime=1.48 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 1027 0 0 0 148 0 0 0 20 0 3 0 333449724 189566976 11582 33554432000 4194304 5964478 140735177107904 140328940209976 4400795 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=185124 CPUtime=1.48 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 9949 0 0 0 139 9 0 0 20 0 3 0 333449724 189566976 11583 33554432000 4194304 5964478 140735177107904 140328929720632 4221120 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 2.99
Current children cumulated vsize (KiB) 185124

[startup+3.10018 s]
/proc/loadavg: 3.85 4.00 3.93 5/182 919
/proc/meminfo: memFree=29445956/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=185268 CPUtime=6.18 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 12616 0 0 0 609 9 0 0 20 0 3 0 333449724 189714432 12457 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 46317 12457 207 433 0 45881 0
[pid=916/tid=917] ppid=914 vsize=185268 CPUtime=3.08 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 1545 0 0 0 308 0 0 0 20 0 3 0 333449724 189714432 12457 33554432000 4194304 5964478 140735177107904 140328940210168 4378992 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=185268 CPUtime=3.08 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 10305 0 0 0 299 9 0 0 20 0 3 0 333449724 189714432 12457 33554432000 4194304 5964478 140735177107904 140328929720440 4400087 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 6.18
Current children cumulated vsize (KiB) 185268

[startup+6.3002 s]
/proc/loadavg: 3.87 4.00 3.93 5/182 919
/proc/meminfo: memFree=29439388/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=185720 CPUtime=12.59 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 13646 0 0 0 1249 10 0 0 20 0 3 0 333449724 190177280 13487 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 46430 13488 207 433 0 45994 0
[pid=916/tid=917] ppid=914 vsize=185720 CPUtime=6.28 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 2004 0 0 0 628 0 0 0 20 0 3 0 333449724 190177280 13488 33554432000 4194304 5964478 140735177107904 140328940210216 4400103 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=185720 CPUtime=6.28 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 10877 0 0 0 619 9 0 0 20 0 3 0 333449724 190177280 13488 33554432000 4194304 5964478 140735177107904 140328929720632 4400432 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 12.59
Current children cumulated vsize (KiB) 185720

[startup+12.7003 s]
/proc/loadavg: 3.88 4.00 3.93 5/182 919
/proc/meminfo: memFree=29431096/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=186204 CPUtime=25.38 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 15413 0 0 0 2527 11 0 0 20 0 3 0 333449724 190672896 15254 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 46551 15254 207 433 0 46115 0
[pid=916/tid=917] ppid=914 vsize=186204 CPUtime=12.69 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 2845 0 0 0 1267 2 0 0 20 0 3 0 333449724 190672896 15254 33554432000 4194304 5964478 140735177107904 140328940209976 4221120 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=186204 CPUtime=12.68 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 11802 0 0 0 1258 10 0 0 20 0 3 0 333449724 190672896 15254 33554432000 4194304 5964478 140735177107904 140328929720632 4400460 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 25.38
Current children cumulated vsize (KiB) 186204

[startup+25.5003 s]
/proc/loadavg: 3.90 4.00 3.93 5/181 919
/proc/meminfo: memFree=29409768/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=188588 CPUtime=50.98 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 18177 0 0 0 5085 13 0 0 20 0 3 0 333449724 193114112 18018 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 47147 18018 207 433 0 46711 0
[pid=916/tid=917] ppid=914 vsize=188588 CPUtime=25.49 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 4311 0 0 0 2546 3 0 0 20 0 3 0 333449724 193114112 18018 33554432000 4194304 5964478 140735177107904 140328940209976 4400067 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=188588 CPUtime=25.48 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 13100 0 0 0 2537 11 0 0 20 0 3 0 333449724 193114112 18018 33554432000 4194304 5964478 140735177107904 140328929720632 4400795 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 50.98
Current children cumulated vsize (KiB) 188588

[startup+51.1004 s]
/proc/loadavg: 3.93 4.00 3.93 5/181 919
/proc/meminfo: memFree=29290632/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=192600 CPUtime=102.17 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 21785 0 0 0 10200 17 0 0 20 0 3 0 333449724 197222400 21563 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 48150 21563 207 433 0 47714 0
[pid=916/tid=917] ppid=914 vsize=192600 CPUtime=51.08 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 6202 0 0 0 5104 4 0 0 20 0 3 0 333449724 197222400 21563 33554432000 4194304 5964478 140735177107904 140328940209976 4400115 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=192600 CPUtime=51.08 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 14817 0 0 0 5094 14 0 0 20 0 3 0 333449724 197222400 21563 33554432000 4194304 5964478 140735177107904 140328929720632 4400097 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 102.17
Current children cumulated vsize (KiB) 192600

[startup+102.3 s]
/proc/loadavg: 3.97 4.00 3.93 5/181 919
/proc/meminfo: memFree=29191228/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=196852 CPUtime=204.56 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 25301 0 0 0 20436 20 0 0 20 0 3 0 333449724 201576448 25079 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 49213 25079 207 433 0 48777 0
[pid=916/tid=917] ppid=914 vsize=196852 CPUtime=102.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 8631 0 0 0 10222 7 0 0 20 0 3 0 333449724 201576448 25079 33554432000 4194304 5964478 140735177107904 140328940210216 4400087 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=196852 CPUtime=102.29 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 15904 0 0 0 10213 16 0 0 20 0 3 0 333449724 201576448 25079 33554432000 4194304 5964478 140735177107904 140328929720680 4400432 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 204.56
Current children cumulated vsize (KiB) 196852

[startup+162.3 s]
/proc/loadavg: 3.99 4.00 3.93 5/181 920
/proc/meminfo: memFree=29175880/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=199604 CPUtime=324.54 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 27924 0 0 0 32431 23 0 0 20 0 3 0 333449724 204394496 27702 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 49901 27702 207 433 0 49465 0
[pid=916/tid=917] ppid=914 vsize=199604 CPUtime=162.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 10131 0 0 0 16220 9 0 0 20 0 3 0 333449724 204394496 27702 33554432000 4194304 5964478 140735177107904 140328940209976 4400089 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=199604 CPUtime=162.28 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 17027 0 0 0 16209 19 0 0 20 0 3 0 333449724 204394496 27702 33554432000 4194304 5964478 140735177107904 140328929720632 4400432 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 324.54
Current children cumulated vsize (KiB) 199604

[startup+222.307 s]
/proc/loadavg: 3.99 4.00 3.93 5/181 920
/proc/meminfo: memFree=29133332/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=206680 CPUtime=444.53 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 32281 0 0 0 44424 29 0 0 20 0 3 0 333449724 211640320 32386 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 51670 32386 207 433 0 51234 0
[pid=916/tid=917] ppid=914 vsize=206680 CPUtime=222.3 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 12899 0 0 0 22217 13 0 0 20 0 3 0 333449724 211640320 32386 33554432000 4194304 5964478 140735177107904 140328940209976 4220672 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=206680 CPUtime=222.29 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 18616 0 0 0 22207 22 0 0 20 0 3 0 333449724 211640320 32386 33554432000 4194304 5964478 140735177107904 140328929720680 4220672 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 444.53
Current children cumulated vsize (KiB) 206680

################
# More data... #
################

/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 14442 0 0 0 28212 16 0 0 20 0 3 0 333449724 214069248 35272 33554432000 4194304 5964478 140735177107904 140328940210216 4399920 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=209052 CPUtime=282.27 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 19717 0 0 0 28202 25 0 0 20 0 3 0 333449724 214069248 35272 33554432000 4194304 5964478 140735177107904 140328929720632 4399920 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 564.47
Current children cumulated vsize (KiB) 209052

[startup+342.3 s]
/proc/loadavg: 3.99 4.00 3.93 5/181 920
/proc/meminfo: memFree=28987004/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=213240 CPUtime=684.45 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 38933 0 0 0 68407 38 0 0 20 0 3 0 333449724 218357760 40479 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 53310 40479 207 433 0 52874 0
[pid=916/tid=917] ppid=914 vsize=213240 CPUtime=342.28 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 17499 0 0 0 34207 21 0 0 20 0 3 0 333449724 218357760 40479 33554432000 4194304 5964478 140735177107904 140328940210168 4221120 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=213240 CPUtime=342.27 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 20668 0 0 0 34200 27 0 0 20 0 3 0 333449724 218357760 40479 33554432000 4194304 5964478 140735177107904 140328929720632 4400542 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 684.45
Current children cumulated vsize (KiB) 213240

[startup+402.3 s]
/proc/loadavg: 4.04 4.01 3.93 5/181 920
/proc/meminfo: memFree=28815216/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=217488 CPUtime=804.43 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 42480 0 0 0 80401 42 0 0 20 0 3 0 333449724 222707712 44021 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 54372 44021 207 433 0 53936 0
[pid=916/tid=917] ppid=914 vsize=217488 CPUtime=402.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 20057 0 0 0 40204 25 0 0 20 0 3 0 333449724 222707712 44021 33554432000 4194304 5964478 140735177107904 140328940210168 4400542 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=217488 CPUtime=402.27 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 21657 0 0 0 40197 30 0 0 20 0 3 0 333449724 222707712 44021 33554432000 4194304 5964478 140735177107904 140328929720680 4399920 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 804.43
Current children cumulated vsize (KiB) 217488

[startup+462.3 s]
/proc/loadavg: 4.05 4.02 3.94 5/181 921
/proc/meminfo: memFree=28691500/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=284840 CPUtime=924.41 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 44737 0 0 0 92396 45 0 0 20 0 3 0 333449724 291676160 46278 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 71210 46278 207 433 0 70774 0
[pid=916/tid=917] ppid=914 vsize=284840 CPUtime=462.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 21448 0 0 0 46201 28 0 0 20 0 3 0 333449724 291676160 46278 33554432000 4194304 5964478 140735177107904 140328940209976 4221120 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=284840 CPUtime=462.27 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 22523 0 0 0 46195 32 0 0 20 0 3 0 333449724 291676160 46278 33554432000 4194304 5964478 140735177107904 140328929720632 4399920 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 924.41
Current children cumulated vsize (KiB) 284840

[startup+522.3 s]
/proc/loadavg: 4.02 4.02 3.94 5/181 921
/proc/meminfo: memFree=28616196/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=289076 CPUtime=1044.39 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 48027 0 0 0 104390 49 0 0 20 0 3 0 333449724 296013824 49568 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 72269 49568 207 433 0 71833 0
[pid=916/tid=917] ppid=914 vsize=289076 CPUtime=522.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 23267 0 0 0 52199 30 0 0 20 0 3 0 333449724 296013824 49568 33554432000 4194304 5964478 140735177107904 140328940210168 4400528 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=289076 CPUtime=522.27 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 23994 0 0 0 52193 34 0 0 20 0 3 0 333449724 296013824 49568 33554432000 4194304 5964478 140735177107904 140328929720440 4399920 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1044.39
Current children cumulated vsize (KiB) 289076

[startup+582.3 s]
/proc/loadavg: 4.00 4.01 3.94 5/182 924
/proc/meminfo: memFree=28556080/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=357936 CPUtime=1164.37 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 51347 0 0 0 116383 54 0 0 20 0 3 0 333449724 366526464 52888 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 89484 52888 207 433 0 89048 0
[pid=916/tid=917] ppid=914 vsize=357936 CPUtime=582.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 25248 0 0 0 58195 34 0 0 20 0 3 0 333449724 366526464 52888 33554432000 4194304 5964478 140735177107904 140328940210168 4221120 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=357936 CPUtime=582.26 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 25333 0 0 0 58189 37 0 0 20 0 3 0 333449724 366526464 52888 33554432000 4194304 5964478 140735177107904 140328929720632 4399920 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1164.37
Current children cumulated vsize (KiB) 357936

[startup+642.3 s]
/proc/loadavg: 4.00 4.00 3.94 5/181 924
/proc/meminfo: memFree=28513184/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=363204 CPUtime=1284.36 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 52969 0 0 0 128380 56 0 0 20 0 3 0 333449724 371920896 54492 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 90801 54492 207 433 0 90365 0
[pid=916/tid=917] ppid=914 vsize=363204 CPUtime=642.3 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 25477 0 0 0 64194 36 0 0 20 0 3 0 333449724 371920896 54492 33554432000 4194304 5964478 140735177107904 140328940210216 4400115 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=363204 CPUtime=642.26 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 26726 0 0 0 64187 39 0 0 20 0 3 0 333449724 371920896 54492 33554432000 4194304 5964478 140735177107904 140328929720504 4399920 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1284.36
Current children cumulated vsize (KiB) 363204

[startup+702.3 s]
/proc/loadavg: 4.00 4.00 3.94 5/181 924
/proc/meminfo: memFree=28504220/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=363384 CPUtime=1404.33 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 55108 0 0 0 140374 59 0 0 20 0 3 0 333449724 372105216 56639 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 90846 56639 207 433 0 90410 0
[pid=916/tid=917] ppid=914 vsize=363384 CPUtime=702.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 27615 0 0 0 70190 39 0 0 20 0 3 0 333449724 372105216 56639 33554432000 4194304 5964478 140735177107904 140328940209976 4400557 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=363384 CPUtime=702.26 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 26727 0 0 0 70186 40 0 0 20 0 3 0 333449724 372105216 56639 33554432000 4194304 5964478 140735177107904 140328929720504 4400056 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1404.33
Current children cumulated vsize (KiB) 363384

[startup+762.3 s]
/proc/loadavg: 4.00 4.00 3.94 5/182 936
/proc/meminfo: memFree=28409976/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=339488 CPUtime=1524.31 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 55212 0 0 0 152372 59 0 0 20 0 3 0 333449724 347635712 51823 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 84872 51823 209 433 0 84436 0
[pid=916/tid=917] ppid=914 vsize=339488 CPUtime=762.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 27683 0 0 0 76188 41 0 0 20 0 3 0 333449724 347635712 51823 33554432000 4194304 5964478 140735177107904 140328940211128 4400078 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=339488 CPUtime=762.27 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 26763 0 0 0 76186 41 0 0 20 0 3 0 333449724 347635712 51823 33554432000 4194304 5964478 140735177107904 140328929720680 4399920 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1524.31
Current children cumulated vsize (KiB) 339488

[startup+822.3 s]
/proc/loadavg: 3.92 3.97 3.93 5/181 979
/proc/meminfo: memFree=29514360/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=340108 CPUtime=1644.28 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 55348 0 0 0 164369 59 0 0 20 0 3 0 333449724 348270592 51985 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 85027 51985 209 433 0 84591 0
[pid=916/tid=917] ppid=914 vsize=340108 CPUtime=822.29 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 27683 0 0 0 82187 42 0 0 20 0 3 0 333449724 348270592 51985 33554432000 4194304 5964478 140735177107904 140328940211128 4400557 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=340108 CPUtime=822.25 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 26899 0 0 0 82183 42 0 0 20 0 3 0 333449724 348270592 51985 33554432000 4194304 5964478 140735177107904 140328929720680 4400115 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1644.28
Current children cumulated vsize (KiB) 340108

[startup+882.307 s]
/proc/loadavg: 4.02 3.99 3.93 5/181 979
/proc/meminfo: memFree=29489556/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=340500 CPUtime=1764.27 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 55404 0 0 0 176368 59 0 0 20 0 3 0 333449724 348672000 52041 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 85125 52041 209 433 0 84689 0
[pid=916/tid=917] ppid=914 vsize=340500 CPUtime=882.3 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 27683 0 0 0 88187 43 0 0 20 0 3 0 333449724 348672000 52041 33554432000 4194304 5964478 140735177107904 140328940211128 4399920 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=340500 CPUtime=882.26 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 26955 0 0 0 88183 43 0 0 20 0 3 0 333449724 348672000 52041 33554432000 4194304 5964478 140735177107904 140328929720680 4399938 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1764.27
Current children cumulated vsize (KiB) 340500



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+900.2 s]
/proc/loadavg: 4.01 3.99 3.93 5/181 979
/proc/meminfo: memFree=29486952/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=340608 CPUtime=1800.06 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 55413 0 0 0 179947 59 0 0 20 0 3 0 333449724 348782592 52050 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 85152 52050 209 433 0 84716 0
[pid=916/tid=917] ppid=914 vsize=340608 CPUtime=900.2 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 27683 0 0 0 89977 43 0 0 20 0 3 0 333449724 348782592 52050 33554432000 4194304 5964478 140735177107904 140328940211128 4399920 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=340608 CPUtime=900.16 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 26964 0 0 0 89973 43 0 0 20 0 3 0 333449724 348782592 52050 33554432000 4194304 5964478 140735177107904 140328929720680 4400091 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1800.06
Current children cumulated vsize (KiB) 340608

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+900.2 s]
/proc/loadavg: 4.01 3.99 3.93 5/181 979
/proc/meminfo: memFree=29486952/32873648 swapFree=7116/7116
[pid=916] ppid=914 vsize=340608 CPUtime=1800.06 cores=0,2,4,6
/proc/916/stat : 916 (pwbo2.02) S 914 916 32611 0 -1 4202496 55413 0 0 0 179947 59 0 0 20 0 3 0 333449724 348782592 52050 33554432000 4194304 5964478 140735177107904 140735177107424 4546685 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/916/statm: 85152 52050 209 433 0 84716 0
[pid=916/tid=917] ppid=914 vsize=340608 CPUtime=900.2 cores=0,2,4,6
/proc/916/task/917/stat : 917 (pwbo2.02) R 914 916 32611 0 -1 4202560 27683 0 0 0 89977 43 0 0 20 0 3 0 333449724 348782592 52050 33554432000 4194304 5964478 140735177107904 140328940211128 4399920 0 0 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=916/tid=918] ppid=914 vsize=340608 CPUtime=900.16 cores=0,2,4,6
/proc/916/task/918/stat : 918 (pwbo2.02) R 914 916 32611 0 -1 4202560 26964 0 0 0 89973 43 0 0 20 0 3 0 333449724 348782592 52050 33554432000 4194304 5964478 140735177107904 140328929720680 4400091 0 0 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1800.06
Current children cumulated vsize (KiB) 340608

Child status: 0
Real time (s): 900.215
CPU time (s): 1800.41
CPU user time (s): 1799.52
CPU system time (s): 0.892864
CPU usage (%): 199.998
Max. virtual memory (cumulated for all children) (KiB): 363528

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1799.52
system time used= 0.892864
maximum resident set size= 226836
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 55414
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 230
involuntary context switches= 1961

runsolver used 1.77173 second user time and 5.14322 second system time

The end

Launcher Data 

Begin job on node127 at 2012-06-02 23:40:43
IDJOB=3728789
IDBENCH=79152
IDSOLVER=2316
FILE ID=node127/3728789-1338673243
RUNJOBID= node127-1338669632-503
PBS_JOBID= 14636489
Free space on /tmp= 70480 MiB

SOLVER NAME= pwbo 2.02
BENCH NAME= PB10/normalized-PB10/SOFT-SMALLINT-LIN/PB06/submitted-PB05/aloul/FPGA_SAT05/normalized-chnl10_11_pb.cnf.cr--soft-0-100-0.wbo
COMMAND LINE= pwbo2.02 -file-format=wbo -verbosity=0 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3728789-1338673243/watcher-3728789-1338673243 -o /tmp/evaluation-result-3728789-1338673243/solver-3728789-1338673243 -C 1800 -W 1900 -M 15500  pwbo2.02 -file-format=wbo -verbosity=0 HOME/instance-3728789-1338673243.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 9e134c91eb84e264e2a3479232b34054
RANDOM SEED=429246248

node127.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.94
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.348
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        29493016 kB
Buffers:          379572 kB
Cached:          1826380 kB
SwapCached:         1500 kB
Active:          1276588 kB
Inactive:        1223856 kB
Active(anon):     288504 kB
Inactive(anon):     7932 kB
Active(file):     988084 kB
Inactive(file):  1215924 kB
Unevictable:        7116 kB
Mlocked:            7116 kB
SwapTotal:      67108856 kB
SwapFree:       67104404 kB
Dirty:              5468 kB
Writeback:             0 kB
AnonPages:        299696 kB
Mapped:            11476 kB
Shmem:                92 kB
Slab:             744488 kB
SReclaimable:     188720 kB
SUnreclaim:       555768 kB
KernelStack:        1488 kB
PageTables:         4656 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:     159460 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70472 MiB
End job on node127 at 2012-06-02 23:55:45