Trace number 3731664

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
PB10: pb_cplex 2010-06-29? (TO) 1800.36 578.416

General information on the benchmark

Namenormalized-PB06/OPT-SMALLINT/submitted-PB06/liu/
wnq/normalized-wnq_n100_mw99_10.opb.PB06.opb
MD5SUMfad34197fa8b5ebd519d12750a352ad5
Bench CategoryOPT-SMALLINT (optimisation, small integers)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark668
Best CPU time to get the best result obtained on this benchmark1800.2
Has Objective FunctionYES
SatisfiableYES
(Un)Satisfiability was provedYES
Best value of the objective function 480
Optimality of the best value was proved NO
Number of variables10000
Total number of constraints656900
Number of constraints which are clauses656900
Number of constraints which are cardinality constraints (but not clauses)0
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint2
Maximum length of a constraint100
Number of terms in the objective function 10000
Biggest coefficient in the objective function 99
Number of bits for the biggest coefficient in the objective function 7
Sum of the numbers in the objective function 495457
Number of bits of the sum of numbers in the objective function 19
Biggest number in a constraint 99
Number of bits of the biggest number in a constraint 7
Biggest sum of numbers in a constraint 495457
Number of bits of the biggest sum of numbers19
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing opb file format. File HOME/instance-3731664-1339246936.opb.
0.00/0.00	c Instance file HOME/instance-3731664-1339246936.opb
0.00/0.00	c File size is 17233664 bytes.
11.79/11.84	c Highest Coefficient sum: 101
11.79/11.84	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
12.09/12.18	c ============================[ Problem Statistics ]=============================
12.09/12.18	c |                                                                             |
12.09/12.18	c |  Parsing time:          12.07        s                                      |
12.09/12.18	c |  Number Variables:      10000                                               |
12.19/12.28	c |  Number Constraints:    656900                                              |
12.19/12.28	c ===============================================================================

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3731664-1339246936/watcher-3731664-1339246936 -o /tmp/evaluation-result-3731664-1339246936/solver-3731664-1339246936 -C 1800 -W 1900 -M 15500 run HOME/instance-3731664-1339246936.opb 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.23 0.75 0.26 3/175 27220
/proc/meminfo: memFree=28241832/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0

[startup+0.036698 s]
/proc/loadavg: 2.23 0.75 0.26 3/175 27220
/proc/meminfo: memFree=28241832/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.100209 s]
/proc/loadavg: 2.23 0.75 0.26 3/175 27220
/proc/meminfo: memFree=28241832/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.300179 s]
/proc/loadavg: 2.23 0.75 0.26 3/175 27220
/proc/meminfo: memFree=28241832/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.700177 s]
/proc/loadavg: 2.23 0.75 0.26 3/175 27220
/proc/meminfo: memFree=28241832/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+1.50021 s]
/proc/loadavg: 2.23 0.75 0.26 3/177 27222
/proc/meminfo: memFree=28239692/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=32000 CPUtime=1.49 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 974 0 0 0 149 0 0 0 20 0 1 0 390822229 32768000 930 33554432000 134512640 146736320 4289304048 4289261868 140528657 0 0 4096 16387 0 0 0 17 0 0 0 0 0 0
/proc/27221/statm: 8000 930 308 2985 0 726 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 41148

[startup+3.10021 s]
/proc/loadavg: 2.23 0.75 0.26 3/177 27222
/proc/meminfo: memFree=28238560/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=32132 CPUtime=3.09 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 1009 0 0 0 309 0 0 0 20 0 1 0 390822229 32903168 965 33554432000 134512640 146736320 4289304048 4289262000 140683279 0 0 4096 16387 0 0 0 17 0 0 0 0 0 0
/proc/27221/statm: 8033 965 308 2985 0 759 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 41280

[startup+6.30027 s]
/proc/loadavg: 2.21 0.77 0.27 3/177 27222
/proc/meminfo: memFree=28330020/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=32264 CPUtime=6.29 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 1043 0 0 0 629 0 0 0 20 0 1 0 390822229 33038336 999 33554432000 134512640 146736320 4289304048 4289261872 140528792 0 0 4096 16387 0 0 0 17 2 0 0 0 0 0
/proc/27221/statm: 8066 999 308 2985 0 792 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 41412

[startup+12.7003 s]
/proc/loadavg: 2.19 0.79 0.28 3/177 27222
/proc/meminfo: memFree=28062944/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=307764 CPUtime=12.69 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 66883 0 0 0 1255 14 0 0 20 0 1 0 390822229 315150336 70796 33554432000 134512640 146736320 4289304048 4289302120 138686278 0 0 4096 16387 0 0 0 17 2 0 0 0 0 0
/proc/27221/statm: 76941 70796 621 2985 0 73875 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 316912

[startup+25.5004 s]
/proc/loadavg: 2.66 0.96 0.34 10/184 27243
/proc/meminfo: memFree=27969152/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=396472 CPUtime=43.58 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) S 27220 27220 26837 0 -1 4202496 119954 0 0 0 4317 41 0 0 20 0 8 0 390822229 405987328 70188 33554432000 134512640 146736320 4289304048 4289300432 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/27221/statm: 99118 70188 960 2985 0 96052 0
[pid=27221/tid=27237] ppid=27220 vsize=396472 CPUtime=2.46 cores=0,2,4,6
/proc/27221/task/27237/stat : 27237 (pb_cplex) S 27220 27220 26837 0 -1 4202560 1383 0 0 0 246 0 0 0 20 0 8 0 390823984 405987328 70188 33554432000 134512640 146736320 4289304048 3988344464 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27238] ppid=27220 vsize=396472 CPUtime=4.48 cores=0,2,4,6
/proc/27221/task/27238/stat : 27238 (pb_cplex) S 27220 27220 26837 0 -1 4202560 1753 0 0 0 446 2 0 0 20 0 8 0 390823984 405987328 70188 33554432000 134512640 146736320 4289304048 4075749008 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27239] ppid=27220 vsize=396472 CPUtime=1.86 cores=0,2,4,6
/proc/27221/task/27239/stat : 27239 (pb_cplex) R 27220 27220 26837 0 -1 4202560 1237 0 0 0 186 0 0 0 20 0 8 0 390823984 405987328 70188 33554432000 134512640 146736320 4289304048 4093701124 136385175 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27240] ppid=27220 vsize=396472 CPUtime=5.72 cores=0,2,4,6
/proc/27221/task/27240/stat : 27240 (pb_cplex) R 27220 27220 26837 0 -1 4202560 1317 0 0 0 572 0 0 0 20 0 8 0 390823984 405987328 70188 33554432000 134512640 146736320 4289304048 4052308024 136475775 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27241] ppid=27220 vsize=396472 CPUtime=3.05 cores=0,2,4,6
/proc/27221/task/27241/stat : 27241 (pb_cplex) S 27220 27220 26837 0 -1 4202560 1455 0 0 0 305 0 0 0 20 0 8 0 390823984 405987328 70188 33554432000 134512640 146736320 4289304048 4019801744 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27242] ppid=27220 vsize=396472 CPUtime=1.87 cores=0,2,4,6
/proc/27221/task/27242/stat : 27242 (pb_cplex) S 27220 27220 26837 0 -1 4202560 1209 0 0 0 187 0 0 0 20 0 8 0 390823984 405987328 70188 33554432000 134512640 146736320 4289304048 4009315984 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27243] ppid=27220 vsize=396472 CPUtime=2.94 cores=0,2,4,6
/proc/27221/task/27243/stat : 27243 (pb_cplex) S 27220 27220 26837 0 -1 4202560 1466 0 0 0 294 0 0 0 20 0 8 0 390823984 405987328 70188 33554432000 134512640 146736320 4289304048 3998830224 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 43.58
Current children cumulated vsize (KiB) 405620

[startup+51.1004 s]
/proc/loadavg: 3.58 1.29 0.47 10/184 27292
/proc/meminfo: memFree=27643244/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=782060 CPUtime=102.25 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 180439 0 0 0 10049 176 0 0 20 0 8 0 390822229 800829440 152220 33554432000 134512640 146736320 4289304048 4289297704 144757825 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/27221/statm: 195515 152220 977 2985 0 192449 0
[pid=27221/tid=27286] ppid=27220 vsize=782060 CPUtime=4.64 cores=0,2,4,6
/proc/27221/task/27286/stat : 27286 (pb_cplex) R 27220 27220 26837 0 -1 4202560 2739 0 0 0 462 2 0 0 20 0 8 0 390826224 800829440 152220 33554432000 134512640 146736320 4289304048 3913554000 135460433 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27287] ppid=27220 vsize=782060 CPUtime=3.44 cores=0,2,4,6
/proc/27221/task/27287/stat : 27287 (pb_cplex) R 27220 27220 26837 0 -1 4202560 1215 0 0 0 342 2 0 0 20 0 8 0 390826224 800829440 152220 33554432000 134512640 146736320 4289304048 3924041948 136009768 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27288] ppid=27220 vsize=782060 CPUtime=4.28 cores=0,2,4,6
/proc/27221/task/27288/stat : 27288 (pb_cplex) R 27220 27220 26837 0 -1 4202560 36 0 0 0 427 1 0 0 20 0 8 0 390826224 800829440 152220 33554432000 134512640 146736320 4289304048 3934527580 135855117 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27289] ppid=27220 vsize=782060 CPUtime=3.91 cores=0,2,4,6
/proc/27221/task/27289/stat : 27289 (pb_cplex) R 27220 27220 26837 0 -1 4202560 695 0 0 0 391 0 0 0 20 0 8 0 390826224 800829440 152220 33554432000 134512640 146736320 4289304048 4052307336 136000767 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27290] ppid=27220 vsize=782060 CPUtime=4.77 cores=0,2,4,6
/proc/27221/task/27290/stat : 27290 (pb_cplex) R 27220 27220 26837 0 -1 4202560 1243 0 0 0 477 0 0 0 20 0 8 0 390826224 800829440 152220 33554432000 134512640 146736320 4289304048 3593699520 136471165 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27291] ppid=27220 vsize=782060 CPUtime=4.98 cores=0,2,4,6
/proc/27221/task/27291/stat : 27291 (pb_cplex) R 27220 27220 26837 0 -1 4202560 1203 0 0 0 498 0 0 0 20 0 8 0 390826224 800829440 152220 33554432000 134512640 146736320 4289304048 3583213780 135846438 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27292] ppid=27220 vsize=782060 CPUtime=3.75 cores=0,2,4,6
/proc/27221/task/27292/stat : 27292 (pb_cplex) R 27220 27220 26837 0 -1 4202560 766 0 0 0 373 2 0 0 20 0 8 0 390826224 800829440 152220 33554432000 134512640 146736320 4289304048 3572728808 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 102.25
Current children cumulated vsize (KiB) 791208

[startup+102.306 s]
/proc/loadavg: 5.19 2.14 0.80 3/184 27292
/proc/meminfo: memFree=27563388/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=828140 CPUtime=265.49 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 368528 0 0 0 26212 337 0 0 20 0 8 0 390822229 848015360 163919 33554432000 134512640 146736320 4289304048 4289297704 144757825 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/27221/statm: 207035 163919 977 2985 0 203969 0
[pid=27221/tid=27286] ppid=27220 vsize=828140 CPUtime=25.38 cores=0,2,4,6
/proc/27221/task/27286/stat : 27286 (pb_cplex) R 27220 27220 26837 0 -1 4202560 25650 0 0 0 2517 21 0 0 20 0 8 0 390826224 848015360 163919 33554432000 134512640 146736320 4289304048 3913556036 136385184 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27287] ppid=27220 vsize=828140 CPUtime=22.72 cores=0,2,4,6
/proc/27221/task/27287/stat : 27287 (pb_cplex) R 27220 27220 26837 0 -1 4202560 24443 0 0 0 2248 24 0 0 20 0 8 0 390826224 848015360 163919 33554432000 134512640 146736320 4289304048 3924042320 136403112 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27288] ppid=27220 vsize=828140 CPUtime=24.71 cores=0,2,4,6
/proc/27221/task/27288/stat : 27288 (pb_cplex) R 27220 27220 26837 0 -1 4202560 13028 0 0 0 2457 14 0 0 20 0 8 0 390826224 848015360 163919 33554432000 134512640 146736320 4289304048 3934528388 135850301 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27289] ppid=27220 vsize=828140 CPUtime=24.76 cores=0,2,4,6
/proc/27221/task/27289/stat : 27289 (pb_cplex) R 27220 27220 26837 0 -1 4202560 27336 0 0 0 2456 20 0 0 20 0 8 0 390826224 848015360 163919 33554432000 134512640 146736320 4289304048 4052308968 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27290] ppid=27220 vsize=828140 CPUtime=25.21 cores=0,2,4,6
/proc/27221/task/27290/stat : 27290 (pb_cplex) R 27220 27220 26837 0 -1 4202560 33434 0 0 0 2498 23 0 0 20 0 8 0 390826224 848015360 163919 33554432000 134512640 146736320 4289304048 3593699536 136008065 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27291] ppid=27220 vsize=828140 CPUtime=25.48 cores=0,2,4,6
/proc/27221/task/27291/stat : 27291 (pb_cplex) R 27220 27220 26837 0 -1 4202560 31101 0 0 0 2520 28 0 0 20 0 8 0 390826224 848015360 163919 33554432000 134512640 146736320 4289304048 3583214568 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27292] ppid=27220 vsize=828140 CPUtime=24.84 cores=0,2,4,6
/proc/27221/task/27292/stat : 27292 (pb_cplex) R 27220 27220 26837 0 -1 4202560 7102 0 0 0 2465 19 0 0 20 0 8 0 390826224 848015360 163919 33554432000 134512640 146736320 4289304048 3572728708 135850322 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 265.49
Current children cumulated vsize (KiB) 837288

[startup+162.3 s]
/proc/loadavg: 5.90 2.95 1.16 10/180 27328
/proc/meminfo: memFree=29037724/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=798400 CPUtime=464.63 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 657838 0 0 0 45960 503 0 0 20 0 8 0 390822229 817561600 166552 33554432000 134512640 146736320 4289304048 4289296616 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/27221/statm: 199600 166552 1000 2985 0 196534 0
[pid=27221/tid=27315] ppid=27220 vsize=798400 CPUtime=2.85 cores=0,2,4,6

################
# More data... #
################

/proc/27221/statm: 242183 207669 1002 2985 0 239117 0
[pid=27221/tid=27315] ppid=27220 vsize=968732 CPUtime=54.55 cores=0,2,4,6
/proc/27221/task/27315/stat : 27315 (pb_cplex) R 27220 27220 26837 0 -1 4202560 129958 0 0 0 5399 56 0 0 20 0 8 0 390837685 991981568 207669 33554432000 134512640 146736320 4289304048 3940641928 137416408 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27316] ppid=27220 vsize=968732 CPUtime=54.36 cores=0,2,4,6
/proc/27221/task/27316/stat : 27316 (pb_cplex) S 27220 27220 26837 0 -1 4202560 135673 0 0 0 5373 63 0 0 20 0 8 0 390837685 991981568 207669 33554432000 134512640 146736320 4289304048 3951132304 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27317] ppid=27220 vsize=968732 CPUtime=54.33 cores=0,2,4,6
/proc/27221/task/27317/stat : 27317 (pb_cplex) S 27220 27220 26837 0 -1 4202560 103025 0 0 0 5371 62 0 0 20 0 8 0 390837685 991981568 207669 33554432000 134512640 146736320 4289304048 4016041616 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27318] ppid=27220 vsize=968732 CPUtime=52.75 cores=0,2,4,6
/proc/27221/task/27318/stat : 27318 (pb_cplex) S 27220 27220 26837 0 -1 4202560 102945 0 0 0 5227 48 0 0 20 0 8 0 390837685 991981568 207669 33554432000 134512640 146736320 4289304048 4052311696 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27319] ppid=27220 vsize=968732 CPUtime=54.49 cores=0,2,4,6
/proc/27221/task/27319/stat : 27319 (pb_cplex) S 27220 27220 26837 0 -1 4202560 133528 0 0 0 5394 55 0 0 20 0 8 0 390837685 991981568 207669 33554432000 134512640 146736320 4289304048 4111629968 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27320] ppid=27220 vsize=968732 CPUtime=52.14 cores=0,2,4,6
/proc/27221/task/27320/stat : 27320 (pb_cplex) S 27220 27220 26837 0 -1 4202560 140409 0 0 0 5155 59 0 0 20 0 8 0 390837685 991981568 207669 33554432000 134512640 146736320 4289304048 4093705872 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27321] ppid=27220 vsize=968732 CPUtime=54.47 cores=0,2,4,6
/proc/27221/task/27321/stat : 27321 (pb_cplex) S 27220 27220 26837 0 -1 4202560 91774 0 0 0 5401 46 0 0 20 0 8 0 390837685 991981568 207669 33554432000 134512640 146736320 4289304048 4075749008 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 872.83
Current children cumulated vsize (KiB) 977880

[startup+342.3 s]
/proc/loadavg: 11.63 6.75 2.93 16/191 27384
/proc/meminfo: memFree=28743080/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=1034864 CPUtime=1052.54 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 1833803 0 0 0 104174 1080 0 0 20 0 8 0 390822229 1059700736 225485 33554432000 134512640 146736320 4289304048 4289296760 136475859 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/27221/statm: 258716 225485 1002 2985 0 255650 0
[pid=27221/tid=27315] ppid=27220 vsize=1034864 CPUtime=76.29 cores=0,2,4,6
/proc/27221/task/27315/stat : 27315 (pb_cplex) R 27220 27220 26837 0 -1 4202560 144070 0 0 0 7556 73 0 0 20 0 8 0 390837685 1059700736 225485 33554432000 134512640 146736320 4289304048 3940643816 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27316] ppid=27220 vsize=1034864 CPUtime=77.06 cores=0,2,4,6
/proc/27221/task/27316/stat : 27316 (pb_cplex) R 27220 27220 26837 0 -1 4202560 175016 0 0 0 7620 86 0 0 20 0 8 0 390837685 1059700736 225485 33554432000 134512640 146736320 4289304048 3951128632 136475785 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27317] ppid=27220 vsize=1034864 CPUtime=76.93 cores=0,2,4,6
/proc/27221/task/27317/stat : 27317 (pb_cplex) R 27220 27220 26837 0 -1 4202560 111595 0 0 0 7616 77 0 0 20 0 8 0 390837685 1059700736 225485 33554432000 134512640 146736320 4289304048 4016037052 135526845 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27318] ppid=27220 vsize=1034864 CPUtime=75.14 cores=0,2,4,6
/proc/27221/task/27318/stat : 27318 (pb_cplex) R 27220 27220 26837 0 -1 4202560 121724 0 0 0 7449 65 0 0 20 0 8 0 390837685 1059700736 225485 33554432000 134512640 146736320 4289304048 4052308060 135855162 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27319] ppid=27220 vsize=1034864 CPUtime=76.78 cores=0,2,4,6
/proc/27221/task/27319/stat : 27319 (pb_cplex) R 27220 27220 26837 0 -1 4202560 161510 0 0 0 7602 76 0 0 20 0 8 0 390837685 1059700736 225485 33554432000 134512640 146736320 4289304048 4111626448 136008035 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27320] ppid=27220 vsize=1034864 CPUtime=74.22 cores=0,2,4,6
/proc/27221/task/27320/stat : 27320 (pb_cplex) R 27220 27220 26837 0 -1 4202560 154881 0 0 0 7346 76 0 0 20 0 8 0 390837685 1059700736 225485 33554432000 134512640 146736320 4289304048 4093702236 135855165 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27321] ppid=27220 vsize=1034864 CPUtime=77.13 cores=0,2,4,6
/proc/27221/task/27321/stat : 27321 (pb_cplex) R 27220 27220 26837 0 -1 4202560 123827 0 0 0 7648 65 0 0 20 0 8 0 390837685 1059700736 225485 33554432000 134512640 146736320 4289304048 4075746280 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1052.54
Current children cumulated vsize (KiB) 1044012

[startup+402.3 s]
/proc/loadavg: 12.06 7.72 3.50 12/191 27384
/proc/meminfo: memFree=28670780/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=1102884 CPUtime=1245.14 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) S 27220 27220 26837 0 -1 4202496 1986539 0 0 0 123282 1232 0 0 20 0 8 0 390822229 1129353216 243028 33554432000 134512640 146736320 4289304048 4289300432 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/27221/statm: 275721 243028 1002 2985 0 272655 0
[pid=27221/tid=27315] ppid=27220 vsize=1102884 CPUtime=99.83 cores=0,2,4,6
/proc/27221/task/27315/stat : 27315 (pb_cplex) S 27220 27220 26837 0 -1 4202560 148317 0 0 0 9894 89 0 0 20 0 8 0 390837685 1129353216 243028 33554432000 134512640 146736320 4289304048 3940646544 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27316] ppid=27220 vsize=1102884 CPUtime=100.34 cores=0,2,4,6
/proc/27221/task/27316/stat : 27316 (pb_cplex) S 27220 27220 26837 0 -1 4202560 182142 0 0 0 9930 104 0 0 20 0 8 0 390837685 1129353216 243028 33554432000 134512640 146736320 4289304048 3951132304 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27317] ppid=27220 vsize=1102884 CPUtime=100.89 cores=0,2,4,6
/proc/27221/task/27317/stat : 27317 (pb_cplex) S 27220 27220 26837 0 -1 4202560 129945 0 0 0 9991 98 0 0 20 0 8 0 390837685 1129353216 243028 33554432000 134512640 146736320 4289304048 4016041616 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27318] ppid=27220 vsize=1102884 CPUtime=100.02 cores=0,2,4,6
/proc/27221/task/27318/stat : 27318 (pb_cplex) R 27220 27220 26837 0 -1 4202560 138506 0 0 0 9917 85 0 0 20 0 8 0 390837685 1129353216 243028 33554432000 134512640 146736320 4289304048 4052308024 136475827 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27319] ppid=27220 vsize=1102884 CPUtime=99.85 cores=0,2,4,6
/proc/27221/task/27319/stat : 27319 (pb_cplex) S 27220 27220 26837 0 -1 4202560 177264 0 0 0 9890 95 0 0 20 0 8 0 390837685 1129353216 243028 33554432000 134512640 146736320 4289304048 4111629968 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27320] ppid=27220 vsize=1102884 CPUtime=98.5 cores=0,2,4,6
/proc/27221/task/27320/stat : 27320 (pb_cplex) S 27220 27220 26837 0 -1 4202560 161557 0 0 0 9759 91 0 0 20 0 8 0 390837685 1129353216 243028 33554432000 134512640 146736320 4289304048 4093705872 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27321] ppid=27220 vsize=1102884 CPUtime=100.63 cores=0,2,4,6
/proc/27221/task/27321/stat : 27321 (pb_cplex) S 27220 27220 26837 0 -1 4202560 152418 0 0 0 9978 85 0 0 20 0 8 0 390837685 1129353216 243028 33554432000 134512640 146736320 4289304048 4075749008 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1245.14
Current children cumulated vsize (KiB) 1112032

[startup+462.3 s]
/proc/loadavg: 12.42 8.63 4.08 11/191 27385
/proc/meminfo: memFree=28590480/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=1175888 CPUtime=1438.51 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) S 27220 27220 26837 0 -1 4202496 2101719 0 0 0 142477 1374 0 0 20 0 8 0 390822229 1204109312 262823 33554432000 134512640 146736320 4289304048 4289300432 144580073 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/27221/statm: 293972 262823 1002 2985 0 290906 0
[pid=27221/tid=27315] ppid=27220 vsize=1175888 CPUtime=124.14 cores=0,2,4,6
/proc/27221/task/27315/stat : 27315 (pb_cplex) S 27220 27220 26837 0 -1 4202560 158221 0 0 0 12306 108 0 0 20 0 8 0 390837685 1204109312 262823 33554432000 134512640 146736320 4289304048 3940646544 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27316] ppid=27220 vsize=1175888 CPUtime=124.7 cores=0,2,4,6
/proc/27221/task/27316/stat : 27316 (pb_cplex) S 27220 27220 26837 0 -1 4202560 199135 0 0 0 12348 122 0 0 20 0 8 0 390837685 1204109312 262823 33554432000 134512640 146736320 4289304048 3951132304 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27317] ppid=27220 vsize=1175888 CPUtime=124.08 cores=0,2,4,6
/proc/27221/task/27317/stat : 27317 (pb_cplex) S 27220 27220 26837 0 -1 4202560 133246 0 0 0 12294 114 0 0 20 0 8 0 390837685 1204109312 262823 33554432000 134512640 146736320 4289304048 4016041616 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27318] ppid=27220 vsize=1175888 CPUtime=123.75 cores=0,2,4,6
/proc/27221/task/27318/stat : 27318 (pb_cplex) S 27220 27220 26837 0 -1 4202560 142916 0 0 0 12272 103 0 0 20 0 8 0 390837685 1204109312 262823 33554432000 134512640 146736320 4289304048 4052311696 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27319] ppid=27220 vsize=1175888 CPUtime=124.15 cores=0,2,4,6
/proc/27221/task/27319/stat : 27319 (pb_cplex) R 27220 27220 26837 0 -1 4202560 194950 0 0 0 12303 112 0 0 20 0 8 0 390837685 1204109312 262823 33554432000 134512640 146736320 4289304048 4111628360 137021756 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27320] ppid=27220 vsize=1175888 CPUtime=122.09 cores=0,2,4,6
/proc/27221/task/27320/stat : 27320 (pb_cplex) S 27220 27220 26837 0 -1 4202560 172266 0 0 0 12099 110 0 0 20 0 8 0 390837685 1204109312 262823 33554432000 134512640 146736320 4289304048 4093705872 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27321] ppid=27220 vsize=1175888 CPUtime=124.74 cores=0,2,4,6
/proc/27221/task/27321/stat : 27321 (pb_cplex) R 27220 27220 26837 0 -1 4202560 155927 0 0 0 12376 98 0 0 20 0 8 0 390837685 1204109312 262823 33554432000 134512640 146736320 4289304048 4075746280 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 1438.51
Current children cumulated vsize (KiB) 1185036

[startup+522.3 s]
/proc/loadavg: 12.75 9.41 4.63 16/192 27388
/proc/meminfo: memFree=28471060/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=1236236 CPUtime=1618.82 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 2197634 0 0 0 160373 1509 0 0 20 0 8 0 390822229 1265905664 278742 33554432000 134512640 146736320 4289304048 4289296744 139044510 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/27221/statm: 309059 278742 1002 2985 0 305993 0
[pid=27221/tid=27315] ppid=27220 vsize=1236236 CPUtime=147.58 cores=0,2,4,6
/proc/27221/task/27315/stat : 27315 (pb_cplex) R 27220 27220 26837 0 -1 4202560 164662 0 0 0 14632 126 0 0 20 0 8 0 390837685 1265905664 278742 33554432000 134512640 146736320 4289304048 3940643036 136389120 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=27221/tid=27316] ppid=27220 vsize=1236236 CPUtime=146.76 cores=0,2,4,6
/proc/27221/task/27316/stat : 27316 (pb_cplex) R 27220 27220 26837 0 -1 4202560 211319 0 0 0 14537 139 0 0 20 0 8 0 390837685 1265905664 278742 33554432000 134512640 146736320 4289304048 3951127740 135453275 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27317] ppid=27220 vsize=1236236 CPUtime=146.03 cores=0,2,4,6
/proc/27221/task/27317/stat : 27317 (pb_cplex) R 27220 27220 26837 0 -1 4202560 134799 0 0 0 14474 129 0 0 20 0 8 0 390837685 1265905664 278742 33554432000 134512640 146736320 4289304048 4016038788 135851007 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27318] ppid=27220 vsize=1236236 CPUtime=146.85 cores=0,2,4,6
/proc/27221/task/27318/stat : 27318 (pb_cplex) R 27220 27220 26837 0 -1 4202560 146013 0 0 0 14567 118 0 0 20 0 8 0 390837685 1265905664 278742 33554432000 134512640 146736320 4289304048 4052307968 136018936 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27319] ppid=27220 vsize=1236236 CPUtime=145.75 cores=0,2,4,6
/proc/27221/task/27319/stat : 27319 (pb_cplex) R 27220 27220 26837 0 -1 4202560 204014 0 0 0 14444 131 0 0 20 0 8 0 390837685 1265905664 278742 33554432000 134512640 146736320 4289304048 4111627240 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27320] ppid=27220 vsize=1236236 CPUtime=144.62 cores=0,2,4,6
/proc/27221/task/27320/stat : 27320 (pb_cplex) R 27220 27220 26837 0 -1 4202560 182674 0 0 0 14334 128 0 0 20 0 8 0 390837685 1265905664 278742 33554432000 134512640 146736320 4289304048 4093702216 139043348 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27321] ppid=27220 vsize=1236236 CPUtime=145.41 cores=0,2,4,6
/proc/27221/task/27321/stat : 27321 (pb_cplex) R 27220 27220 26837 0 -1 4202560 165626 0 0 0 14424 117 0 0 20 0 8 0 390837685 1265905664 278742 33554432000 134512640 146736320 4289304048 4075746280 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 1618.82
Current children cumulated vsize (KiB) 1245384



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+578.4 s]
/proc/loadavg: 12.16 9.81 5.04 17/191 27388
/proc/meminfo: memFree=28406196/32873844 swapFree=7120/7120
[pid=27220] ppid=27218 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/27220/stat : 27220 (run) S 27218 27220 26837 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 390822229 9367552 270 33554432000 4194304 5064252 140736987877776 140736987876688 235374536270 0 65536 4100 65538 18446744071579302229 0 0 17 2 0 0 0 0 0
/proc/27220/statm: 2287 270 229 213 0 72 0
[pid=27221] ppid=27220 vsize=1300204 CPUtime=1800.36 cores=0,2,4,6
/proc/27221/stat : 27221 (pb_cplex) R 27220 27220 26837 0 -1 4202496 2386082 0 0 0 178379 1657 0 0 20 0 8 0 390822229 1331408896 295232 33554432000 134512640 146736320 4289304048 4289297604 135850301 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/27221/statm: 325051 295232 1002 2985 0 321985 0
[pid=27221/tid=27315] ppid=27220 vsize=1300204 CPUtime=169.76 cores=0,2,4,6
/proc/27221/task/27315/stat : 27315 (pb_cplex) R 27220 27220 26837 0 -1 4202560 205523 0 0 0 16829 147 0 0 20 0 8 0 390837685 1331408896 295232 33554432000 134512640 146736320 4289304048 3940642884 136385175 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27316] ppid=27220 vsize=1300204 CPUtime=170.67 cores=0,2,4,6
/proc/27221/task/27316/stat : 27316 (pb_cplex) R 27220 27220 26837 0 -1 4202560 230281 0 0 0 16911 156 0 0 20 0 8 0 390837685 1331408896 295232 33554432000 134512640 146736320 4289304048 3951128668 135854509 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27317] ppid=27220 vsize=1300204 CPUtime=168.78 cores=0,2,4,6
/proc/27221/task/27317/stat : 27317 (pb_cplex) R 27220 27220 26837 0 -1 4202560 176556 0 0 0 16727 151 0 0 20 0 8 0 390837685 1331408896 295232 33554432000 134512640 146736320 4289304048 4016038888 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27318] ppid=27220 vsize=1300204 CPUtime=168.99 cores=0,2,4,6
/proc/27221/task/27318/stat : 27318 (pb_cplex) R 27220 27220 26837 0 -1 4202560 154541 0 0 0 16764 135 0 0 20 0 8 0 390837685 1331408896 295232 33554432000 134512640 146736320 4289304048 4052308024 136475827 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=27221/tid=27319] ppid=27220 vsize=1300204 CPUtime=168.35 cores=0,2,4,6
/proc/27221/task/27319/stat : 27319 (pb_cplex) R 27220 27220 26837 0 -1 4202560 233698 0 0 0 16684 151 0 0 20 0 8 0 390837685 1331408896 295232 33554432000 134512640 146736320 4289304048 4111626696 136000764 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=27221/tid=27320] ppid=27220 vsize=1300204 CPUtime=166.34 cores=0,2,4,6
/proc/27221/task/27320/stat : 27320 (pb_cplex) R 27220 27220 26837 0 -1 4202560 183191 0 0 0 16493 141 0 0 20 0 8 0 390837685 1331408896 295232 33554432000 134512640 146736320 4289304048 4093702352 136008167 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=27221/tid=27321] ppid=27220 vsize=1300204 CPUtime=166.99 cores=0,2,4,6
/proc/27221/task/27321/stat : 27321 (pb_cplex) R 27220 27220 26837 0 -1 4202560 165674 0 0 0 16568 131 0 0 20 0 8 0 390837685 1331408896 295232 33554432000 134512640 146736320 4289304048 4075746896 135921211 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1800.36
Current children cumulated vsize (KiB) 1309352

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 27220 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=0
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=999
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1800.36
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 578.416
CPU time (s): 1800.36
CPU user time (s): 1783.79
CPU system time (s): 16.57
CPU usage (%): 311.257
Max. virtual memory (cumulated for all children) (KiB): 1309764

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0
system time used= 0.000999
maximum resident set size= 1080
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 355
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 1

runsolver used 2.40163 second user time and 5.46717 second system time

The end

Launcher Data 

Begin job on node104 at 2012-06-09 15:02:17
IDJOB=3731664
IDBENCH=1974
IDSOLVER=2325
FILE ID=node104/3731664-1339246936
RUNJOBID= node104-1339246834-26853
PBS_JOBID= 14644377
Free space on /tmp= 69444 MiB

SOLVER NAME= PB10: pb_cplex 2010-06-29
BENCH NAME= PB06//final/normalized-PB06/OPT-SMALLINT/submitted-PB06/liu/wnq/normalized-wnq_n100_mw99_10.opb.PB06.opb
COMMAND LINE= run BENCHNAME
CONVERSION SCRIPT= PBconversionToLinear BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3731664-1339246936/watcher-3731664-1339246936 -o /tmp/evaluation-result-3731664-1339246936/solver-3731664-1339246936 -C 1800 -W 1900 -M 15500  run HOME/instance-3731664-1339246936.opb

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= fad34197fa8b5ebd519d12750a352ad5
RANDOM SEED=1256609011

node104.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.53
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.266
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        28241832 kB
Buffers:          708084 kB
Cached:          1500576 kB
SwapCached:         1864 kB
Active:          3577396 kB
Inactive:          94456 kB
Active(anon):    1459384 kB
Inactive(anon):     5824 kB
Active(file):    2118012 kB
Inactive(file):    88632 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67102148 kB
Dirty:             33368 kB
Writeback:             0 kB
AnonPages:       1470584 kB
Mapped:            12656 kB
Shmem:                 4 kB
Slab:             815592 kB
SReclaimable:     257336 kB
SUnreclaim:       558256 kB
KernelStack:        1440 kB
PageTables:         7152 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    2750204 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1159168 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

CONVERSION COMMAND LINE= PBconversionToLinear HOME/instance-3731664-1339246936.opb

Free space on /tmp at the end= 69460 MiB
End job on node104 at 2012-06-09 15:11:55