Trace number 3734931

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
PB10: pb_cplex 2010-06-29? (TO) 1800.02 1165.92

General information on the benchmark

Namenormalized-PB06/OPT-SMALLINT/submitted-PB06/
manquiho/market-split/normalized-opt-market-split_8_70_1.opb
MD5SUM0263d0a7c5e3320bd7cce9534c14bf27
Bench CategoryOPT-SMALLINT (optimisation, small integers)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark15
Best CPU time to get the best result obtained on this benchmark1792.11
Has Objective FunctionYES
SatisfiableYES
(Un)Satisfiability was provedYES
Best value of the objective function 32
Optimality of the best value was proved NO
Number of variables198
Total number of constraints16
Number of constraints which are clauses0
Number of constraints which are cardinality constraints (but not clauses)0
Number of constraints which are nor clauses,nor cardinality constraints16
Minimum length of a constraint85
Maximum length of a constraint86
Number of terms in the objective function 128
Biggest coefficient in the objective function 128
Number of bits for the biggest coefficient in the objective function 8
Sum of the numbers in the objective function 4080
Number of bits of the sum of numbers in the objective function 12
Biggest number in a constraint 1884
Number of bits of the biggest number in a constraint 11
Biggest sum of numbers in a constraint 6162
Number of bits of the biggest sum of numbers13
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing opb file format. File HOME/instance-3734931-1339247666.opb.
0.00/0.00	c Instance file HOME/instance-3734931-1339247666.opb
0.00/0.00	c File size is 13239 bytes.
0.00/0.01	c Highest Coefficient sum: 6162
0.00/0.01	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.02	c ============================[ Problem Statistics ]=============================
0.00/0.02	c |                                                                             |
0.00/0.02	c |  Parsing time:          0.01         s                                      |
0.00/0.02	c |  Number Variables:      198                                                 |
0.00/0.02	c |  Number Constraints:    16                                                  |
0.00/0.02	c ===============================================================================
1800.02/1165.91	c Remaining time 1799.99 sec.
1800.02/1165.91	c 
1800.02/1165.91	c *** INTERRUPTED ***
1800.02/1165.91	s UNKNOWN
1800.02/1165.91	c Total CPU time              : 1699.15 s
1800.02/1165.91	c Memory used           : 279.69 MB
1800.02/1165.91	c CPU time              : 1699.15 s
1800.02/1165.91	c 
1800.02/1165.91	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3734931-1339247666/watcher-3734931-1339247666 -o /tmp/evaluation-result-3734931-1339247666/solver-3734931-1339247666 -C 1800 -W 1900 -M 15500 run HOME/instance-3734931-1339247666.opb 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 12.65 11.79 7.12 10/184 20389
/proc/meminfo: memFree=30900456/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0

[startup+0.0791809 s]
/proc/loadavg: 12.65 11.79 7.12 10/184 20389
/proc/meminfo: memFree=30900456/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.100192 s]
/proc/loadavg: 12.65 11.79 7.12 10/184 20389
/proc/meminfo: memFree=30900456/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.300153 s]
/proc/loadavg: 12.65 11.79 7.12 10/184 20389
/proc/meminfo: memFree=30900456/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.700145 s]
/proc/loadavg: 12.65 11.79 7.12 10/184 20389
/proc/meminfo: memFree=30900456/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+1.50103 s]
/proc/loadavg: 12.67 11.81 7.15 16/193 20440
/proc/meminfo: memFree=30879596/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=99128 CPUtime=3.97 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) S 20389 20389 20024 0 -1 4202496 13487 0 0 0 374 23 0 0 20 0 8 0 390892586 101507072 5436 33554432000 134512640 146736320 4293789120 4293785488 144580073 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/20390/statm: 24782 5436 972 2985 0 21716 0
[pid=20390/tid=20434] ppid=20389 vsize=99128 CPUtime=0.23 cores=0,2,4,6
/proc/20390/task/20434/stat : 20434 (pb_cplex) S 20389 20389 20024 0 -1 4202560 408 0 0 0 21 2 0 0 20 0 8 0 390892657 101507072 5436 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20435] ppid=20389 vsize=99128 CPUtime=0.22 cores=0,2,4,6
/proc/20390/task/20435/stat : 20435 (pb_cplex) S 20389 20389 20024 0 -1 4202560 12 0 0 0 22 0 0 0 20 0 8 0 390892657 101507072 5436 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20436] ppid=20389 vsize=99128 CPUtime=0.23 cores=0,2,4,6
/proc/20390/task/20436/stat : 20436 (pb_cplex) S 20389 20389 20024 0 -1 4202560 122 0 0 0 23 0 0 0 20 0 8 0 390892657 101507072 5436 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20437] ppid=20389 vsize=99128 CPUtime=0.2 cores=0,2,4,6
/proc/20390/task/20437/stat : 20437 (pb_cplex) R 20389 20389 20024 0 -1 4202560 332 0 0 0 19 1 0 0 20 0 8 0 390892657 101507072 5436 33554432000 134512640 146736320 4293789120 4120060904 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20438] ppid=20389 vsize=99128 CPUtime=0.18 cores=0,2,4,6
/proc/20390/task/20438/stat : 20438 (pb_cplex) R 20389 20389 20024 0 -1 4202560 138 0 0 0 17 1 0 0 20 0 8 0 390892657 101507072 5436 33554432000 134512640 146736320 4293789120 4106217688 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20439] ppid=20389 vsize=99128 CPUtime=0.24 cores=0,2,4,6
/proc/20390/task/20439/stat : 20439 (pb_cplex) S 20389 20389 20024 0 -1 4202560 499 0 0 0 22 2 0 0 20 0 8 0 390892657 101507072 5436 33554432000 134512640 146736320 4293789120 4095733392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20440] ppid=20389 vsize=99128 CPUtime=0.23 cores=0,2,4,6
/proc/20390/task/20440/stat : 20440 (pb_cplex) R 20389 20389 20024 0 -1 4202560 330 0 0 0 21 2 0 0 20 0 8 0 390892657 101507072 5436 33554432000 134512640 146736320 4293789120 4070566508 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 3.97
Current children cumulated vsize (KiB) 108276

[startup+3.10033 s]
/proc/loadavg: 12.67 11.81 7.15 10/193 20440
/proc/meminfo: memFree=30875008/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=99128 CPUtime=6.29 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) S 20389 20389 20024 0 -1 4202496 15669 0 0 0 593 36 0 0 20 0 8 0 390892586 101507072 5747 33554432000 134512640 146736320 4293789120 4293785488 144580073 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/20390/statm: 24782 5747 972 2985 0 21716 0
[pid=20390/tid=20434] ppid=20389 vsize=99128 CPUtime=0.48 cores=0,2,4,6
/proc/20390/task/20434/stat : 20434 (pb_cplex) S 20389 20389 20024 0 -1 4202560 538 0 0 0 45 3 0 0 20 0 8 0 390892657 101507072 5747 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20435] ppid=20389 vsize=99128 CPUtime=0.53 cores=0,2,4,6
/proc/20390/task/20435/stat : 20435 (pb_cplex) S 20389 20389 20024 0 -1 4202560 339 0 0 0 50 3 0 0 20 0 8 0 390892657 101507072 5747 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20436] ppid=20389 vsize=99128 CPUtime=0.51 cores=0,2,4,6
/proc/20390/task/20436/stat : 20436 (pb_cplex) S 20389 20389 20024 0 -1 4202560 249 0 0 0 49 2 0 0 20 0 8 0 390892657 101507072 5747 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20437] ppid=20389 vsize=99128 CPUtime=0.56 cores=0,2,4,6
/proc/20390/task/20437/stat : 20437 (pb_cplex) S 20389 20389 20024 0 -1 4202560 1061 0 0 0 52 4 0 0 20 0 8 0 390892657 101507072 5747 33554432000 134512640 146736320 4293789120 4120063632 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=20390/tid=20438] ppid=20389 vsize=99128 CPUtime=0.35 cores=0,2,4,6
/proc/20390/task/20438/stat : 20438 (pb_cplex) R 20389 20389 20024 0 -1 4202560 356 0 0 0 33 2 0 0 20 0 8 0 390892657 101507072 5747 33554432000 134512640 146736320 4293789120 4106216424 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20439] ppid=20389 vsize=99128 CPUtime=0.54 cores=0,2,4,6
/proc/20390/task/20439/stat : 20439 (pb_cplex) S 20389 20389 20024 0 -1 4202560 913 0 0 0 51 3 0 0 20 0 8 0 390892657 101507072 5747 33554432000 134512640 146736320 4293789120 4095733392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20440] ppid=20389 vsize=99128 CPUtime=0.54 cores=0,2,4,6
/proc/20390/task/20440/stat : 20440 (pb_cplex) S 20389 20389 20024 0 -1 4202560 563 0 0 0 51 3 0 0 20 0 8 0 390892657 101507072 5747 33554432000 134512640 146736320 4293789120 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 108276

[startup+6.30034 s]
/proc/loadavg: 12.67 11.81 7.15 16/193 20440
/proc/meminfo: memFree=30868688/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=104248 CPUtime=16.7 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) R 20389 20389 20024 0 -1 4202496 25303 0 0 0 1588 82 0 0 20 0 8 0 390892586 106749952 6414 33554432000 134512640 146736320 4293789120 4293779664 137151261 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/20390/statm: 26062 6414 982 2985 0 22996 0
[pid=20390/tid=20434] ppid=20389 vsize=104248 CPUtime=1.72 cores=0,2,4,6
/proc/20390/task/20434/stat : 20434 (pb_cplex) R 20389 20389 20024 0 -1 4202560 1826 0 0 0 163 9 0 0 20 0 8 0 390892657 106749952 6414 33554432000 134512640 146736320 4293789120 4130543680 137149361 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20435] ppid=20389 vsize=104248 CPUtime=1.63 cores=0,2,4,6
/proc/20390/task/20435/stat : 20435 (pb_cplex) R 20389 20389 20024 0 -1 4202560 1433 0 0 0 155 8 0 0 20 0 8 0 390892657 106749952 6414 33554432000 134512640 146736320 4293789120 4141027504 137176000 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20436] ppid=20389 vsize=104248 CPUtime=1.63 cores=0,2,4,6
/proc/20390/task/20436/stat : 20436 (pb_cplex) R 20389 20389 20024 0 -1 4202560 1298 0 0 0 155 8 0 0 20 0 8 0 390892657 106749952 6414 33554432000 134512640 146736320 4293789120 4151515100 144785853 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=20390/tid=20437] ppid=20389 vsize=104248 CPUtime=1.92 cores=0,2,4,6
/proc/20390/task/20437/stat : 20437 (pb_cplex) R 20389 20389 20024 0 -1 4202560 1796 0 0 0 183 9 0 0 20 0 8 0 390892657 106749952 6414 33554432000 134512640 146736320 4293789120 4120056732 144785853 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20438] ppid=20389 vsize=104248 CPUtime=1.66 cores=0,2,4,6
/proc/20390/task/20438/stat : 20438 (pb_cplex) R 20389 20389 20024 0 -1 4202560 1810 0 0 0 156 10 0 0 20 0 8 0 390892657 106749952 6414 33554432000 134512640 146736320 4293789120 4106211588 135850298 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20439] ppid=20389 vsize=104248 CPUtime=1.82 cores=0,2,4,6
/proc/20390/task/20439/stat : 20439 (pb_cplex) R 20389 20389 20024 0 -1 4202560 1871 0 0 0 174 8 0 0 20 0 8 0 390892657 106749952 6414 33554432000 134512640 146736320 4293789120 4095727280 137371870 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=20390/tid=20440] ppid=20389 vsize=104248 CPUtime=2.06 cores=0,2,4,6
/proc/20390/task/20440/stat : 20440 (pb_cplex) R 20389 20389 20024 0 -1 4202560 1658 0 0 0 198 8 0 0 20 0 8 0 390892657 106749952 6414 33554432000 134512640 146736320 4293789120 4070561856 137150826 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 16.7
Current children cumulated vsize (KiB) 113396

[startup+12.7003 s]
/proc/loadavg: 12.55 11.81 7.20 10/193 20440
/proc/meminfo: memFree=30868248/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=104248 CPUtime=34.87 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) S 20389 20389 20024 0 -1 4202496 36332 0 0 0 3324 163 0 0 20 0 8 0 390892586 106749952 6316 33554432000 134512640 146736320 4293789120 4293785488 144580073 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/20390/statm: 26062 6316 982 2985 0 22996 0
[pid=20390/tid=20434] ppid=20389 vsize=104248 CPUtime=4.1 cores=0,2,4,6
/proc/20390/task/20434/stat : 20434 (pb_cplex) R 20389 20389 20024 0 -1 4202560 3987 0 0 0 390 20 0 0 20 0 8 0 390892657 106749952 6316 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20435] ppid=20389 vsize=104248 CPUtime=4.25 cores=0,2,4,6
/proc/20390/task/20435/stat : 20435 (pb_cplex) S 20389 20389 20024 0 -1 4202560 2570 0 0 0 407 18 0 0 20 0 8 0 390892657 106749952 6316 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20436] ppid=20389 vsize=104248 CPUtime=4.12 cores=0,2,4,6
/proc/20390/task/20436/stat : 20436 (pb_cplex) S 20389 20389 20024 0 -1 4202560 2550 0 0 0 391 21 0 0 20 0 8 0 390892657 106749952 6316 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20437] ppid=20389 vsize=104248 CPUtime=3.98 cores=0,2,4,6
/proc/20390/task/20437/stat : 20437 (pb_cplex) S 20389 20389 20024 0 -1 4202560 2790 0 0 0 380 18 0 0 20 0 8 0 390892657 106749952 6316 33554432000 134512640 146736320 4293789120 4120063632 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20438] ppid=20389 vsize=104248 CPUtime=3.79 cores=0,2,4,6
/proc/20390/task/20438/stat : 20438 (pb_cplex) S 20389 20389 20024 0 -1 4202560 2971 0 0 0 361 18 0 0 20 0 8 0 390892657 106749952 6316 33554432000 134512640 146736320 4293789120 4106219152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20439] ppid=20389 vsize=104248 CPUtime=3.99 cores=0,2,4,6
/proc/20390/task/20439/stat : 20439 (pb_cplex) S 20389 20389 20024 0 -1 4202560 3239 0 0 0 381 18 0 0 20 0 8 0 390892657 106749952 6316 33554432000 134512640 146736320 4293789120 4095733392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20440] ppid=20389 vsize=104248 CPUtime=4.02 cores=0,2,4,6
/proc/20390/task/20440/stat : 20440 (pb_cplex) R 20389 20389 20024 0 -1 4202560 2843 0 0 0 385 17 0 0 20 0 8 0 390892657 106749952 6316 33554432000 134512640 146736320 4293789120 4070560064 144785853 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 34.87
Current children cumulated vsize (KiB) 113396

[startup+25.5004 s]
/proc/loadavg: 12.17 11.75 7.23 5/193 20475
/proc/meminfo: memFree=30865468/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=106704 CPUtime=55.23 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) R 20389 20389 20024 0 -1 4202496 80764 0 0 0 5251 272 0 0 20 0 8 0 390892586 109264896 5792 33554432000 134512640 146736320 4293789120 4293779876 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/20390/statm: 26676 5792 983 2985 0 23610 0
[pid=20390/tid=20469] ppid=20389 vsize=106704 CPUtime=1.23 cores=0,2,4,6
/proc/20390/task/20469/stat : 20469 (pb_cplex) S 20389 20389 20024 0 -1 4202560 5065 0 0 0 118 5 0 0 20 0 8 0 390894598 109264896 5792 33554432000 134512640 146736320 4293789120 4088606352 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20470] ppid=20389 vsize=106704 CPUtime=0.94 cores=0,2,4,6
/proc/20390/task/20470/stat : 20470 (pb_cplex) S 20389 20389 20024 0 -1 4202560 1778 0 0 0 89 5 0 0 20 0 8 0 390894598 109264896 5792 33554432000 134512640 146736320 4293789120 4099092112 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20471] ppid=20389 vsize=106704 CPUtime=1.35 cores=0,2,4,6
/proc/20390/task/20471/stat : 20471 (pb_cplex) S 20389 20389 20024 0 -1 4202560 5771 0 0 0 130 5 0 0 20 0 8 0 390894598 109264896 5792 33554432000 134512640 146736320 4293789120 4109577872 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20472] ppid=20389 vsize=106704 CPUtime=1.23 cores=0,2,4,6
/proc/20390/task/20472/stat : 20472 (pb_cplex) S 20389 20389 20024 0 -1 4202560 4632 0 0 0 118 5 0 0 20 0 8 0 390894598 109264896 5792 33554432000 134512640 146736320 4293789120 4120063632 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20473] ppid=20389 vsize=106704 CPUtime=1.03 cores=0,2,4,6
/proc/20390/task/20473/stat : 20473 (pb_cplex) S 20389 20389 20024 0 -1 4202560 3669 0 0 0 98 5 0 0 20 0 8 0 390894598 109264896 5792 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20474] ppid=20389 vsize=106704 CPUtime=1.27 cores=0,2,4,6
/proc/20390/task/20474/stat : 20474 (pb_cplex) S 20389 20389 20024 0 -1 4202560 4781 0 0 0 122 5 0 0 20 0 8 0 390894598 109264896 5792 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20475] ppid=20389 vsize=106704 CPUtime=1.08 cores=0,2,4,6
/proc/20390/task/20475/stat : 20475 (pb_cplex) S 20389 20389 20024 0 -1 4202560 4253 0 0 0 105 3 0 0 20 0 8 0 390894598 109264896 5792 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 55.23
Current children cumulated vsize (KiB) 115852

[startup+51.1003 s]
/proc/loadavg: 9.10 11.06 7.12 6/186 20475
/proc/meminfo: memFree=31138444/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0

################
# More data... #
################

/proc/20390/statm: 55014 35238 985 2985 0 51948 0
[pid=20390/tid=20519] ppid=20389 vsize=220056 CPUtime=113.87 cores=0,2,4,6
/proc/20390/task/20519/stat : 20519 (pb_cplex) S 20389 20389 20024 0 -1 4202560 95229 0 0 0 10690 697 0 0 20 0 8 0 390924812 225337344 35238 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20520] ppid=20389 vsize=220056 CPUtime=112.67 cores=0,2,4,6
/proc/20390/task/20520/stat : 20520 (pb_cplex) R 20389 20389 20024 0 -1 4202560 93211 0 0 0 10561 706 0 0 20 0 8 0 390924812 225337344 35238 33554432000 134512640 146736320 4293789120 4141032424 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20521] ppid=20389 vsize=220056 CPUtime=111.77 cores=0,2,4,6
/proc/20390/task/20521/stat : 20521 (pb_cplex) S 20389 20389 20024 0 -1 4202560 94662 0 0 0 10471 706 0 0 20 0 8 0 390924812 225337344 35238 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20522] ppid=20389 vsize=220056 CPUtime=111.19 cores=0,2,4,6
/proc/20390/task/20522/stat : 20522 (pb_cplex) S 20389 20389 20024 0 -1 4202560 93789 0 0 0 10429 690 0 0 20 0 8 0 390924812 225337344 35238 33554432000 134512640 146736320 4293789120 4120063632 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20523] ppid=20389 vsize=220056 CPUtime=113.45 cores=0,2,4,6
/proc/20390/task/20523/stat : 20523 (pb_cplex) S 20389 20389 20024 0 -1 4202560 98115 0 0 0 10652 693 0 0 20 0 8 0 390924812 225337344 35238 33554432000 134512640 146736320 4293789120 4106219152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 1 0 0
[pid=20390/tid=20524] ppid=20389 vsize=220056 CPUtime=111.81 cores=0,2,4,6
/proc/20390/task/20524/stat : 20524 (pb_cplex) S 20389 20389 20024 0 -1 4202560 97320 0 0 0 10486 695 0 0 20 0 8 0 390924812 225337344 35238 33554432000 134512640 146736320 4293789120 4095733392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=20390/tid=20525] ppid=20389 vsize=220056 CPUtime=111.42 cores=0,2,4,6
/proc/20390/task/20525/stat : 20525 (pb_cplex) S 20389 20389 20024 0 -1 4202560 93695 0 0 0 10458 684 0 0 20 0 8 0 390924812 225337344 35238 33554432000 134512640 146736320 4293789120 4060081808 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1465.4
Current children cumulated vsize (KiB) 229204

[startup+1002.3 s]
/proc/loadavg: 5.19 5.35 5.97 8/193 20839
/proc/meminfo: memFree=30956180/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=227652 CPUtime=1553.28 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) S 20389 20389 20024 0 -1 4202496 1885846 0 0 0 146704 8624 0 0 20 0 8 0 390892586 233115648 36972 33554432000 134512640 146736320 4293789120 4293785488 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/20390/statm: 56913 36972 985 2985 0 53847 0
[pid=20390/tid=20519] ppid=20389 vsize=227652 CPUtime=124.43 cores=0,2,4,6
/proc/20390/task/20519/stat : 20519 (pb_cplex) S 20389 20389 20024 0 -1 4202560 102598 0 0 0 11685 758 0 0 20 0 8 0 390924812 233115648 36972 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20520] ppid=20389 vsize=227652 CPUtime=123.49 cores=0,2,4,6
/proc/20390/task/20520/stat : 20520 (pb_cplex) S 20389 20389 20024 0 -1 4202560 102038 0 0 0 11577 772 0 0 20 0 8 0 390924812 233115648 36972 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20521] ppid=20389 vsize=227652 CPUtime=122.34 cores=0,2,4,6
/proc/20390/task/20521/stat : 20521 (pb_cplex) S 20389 20389 20024 0 -1 4202560 103471 0 0 0 11458 776 0 0 20 0 8 0 390924812 233115648 36972 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20522] ppid=20389 vsize=227652 CPUtime=121.83 cores=0,2,4,6
/proc/20390/task/20522/stat : 20522 (pb_cplex) S 20389 20389 20024 0 -1 4202560 102801 0 0 0 11425 758 0 0 20 0 8 0 390924812 233115648 36972 33554432000 134512640 146736320 4293789120 4120063632 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20523] ppid=20389 vsize=227652 CPUtime=124.36 cores=0,2,4,6
/proc/20390/task/20523/stat : 20523 (pb_cplex) S 20389 20389 20024 0 -1 4202560 107039 0 0 0 11673 763 0 0 20 0 8 0 390924812 233115648 36972 33554432000 134512640 146736320 4293789120 4106219152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 1 0 0
[pid=20390/tid=20524] ppid=20389 vsize=227652 CPUtime=122.26 cores=0,2,4,6
/proc/20390/task/20524/stat : 20524 (pb_cplex) S 20389 20389 20024 0 -1 4202560 105217 0 0 0 11469 757 0 0 20 0 8 0 390924812 233115648 36972 33554432000 134512640 146736320 4293789120 4095733392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20525] ppid=20389 vsize=227652 CPUtime=122.02 cores=0,2,4,6
/proc/20390/task/20525/stat : 20525 (pb_cplex) R 20389 20389 20024 0 -1 4202560 101988 0 0 0 11452 750 0 0 20 0 8 0 390924812 233115648 36972 33554432000 134512640 146736320 4293789120 4060081808 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 1553.28
Current children cumulated vsize (KiB) 236800

[startup+1062.3 s]
/proc/loadavg: 4.90 5.22 5.88 3/186 20891
/proc/meminfo: memFree=31059536/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=242832 CPUtime=1641.93 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) R 20389 20389 20024 0 -1 4202496 1969897 0 0 0 155055 9138 0 0 20 0 8 0 390892586 248659968 40953 33554432000 134512640 146736320 4293789120 4293778404 136389777 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/20390/statm: 60708 40953 985 2985 0 57642 0
[pid=20390/tid=20519] ppid=20389 vsize=242832 CPUtime=135.21 cores=0,2,4,6
/proc/20390/task/20519/stat : 20519 (pb_cplex) S 20389 20389 20024 0 -1 4202560 112748 0 0 0 12695 826 0 0 20 0 8 0 390924812 248659968 40953 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=20390/tid=20520] ppid=20389 vsize=242832 CPUtime=134.4 cores=0,2,4,6
/proc/20390/task/20520/stat : 20520 (pb_cplex) S 20389 20389 20024 0 -1 4202560 112220 0 0 0 12602 838 0 0 20 0 8 0 390924812 248659968 40953 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20521] ppid=20389 vsize=242832 CPUtime=132.88 cores=0,2,4,6
/proc/20390/task/20521/stat : 20521 (pb_cplex) S 20389 20389 20024 0 -1 4202560 114549 0 0 0 12452 836 0 0 20 0 8 0 390924812 248659968 40953 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20522] ppid=20389 vsize=242832 CPUtime=132.38 cores=0,2,4,6
/proc/20390/task/20522/stat : 20522 (pb_cplex) S 20389 20389 20024 0 -1 4202560 112892 0 0 0 12415 823 0 0 20 0 8 0 390924812 248659968 40953 33554432000 134512640 146736320 4293789120 4120063632 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20523] ppid=20389 vsize=242832 CPUtime=134.68 cores=0,2,4,6
/proc/20390/task/20523/stat : 20523 (pb_cplex) S 20389 20389 20024 0 -1 4202560 117270 0 0 0 12642 826 0 0 20 0 8 0 390924812 248659968 40953 33554432000 134512640 146736320 4293789120 4106219152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 1 0 0
[pid=20390/tid=20524] ppid=20389 vsize=242832 CPUtime=133.46 cores=0,2,4,6
/proc/20390/task/20524/stat : 20524 (pb_cplex) S 20389 20389 20024 0 -1 4202560 116224 0 0 0 12520 826 0 0 20 0 8 0 390924812 248659968 40953 33554432000 134512640 146736320 4293789120 4095733392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20525] ppid=20389 vsize=242832 CPUtime=132.41 cores=0,2,4,6
/proc/20390/task/20525/stat : 20525 (pb_cplex) S 20389 20389 20024 0 -1 4202560 112819 0 0 0 12428 813 0 0 20 0 8 0 390924812 248659968 40953 33554432000 134512640 146736320 4293789120 4060081808 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1641.93
Current children cumulated vsize (KiB) 251980

[startup+1122.3 s]
/proc/loadavg: 3.29 4.68 5.65 3/186 20891
/proc/meminfo: memFree=31007952/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=270920 CPUtime=1734.11 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) R 20389 20389 20024 0 -1 4202496 2078826 0 0 0 163715 9696 0 0 20 0 8 0 390892586 277422080 47570 33554432000 134512640 146736320 4293789120 4293777604 136385174 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/20390/statm: 67730 47570 985 2985 0 64664 0
[pid=20390/tid=20519] ppid=20389 vsize=270920 CPUtime=147.01 cores=0,2,4,6
/proc/20390/task/20519/stat : 20519 (pb_cplex) S 20389 20389 20024 0 -1 4202560 126994 0 0 0 13802 899 0 0 20 0 8 0 390924812 277422080 47570 33554432000 134512640 146736320 4293789120 4130549392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20520] ppid=20389 vsize=270920 CPUtime=145.78 cores=0,2,4,6
/proc/20390/task/20520/stat : 20520 (pb_cplex) S 20389 20389 20024 0 -1 4202560 126606 0 0 0 13671 907 0 0 20 0 8 0 390924812 277422080 47570 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20521] ppid=20389 vsize=270920 CPUtime=144.57 cores=0,2,4,6
/proc/20390/task/20521/stat : 20521 (pb_cplex) S 20389 20389 20024 0 -1 4202560 127296 0 0 0 13550 907 0 0 20 0 8 0 390924812 277422080 47570 33554432000 134512640 146736320 4293789120 4151520912 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20522] ppid=20389 vsize=270920 CPUtime=143.69 cores=0,2,4,6
/proc/20390/task/20522/stat : 20522 (pb_cplex) S 20389 20389 20024 0 -1 4202560 127076 0 0 0 13475 894 0 0 20 0 8 0 390924812 277422080 47570 33554432000 134512640 146736320 4293789120 4120063632 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20523] ppid=20389 vsize=270920 CPUtime=145.67 cores=0,2,4,6
/proc/20390/task/20523/stat : 20523 (pb_cplex) S 20389 20389 20024 0 -1 4202560 129216 0 0 0 13669 898 0 0 20 0 8 0 390924812 277422080 47570 33554432000 134512640 146736320 4293789120 4106219152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 1 0 0
[pid=20390/tid=20524] ppid=20389 vsize=270920 CPUtime=145.01 cores=0,2,4,6
/proc/20390/task/20524/stat : 20524 (pb_cplex) S 20389 20389 20024 0 -1 4202560 129874 0 0 0 13604 897 0 0 20 0 8 0 390924812 277422080 47570 33554432000 134512640 146736320 4293789120 4095733392 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=20390/tid=20525] ppid=20389 vsize=270920 CPUtime=143.21 cores=0,2,4,6
/proc/20390/task/20525/stat : 20525 (pb_cplex) S 20389 20389 20024 0 -1 4202560 125549 0 0 0 13438 883 0 0 20 0 8 0 390924812 277422080 47570 33554432000 134512640 146736320 4293789120 4060081808 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1734.11
Current children cumulated vsize (KiB) 280068



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1165.9 s]
/proc/loadavg: 2.77 4.38 5.51 8/186 20891
/proc/meminfo: memFree=31005724/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=286400 CPUtime=1800.02 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) R 20389 20389 20024 0 -1 4202496 2147375 0 0 0 169913 10089 0 0 20 0 8 0 390892586 293273600 51309 33554432000 134512640 146736320 4293789120 4293783816 137852813 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/20390/statm: 71600 51309 985 2985 0 68534 0
[pid=20390/tid=20519] ppid=20389 vsize=286400 CPUtime=155.38 cores=0,2,4,6
/proc/20390/task/20519/stat : 20519 (pb_cplex) R 20389 20389 20024 0 -1 4202560 135130 0 0 0 14591 947 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4130548380 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20520] ppid=20389 vsize=286400 CPUtime=153.88 cores=0,2,4,6
/proc/20390/task/20520/stat : 20520 (pb_cplex) S 20389 20389 20024 0 -1 4202560 134731 0 0 0 14428 960 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20521] ppid=20389 vsize=286400 CPUtime=152.96 cores=0,2,4,6
/proc/20390/task/20521/stat : 20521 (pb_cplex) R 20389 20389 20024 0 -1 4202560 136995 0 0 0 14343 953 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4151519652 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20522] ppid=20389 vsize=286400 CPUtime=151.83 cores=0,2,4,6
/proc/20390/task/20522/stat : 20522 (pb_cplex) R 20389 20389 20024 0 -1 4202560 134013 0 0 0 14237 946 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4120063660 144580332 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20523] ppid=20389 vsize=286400 CPUtime=153.69 cores=0,2,4,6
/proc/20390/task/20523/stat : 20523 (pb_cplex) R 20389 20389 20024 0 -1 4202560 136717 0 0 0 14420 949 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4106216424 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 1 0 0
[pid=20390/tid=20524] ppid=20389 vsize=286400 CPUtime=153.32 cores=0,2,4,6
/proc/20390/task/20524/stat : 20524 (pb_cplex) R 20389 20389 20024 0 -1 4202560 138662 0 0 0 14386 946 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4095733420 144580332 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20525] ppid=20389 vsize=286400 CPUtime=151.11 cores=0,2,4,6
/proc/20390/task/20525/stat : 20525 (pb_cplex) R 20389 20389 20024 0 -1 4202560 134136 0 0 0 14180 931 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4060079080 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1800.02
Current children cumulated vsize (KiB) 295548

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1165.9 s]
/proc/loadavg: 2.77 4.38 5.51 8/186 20891
/proc/meminfo: memFree=31005724/32873648 swapFree=7376/7376
[pid=20389] ppid=20387 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/20389/stat : 20389 (run) S 20387 20389 20024 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390892586 9367552 269 33554432000 4194304 5064252 140735260606304 140735260605216 226084152910 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/20389/statm: 2287 269 229 213 0 72 0
[pid=20390] ppid=20389 vsize=286400 CPUtime=1800.02 cores=0,2,4,6
/proc/20390/stat : 20390 (pb_cplex) R 20389 20389 20024 0 -1 4202496 2147375 0 0 0 169913 10089 0 0 20 0 8 0 390892586 293273600 51309 33554432000 134512640 146736320 4293789120 4293783816 137852813 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/20390/statm: 71600 51309 985 2985 0 68534 0
[pid=20390/tid=20519] ppid=20389 vsize=286400 CPUtime=155.38 cores=0,2,4,6
/proc/20390/task/20519/stat : 20519 (pb_cplex) R 20389 20389 20024 0 -1 4202560 135130 0 0 0 14591 947 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4130548380 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20520] ppid=20389 vsize=286400 CPUtime=153.88 cores=0,2,4,6
/proc/20390/task/20520/stat : 20520 (pb_cplex) S 20389 20389 20024 0 -1 4202560 134731 0 0 0 14428 960 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4141035152 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20521] ppid=20389 vsize=286400 CPUtime=152.96 cores=0,2,4,6
/proc/20390/task/20521/stat : 20521 (pb_cplex) R 20389 20389 20024 0 -1 4202560 136995 0 0 0 14343 953 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4151519652 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=20390/tid=20522] ppid=20389 vsize=286400 CPUtime=151.83 cores=0,2,4,6
/proc/20390/task/20522/stat : 20522 (pb_cplex) R 20389 20389 20024 0 -1 4202560 134013 0 0 0 14237 946 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4120063660 144580332 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=20390/tid=20523] ppid=20389 vsize=286400 CPUtime=153.69 cores=0,2,4,6
/proc/20390/task/20523/stat : 20523 (pb_cplex) R 20389 20389 20024 0 -1 4202560 136717 0 0 0 14420 949 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4106216424 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 1 0 0
[pid=20390/tid=20524] ppid=20389 vsize=286400 CPUtime=153.32 cores=0,2,4,6
/proc/20390/task/20524/stat : 20524 (pb_cplex) R 20389 20389 20024 0 -1 4202560 138662 0 0 0 14386 946 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4095733420 144580332 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=20390/tid=20525] ppid=20389 vsize=286400 CPUtime=151.11 cores=0,2,4,6
/proc/20390/task/20525/stat : 20525 (pb_cplex) R 20389 20389 20024 0 -1 4202560 134136 0 0 0 14180 931 0 0 20 0 8 0 390924812 293273600 51309 33554432000 134512640 146736320 4293789120 4060079080 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1800.02
Current children cumulated vsize (KiB) 295548

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 20389 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=0
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=999
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1800.02
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 1165.92
CPU time (s): 1800.02
CPU user time (s): 1699.13
CPU system time (s): 100.89
CPU usage (%): 154.386
Max. virtual memory (cumulated for all children) (KiB): 296572

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0
system time used= 0.000999
maximum resident set size= 1076
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 354
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1
involuntary context switches= 2

runsolver used 5.18321 second user time and 4.5813 second system time

The end

Launcher Data 

Begin job on node120 at 2012-06-09 15:14:26
IDJOB=3734931
IDBENCH=2920
IDSOLVER=2325
FILE ID=node120/3734931-1339247666
RUNJOBID= node120-1339246833-20041
PBS_JOBID= 14644361
Free space on /tmp= 70828 MiB

SOLVER NAME= PB10: pb_cplex 2010-06-29
BENCH NAME= PB06/final/normalized-PB06/OPT-SMALLINT/submitted-PB06/manquiho/market-split/normalized-opt-market-split_8_70_1.opb
COMMAND LINE= run BENCHNAME
CONVERSION SCRIPT= PBconversionToLinear BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3734931-1339247666/watcher-3734931-1339247666 -o /tmp/evaluation-result-3734931-1339247666/solver-3734931-1339247666 -C 1800 -W 1900 -M 15500  run HOME/instance-3734931-1339247666.opb

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 0263d0a7c5e3320bd7cce9534c14bf27
RANDOM SEED=1185775200

node120.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.88
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.456
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        30901100 kB
Buffers:          439312 kB
Cached:           344088 kB
SwapCached:         5856 kB
Active:           819120 kB
Inactive:         331320 kB
Active(anon):     350376 kB
Inactive(anon):    10444 kB
Active(file):     468744 kB
Inactive(file):   320876 kB
Unevictable:        7376 kB
Mlocked:            7376 kB
SwapTotal:      67108856 kB
SwapFree:       67096152 kB
Dirty:             16356 kB
Writeback:             0 kB
AnonPages:        360280 kB
Mapped:            12640 kB
Shmem:                 0 kB
Slab:             683240 kB
SReclaimable:     127488 kB
SUnreclaim:       555752 kB
KernelStack:        1528 kB
PageTables:         4688 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:     524120 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:     98304 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

CONVERSION COMMAND LINE= PBconversionToLinear HOME/instance-3734931-1339247666.opb

Free space on /tmp at the end= 70812 MiB
End job on node120 at 2012-06-09 15:33:54