Trace number 3736611

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
PB10: pb_cplex 2010-06-29? (TO) 1800.23 566.817

General information on the benchmark

Namenormalized-PB07/OPT-SMALLINT-NLC/submittedPB07/roussel/factor-mod-B/
factor-mod-size=8-P0=229-P1=67-P2=149-P3=97-P4=149-P5=211-P6=89-P7=59-B.opb
MD5SUM12f1f3305860fafcd361cd1c61e0d104
Bench CategoryOPT-SMALLINT-NLC (optimisation, small integers, non linear constraints)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark3
Best CPU time to get the best result obtained on this benchmark0.086986
Has Objective FunctionYES
SatisfiableYES
(Un)Satisfiability was provedYES
Best value of the objective function 3
Optimality of the best value was proved YES
Number of variables168
Total number of constraints15
Number of constraints which are clauses0
Number of constraints which are cardinality constraints (but not clauses)0
Number of constraints which are nor clauses,nor cardinality constraints15
Minimum length of a constraint8
Maximum length of a constraint80
Number of terms in the objective function 8
Biggest coefficient in the objective function 128
Number of bits for the biggest coefficient in the objective function 8
Sum of the numbers in the objective function 255
Number of bits of the sum of numbers in the objective function 8
Biggest number in a constraint 32768
Number of bits of the biggest number in a constraint 16
Biggest sum of numbers in a constraint 130560
Number of bits of the biggest sum of numbers17
Number of products (including duplicates)448
Sum of products size (including duplicates)896
Number of different products448
Sum of products size896

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing opb file format. File HOME/instance-3736611-1339248219.opb.
0.00/0.00	c Instance file HOME/instance-3736611-1339248219.opb
0.00/0.00	c File size is 33487 bytes.
0.00/0.01	c Highest Coefficient sum: 130560
0.00/0.01	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.01	c ============================[ Problem Statistics ]=============================
0.00/0.01	c |                                                                             |
0.00/0.01	c |  Parsing time:          0.01         s                                      |
0.00/0.01	c |  Number Variables:      616                                                 |
0.00/0.01	c |  Number Constraints:    911                                                 |
0.00/0.01	c ===============================================================================
1800.23/566.81	c Remaining time 1799.99 sec.
1800.23/566.81	c 
1800.23/566.81	c *** INTERRUPTED ***
1800.23/566.81	s UNKNOWN
1800.23/566.81	c Total CPU time              : 1782.96 s
1800.23/566.81	c Memory used           : 261.09 MB
1800.23/566.81	c CPU time              : 1782.96 s
1800.23/566.81	c 
1800.23/566.81	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3736611-1339248219/watcher-3736611-1339248219 -o /tmp/evaluation-result-3736611-1339248219/solver-3736611-1339248219 -C 1800 -W 1900 -M 15500 run HOME/instance-3736611-1339248219.opb 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.56 8.11 7.18 6/184 23636
/proc/meminfo: memFree=30226596/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0

[startup+0.097032 s]
/proc/loadavg: 4.56 8.11 7.18 6/184 23636
/proc/meminfo: memFree=30226596/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.100232 s]
/proc/loadavg: 4.56 8.11 7.18 6/184 23636
/proc/meminfo: memFree=30226596/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.300232 s]
/proc/loadavg: 4.56 8.11 7.18 6/184 23636
/proc/meminfo: memFree=30226596/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.700179 s]
/proc/loadavg: 4.56 8.11 7.18 6/184 23636
/proc/meminfo: memFree=30226596/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+1.5004 s]
/proc/loadavg: 4.56 8.11 7.18 11/193 23659
/proc/meminfo: memFree=30182388/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=134880 CPUtime=4.02 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) S 23636 23636 22465 0 -1 4202496 32759 0 0 0 384 18 0 0 20 0 8 0 390947795 138117120 13157 33554432000 134512640 146736320 4289841024 4289837392 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/23637/statm: 33720 13157 971 2985 0 30654 0
[pid=23637/tid=23653] ppid=23636 vsize=134880 CPUtime=0.4 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 2515 0 0 0 38 2 0 0 20 0 8 0 390947830 138117120 13157 33554432000 134512640 146736320 4289841024 4131169816 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23654] ppid=23636 vsize=134880 CPUtime=0.4 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) R 23636 23636 22465 0 -1 4202560 2607 0 0 0 38 2 0 0 20 0 8 0 390947830 138117120 13157 33554432000 134512640 146736320 4289841024 4141653940 135450566 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=134880 CPUtime=0.43 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) R 23636 23636 22465 0 -1 4202560 1435 0 0 0 41 2 0 0 20 0 8 0 390947830 138117120 13157 33554432000 134512640 146736320 4289841024 4152142116 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=134880 CPUtime=0.41 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) R 23636 23636 22465 0 -1 4202560 1705 0 0 0 41 0 0 0 20 0 8 0 390947830 138117120 13157 33554432000 134512640 146736320 4289841024 4120023372 136017755 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=134880 CPUtime=0.42 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) R 23636 23636 22465 0 -1 4202560 3821 0 0 0 40 2 0 0 20 0 8 0 390947830 138117120 13157 33554432000 134512640 146736320 4289841024 4099924968 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=134880 CPUtime=0.45 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) R 23636 23636 22465 0 -1 4202560 3744 0 0 0 43 2 0 0 20 0 8 0 390947830 138117120 13157 33554432000 134512640 146736320 4289841024 4089438260 135450974 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=134880 CPUtime=0.38 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) S 23636 23636 22465 0 -1 4202560 4374 0 0 0 37 1 0 0 20 0 8 0 390947830 138117120 13157 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 4.02
Current children cumulated vsize (KiB) 144028

[startup+3.10044 s]
/proc/loadavg: 4.56 8.11 7.18 11/193 23659
/proc/meminfo: memFree=30164416/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=167196 CPUtime=9.01 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) S 23636 23636 22465 0 -1 4202496 53901 0 0 0 876 25 0 0 20 0 8 0 390947795 171208704 20375 33554432000 134512640 146736320 4289841024 4289837392 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/23637/statm: 41799 20375 971 2985 0 38733 0
[pid=23637/tid=23653] ppid=23636 vsize=167196 CPUtime=0.98 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 5339 0 0 0 96 2 0 0 20 0 8 0 390947830 171208704 20375 33554432000 134512640 146736320 4289841024 4131168464 136008423 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23654] ppid=23636 vsize=167196 CPUtime=1 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) S 23636 23636 22465 0 -1 4202560 4155 0 0 0 98 2 0 0 20 0 8 0 390947830 171208704 20375 33554432000 134512640 146736320 4289841024 4141657744 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=167196 CPUtime=1.12 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) S 23636 23636 22465 0 -1 4202560 3128 0 0 0 109 3 0 0 20 0 8 0 390947830 171208704 20375 33554432000 134512640 146736320 4289841024 4152143504 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=167196 CPUtime=1.06 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) S 23636 23636 22465 0 -1 4202560 3596 0 0 0 105 1 0 0 20 0 8 0 390947830 171208704 20375 33554432000 134512640 146736320 4289841024 4120026768 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=167196 CPUtime=1.07 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) S 23636 23636 22465 0 -1 4202560 6380 0 0 0 105 2 0 0 20 0 8 0 390947830 171208704 20375 33554432000 134512640 146736320 4289841024 4099927696 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=167196 CPUtime=1.11 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) S 23636 23636 22465 0 -1 4202560 7226 0 0 0 109 2 0 0 20 0 8 0 390947830 171208704 20375 33554432000 134512640 146736320 4289841024 4089441936 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=167196 CPUtime=1.02 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) S 23636 23636 22465 0 -1 4202560 10610 0 0 0 98 4 0 0 20 0 8 0 390947830 171208704 20375 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 9.01
Current children cumulated vsize (KiB) 176344

[startup+6.3004 s]
/proc/loadavg: 5.15 8.18 7.21 11/193 23659
/proc/meminfo: memFree=30140732/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=186860 CPUtime=19.23 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) S 23636 23636 22465 0 -1 4202496 88789 0 0 0 1888 35 0 0 20 0 8 0 390947795 191344640 25000 33554432000 134512640 146736320 4289841024 4289837392 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/23637/statm: 46715 25000 971 2985 0 43649 0
[pid=23637/tid=23653] ppid=23636 vsize=186860 CPUtime=2.25 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 8538 0 0 0 222 3 0 0 20 0 8 0 390947830 191344640 25000 33554432000 134512640 146736320 4289841024 4131168464 136008309 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23654] ppid=23636 vsize=186860 CPUtime=2.33 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) S 23636 23636 22465 0 -1 4202560 8416 0 0 0 230 3 0 0 20 0 8 0 390947830 191344640 25000 33554432000 134512640 146736320 4289841024 4141657744 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=186860 CPUtime=2.37 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) S 23636 23636 22465 0 -1 4202560 8824 0 0 0 232 5 0 0 20 0 8 0 390947830 191344640 25000 33554432000 134512640 146736320 4289841024 4152143504 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=186860 CPUtime=2.33 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) S 23636 23636 22465 0 -1 4202560 8745 0 0 0 231 2 0 0 20 0 8 0 390947830 191344640 25000 33554432000 134512640 146736320 4289841024 4120026768 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=186860 CPUtime=2.3 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) S 23636 23636 22465 0 -1 4202560 10465 0 0 0 227 3 0 0 20 0 8 0 390947830 191344640 25000 33554432000 134512640 146736320 4289841024 4099927696 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=186860 CPUtime=2.5 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) S 23636 23636 22465 0 -1 4202560 8573 0 0 0 247 3 0 0 20 0 8 0 390947830 191344640 25000 33554432000 134512640 146736320 4289841024 4089441936 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=186860 CPUtime=2.3 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) S 23636 23636 22465 0 -1 4202560 19259 0 0 0 225 5 0 0 20 0 8 0 390947830 191344640 25000 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 19.23
Current children cumulated vsize (KiB) 196008

[startup+12.7004 s]
/proc/loadavg: 5.70 8.24 7.23 14/193 23659
/proc/meminfo: memFree=30094668/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=223364 CPUtime=39.39 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) S 23636 23636 22465 0 -1 4202496 196045 0 0 0 3872 67 0 0 20 0 8 0 390947795 228724736 33336 33554432000 134512640 146736320 4289841024 4289837392 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/23637/statm: 55841 33336 971 2985 0 52775 0
[pid=23637/tid=23653] ppid=23636 vsize=223364 CPUtime=4.2 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) S 23636 23636 22465 0 -1 4202560 15002 0 0 0 415 5 0 0 20 0 8 0 390947830 228724736 33336 33554432000 134512640 146736320 4289841024 4131171984 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23654] ppid=23636 vsize=223364 CPUtime=4.81 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) S 23636 23636 22465 0 -1 4202560 23404 0 0 0 473 8 0 0 20 0 8 0 390947830 228724736 33336 33554432000 134512640 146736320 4289841024 4141657744 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=223364 CPUtime=4.81 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) S 23636 23636 22465 0 -1 4202560 18568 0 0 0 473 8 0 0 20 0 8 0 390947830 228724736 33336 33554432000 134512640 146736320 4289841024 4152143504 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=223364 CPUtime=4.73 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) S 23636 23636 22465 0 -1 4202560 22292 0 0 0 467 6 0 0 20 0 8 0 390947830 228724736 33336 33554432000 134512640 146736320 4289841024 4120026768 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=223364 CPUtime=5.16 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) R 23636 23636 22465 0 -1 4202560 24831 0 0 0 509 7 0 0 20 0 8 0 390947830 228724736 33336 33554432000 134512640 146736320 4289841024 4099923968 136019069 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=223364 CPUtime=4.95 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) S 23636 23636 22465 0 -1 4202560 26391 0 0 0 488 7 0 0 20 0 8 0 390947830 228724736 33336 33554432000 134512640 146736320 4289841024 4089441936 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=223364 CPUtime=5.15 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) S 23636 23636 22465 0 -1 4202560 35771 0 0 0 505 10 0 0 20 0 8 0 390947830 228724736 33336 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 39.39
Current children cumulated vsize (KiB) 232512

[startup+25.5004 s]
/proc/loadavg: 5.48 8.07 7.19 6/193 23659
/proc/meminfo: memFree=30066332/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=253972 CPUtime=75.59 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) R 23636 23636 22465 0 -1 4202496 377572 0 0 0 7440 119 0 0 20 0 8 0 390947795 260067328 40626 33554432000 134512640 146736320 4289841024 4289828224 139039594 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/23637/statm: 63493 40626 992 2985 0 60427 0
[pid=23637/tid=23653] ppid=23636 vsize=253972 CPUtime=8.94 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) S 23636 23636 22465 0 -1 4202560 34471 0 0 0 885 9 0 0 20 0 8 0 390947830 260067328 40626 33554432000 134512640 146736320 4289841024 4131171984 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23654] ppid=23636 vsize=253972 CPUtime=9.27 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) S 23636 23636 22465 0 -1 4202560 42220 0 0 0 913 14 0 0 20 0 8 0 390947830 260067328 40626 33554432000 134512640 146736320 4289841024 4141657744 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=253972 CPUtime=8.98 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) S 23636 23636 22465 0 -1 4202560 39819 0 0 0 886 12 0 0 20 0 8 0 390947830 260067328 40626 33554432000 134512640 146736320 4289841024 4152143504 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=253972 CPUtime=8.62 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) S 23636 23636 22465 0 -1 4202560 46909 0 0 0 849 13 0 0 20 0 8 0 390947830 260067328 40626 33554432000 134512640 146736320 4289841024 4120026768 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=253972 CPUtime=9.4 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) S 23636 23636 22465 0 -1 4202560 41237 0 0 0 928 12 0 0 20 0 8 0 390947830 260067328 40626 33554432000 134512640 146736320 4289841024 4099927696 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=253972 CPUtime=9.33 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) S 23636 23636 22465 0 -1 4202560 45872 0 0 0 920 13 0 0 20 0 8 0 390947830 260067328 40626 33554432000 134512640 146736320 4289841024 4089441936 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=253972 CPUtime=8.96 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) S 23636 23636 22465 0 -1 4202560 60960 0 0 0 880 16 0 0 20 0 8 0 390947830 260067328 40626 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 75.59
Current children cumulated vsize (KiB) 263120

[startup+51.1005 s]
/proc/loadavg: 5.69 7.89 7.15 9/193 23659
/proc/meminfo: memFree=30060552/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0

################
# More data... #
################

/proc/23637/statm: 65592 43104 1010 2985 0 62526 0
[pid=23637/tid=23653] ppid=23636 vsize=262368 CPUtime=129.17 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 229281 0 0 0 12799 118 0 0 20 0 8 0 390947830 268664832 43104 33554432000 134512640 146736320 4289841024 4131168236 135454093 0 2 4096 16387 0 0 0 -1 3 0 0 1 0 0
[pid=23637/tid=23654] ppid=23636 vsize=262368 CPUtime=133.53 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) R 23636 23636 22465 0 -1 4202560 214618 0 0 0 13230 123 0 0 20 0 8 0 390947830 268664832 43104 33554432000 134512640 146736320 4289841024 4141654016 135468509 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=262368 CPUtime=133.7 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) R 23636 23636 22465 0 -1 4202560 297898 0 0 0 13230 140 0 0 20 0 8 0 390947830 268664832 43104 33554432000 134512640 146736320 4289841024 4152140216 136014940 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=262368 CPUtime=134.45 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) R 23636 23636 22465 0 -1 4202560 254767 0 0 0 13317 128 0 0 20 0 8 0 390947830 268664832 43104 33554432000 134512640 146736320 4289841024 4120023296 136490027 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=262368 CPUtime=131.36 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) R 23636 23636 22465 0 -1 4202560 264430 0 0 0 13005 131 0 0 20 0 8 0 390947830 268664832 43104 33554432000 134512640 146736320 4289841024 4099924968 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=262368 CPUtime=131.77 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) R 23636 23636 22465 0 -1 4202560 267325 0 0 0 13051 126 0 0 20 0 8 0 390947830 268664832 43104 33554432000 134512640 146736320 4289841024 4089438648 136015707 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=262368 CPUtime=131.2 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) R 23636 23636 22465 0 -1 4202560 268958 0 0 0 12987 133 0 0 20 0 8 0 390947830 268664832 43104 33554432000 134512640 146736320 4289841024 4070564840 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 1075.04
Current children cumulated vsize (KiB) 271516

[startup+402.3 s]
/proc/loadavg: 7.20 7.59 7.26 8/194 23665
/proc/meminfo: memFree=30050392/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=262868 CPUtime=1263.96 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) R 23636 23636 22465 0 -1 4202496 2325672 0 0 0 125178 1218 0 0 20 0 8 0 390947795 269176832 43120 33554432000 134512640 146736320 4289841024 4289834664 144757825 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 1 0 0
/proc/23637/statm: 65717 43120 1010 2985 0 62651 0
[pid=23637/tid=23653] ppid=23636 vsize=262868 CPUtime=152.18 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 252233 0 0 0 15076 142 0 0 20 0 8 0 390947830 269176832 43120 33554432000 134512640 146736320 4289841024 4131168252 135526778 0 2 4096 16387 0 0 0 -1 3 0 0 1 0 0
[pid=23637/tid=23654] ppid=23636 vsize=262868 CPUtime=156.49 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) R 23636 23636 22465 0 -1 4202560 239164 0 0 0 15502 147 0 0 20 0 8 0 390947830 269176832 43120 33554432000 134512640 146736320 4289841024 4141654916 135850298 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=262868 CPUtime=157.37 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) R 23636 23636 22465 0 -1 4202560 320447 0 0 0 15574 163 0 0 20 0 8 0 390947830 269176832 43120 33554432000 134512640 146736320 4289841024 4152139912 135468239 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=262868 CPUtime=158.55 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) R 23636 23636 22465 0 -1 4202560 273606 0 0 0 15705 150 0 0 20 0 8 0 390947830 269176832 43120 33554432000 134512640 146736320 4289841024 4120021740 135454096 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=262868 CPUtime=154.29 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) R 23636 23636 22465 0 -1 4202560 294163 0 0 0 15276 153 0 0 20 0 8 0 390947830 269176832 43120 33554432000 134512640 146736320 4289841024 4099923948 135454096 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=262868 CPUtime=154.47 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) R 23636 23636 22465 0 -1 4202560 305963 0 0 0 15295 152 0 0 20 0 8 0 390947830 269176832 43120 33554432000 134512640 146736320 4289841024 4089436908 135454128 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=262868 CPUtime=153.85 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) R 23636 23636 22465 0 -1 4202560 296150 0 0 0 15229 156 0 0 20 0 8 0 390947830 269176832 43120 33554432000 134512640 146736320 4289841024 4070564840 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 1263.96
Current children cumulated vsize (KiB) 272016

[startup+462.3 s]
/proc/loadavg: 7.63 7.59 7.27 9/193 23672
/proc/meminfo: memFree=30040064/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=264228 CPUtime=1453.16 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) S 23636 23636 22465 0 -1 4202496 2551474 0 0 0 143915 1401 0 0 20 0 8 0 390947795 270569472 43504 33554432000 134512640 146736320 4289841024 4289837392 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 1 0 0
/proc/23637/statm: 66057 43504 1010 2985 0 62991 0
[pid=23637/tid=23653] ppid=23636 vsize=264228 CPUtime=175.57 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 290418 0 0 0 17389 168 0 0 20 0 8 0 390947830 270569472 43504 33554432000 134512640 146736320 4289841024 4131168856 136402278 0 2 4096 16387 0 0 0 -1 7 0 0 1 0 0
[pid=23637/tid=23654] ppid=23636 vsize=264228 CPUtime=179.19 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) R 23636 23636 22465 0 -1 4202560 261057 0 0 0 17750 169 0 0 20 0 8 0 390947830 270569472 43504 33554432000 134512640 146736320 4289841024 4141654016 135468700 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=264228 CPUtime=180.81 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) R 23636 23636 22465 0 -1 4202560 342348 0 0 0 17894 187 0 0 20 0 8 0 390947830 270569472 43504 33554432000 134512640 146736320 4289841024 4152140776 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=264228 CPUtime=182.63 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) S 23636 23636 22465 0 -1 4202560 291438 0 0 0 18093 170 0 0 20 0 8 0 390947830 270569472 43504 33554432000 134512640 146736320 4289841024 4120026768 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=264228 CPUtime=177.4 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) S 23636 23636 22465 0 -1 4202560 345216 0 0 0 17559 181 0 0 20 0 8 0 390947830 270569472 43504 33554432000 134512640 146736320 4289841024 4099927696 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=264228 CPUtime=177.49 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) S 23636 23636 22465 0 -1 4202560 334264 0 0 0 17576 173 0 0 20 0 8 0 390947830 270569472 43504 33554432000 134512640 146736320 4289841024 4089441936 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=264228 CPUtime=176.77 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) R 23636 23636 22465 0 -1 4202560 315915 0 0 0 17501 176 0 0 20 0 8 0 390947830 270569472 43504 33554432000 134512640 146736320 4289841024 4070563764 135450767 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1453.16
Current children cumulated vsize (KiB) 273376

[startup+522.3 s]
/proc/loadavg: 7.09 7.52 7.27 3/186 23724
/proc/meminfo: memFree=30068468/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=265108 CPUtime=1649.84 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) S 23636 23636 22465 0 -1 4202496 2787257 0 0 0 163392 1592 0 0 20 0 8 0 390947795 271470592 43604 33554432000 134512640 146736320 4289841024 4289837392 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 1 0 0
/proc/23637/statm: 66277 43604 1010 2985 0 63211 0
[pid=23637/tid=23653] ppid=23636 vsize=265108 CPUtime=200.02 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) S 23636 23636 22465 0 -1 4202560 333173 0 0 0 19804 198 0 0 20 0 8 0 390947830 271470592 43604 33554432000 134512640 146736320 4289841024 4131171984 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 1 0 0
[pid=23637/tid=23654] ppid=23636 vsize=265108 CPUtime=202.92 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) S 23636 23636 22465 0 -1 4202560 272530 0 0 0 20105 187 0 0 20 0 8 0 390947830 271470592 43604 33554432000 134512640 146736320 4289841024 4141657744 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=265108 CPUtime=205.61 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) S 23636 23636 22465 0 -1 4202560 376740 0 0 0 20348 213 0 0 20 0 8 0 390947830 271470592 43604 33554432000 134512640 146736320 4289841024 4152143504 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=265108 CPUtime=207.61 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) S 23636 23636 22465 0 -1 4202560 326378 0 0 0 20564 197 0 0 20 0 8 0 390947830 271470592 43604 33554432000 134512640 146736320 4289841024 4120026768 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=265108 CPUtime=201.48 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) R 23636 23636 22465 0 -1 4202560 376784 0 0 0 19941 207 0 0 20 0 8 0 390947830 271470592 43604 33554432000 134512640 146736320 4289841024 4099927724 144580332 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=265108 CPUtime=201.63 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) S 23636 23636 22465 0 -1 4202560 358827 0 0 0 19965 198 0 0 20 0 8 0 390947830 271470592 43604 33554432000 134512640 146736320 4289841024 4089441936 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=265108 CPUtime=200.85 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) S 23636 23636 22465 0 -1 4202560 335046 0 0 0 19890 195 0 0 20 0 8 0 390947830 271470592 43604 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 1649.84
Current children cumulated vsize (KiB) 274256



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+566.8 s]
/proc/loadavg: 6.17 7.21 7.18 10/186 23724
/proc/meminfo: memFree=30054032/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=267356 CPUtime=1800.23 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) R 23636 23636 22465 0 -1 4202496 2942682 0 0 0 178294 1729 0 0 20 0 8 0 390947795 273772544 43965 33554432000 134512640 146736320 4289841024 4289833772 135453913 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 1 0 0
/proc/23637/statm: 66839 43965 1010 2985 0 63773 0
[pid=23637/tid=23653] ppid=23636 vsize=267356 CPUtime=218.56 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 352057 0 0 0 21643 213 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4131168696 136015579 0 2 4096 16387 0 0 0 -1 1 0 0 1 0 0
[pid=23637/tid=23654] ppid=23636 vsize=267356 CPUtime=221.41 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) R 23636 23636 22465 0 -1 4202560 289900 0 0 0 21937 204 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4141657772 144580332 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=267356 CPUtime=224.51 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) R 23636 23636 22465 0 -1 4202560 392574 0 0 0 22221 230 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4152139700 135450610 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=267356 CPUtime=226.81 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) R 23636 23636 22465 0 -1 4202560 343666 0 0 0 22466 215 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4120026796 144580332 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=267356 CPUtime=220.03 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) R 23636 23636 22465 0 -1 4202560 399795 0 0 0 21778 225 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4099924868 135850322 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=267356 CPUtime=220.12 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) R 23636 23636 22465 0 -1 4202560 383462 0 0 0 21795 217 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4089436992 135468690 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=267356 CPUtime=219.36 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) R 23636 23636 22465 0 -1 4202560 357094 0 0 0 21723 213 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1800.23
Current children cumulated vsize (KiB) 276504

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+566.8 s]
/proc/loadavg: 6.17 7.21 7.18 10/186 23724
/proc/meminfo: memFree=30054032/32873648 swapFree=7372/7372
[pid=23636] ppid=23634 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/23636/stat : 23636 (run) S 23634 23636 22465 0 -1 4202496 354 0 0 0 0 0 0 0 20 0 1 0 390947794 9367552 269 33554432000 4194304 5064252 140735631964880 140735631963792 259112199758 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/23636/statm: 2287 269 229 213 0 72 0
[pid=23637] ppid=23636 vsize=267356 CPUtime=1800.23 cores=1,3,5,7
/proc/23637/stat : 23637 (pb_cplex) R 23636 23636 22465 0 -1 4202496 2942682 0 0 0 178294 1729 0 0 20 0 8 0 390947795 273772544 43965 33554432000 134512640 146736320 4289841024 4289833772 135453913 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 1 0 0
/proc/23637/statm: 66839 43965 1010 2985 0 63773 0
[pid=23637/tid=23653] ppid=23636 vsize=267356 CPUtime=218.56 cores=1,3,5,7
/proc/23637/task/23653/stat : 23653 (pb_cplex) R 23636 23636 22465 0 -1 4202560 352057 0 0 0 21643 213 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4131168696 136015579 0 2 4096 16387 0 0 0 -1 1 0 0 1 0 0
[pid=23637/tid=23654] ppid=23636 vsize=267356 CPUtime=221.41 cores=1,3,5,7
/proc/23637/task/23654/stat : 23654 (pb_cplex) R 23636 23636 22465 0 -1 4202560 289900 0 0 0 21937 204 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4141657772 144580332 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23655] ppid=23636 vsize=267356 CPUtime=224.51 cores=1,3,5,7
/proc/23637/task/23655/stat : 23655 (pb_cplex) R 23636 23636 22465 0 -1 4202560 392574 0 0 0 22221 230 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4152139700 135450610 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=23637/tid=23656] ppid=23636 vsize=267356 CPUtime=226.81 cores=1,3,5,7
/proc/23637/task/23656/stat : 23656 (pb_cplex) R 23636 23636 22465 0 -1 4202560 343666 0 0 0 22466 215 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4120026796 144580332 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23657] ppid=23636 vsize=267356 CPUtime=220.03 cores=1,3,5,7
/proc/23637/task/23657/stat : 23657 (pb_cplex) R 23636 23636 22465 0 -1 4202560 399795 0 0 0 21778 225 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4099924868 135850322 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=23637/tid=23658] ppid=23636 vsize=267356 CPUtime=220.12 cores=1,3,5,7
/proc/23637/task/23658/stat : 23658 (pb_cplex) R 23636 23636 22465 0 -1 4202560 383462 0 0 0 21795 217 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4089436992 135468690 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=23637/tid=23659] ppid=23636 vsize=267356 CPUtime=219.36 cores=1,3,5,7
/proc/23637/task/23659/stat : 23659 (pb_cplex) R 23636 23636 22465 0 -1 4202560 357094 0 0 0 21723 213 0 0 20 0 8 0 390947830 273772544 43965 33554432000 134512640 146736320 4289841024 4070567568 144580073 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1800.23
Current children cumulated vsize (KiB) 276504

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 23636 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=0
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=0
# CPU time returned by wait4() is 0
# while last known CPU time is 1800.23
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 566.817
CPU time (s): 1800.23
CPU user time (s): 1782.94
CPU system time (s): 17.29
CPU usage (%): 317.604
Max. virtual memory (cumulated for all children) (KiB): 324320

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0
system time used= 0
maximum resident set size= 1076
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 354
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 2

runsolver used 1.57176 second user time and 3.87441 second system time

The end

Launcher Data 

Begin job on node124 at 2012-06-09 15:23:39
IDJOB=3736611
IDBENCH=48017
IDSOLVER=2325
FILE ID=node124/3736611-1339248219
RUNJOBID= node124-1339246831-22482
PBS_JOBID= 14644357
Free space on /tmp= 70584 MiB

SOLVER NAME= PB10: pb_cplex 2010-06-29
BENCH NAME= PB07/normalized-PB07/OPT-SMALLINT-NLC/submittedPB07/roussel/factor-mod-B/factor-mod-size=8-P0=229-P1=67-P2=149-P3=97-P4=149-P5=211-P6=89-P7=59-B.opb
COMMAND LINE= run BENCHNAME
CONVERSION SCRIPT= PBconversionToLinear BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3736611-1339248219/watcher-3736611-1339248219 -o /tmp/evaluation-result-3736611-1339248219/solver-3736611-1339248219 -C 1800 -W 1900 -M 15500  run HOME/instance-3736611-1339248219.opb

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 12f1f3305860fafcd361cd1c61e0d104
RANDOM SEED=1326513762

node124.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5385.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5385.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5357.53
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5385.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5385.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5385.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.325
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5385.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        30227496 kB
Buffers:          453000 kB
Cached:          1259740 kB
SwapCached:         8612 kB
Active:          1689404 kB
Inactive:          83724 kB
Active(anon):      45456 kB
Inactive(anon):    11320 kB
Active(file):    1643948 kB
Inactive(file):    72404 kB
Unevictable:        7372 kB
Mlocked:            7372 kB
SwapTotal:      67108856 kB
SwapFree:       67092344 kB
Dirty:             16704 kB
Writeback:             0 kB
AnonPages:         53672 kB
Mapped:            12988 kB
Shmem:                 0 kB
Slab:             733304 kB
SReclaimable:     177180 kB
SUnreclaim:       556124 kB
KernelStack:        1512 kB
PageTables:         4048 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:     205128 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:     10240 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

CONVERSION COMMAND LINE= PBconversionToLinear HOME/instance-3736611-1339248219.opb

Free space on /tmp at the end= 70568 MiB
End job on node124 at 2012-06-09 15:33:06