Trace number 3741226

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB10: pb_cplex 2010-06-29? (TO) 1800.36 516.016

General information on the benchmark

Name/DEC-SMALLINT-LIN/heinz/
normalized-neos-849702.opb
MD5SUM1d9496473cab6648d4a43b81cc40ffb9
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark4.78427
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables1737
Total number of constraints1041
Number of constraints which are clauses0
Number of constraints which are cardinality constraints (but not clauses)954
Number of constraints which are nor clauses,nor cardinality constraints87
Minimum length of a constraint9
Maximum length of a constraint32
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 2
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 34
Number of bits of the biggest sum of numbers6
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing opb file format. File HOME/instance-3741226-1339249980.opb.
0.00/0.00	c Instance file HOME/instance-3741226-1339249980.opb
0.00/0.00	c File size is 167971 bytes.
0.00/0.05	c Highest Coefficient sum: 34
0.00/0.05	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.05	c ============================[ Problem Statistics ]=============================
0.00/0.05	c |                                                                             |
0.00/0.05	c |  Parsing time:          0.05         s                                      |
0.00/0.05	c |  Number Variables:      1737                                                |
0.00/0.05	c |  Number Constraints:    1041                                                |
0.00/0.05	c ===============================================================================
1800.36/516.01	c Remaining time 1799.94 sec.
1800.36/516.01	c 
1800.36/516.01	c *** INTERRUPTED ***
1800.36/516.01	s UNKNOWN
1800.36/516.01	c Total CPU time              : 1799.32 s

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3741226-1339249980/watcher-3741226-1339249980 -o /tmp/evaluation-result-3741226-1339249980/solver-3741226-1339249980 -C 1800 -W 1900 -M 15500 run HOME/instance-3741226-1339249980.opb 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.17 3.10 4.87 3/177 20786
/proc/meminfo: memFree=27119492/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0

[startup+0.0893299 s]
/proc/loadavg: 2.17 3.10 4.87 3/177 20786
/proc/meminfo: memFree=27119492/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.10023 s]
/proc/loadavg: 2.17 3.10 4.87 3/177 20786
/proc/meminfo: memFree=27119492/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.300183 s]
/proc/loadavg: 2.17 3.10 4.87 3/177 20786
/proc/meminfo: memFree=27119492/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.700186 s]
/proc/loadavg: 2.17 3.10 4.87 3/177 20786
/proc/meminfo: memFree=27119492/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+1.50022 s]
/proc/loadavg: 2.17 3.10 4.87 3/179 20788
/proc/meminfo: memFree=27060440/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=23440 CPUtime=1.48 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 4420 0 0 0 148 0 0 0 20 0 1 0 391123954 24002560 3018 33554432000 134512640 146736320 4293956112 4293949760 136388081 0 0 4096 16387 0 0 0 17 3 0 0 0 0 0
/proc/20787/statm: 5860 3018 833 2985 0 2794 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 32588

[startup+3.10022 s]
/proc/loadavg: 2.17 3.10 4.87 3/179 20788
/proc/meminfo: memFree=26937560/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=22436 CPUtime=3.09 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 4817 0 0 0 308 1 0 0 20 0 1 0 391123954 22974464 2952 33554432000 134512640 146736320 4293956112 4293950496 136388276 0 0 4096 16387 0 0 0 17 3 0 0 0 0 0
/proc/20787/statm: 5609 2952 853 2985 0 2543 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 31584

[startup+6.30022 s]
/proc/loadavg: 2.16 3.08 4.85 3/179 20788
/proc/meminfo: memFree=27091592/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=25412 CPUtime=6.29 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 5701 0 0 0 628 1 0 0 20 0 1 0 391123954 26021888 3512 33554432000 134512640 146736320 4293956112 4293949944 138768232 0 0 4096 16387 0 0 0 17 3 0 0 0 0 0
/proc/20787/statm: 6353 3512 853 2985 0 3287 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 34560

[startup+12.7002 s]
/proc/loadavg: 2.14 3.07 4.84 3/179 20788
/proc/meminfo: memFree=27087212/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=22204 CPUtime=12.69 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 5929 0 0 0 1268 1 0 0 20 0 1 0 391123954 22736896 2900 33554432000 134512640 146736320 4293956112 4293950336 139046460 0 0 4096 16387 0 0 0 17 3 0 0 0 0 0
/proc/20787/statm: 5551 2900 857 2985 0 2485 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 31352

[startup+25.5004 s]
/proc/loadavg: 2.18 3.03 4.80 10/186 20795
/proc/meminfo: memFree=27050588/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=134420 CPUtime=33.76 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 14149 0 0 0 3373 3 0 0 20 0 8 0 391123954 137646080 10637 33554432000 134512640 146736320 4293956112 4293948944 136008039 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/20787/statm: 33605 10637 898 2985 0 30539 0
[pid=20787/tid=20789] ppid=20786 vsize=134420 CPUtime=1.36 cores=1,3,5,7
/proc/20787/task/20789/stat : 20789 (pb_cplex) R 20786 20786 19841 0 -1 4202560 484 0 0 0 136 0 0 0 20 0 8 0 391125816 137646080 10637 33554432000 134512640 146736320 4293956112 4141334748 136389135 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20790] ppid=20786 vsize=134420 CPUtime=1.36 cores=1,3,5,7
/proc/20787/task/20790/stat : 20790 (pb_cplex) R 20786 20786 19841 0 -1 4202560 441 0 0 0 136 0 0 0 20 0 8 0 391125816 137646080 10637 33554432000 134512640 146736320 4293956112 4130849360 136403112 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20791] ppid=20786 vsize=134420 CPUtime=1.45 cores=1,3,5,7
/proc/20787/task/20791/stat : 20791 (pb_cplex) R 20786 20786 19841 0 -1 4202560 538 0 0 0 145 0 0 0 20 0 8 0 391125816 137646080 10637 33554432000 134512640 146736320 4293956112 4120361792 138462680 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20792] ppid=20786 vsize=134420 CPUtime=1.37 cores=1,3,5,7
/proc/20787/task/20792/stat : 20792 (pb_cplex) R 20786 20786 19841 0 -1 4202560 524 0 0 0 137 0 0 0 20 0 8 0 391125816 137646080 10637 33554432000 134512640 146736320 4293956112 4109877340 135855170 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20793] ppid=20786 vsize=134420 CPUtime=1.35 cores=1,3,5,7
/proc/20787/task/20793/stat : 20793 (pb_cplex) R 20786 20786 19841 0 -1 4202560 558 0 0 0 135 0 0 0 20 0 8 0 391125816 137646080 10637 33554432000 134512640 146736320 4293956112 4099391560 139042883 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20794] ppid=20786 vsize=134420 CPUtime=1.39 cores=1,3,5,7
/proc/20787/task/20794/stat : 20794 (pb_cplex) R 20786 20786 19841 0 -1 4202560 478 0 0 0 139 0 0 0 20 0 8 0 391125816 137646080 10637 33554432000 134512640 146736320 4293956112 4088904144 135461145 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20795] ppid=20786 vsize=134420 CPUtime=1.32 cores=1,3,5,7
/proc/20787/task/20795/stat : 20795 (pb_cplex) R 20786 20786 19841 0 -1 4202560 512 0 0 0 132 0 0 0 20 0 8 0 391125816 137646080 10637 33554432000 134512640 146736320 4293956112 4078420424 136000767 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 33.76
Current children cumulated vsize (KiB) 143568

[startup+51.1004 s]
/proc/loadavg: 4.15 3.43 4.88 6/186 20795
/proc/meminfo: memFree=27046516/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=135616 CPUtime=129.93 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) S 20786 20786 19841 0 -1 4202496 14246 0 0 0 12984 9 0 0 20 0 8 0 391123954 138870784 10732 33554432000 134512640 146736320 4293956112 4293952464 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/20787/statm: 33904 10732 899 2985 0 30838 0
[pid=20787/tid=20789] ppid=20786 vsize=135616 CPUtime=13.64 cores=1,3,5,7
/proc/20787/task/20789/stat : 20789 (pb_cplex) R 20786 20786 19841 0 -1 4202560 492 0 0 0 1364 0 0 0 20 0 8 0 391125816 138870784 10732 33554432000 134512640 146736320 4293956112 4141334528 136019021 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20790] ppid=20786 vsize=135616 CPUtime=12.42 cores=1,3,5,7
/proc/20787/task/20790/stat : 20790 (pb_cplex) R 20786 20786 19841 0 -1 4202560 464 0 0 0 1242 0 0 0 20 0 8 0 391125816 138870784 10732 33554432000 134512640 146736320 4293956112 4130849360 136403109 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20791] ppid=20786 vsize=135616 CPUtime=13.12 cores=1,3,5,7
/proc/20787/task/20791/stat : 20791 (pb_cplex) S 20786 20786 19841 0 -1 4202560 549 0 0 0 1312 0 0 0 20 0 8 0 391125816 138870784 10732 33554432000 134512640 146736320 4293956112 4120366736 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20792] ppid=20786 vsize=135616 CPUtime=13.16 cores=1,3,5,7
/proc/20787/task/20792/stat : 20792 (pb_cplex) S 20786 20786 19841 0 -1 4202560 533 0 0 0 1316 0 0 0 20 0 8 0 391125816 138870784 10732 33554432000 134512640 146736320 4293956112 4109880976 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20793] ppid=20786 vsize=135616 CPUtime=15.34 cores=1,3,5,7
/proc/20787/task/20793/stat : 20793 (pb_cplex) S 20786 20786 19841 0 -1 4202560 560 0 0 0 1532 2 0 0 20 0 8 0 391125816 138870784 10732 33554432000 134512640 146736320 4293956112 4099395216 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20794] ppid=20786 vsize=135616 CPUtime=14.48 cores=1,3,5,7
/proc/20787/task/20794/stat : 20794 (pb_cplex) S 20786 20786 19841 0 -1 4202560 506 0 0 0 1448 0 0 0 20 0 8 0 391125816 138870784 10732 33554432000 134512640 146736320 4293956112 4088909456 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20795] ppid=20786 vsize=135616 CPUtime=13.83 cores=1,3,5,7
/proc/20787/task/20795/stat : 20795 (pb_cplex) R 20786 20786 19841 0 -1 4202560 513 0 0 0 1383 0 0 0 20 0 8 0 391125816 138870784 10732 33554432000 134512640 146736320 4293956112 4078419968 136018974 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 129.93
Current children cumulated vsize (KiB) 144764

[startup+102.306 s]
/proc/loadavg: 3.76 3.49 4.83 3/179 20795
/proc/meminfo: memFree=27028256/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=120996 CPUtime=222.45 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 21935 0 0 0 22223 22 0 0 20 0 1 0 391123954 123899904 14387 33554432000 134512640 146736320 4293956112 4293951192 144757825 0 0 4096 16387 0 0 0 17 5 0 0 0 0 0
/proc/20787/statm: 30249 14387 902 2985 0 27183 0
Current children cumulated CPU time (s) 222.45
Current children cumulated vsize (KiB) 130144

[startup+162.3 s]
/proc/loadavg: 6.09 4.15 4.96 3/186 20809
/proc/meminfo: memFree=26845468/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=405740 CPUtime=420.83 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 92372 0 0 0 42032 51 0 0 20 0 8 0 391123954 415477760 61821 33554432000 134512640 146736320 4293956112 4293948944 136008309 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/20787/statm: 101435 61821 913 2985 0 98369 0
[pid=20787/tid=20803] ppid=20786 vsize=405740 CPUtime=28.47 cores=1,3,5,7
/proc/20787/task/20803/stat : 20803 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3927 0 0 0 2845 2 0 0 20 0 8 0 391134395 415477760 61821 33554432000 134512640 146736320 4293956112 4021805120 136382627 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20804] ppid=20786 vsize=405740 CPUtime=22.78 cores=1,3,5,7
/proc/20787/task/20804/stat : 20804 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3715 0 0 0 2276 2 0 0 20 0 8 0 391134395 415477760 61821 33554432000 134512640 146736320 4293956112 4033023176 135468104 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20805] ppid=20786 vsize=405740 CPUtime=25.19 cores=1,3,5,7
/proc/20787/task/20805/stat : 20805 (pb_cplex) R 20786 20786 19841 0 -1 4202560 2391 0 0 0 2517 2 0 0 20 0 8 0 391134395 415477760 61821 33554432000 134512640 146736320 4293956112 4043509980 136389140 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20806] ppid=20786 vsize=405740 CPUtime=23.13 cores=1,3,5,7
/proc/20787/task/20806/stat : 20806 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4111 0 0 0 2310 3 0 0 20 0 8 0 391134395 415477760 61821 33554432000 134512640 146736320 4293956112 4109877304 136476016 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20807] ppid=20786 vsize=405740 CPUtime=20.63 cores=1,3,5,7
/proc/20787/task/20807/stat : 20807 (pb_cplex) R 20786 20786 19841 0 -1 4202560 2699 0 0 0 2061 2 0 0 20 0 8 0 391134395 415477760 61821 33554432000 134512640 146736320 4293956112 3820788948 135455302 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20808] ppid=20786 vsize=405740 CPUtime=22.81 cores=1,3,5,7
/proc/20787/task/20808/stat : 20808 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3058 0 0 0 2279 2 0 0 20 0 8 0 391134395 415477760 61821 33554432000 134512640 146736320 4293956112 3810305000 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20809] ppid=20786 vsize=405740 CPUtime=24.78 cores=1,3,5,7
/proc/20787/task/20809/stat : 20809 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3120 0 0 0 2475 3 0 0 20 0 8 0 391134395 415477760 61821 33554432000 134512640 146736320 4293956112 3799817024 138462602 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 420.83

################
# More data... #
################

/proc/20787/statm: 101085 61779 919 2985 0 98019 0
[pid=20787/tid=20803] ppid=20786 vsize=404340 CPUtime=57.96 cores=1,3,5,7
/proc/20787/task/20803/stat : 20803 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3991 0 0 0 5792 4 0 0 20 0 8 0 391134395 414044160 61779 33554432000 134512640 146736320 4293956112 4021805512 136000412 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20804] ppid=20786 vsize=404340 CPUtime=52.14 cores=1,3,5,7
/proc/20787/task/20804/stat : 20804 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4666 0 0 0 5211 3 0 0 20 0 8 0 391134395 414044160 61779 33554432000 134512640 146736320 4293956112 4033024208 136008234 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20805] ppid=20786 vsize=404340 CPUtime=55.44 cores=1,3,5,7
/proc/20787/task/20805/stat : 20805 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3124 0 0 0 5540 4 0 0 20 0 8 0 391134395 414044160 61779 33554432000 134512640 146736320 4293956112 4043509828 136385174 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20806] ppid=20786 vsize=404340 CPUtime=52.02 cores=1,3,5,7
/proc/20787/task/20806/stat : 20806 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4837 0 0 0 5198 4 0 0 20 0 8 0 391134395 414044160 61779 33554432000 134512640 146736320 4293956112 4109876216 136476109 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20807] ppid=20786 vsize=404340 CPUtime=49.9 cores=1,3,5,7
/proc/20787/task/20807/stat : 20807 (pb_cplex) R 20786 20786 19841 0 -1 4202560 2920 0 0 0 4986 4 0 0 20 0 8 0 391134395 414044160 61779 33554432000 134512640 146736320 4293956112 3820789968 136008053 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20808] ppid=20786 vsize=404340 CPUtime=51.99 cores=1,3,5,7
/proc/20787/task/20808/stat : 20808 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3255 0 0 0 5195 4 0 0 20 0 8 0 391134395 414044160 61779 33554432000 134512640 146736320 4293956112 3810304456 136000422 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20809] ppid=20786 vsize=404340 CPUtime=54.28 cores=1,3,5,7
/proc/20787/task/20809/stat : 20809 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4232 0 0 0 5424 4 0 0 20 0 8 0 391134395 414044160 61779 33554432000 134512640 146736320 4293956112 3799818308 136385185 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 656.23
Current children cumulated vsize (KiB) 413488

[startup+282.3 s]
/proc/loadavg: 8.31 5.58 5.38 10/186 20810
/proc/meminfo: memFree=26827532/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=405792 CPUtime=888.69 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 106107 0 0 0 88797 72 0 0 20 0 8 0 391123954 415531008 61988 33554432000 134512640 146736320 4293956112 4293948240 136403117 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/20787/statm: 101448 61988 919 2985 0 98382 0
[pid=20787/tid=20803] ppid=20786 vsize=405792 CPUtime=86.06 cores=1,3,5,7
/proc/20787/task/20803/stat : 20803 (pb_cplex) R 20786 20786 19841 0 -1 4202560 8455 0 0 0 8599 7 0 0 20 0 8 0 391134395 415531008 61988 33554432000 134512640 146736320 4293956112 4021805264 136008328 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20804] ppid=20786 vsize=405792 CPUtime=81.82 cores=1,3,5,7
/proc/20787/task/20804/stat : 20804 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4678 0 0 0 8177 5 0 0 20 0 8 0 391134395 415531008 61988 33554432000 134512640 146736320 4293956112 4033022680 135466467 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20805] ppid=20786 vsize=405792 CPUtime=84.24 cores=1,3,5,7
/proc/20787/task/20805/stat : 20805 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3126 0 0 0 8418 6 0 0 20 0 8 0 391134395 415531008 61988 33554432000 134512640 146736320 4293956112 4043509968 136008042 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20806] ppid=20786 vsize=405792 CPUtime=81.46 cores=1,3,5,7
/proc/20787/task/20806/stat : 20806 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4838 0 0 0 8140 6 0 0 20 0 8 0 391134395 415531008 61988 33554432000 134512640 146736320 4293956112 4109877456 136008050 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20807] ppid=20786 vsize=405792 CPUtime=79.3 cores=1,3,5,7
/proc/20787/task/20807/stat : 20807 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4891 0 0 0 7924 6 0 0 20 0 8 0 391134395 415531008 61988 33554432000 134512640 146736320 4293956112 3820788744 136474274 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20808] ppid=20786 vsize=405792 CPUtime=81.18 cores=1,3,5,7
/proc/20787/task/20808/stat : 20808 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3276 0 0 0 8113 5 0 0 20 0 8 0 391134395 415531008 61988 33554432000 134512640 146736320 4293956112 3810304092 135855174 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20809] ppid=20786 vsize=405792 CPUtime=83.63 cores=1,3,5,7
/proc/20787/task/20809/stat : 20809 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4374 0 0 0 8357 6 0 0 20 0 8 0 391134395 415531008 61988 33554432000 134512640 146736320 4293956112 3799818448 136008160 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 888.69
Current children cumulated vsize (KiB) 414940

[startup+342.3 s]
/proc/loadavg: 8.41 6.11 5.58 10/186 20810
/proc/meminfo: memFree=26821384/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=405792 CPUtime=1122.33 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 110862 0 0 0 112151 82 0 0 20 0 8 0 391123954 415531008 62124 33554432000 134512640 146736320 4293956112 4293948736 136019043 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/20787/statm: 101448 62124 919 2985 0 98382 0
[pid=20787/tid=20803] ppid=20786 vsize=405792 CPUtime=115.41 cores=1,3,5,7
/proc/20787/task/20803/stat : 20803 (pb_cplex) R 20786 20786 19841 0 -1 4202560 12148 0 0 0 11532 9 0 0 20 0 8 0 391134395 415531008 62124 33554432000 134512640 146736320 4293956112 4021804872 135526398 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20804] ppid=20786 vsize=405792 CPUtime=111.1 cores=1,3,5,7
/proc/20787/task/20804/stat : 20804 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4824 0 0 0 11104 6 0 0 20 0 8 0 391134395 415531008 62124 33554432000 134512640 146736320 4293956112 4033024056 136475798 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20805] ppid=20786 vsize=405792 CPUtime=112.69 cores=1,3,5,7
/proc/20787/task/20805/stat : 20805 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3186 0 0 0 11263 6 0 0 20 0 8 0 391134395 415531008 62124 33554432000 134512640 146736320 4293956112 4043509968 136008065 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20806] ppid=20786 vsize=405792 CPUtime=111.37 cores=1,3,5,7
/proc/20787/task/20806/stat : 20806 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4963 0 0 0 11130 7 0 0 20 0 8 0 391134395 415531008 62124 33554432000 134512640 146736320 4293956112 4109877704 136001700 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20807] ppid=20786 vsize=405792 CPUtime=108.64 cores=1,3,5,7
/proc/20787/task/20807/stat : 20807 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5433 0 0 0 10856 8 0 0 20 0 8 0 391134395 415531008 62124 33554432000 134512640 146736320 4293956112 3820789928 135983214 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20808] ppid=20786 vsize=405792 CPUtime=110.7 cores=1,3,5,7
/proc/20787/task/20808/stat : 20808 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3276 0 0 0 11063 7 0 0 20 0 8 0 391134395 415531008 62124 33554432000 134512640 146736320 4293956112 3810304092 135855179 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20809] ppid=20786 vsize=405792 CPUtime=112.68 cores=1,3,5,7
/proc/20787/task/20809/stat : 20809 (pb_cplex) R 20786 20786 19841 0 -1 4202560 4512 0 0 0 11261 7 0 0 20 0 8 0 391134395 415531008 62124 33554432000 134512640 146736320 4293956112 3799818448 136008039 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 1122.33
Current children cumulated vsize (KiB) 414940

[startup+402.3 s]
/proc/loadavg: 8.56 6.58 5.77 10/186 20810
/proc/meminfo: memFree=26796864/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=408864 CPUtime=1356.81 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 114893 0 0 0 135590 91 0 0 20 0 8 0 391123954 418676736 62852 33554432000 134512640 146736320 4293956112 4293948944 136008168 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/20787/statm: 102216 62852 919 2985 0 99150 0
[pid=20787/tid=20803] ppid=20786 vsize=408864 CPUtime=145.08 cores=1,3,5,7
/proc/20787/task/20803/stat : 20803 (pb_cplex) R 20786 20786 19841 0 -1 4202560 14604 0 0 0 14497 11 0 0 20 0 8 0 391134395 418676736 62852 33554432000 134512640 146736320 4293956112 4021805648 136403117 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20804] ppid=20786 vsize=408864 CPUtime=141.3 cores=1,3,5,7
/proc/20787/task/20804/stat : 20804 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5174 0 0 0 14123 7 0 0 20 0 8 0 391134395 418676736 62852 33554432000 134512640 146736320 4293956112 4033024056 136475826 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20805] ppid=20786 vsize=408864 CPUtime=143.1 cores=1,3,5,7
/proc/20787/task/20805/stat : 20805 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3584 0 0 0 14301 9 0 0 20 0 8 0 391134395 418676736 62852 33554432000 134512640 146736320 4293956112 4043508916 138462832 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20806] ppid=20786 vsize=408864 CPUtime=140.31 cores=1,3,5,7
/proc/20787/task/20806/stat : 20806 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5109 0 0 0 14022 9 0 0 20 0 8 0 391134395 418676736 62852 33554432000 134512640 146736320 4293956112 4109877524 136003848 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20807] ppid=20786 vsize=408864 CPUtime=136.75 cores=1,3,5,7
/proc/20787/task/20807/stat : 20807 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5433 0 0 0 13665 10 0 0 20 0 8 0 391134395 418676736 62852 33554432000 134512640 146736320 4293956112 3820789968 136008116 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20808] ppid=20786 vsize=408864 CPUtime=139.36 cores=1,3,5,7
/proc/20787/task/20808/stat : 20808 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3348 0 0 0 13927 9 0 0 20 0 8 0 391134395 418676736 62852 33554432000 134512640 146736320 4293956112 3810304068 136385184 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20809] ppid=20786 vsize=408864 CPUtime=142.09 cores=1,3,5,7
/proc/20787/task/20809/stat : 20809 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5089 0 0 0 14200 9 0 0 20 0 8 0 391134395 418676736 62852 33554432000 134512640 146736320 4293956112 3799818696 136000404 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1356.81
Current children cumulated vsize (KiB) 418012

[startup+462.3 s]
/proc/loadavg: 8.63 6.97 5.95 5/187 20813
/proc/meminfo: memFree=26795300/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=406816 CPUtime=1589.08 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 117330 0 0 0 158808 100 0 0 20 0 8 0 391123954 416579584 62757 33554432000 134512640 146736320 4293956112 4293948944 136008061 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/20787/statm: 101704 62757 919 2985 0 98638 0
[pid=20787/tid=20803] ppid=20786 vsize=406816 CPUtime=174.01 cores=1,3,5,7
/proc/20787/task/20803/stat : 20803 (pb_cplex) R 20786 20786 19841 0 -1 4202560 16924 0 0 0 17388 13 0 0 20 0 8 0 391134395 416579584 62757 33554432000 134512640 146736320 4293956112 4021805512 136001703 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20804] ppid=20786 vsize=406816 CPUtime=171.39 cores=1,3,5,7
/proc/20787/task/20804/stat : 20804 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5174 0 0 0 17131 8 0 0 20 0 8 0 391134395 416579584 62757 33554432000 134512640 146736320 4293956112 4033024072 139042897 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20805] ppid=20786 vsize=406816 CPUtime=171.49 cores=1,3,5,7
/proc/20787/task/20805/stat : 20805 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3585 0 0 0 17139 10 0 0 20 0 8 0 391134395 416579584 62757 33554432000 134512640 146736320 4293956112 4043509760 136018922 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20806] ppid=20786 vsize=406816 CPUtime=169.26 cores=1,3,5,7
/proc/20787/task/20806/stat : 20806 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5117 0 0 0 16916 10 0 0 20 0 8 0 391134395 416579584 62757 33554432000 134512640 146736320 4293956112 4109877320 139042897 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20807] ppid=20786 vsize=406816 CPUtime=166.01 cores=1,3,5,7
/proc/20787/task/20807/stat : 20807 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5436 0 0 0 16589 12 0 0 20 0 8 0 391134395 416579584 62757 33554432000 134512640 146736320 4293956112 3820789828 136385175 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20808] ppid=20786 vsize=406816 CPUtime=168.97 cores=1,3,5,7
/proc/20787/task/20808/stat : 20808 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3348 0 0 0 16887 10 0 0 20 0 8 0 391134395 416579584 62757 33554432000 134512640 146736320 4293956112 3810304224 139028318 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20809] ppid=20786 vsize=406816 CPUtime=170.61 cores=1,3,5,7
/proc/20787/task/20809/stat : 20809 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5141 0 0 0 17051 10 0 0 20 0 8 0 391134395 416579584 62757 33554432000 134512640 146736320 4293956112 3799818352 136384892 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 1589.08
Current children cumulated vsize (KiB) 415964



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+516 s]
/proc/loadavg: 8.49 7.20 6.09 10/187 20837
/proc/meminfo: memFree=26791764/32873648 swapFree=7628/7628
[pid=20786] ppid=20784 vsize=9148 CPUtime=0 cores=1,3,5,7
/proc/20786/stat : 20786 (run) S 20784 20786 19841 0 -1 4202496 356 0 0 0 0 0 0 0 20 0 1 0 391123954 9367552 270 33554432000 4194304 5064252 140733430863152 140733430862064 217410332238 0 65536 4100 65538 18446744071579302229 0 0 17 5 0 0 0 0 0
/proc/20786/statm: 2287 270 229 213 0 72 0
[pid=20787] ppid=20786 vsize=405844 CPUtime=1800.36 cores=1,3,5,7
/proc/20787/stat : 20787 (pb_cplex) R 20786 20786 19841 0 -1 4202496 121643 0 0 0 179928 108 0 0 20 0 8 0 391123954 415584256 62652 33554432000 134512640 146736320 4293956112 4293948784 136384852 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0 0 0
/proc/20787/statm: 101461 62652 919 2985 0 98395 0
[pid=20787/tid=20803] ppid=20786 vsize=405844 CPUtime=200.48 cores=1,3,5,7
/proc/20787/task/20803/stat : 20803 (pb_cplex) R 20786 20786 19841 0 -1 4202560 18962 0 0 0 20033 15 0 0 20 0 8 0 391134395 415584256 62652 33554432000 134512640 146736320 4293956112 4021805168 136384883 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
[pid=20787/tid=20804] ppid=20786 vsize=405844 CPUtime=197.78 cores=1,3,5,7
/proc/20787/task/20804/stat : 20804 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5174 0 0 0 19768 10 0 0 20 0 8 0 391134395 415584256 62652 33554432000 134512640 146736320 4293956112 4033024592 136403112 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20805] ppid=20786 vsize=405844 CPUtime=198.01 cores=1,3,5,7
/proc/20787/task/20805/stat : 20805 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3585 0 0 0 19789 12 0 0 20 0 8 0 391134395 415584256 62652 33554432000 134512640 146736320 4293956112 4043510036 136003825 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20806] ppid=20786 vsize=405844 CPUtime=195.37 cores=1,3,5,7
/proc/20787/task/20806/stat : 20806 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5117 0 0 0 19527 10 0 0 20 0 8 0 391134395 415584256 62652 33554432000 134512640 146736320 4293956112 4109877704 136001676 0 2 4096 16387 0 0 0 -1 1 0 0 0 0 0
[pid=20787/tid=20807] ppid=20786 vsize=405844 CPUtime=192.62 cores=1,3,5,7
/proc/20787/task/20807/stat : 20807 (pb_cplex) R 20786 20786 19841 0 -1 4202560 5436 0 0 0 19250 12 0 0 20 0 8 0 391134395 415584256 62652 33554432000 134512640 146736320 4293956112 3820789852 135855179 0 2 4096 16387 0 0 0 -1 7 0 0 0 0 0
[pid=20787/tid=20808] ppid=20786 vsize=405844 CPUtime=195.02 cores=1,3,5,7
/proc/20787/task/20808/stat : 20808 (pb_cplex) R 20786 20786 19841 0 -1 4202560 3349 0 0 0 19492 10 0 0 20 0 8 0 391134395 415584256 62652 33554432000 134512640 146736320 4293956112 3810304208 136008441 0 2 4096 16387 0 0 0 -1 3 0 0 0 0 0
[pid=20787/tid=20809] ppid=20786 vsize=405844 CPUtime=197.38 cores=1,3,5,7
/proc/20787/task/20809/stat : 20809 (pb_cplex) R 20786 20786 19841 0 -1 4202560 6232 0 0 0 19727 11 0 0 20 0 8 0 391134395 415584256 62652 33554432000 134512640 146736320 4293956112 3799819288 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0 0 0
Current children cumulated CPU time (s) 1800.36
Current children cumulated vsize (KiB) 414992

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 20786 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=0
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=999
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1800.36
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 516.016
CPU time (s): 1800.36
CPU user time (s): 1799.28
CPU system time (s): 1.08
CPU usage (%): 348.896
Max. virtual memory (cumulated for all children) (KiB): 419036

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0
system time used= 0.000999
maximum resident set size= 1080
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 356
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 1

runsolver used 2.80057 second user time and 1.9847 second system time

The end

Launcher Data 

Begin job on node122 at 2012-06-09 15:53:00
IDJOB=3741226
IDBENCH=89820
IDSOLVER=2325
FILE ID=node122/3741226-1339249980
RUNJOBID= node122-1339246833-19859
PBS_JOBID= 14644359
Free space on /tmp= 64376 MiB

SOLVER NAME= PB10: pb_cplex 2010-06-29
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/heinz/normalized-neos-849702.opb
COMMAND LINE= run BENCHNAME
CONVERSION SCRIPT= PBconversionToLinear BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3741226-1339249980/watcher-3741226-1339249980 -o /tmp/evaluation-result-3741226-1339249980/solver-3741226-1339249980 -C 1800 -W 1900 -M 15500  run HOME/instance-3741226-1339249980.opb

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1d9496473cab6648d4a43b81cc40ffb9
RANDOM SEED=1867392880

node122.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5438.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        27120028 kB
Buffers:          230732 kB
Cached:          3334800 kB
SwapCached:         1868 kB
Active:          3159696 kB
Inactive:        1634136 kB
Active(anon):    1225220 kB
Inactive(anon):     4364 kB
Active(file):    1934476 kB
Inactive(file):  1629772 kB
Unevictable:        7628 kB
Mlocked:            7628 kB
SwapTotal:      67108856 kB
SwapFree:       67099792 kB
Dirty:             16304 kB
Writeback:             0 kB
AnonPages:       1389280 kB
Mapped:            12056 kB
Shmem:                 0 kB
Slab:             819568 kB
SReclaimable:     263596 kB
SUnreclaim:       555972 kB
KernelStack:        1488 kB
PageTables:         6728 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    2349292 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:    856064 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

CONVERSION COMMAND LINE= PBconversionToLinear HOME/instance-3741226-1339249980.opb

Free space on /tmp at the end= 64360 MiB
End job on node122 at 2012-06-09 16:01:36