Trace number 3736468

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerobjective functionCPU timeWall clock time
PB10: pb_cplex 2010-06-29? (TO) 1800.05 679.217

General information on the benchmark

Namenormalized-PB07/OPT-SMALLINT-NLC/submittedPB07/roussel/factor-mod-B/
factor-mod-size=7-P0=97-P1=37-P2=107-P3=53-P4=53-B.opb
MD5SUM3c1c5627c2ecb7d046a74fd6c2fb2be3
Bench CategoryOPT-SMALLINT-NLC (optimisation, small integers, non linear constraints)
Best result obtained on this benchmarkOPT
Best value of the objective obtained on this benchmark3
Best CPU time to get the best result obtained on this benchmark0.035993
Has Objective FunctionYES
SatisfiableYES
(Un)Satisfiability was provedYES
Best value of the objective function 3
Optimality of the best value was proved YES
Number of variables84
Total number of constraints9
Number of constraints which are clauses0
Number of constraints which are cardinality constraints (but not clauses)0
Number of constraints which are nor clauses,nor cardinality constraints9
Minimum length of a constraint7
Maximum length of a constraint63
Number of terms in the objective function 7
Biggest coefficient in the objective function 64
Number of bits for the biggest coefficient in the objective function 7
Sum of the numbers in the objective function 127
Number of bits of the sum of numbers in the objective function 7
Biggest number in a constraint 8192
Number of bits of the biggest number in a constraint 14
Biggest sum of numbers in a constraint 32512
Number of bits of the biggest sum of numbers15
Number of products (including duplicates)196
Sum of products size (including duplicates)392
Number of different products196
Sum of products size392

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing opb file format. File HOME/instance-3736468-1339248115.opb.
0.00/0.00	c Instance file HOME/instance-3736468-1339248115.opb
0.00/0.00	c File size is 14533 bytes.
0.00/0.00	c Highest Coefficient sum: 32512
0.00/0.00	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.00	c ============================[ Problem Statistics ]=============================
0.00/0.00	c |                                                                             |
0.00/0.00	c |  Parsing time:          0.00         s                                      |
0.00/0.00	c |  Number Variables:      280                                                 |
0.00/0.00	c |  Number Constraints:    401                                                 |
0.00/0.00	c ===============================================================================
1799.75/679.21	c Remaining time 1800.00 sec.
1799.75/679.21	c 
1799.75/679.21	c *** INTERRUPTED ***
1799.75/679.21	s UNKNOWN
1799.75/679.21	c Total CPU time              : 1772.8 s
1799.75/679.21	c Memory used           : 176.95 MB
1799.75/679.21	c CPU time              : 1772.8 s
1799.75/679.21	c 
1799.75/679.21	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3736468-1339248115/watcher-3736468-1339248115 -o /tmp/evaluation-result-3736468-1339248115/solver-3736468-1339248115 -C 1800 -W 1900 -M 15500 run HOME/instance-3736468-1339248115.opb 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 8.22 6.88 3/186 2423
/proc/meminfo: memFree=30360440/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0

[startup+0.095377 s]
/proc/loadavg: 8.11 8.22 6.88 3/186 2423
/proc/meminfo: memFree=30360440/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.100204 s]
/proc/loadavg: 8.11 8.22 6.88 3/186 2423
/proc/meminfo: memFree=30360440/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.300175 s]
/proc/loadavg: 8.11 8.22 6.88 3/186 2423
/proc/meminfo: memFree=30360440/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+0.700162 s]
/proc/loadavg: 8.11 8.22 6.88 3/186 2423
/proc/meminfo: memFree=30360440/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9148

[startup+1.5004 s]
/proc/loadavg: 7.78 8.15 6.87 10/195 2460
/proc/meminfo: memFree=30323380/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=125644 CPUtime=3.92 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) R 2423 2423 1671 0 -1 4202496 31250 0 0 0 375 17 0 0 20 0 8 0 34778405 128659456 11567 33554432000 134512640 146736320 4291399696 4291396076 144580332 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/2424/statm: 31411 11567 971 2985 0 28345 0
[pid=2424/tid=2454] ppid=2423 vsize=125644 CPUtime=0.39 cores=0,2,4,6
/proc/2424/task/2454/stat : 2454 (pb_cplex) R 2423 2423 1671 0 -1 4202560 1620 0 0 0 39 0 0 0 20 0 8 0 34778427 128659456 11567 33554432000 134512640 146736320 4291399696 4130574472 135468233 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2455] ppid=2423 vsize=125644 CPUtime=0.42 cores=0,2,4,6
/proc/2424/task/2455/stat : 2455 (pb_cplex) R 2423 2423 1671 0 -1 4202560 1717 0 0 0 40 2 0 0 20 0 8 0 34778427 128659456 11567 33554432000 134512640 146736320 4291399696 4141060304 136008157 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2456] ppid=2423 vsize=125644 CPUtime=0.43 cores=0,2,4,6
/proc/2424/task/2456/stat : 2456 (pb_cplex) R 2423 2423 1671 0 -1 4202560 2098 0 0 0 43 0 0 0 20 0 8 0 34778427 128659456 11567 33554432000 134512640 146736320 4291399696 4151546856 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2457] ppid=2423 vsize=125644 CPUtime=0.48 cores=0,2,4,6
/proc/2424/task/2457/stat : 2457 (pb_cplex) R 2423 2423 1671 0 -1 4202560 3894 0 0 0 47 1 0 0 20 0 8 0 34778427 128659456 11567 33554432000 134512640 146736320 4291399696 4119897064 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2458] ppid=2423 vsize=125644 CPUtime=0.39 cores=0,2,4,6
/proc/2424/task/2458/stat : 2458 (pb_cplex) R 2423 2423 1671 0 -1 4202560 3112 0 0 0 39 0 0 0 20 0 8 0 34778427 128659456 11567 33554432000 134512640 146736320 4291399696 4104122000 144580073 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2459] ppid=2423 vsize=125644 CPUtime=0.41 cores=0,2,4,6
/proc/2424/task/2459/stat : 2459 (pb_cplex) S 2423 2423 1671 0 -1 4202560 4125 0 0 0 41 0 0 0 20 0 8 0 34778428 128659456 11567 33554432000 134512640 146736320 4291399696 4093636240 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2460] ppid=2423 vsize=125644 CPUtime=0.4 cores=0,2,4,6
/proc/2424/task/2460/stat : 2460 (pb_cplex) S 2423 2423 1671 0 -1 4202560 4129 0 0 0 38 2 0 0 20 0 8 0 34778428 128659456 11567 33554432000 134512640 146736320 4291399696 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 3.92
Current children cumulated vsize (KiB) 134792

[startup+3.10041 s]
/proc/loadavg: 7.78 8.15 6.87 7/195 2460
/proc/meminfo: memFree=30307880/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=151972 CPUtime=8.81 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) R 2423 2423 1671 0 -1 4202496 54889 0 0 0 855 26 0 0 20 0 8 0 34778405 155619328 17164 33554432000 134512640 146736320 4291399696 4291392528 136008306 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2424/statm: 37993 17164 971 2985 0 34927 0
[pid=2424/tid=2454] ppid=2423 vsize=151972 CPUtime=0.96 cores=0,2,4,6
/proc/2424/task/2454/stat : 2454 (pb_cplex) R 2423 2423 1671 0 -1 4202560 4025 0 0 0 93 3 0 0 20 0 8 0 34778427 155619328 17164 33554432000 134512640 146736320 4291399696 4130573036 135454093 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2455] ppid=2423 vsize=151972 CPUtime=1.03 cores=0,2,4,6
/proc/2424/task/2455/stat : 2455 (pb_cplex) R 2423 2423 1671 0 -1 4202560 3560 0 0 0 100 3 0 0 20 0 8 0 34778427 155619328 17164 33554432000 134512640 146736320 4291399696 4141060092 135526884 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2456] ppid=2423 vsize=151972 CPUtime=1.02 cores=0,2,4,6
/proc/2424/task/2456/stat : 2456 (pb_cplex) R 2423 2423 1671 0 -1 4202560 3883 0 0 0 100 2 0 0 20 0 8 0 34778427 155619328 17164 33554432000 134512640 146736320 4291399696 4151546756 135851010 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2457] ppid=2423 vsize=151972 CPUtime=1.07 cores=0,2,4,6
/proc/2424/task/2457/stat : 2457 (pb_cplex) R 2423 2423 1671 0 -1 4202560 6140 0 0 0 104 3 0 0 20 0 8 0 34778427 155619328 17164 33554432000 134512640 146736320 4291399696 4119896192 136494120 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2458] ppid=2423 vsize=151972 CPUtime=1.01 cores=0,2,4,6
/proc/2424/task/2458/stat : 2458 (pb_cplex) R 2423 2423 1671 0 -1 4202560 8513 0 0 0 99 2 0 0 20 0 8 0 34778427 155619328 17164 33554432000 134512640 146736320 4291399696 4104118464 135468673 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2459] ppid=2423 vsize=151972 CPUtime=0.98 cores=0,2,4,6
/proc/2424/task/2459/stat : 2459 (pb_cplex) R 2423 2423 1671 0 -1 4202560 7083 0 0 0 96 2 0 0 20 0 8 0 34778428 155619328 17164 33554432000 134512640 146736320 4291399696 4093633412 135851010 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2460] ppid=2423 vsize=151972 CPUtime=1.12 cores=0,2,4,6
/proc/2424/task/2460/stat : 2460 (pb_cplex) R 2423 2423 1671 0 -1 4202560 8832 0 0 0 109 3 0 0 20 0 8 0 34778428 155619328 17164 33554432000 134512640 146736320 4291399696 4070567568 144580073 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 8.81
Current children cumulated vsize (KiB) 161120

[startup+6.30039 s]
/proc/loadavg: 7.78 8.15 6.87 4/195 2460
/proc/meminfo: memFree=30287188/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=158604 CPUtime=17.39 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) R 2423 2423 1671 0 -1 4202496 67715 0 0 0 1702 37 0 0 20 0 8 0 34778405 162410496 18922 33554432000 134512640 146736320 4291399696 4291385948 144785853 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2424/statm: 39651 18922 993 2985 0 36585 0
[pid=2424/tid=2454] ppid=2423 vsize=158604 CPUtime=1.94 cores=0,2,4,6
/proc/2424/task/2454/stat : 2454 (pb_cplex) S 2423 2423 1671 0 -1 4202560 4685 0 0 0 190 4 0 0 20 0 8 0 34778427 162410496 18922 33554432000 134512640 146736320 4291399696 4130578064 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2455] ppid=2423 vsize=158604 CPUtime=2.02 cores=0,2,4,6
/proc/2424/task/2455/stat : 2455 (pb_cplex) S 2423 2423 1671 0 -1 4202560 5952 0 0 0 197 5 0 0 20 0 8 0 34778427 162410496 18922 33554432000 134512640 146736320 4291399696 4141063824 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2456] ppid=2423 vsize=158604 CPUtime=2.17 cores=0,2,4,6
/proc/2424/task/2456/stat : 2456 (pb_cplex) S 2423 2423 1671 0 -1 4202560 4926 0 0 0 214 3 0 0 20 0 8 0 34778427 162410496 18922 33554432000 134512640 146736320 4291399696 4151549584 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2457] ppid=2423 vsize=158604 CPUtime=2.12 cores=0,2,4,6
/proc/2424/task/2457/stat : 2457 (pb_cplex) S 2423 2423 1671 0 -1 4202560 7645 0 0 0 208 4 0 0 20 0 8 0 34778427 162410496 18922 33554432000 134512640 146736320 4291399696 4119899792 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2458] ppid=2423 vsize=158604 CPUtime=2.01 cores=0,2,4,6
/proc/2424/task/2458/stat : 2458 (pb_cplex) S 2423 2423 1671 0 -1 4202560 11010 0 0 0 197 4 0 0 20 0 8 0 34778427 162410496 18922 33554432000 134512640 146736320 4291399696 4104122000 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2459] ppid=2423 vsize=158604 CPUtime=1.89 cores=0,2,4,6
/proc/2424/task/2459/stat : 2459 (pb_cplex) S 2423 2423 1671 0 -1 4202560 9563 0 0 0 185 4 0 0 20 0 8 0 34778428 162410496 18922 33554432000 134512640 146736320 4291399696 4093636240 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2460] ppid=2423 vsize=158604 CPUtime=2.1 cores=0,2,4,6
/proc/2424/task/2460/stat : 2460 (pb_cplex) S 2423 2423 1671 0 -1 4202560 10101 0 0 0 206 4 0 0 20 0 8 0 34778428 162410496 18922 33554432000 134512640 146736320 4291399696 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 17.39
Current children cumulated vsize (KiB) 167752

[startup+12.7004 s]
/proc/loadavg: 7.14 7.99 6.83 12/195 2474
/proc/meminfo: memFree=30299784/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=152336 CPUtime=36.19 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) S 2423 2423 1671 0 -1 4202496 111972 0 0 0 3539 80 0 0 20 0 8 0 34778405 155992064 17314 33554432000 134512640 146736320 4291399696 4291396048 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2424/statm: 38083 17314 1003 2985 0 35017 0
[pid=2424/tid=2468] ppid=2423 vsize=152332 CPUtime=1.11 cores=0,2,4,6
/proc/2424/task/2468/stat : 2468 (pb_cplex) R 2423 2423 1671 0 -1 4202560 2599 0 0 0 109 2 0 0 20 0 8 0 34779384 155987968 17314 33554432000 134512640 146736320 4291399696 4130567596 144797937 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2469] ppid=2423 vsize=152332 CPUtime=1.09 cores=0,2,4,6
/proc/2424/task/2469/stat : 2469 (pb_cplex) R 2423 2423 1671 0 -1 4202560 3370 0 0 0 107 2 0 0 20 0 8 0 34779384 155987968 17314 33554432000 134512640 146736320 4291399696 4141060304 136008455 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2470] ppid=2423 vsize=152332 CPUtime=1.04 cores=0,2,4,6
/proc/2424/task/2470/stat : 2470 (pb_cplex) R 2423 2423 1671 0 -1 4202560 2413 0 0 0 102 2 0 0 20 0 8 0 34779384 155987968 17314 33554432000 134512640 146736320 4291399696 4151545912 136015707 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2471] ppid=2423 vsize=152332 CPUtime=1.02 cores=0,2,4,6
/proc/2424/task/2471/stat : 2471 (pb_cplex) R 2423 2423 1671 0 -1 4202560 2660 0 0 0 100 2 0 0 20 0 8 0 34779384 155987968 17314 33554432000 134512640 146736320 4291399696 4119896104 139041202 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2472] ppid=2423 vsize=152332 CPUtime=1.13 cores=0,2,4,6
/proc/2424/task/2472/stat : 2472 (pb_cplex) R 2423 2423 1671 0 -1 4202560 2579 0 0 0 111 2 0 0 20 0 8 0 34779384 155987968 17314 33554432000 134512640 146736320 4291399696 4091535184 136008245 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2473] ppid=2423 vsize=152332 CPUtime=1.02 cores=0,2,4,6
/proc/2424/task/2473/stat : 2473 (pb_cplex) R 2423 2423 1671 0 -1 4202560 3451 0 0 0 101 1 0 0 20 0 8 0 34779384 155987968 17314 33554432000 134512640 146736320 4291399696 4070563896 136015696 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2474] ppid=2423 vsize=152332 CPUtime=1 cores=0,2,4,6
/proc/2424/task/2474/stat : 2474 (pb_cplex) R 2423 2423 1671 0 -1 4202560 1779 0 0 0 99 1 0 0 20 0 8 0 34779384 155987968 17314 33554432000 134512640 146736320 4291399696 4043301864 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 36.19
Current children cumulated vsize (KiB) 161484

[startup+25.5004 s]
/proc/loadavg: 6.78 7.89 6.81 8/195 2474
/proc/meminfo: memFree=30288276/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=157928 CPUtime=76.52 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) R 2423 2423 1671 0 -1 4202496 158564 0 0 0 7524 128 0 0 20 0 8 0 34778405 161718272 18468 33554432000 134512640 146736320 4291399696 4291390964 135451459 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2424/statm: 39482 18468 1023 2985 0 36416 0
[pid=2424/tid=2468] ppid=2423 vsize=157928 CPUtime=6.01 cores=0,2,4,6
/proc/2424/task/2468/stat : 2468 (pb_cplex) R 2423 2423 1671 0 -1 4202560 6995 0 0 0 592 9 0 0 20 0 8 0 34779384 161718272 18468 33554432000 134512640 146736320 4291399696 4130574952 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2469] ppid=2423 vsize=157928 CPUtime=6.11 cores=0,2,4,6
/proc/2424/task/2469/stat : 2469 (pb_cplex) S 2423 2423 1671 0 -1 4202560 6119 0 0 0 603 8 0 0 20 0 8 0 34779384 161718272 18468 33554432000 134512640 146736320 4291399696 4141063824 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2470] ppid=2423 vsize=157928 CPUtime=6.11 cores=0,2,4,6
/proc/2424/task/2470/stat : 2470 (pb_cplex) S 2423 2423 1671 0 -1 4202560 14742 0 0 0 602 9 0 0 20 0 8 0 34779384 161718272 18468 33554432000 134512640 146736320 4291399696 4151549584 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2471] ppid=2423 vsize=157928 CPUtime=5.85 cores=0,2,4,6
/proc/2424/task/2471/stat : 2471 (pb_cplex) R 2423 2423 1671 0 -1 4202560 8333 0 0 0 578 7 0 0 20 0 8 0 34779384 161718272 18468 33554432000 134512640 146736320 4291399696 4119896044 135454079 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2472] ppid=2423 vsize=157928 CPUtime=6.12 cores=0,2,4,6
/proc/2424/task/2472/stat : 2472 (pb_cplex) R 2423 2423 1671 0 -1 4202560 10454 0 0 0 603 9 0 0 20 0 8 0 34779384 161718272 18468 33554432000 134512640 146736320 4291399696 4091535976 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2473] ppid=2423 vsize=157928 CPUtime=5.94 cores=0,2,4,6
/proc/2424/task/2473/stat : 2473 (pb_cplex) S 2423 2423 1671 0 -1 4202560 10339 0 0 0 588 6 0 0 20 0 8 0 34779384 161718272 18468 33554432000 134512640 146736320 4291399696 4070567568 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2474] ppid=2423 vsize=157928 CPUtime=6.02 cores=0,2,4,6
/proc/2424/task/2474/stat : 2474 (pb_cplex) R 2423 2423 1671 0 -1 4202560 4434 0 0 0 596 6 0 0 20 0 8 0 34779384 161718272 18468 33554432000 134512640 146736320 4291399696 4043301864 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 76.52
Current children cumulated vsize (KiB) 167076

[startup+51.1004 s]
/proc/loadavg: 7.22 7.92 6.85 10/195 2474
/proc/meminfo: memFree=30288888/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0

################
# More data... #
################

[pid=2424/tid=2487] ppid=2423 vsize=169012 CPUtime=77.16 cores=0,2,4,6
/proc/2424/task/2487/stat : 2487 (pb_cplex) R 2423 2423 1671 0 -1 4202560 25354 0 0 0 7594 122 0 0 20 0 8 0 34793917 173068288 21316 33554432000 134512640 146736320 4291399696 4065488528 144580073 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2488] ppid=2423 vsize=169012 CPUtime=75.75 cores=0,2,4,6
/proc/2424/task/2488/stat : 2488 (pb_cplex) S 2423 2423 1671 0 -1 4202560 47852 0 0 0 7448 127 0 0 20 0 8 0 34793917 173068288 21316 33554432000 134512640 146736320 4291399696 4043304592 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2489] ppid=2423 vsize=169012 CPUtime=76.36 cores=0,2,4,6
/proc/2424/task/2489/stat : 2489 (pb_cplex) S 2423 2423 1671 0 -1 4202560 42205 0 0 0 7511 125 0 0 20 0 8 0 34793917 173068288 21316 33554432000 134512640 146736320 4291399696 4032818832 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1092.88
Current children cumulated vsize (KiB) 178160

[startup+462.304 s]
/proc/loadavg: 3.25 4.87 5.86 4/195 2491
/proc/meminfo: memFree=30277728/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=169304 CPUtime=1238.76 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) R 2423 2423 1671 0 -1 4202496 712922 0 0 0 122055 1821 0 0 20 0 8 0 34778405 173367296 21325 33554432000 134512640 146736320 4291399696 4291393880 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2424/statm: 42326 21325 1023 2985 0 39260 0
[pid=2424/tid=2483] ppid=2423 vsize=169304 CPUtime=88.99 cores=0,2,4,6
/proc/2424/task/2483/stat : 2483 (pb_cplex) R 2423 2423 1671 0 -1 4202560 31750 0 0 0 8751 148 0 0 20 0 8 0 34793917 173367296 21325 33554432000 134512640 146736320 4291399696 4130575072 136002742 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2484] ppid=2423 vsize=169304 CPUtime=95.53 cores=0,2,4,6
/proc/2424/task/2484/stat : 2484 (pb_cplex) R 2423 2423 1671 0 -1 4202560 55054 0 0 0 9403 150 0 0 20 0 8 0 34793917 173367296 21325 33554432000 134512640 146736320 4291399696 4141059996 136389732 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2485] ppid=2423 vsize=169304 CPUtime=94.3 cores=0,2,4,6
/proc/2424/task/2485/stat : 2485 (pb_cplex) R 2423 2423 1671 0 -1 4202560 54326 0 0 0 9282 148 0 0 20 0 8 0 34793917 173367296 21325 33554432000 134512640 146736320 4291399696 4151544500 135450566 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2486] ppid=2423 vsize=169304 CPUtime=93.45 cores=0,2,4,6
/proc/2424/task/2486/stat : 2486 (pb_cplex) S 2423 2423 1671 0 -1 4202560 41988 0 0 0 9197 148 0 0 20 0 8 0 34793917 173367296 21325 33554432000 134512640 146736320 4291399696 4119899792 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2487] ppid=2423 vsize=169304 CPUtime=96.91 cores=0,2,4,6
/proc/2424/task/2487/stat : 2487 (pb_cplex) R 2423 2423 1671 0 -1 4202560 54763 0 0 0 9537 154 0 0 20 0 8 0 34793917 173367296 21325 33554432000 134512640 146736320 4291399696 4065486360 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2488] ppid=2423 vsize=169304 CPUtime=92.6 cores=0,2,4,6
/proc/2424/task/2488/stat : 2488 (pb_cplex) R 2423 2423 1671 0 -1 4202560 54122 0 0 0 9106 154 0 0 20 0 8 0 34793917 173367296 21325 33554432000 134512640 146736320 4291399696 4043301480 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2489] ppid=2423 vsize=169304 CPUtime=95.16 cores=0,2,4,6
/proc/2424/task/2489/stat : 2489 (pb_cplex) R 2423 2423 1671 0 -1 4202560 85457 0 0 0 9359 157 0 0 20 0 8 0 34793917 173367296 21325 33554432000 134512640 146736320 4291399696 4032814536 135526548 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 1238.76
Current children cumulated vsize (KiB) 178452

[startup+522.3 s]
/proc/loadavg: 3.93 4.77 5.76 5/196 2494
/proc/meminfo: memFree=30274496/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=173400 CPUtime=1376.85 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) R 2423 2423 1671 0 -1 4202496 864727 0 0 0 135653 2032 0 0 20 0 8 0 34778405 177561600 22112 33554432000 134512640 146736320 4291399696 4291392760 136015583 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/2424/statm: 43350 22112 1023 2985 0 40284 0
[pid=2424/tid=2483] ppid=2423 vsize=173400 CPUtime=105.08 cores=0,2,4,6
/proc/2424/task/2483/stat : 2483 (pb_cplex) S 2423 2423 1671 0 -1 4202560 58906 0 0 0 10334 174 0 0 20 0 8 0 34793917 177561600 22112 33554432000 134512640 146736320 4291399696 4130578064 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2484] ppid=2423 vsize=173400 CPUtime=112.15 cores=0,2,4,6
/proc/2424/task/2484/stat : 2484 (pb_cplex) S 2423 2423 1671 0 -1 4202560 74706 0 0 0 11037 178 0 0 20 0 8 0 34793917 177561600 22112 33554432000 134512640 146736320 4291399696 4141063824 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2485] ppid=2423 vsize=173400 CPUtime=111.5 cores=0,2,4,6
/proc/2424/task/2485/stat : 2485 (pb_cplex) S 2423 2423 1671 0 -1 4202560 81348 0 0 0 10974 176 0 0 20 0 8 0 34793917 177561600 22112 33554432000 134512640 146736320 4291399696 4151549584 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2486] ppid=2423 vsize=173400 CPUtime=110.24 cores=0,2,4,6
/proc/2424/task/2486/stat : 2486 (pb_cplex) S 2423 2423 1671 0 -1 4202560 66930 0 0 0 10849 175 0 0 20 0 8 0 34793917 177561600 22112 33554432000 134512640 146736320 4291399696 4119899792 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2487] ppid=2423 vsize=173400 CPUtime=114.44 cores=0,2,4,6
/proc/2424/task/2487/stat : 2487 (pb_cplex) S 2423 2423 1671 0 -1 4202560 62359 0 0 0 11266 178 0 0 20 0 8 0 34793917 177561600 22112 33554432000 134512640 146736320 4291399696 4065488528 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2488] ppid=2423 vsize=173400 CPUtime=110.12 cores=0,2,4,6
/proc/2424/task/2488/stat : 2488 (pb_cplex) S 2423 2423 1671 0 -1 4202560 76024 0 0 0 10834 178 0 0 20 0 8 0 34793917 177561600 22112 33554432000 134512640 146736320 4291399696 4043304592 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2489] ppid=2423 vsize=173400 CPUtime=112.24 cores=0,2,4,6
/proc/2424/task/2489/stat : 2489 (pb_cplex) S 2423 2423 1671 0 -1 4202560 100433 0 0 0 11039 185 0 0 20 0 8 0 34793917 177561600 22112 33554432000 134512640 146736320 4291399696 4032818832 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1376.85
Current children cumulated vsize (KiB) 182548

[startup+582.3 s]
/proc/loadavg: 3.59 4.52 5.61 3/195 2494
/proc/meminfo: memFree=30273388/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=173556 CPUtime=1525.26 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) S 2423 2423 1671 0 -1 4202496 962516 0 0 0 150254 2272 0 0 20 0 8 0 34778405 177721344 22330 33554432000 134512640 146736320 4291399696 4291396048 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2424/statm: 43389 22330 1023 2985 0 40323 0
[pid=2424/tid=2483] ppid=2423 vsize=173556 CPUtime=122.07 cores=0,2,4,6
/proc/2424/task/2483/stat : 2483 (pb_cplex) R 2423 2423 1671 0 -1 4202560 68137 0 0 0 12003 204 0 0 20 0 8 0 34793917 177721344 22330 33554432000 134512640 146736320 4291399696 4130575896 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2484] ppid=2423 vsize=173556 CPUtime=131.06 cores=0,2,4,6
/proc/2424/task/2484/stat : 2484 (pb_cplex) S 2423 2423 1671 0 -1 4202560 82601 0 0 0 12896 210 0 0 20 0 8 0 34793917 177721344 22330 33554432000 134512640 146736320 4291399696 4141063824 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2485] ppid=2423 vsize=173556 CPUtime=131.21 cores=0,2,4,6
/proc/2424/task/2485/stat : 2485 (pb_cplex) R 2423 2423 1671 0 -1 4202560 104252 0 0 0 12912 209 0 0 20 0 8 0 34793917 177721344 22330 33554432000 134512640 146736320 4291399696 4151545912 136015684 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2486] ppid=2423 vsize=173556 CPUtime=128.3 cores=0,2,4,6
/proc/2424/task/2486/stat : 2486 (pb_cplex) S 2423 2423 1671 0 -1 4202560 89162 0 0 0 12625 205 0 0 20 0 8 0 34793917 177721344 22330 33554432000 134512640 146736320 4291399696 4119899792 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2487] ppid=2423 vsize=173556 CPUtime=132.03 cores=0,2,4,6
/proc/2424/task/2487/stat : 2487 (pb_cplex) R 2423 2423 1671 0 -1 4202560 81863 0 0 0 12995 208 0 0 20 0 8 0 34793917 177721344 22330 33554432000 134512640 146736320 4291399696 4065485816 136364238 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2488] ppid=2423 vsize=173556 CPUtime=128.2 cores=0,2,4,6
/proc/2424/task/2488/stat : 2488 (pb_cplex) R 2423 2423 1671 0 -1 4202560 82820 0 0 0 12614 206 0 0 20 0 8 0 34793917 177721344 22330 33554432000 134512640 146736320 4291399696 4043302040 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2489] ppid=2423 vsize=173556 CPUtime=130.59 cores=0,2,4,6
/proc/2424/task/2489/stat : 2489 (pb_cplex) R 2423 2423 1671 0 -1 4202560 106191 0 0 0 12843 216 0 0 20 0 8 0 34793917 177721344 22330 33554432000 134512640 146736320 4291399696 4032815720 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 1525.26
Current children cumulated vsize (KiB) 182704

[startup+642.3 s]
/proc/loadavg: 3.86 4.45 5.52 8/195 2494
/proc/meminfo: memFree=30269972/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=177916 CPUtime=1696.27 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) S 2423 2423 1671 0 -1 4202496 992642 0 0 0 167085 2542 0 0 20 0 8 0 34778405 182185984 23231 33554432000 134512640 146736320 4291399696 4291396048 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/2424/statm: 44479 23231 1023 2985 0 41413 0
[pid=2424/tid=2483] ppid=2423 vsize=177916 CPUtime=142.44 cores=0,2,4,6
/proc/2424/task/2483/stat : 2483 (pb_cplex) S 2423 2423 1671 0 -1 4202560 72546 0 0 0 14005 239 0 0 20 0 8 0 34793917 182185984 23231 33554432000 134512640 146736320 4291399696 4130578064 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2484] ppid=2423 vsize=177916 CPUtime=151.91 cores=0,2,4,6
/proc/2424/task/2484/stat : 2484 (pb_cplex) S 2423 2423 1671 0 -1 4202560 84930 0 0 0 14948 243 0 0 20 0 8 0 34793917 182185984 23231 33554432000 134512640 146736320 4291399696 4141063824 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2485] ppid=2423 vsize=177916 CPUtime=152.06 cores=0,2,4,6
/proc/2424/task/2485/stat : 2485 (pb_cplex) S 2423 2423 1671 0 -1 4202560 112762 0 0 0 14960 246 0 0 20 0 8 0 34793917 182185984 23231 33554432000 134512640 146736320 4291399696 4151549584 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2486] ppid=2423 vsize=177916 CPUtime=149.05 cores=0,2,4,6
/proc/2424/task/2486/stat : 2486 (pb_cplex) R 2423 2423 1671 0 -1 4202560 91532 0 0 0 14666 239 0 0 20 0 8 0 34793917 182185984 23231 33554432000 134512640 146736320 4291399696 4119894180 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2487] ppid=2423 vsize=177916 CPUtime=152.79 cores=0,2,4,6
/proc/2424/task/2487/stat : 2487 (pb_cplex) S 2423 2423 1671 0 -1 4202560 86141 0 0 0 15036 243 0 0 20 0 8 0 34793917 182185984 23231 33554432000 134512640 146736320 4291399696 4065488528 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2488] ppid=2423 vsize=177916 CPUtime=149.17 cores=0,2,4,6
/proc/2424/task/2488/stat : 2488 (pb_cplex) S 2423 2423 1671 0 -1 4202560 84668 0 0 0 14679 238 0 0 20 0 8 0 34793917 182185984 23231 33554432000 134512640 146736320 4291399696 4043304592 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2489] ppid=2423 vsize=177916 CPUtime=151.6 cores=0,2,4,6
/proc/2424/task/2489/stat : 2489 (pb_cplex) S 2423 2423 1671 0 -1 4202560 108627 0 0 0 14907 253 0 0 20 0 8 0 34793917 182185984 23231 33554432000 134512640 146736320 4291399696 4032818832 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 1696.27
Current children cumulated vsize (KiB) 187064



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+679.2 s]
/proc/loadavg: 3.98 4.42 5.47 9/195 2495
/proc/meminfo: memFree=30266128/32873844 swapFree=7124/7124
[pid=2423] ppid=2421 vsize=9148 CPUtime=0 cores=0,2,4,6
/proc/2423/stat : 2423 (run) S 2421 2423 1671 0 -1 4202496 355 0 0 0 0 0 0 0 20 0 1 0 34778405 9367552 270 33554432000 4194304 5064252 140734041554416 140734041553328 215143310926 0 65536 4100 65538 18446744071579302229 0 0 17 0 0 0 0 0 0
/proc/2423/statm: 2287 270 229 213 0 72 0
[pid=2424] ppid=2423 vsize=181196 CPUtime=1799.75 cores=0,2,4,6
/proc/2424/stat : 2424 (pb_cplex) R 2423 2423 1671 0 -1 4202496 1009881 0 0 0 177278 2697 0 0 20 0 8 0 34778405 185544704 24213 33554432000 134512640 146736320 4291399696 4291396076 144580332 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0 0 0
/proc/2424/statm: 45299 24213 1023 2985 0 42233 0
[pid=2424/tid=2483] ppid=2423 vsize=181196 CPUtime=154.73 cores=0,2,4,6
/proc/2424/task/2483/stat : 2483 (pb_cplex) R 2423 2423 1671 0 -1 4202560 74600 0 0 0 15215 258 0 0 20 0 8 0 34793917 185544704 24213 33554432000 134512640 146736320 4291399696 4130574952 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0 0 0
[pid=2424/tid=2484] ppid=2423 vsize=181196 CPUtime=164.3 cores=0,2,4,6
/proc/2424/task/2484/stat : 2484 (pb_cplex) R 2423 2423 1671 0 -1 4202560 87100 0 0 0 16167 263 0 0 20 0 8 0 34793917 185544704 24213 33554432000 134512640 146736320 4291399696 4141059764 135450735 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2485] ppid=2423 vsize=181196 CPUtime=164.57 cores=0,2,4,6
/proc/2424/task/2485/stat : 2485 (pb_cplex) S 2423 2423 1671 0 -1 4202560 115113 0 0 0 16192 265 0 0 20 0 8 0 34793917 185544704 24213 33554432000 134512640 146736320 4291399696 4151549584 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=2424/tid=2486] ppid=2423 vsize=181196 CPUtime=162.21 cores=0,2,4,6
/proc/2424/task/2486/stat : 2486 (pb_cplex) R 2423 2423 1671 0 -1 4202560 92271 0 0 0 15963 258 0 0 20 0 8 0 34793917 185544704 24213 33554432000 134512640 146736320 4291399696 4119896680 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2487] ppid=2423 vsize=181196 CPUtime=165.5 cores=0,2,4,6
/proc/2424/task/2487/stat : 2487 (pb_cplex) S 2423 2423 1671 0 -1 4202560 88635 0 0 0 16285 265 0 0 20 0 8 0 34793917 185544704 24213 33554432000 134512640 146736320 4291399696 4065488528 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=2424/tid=2488] ppid=2423 vsize=181196 CPUtime=161.99 cores=0,2,4,6
/proc/2424/task/2488/stat : 2488 (pb_cplex) R 2423 2423 1671 0 -1 4202560 89188 0 0 0 15943 256 0 0 20 0 8 0 34793917 185544704 24213 33554432000 134512640 146736320 4291399696 4043301304 136015696 0 2 4096 16387 0 0 0 -1 6 0 0 0 0 0
[pid=2424/tid=2489] ppid=2423 vsize=181196 CPUtime=164.12 cores=0,2,4,6
/proc/2424/task/2489/stat : 2489 (pb_cplex) S 2423 2423 1671 0 -1 4202560 110507 0 0 0 16140 272 0 0 20 0 8 0 34793917 185544704 24213 33554432000 134512640 146736320 4291399696 4032818832 144580073 0 2 4096 16387 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 1799.75
Current children cumulated vsize (KiB) 190344

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2423 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=0
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=999
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1799.75
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...


# WARNING:
# The CPU time of some children was not reported to their father
# (probably because of a missing or aborted wait()).
# This 'lost CPU time' is added to the watched process CPU time.
#  lost CPU time (s): 0.300018
#  lost CPU user time (s): 0.380005
#  lost CPU system time (s): -0.0799999

Real time (s): 679.217
CPU time (s): 1800.05
CPU user time (s): 1773.16
CPU system time (s): 26.89
CPU usage (%): 265.018
Max. virtual memory (cumulated for all children) (KiB): 190344

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0
system time used= 0.000999
maximum resident set size= 1080
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 355
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 1

runsolver used 3.05054 second user time and 3.22551 second system time

The end

Launcher Data 

Begin job on node133 at 2012-06-09 15:21:55
IDJOB=3736468
IDBENCH=47925
IDSOLVER=2325
FILE ID=node133/3736468-1339248115
RUNJOBID= node133-1339246833-1688
PBS_JOBID= 14644348
Free space on /tmp= 68380 MiB

SOLVER NAME= PB10: pb_cplex 2010-06-29
BENCH NAME= PB07/normalized-PB07/OPT-SMALLINT-NLC/submittedPB07/roussel/factor-mod-B/factor-mod-size=7-P0=97-P1=37-P2=107-P3=53-P4=53-B.opb
COMMAND LINE= run BENCHNAME
CONVERSION SCRIPT= PBconversionToLinear BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3736468-1339248115/watcher-3736468-1339248115 -o /tmp/evaluation-result-3736468-1339248115/solver-3736468-1339248115 -C 1800 -W 1900 -M 15500  run HOME/instance-3736468-1339248115.opb

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 3c1c5627c2ecb7d046a74fd6c2fb2be3
RANDOM SEED=823654895

node133.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.56
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.52
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.139
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        30361620 kB
Buffers:          115844 kB
Cached:          1526496 kB
SwapCached:         2212 kB
Active:          1628392 kB
Inactive:          32080 kB
Active(anon):      17344 kB
Inactive(anon):     3488 kB
Active(file):    1611048 kB
Inactive(file):    28592 kB
Unevictable:        7124 kB
Mlocked:            7124 kB
SwapTotal:      67108856 kB
SwapFree:       67100208 kB
Dirty:             16824 kB
Writeback:             0 kB
AnonPages:         24188 kB
Mapped:            11672 kB
Shmem:                 0 kB
Slab:             713600 kB
SReclaimable:     154644 kB
SUnreclaim:       558956 kB
KernelStack:        1528 kB
PageTables:         4224 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:     200304 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346376 kB
VmallocChunk:   34341987860 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

CONVERSION COMMAND LINE= PBconversionToLinear HOME/instance-3736468-1339248115.opb

Free space on /tmp at the end= 68364 MiB
End job on node133 at 2012-06-09 15:33:14