Trace number 2697992

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.13 483.986

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900546417--soft-33-100-0.wbo
MD5SUM4d7369f2c7754df76e462e1a1e9c34c5
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark151.682
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints1857
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 93863
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 93862
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 689
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.05	c pb_cplex 1.0 - 20100701
0.00/0.05	c Parsing wbo file format. File HOME/instance-2697992-1278055214.wbo.
0.00/0.05	c Instance file HOME/instance-2697992-1278055214.wbo
0.00/0.05	c File size is 92583 bytes.
0.69/1.56	c Highest Coefficient sum: 689
0.69/1.56	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.69/1.56	c ============================[ Problem Statistics ]=============================
0.69/1.56	c |                                                                             |
0.69/1.56	c |  Parsing time:          0.69         s                                      |
0.69/1.56	c |  Number Variables:      2088                                                |
0.69/1.56	c |  Number Constraints:    2708                                                |
0.69/1.56	c ===============================================================================
1800.13/483.98	c Remaining time 1799.31 sec.
1800.13/483.98	c

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697992-1278055214/watcher-2697992-1278055214 -o /tmp/evaluation-result-2697992-1278055214/solver-2697992-1278055214 -C 1800 -W 2000 -M 3800 run HOME/instance-2697992-1278055214.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.51 6.55 2.98 3/190 21185
/proc/meminfo: memFree=31314416/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0

[startup+0.149799 s]
/proc/loadavg: 9.51 6.55 2.98 3/190 21185
/proc/meminfo: memFree=31314416/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.228615 s]
/proc/loadavg: 9.51 6.55 2.98 3/190 21185
/proc/meminfo: memFree=31314416/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.360741 s]
/proc/loadavg: 9.51 6.55 2.98 3/190 21185
/proc/meminfo: memFree=31314416/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.701318 s]
/proc/loadavg: 9.51 6.55 2.98 3/190 21185
/proc/meminfo: memFree=31314416/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.55615 s]
/proc/loadavg: 9.51 6.55 2.98 4/192 21187
/proc/meminfo: memFree=31314444/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=18376 CPUtime=0.69
/proc/21186/stat : 21186 (pb_cplex) R 21185 21185 20819 0 -1 4202496 1694 0 0 0 69 0 0 0 25 0 1 0 146817062 18817024 1575 4089446400 134512640 146736320 4290474704 18446744073709551615 144716523 0 0 4096 16387 0 0 0 17 0 0 0 0
/proc/21186/statm: 4594 1575 730 2985 0 1136 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 27076

[startup+3.10093 s]
/proc/loadavg: 9.39 6.57 3.00 8/192 21187
/proc/meminfo: memFree=31306012/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=133548 CPUtime=2.68
/proc/21186/stat : 21186 (pb_cplex) R 21185 21185 20819 0 -1 4202496 14417 0 0 0 264 4 0 0 17 0 8 0 146817062 136753152 10198 4089446400 134512640 146736320 4290474704 18446744073709551615 136475798 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/21186/statm: 33387 10198 1068 2985 0 29929 0
Current children cumulated CPU time (s) 2.68
Current children cumulated vsize (KiB) 142248

[startup+6.38827 s]
/proc/loadavg: 9.39 6.57 3.00 3/199 21292
/proc/meminfo: memFree=31289084/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=141004 CPUtime=10.5
/proc/21186/stat : 21186 (pb_cplex) R 21185 21185 20819 0 -1 4202496 25854 0 0 0 1038 12 0 0 15 0 8 0 146817062 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21186/statm: 35251 12856 1081 2985 0 31793 0
[pid=21186/tid=21258] ppid=21185 vsize=141004 CPUtime=0.73
/proc/21186/task/21258/stat : 21258 (pb_cplex) R 21185 21185 20819 0 -1 4202560 361 0 0 0 73 0 0 0 16 0 8 0 146817491 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 139028313 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21186/tid=21259] ppid=21185 vsize=141004 CPUtime=0.69
/proc/21186/task/21259/stat : 21259 (pb_cplex) R 21185 21185 20819 0 -1 4202560 730 0 0 0 69 0 0 0 18 0 8 0 146817491 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 136007643 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21186/tid=21260] ppid=21185 vsize=141004 CPUtime=0.38
/proc/21186/task/21260/stat : 21260 (pb_cplex) R 21185 21185 20819 0 -1 4202560 208 0 0 0 38 0 0 0 15 0 8 0 146817491 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21186/tid=21261] ppid=21185 vsize=141004 CPUtime=0.63
/proc/21186/task/21261/stat : 21261 (pb_cplex) R 21185 21185 20819 0 -1 4202560 54 0 0 0 63 0 0 0 17 0 8 0 146817491 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 136400583 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21186/tid=21262] ppid=21185 vsize=141004 CPUtime=0.56
/proc/21186/task/21262/stat : 21262 (pb_cplex) R 21185 21185 20819 0 -1 4202560 337 0 0 0 56 0 0 0 16 0 8 0 146817491 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21186/tid=21263] ppid=21185 vsize=141004 CPUtime=0.68
/proc/21186/task/21263/stat : 21263 (pb_cplex) R 21185 21185 20819 0 -1 4202560 112 0 0 0 68 0 0 0 18 0 8 0 146817491 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 144580332 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21264] ppid=21185 vsize=141004 CPUtime=0.42
/proc/21186/task/21264/stat : 21264 (pb_cplex) R 21185 21185 20819 0 -1 4202560 47 0 0 0 42 0 0 0 15 0 8 0 146817491 144388096 12856 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 10.5
Current children cumulated vsize (KiB) 149704

[startup+12.7071 s]
/proc/loadavg: 9.25 6.63 3.06 9/199 21292
/proc/meminfo: memFree=31264780/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=145100 CPUtime=33.75
/proc/21186/stat : 21186 (pb_cplex) S 21185 21185 20819 0 -1 4202496 33173 0 0 0 3340 35 0 0 15 0 8 0 146817062 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21186/statm: 36275 14065 1080 2985 0 32817 0
[pid=21186/tid=21258] ppid=21185 vsize=145100 CPUtime=3.53
/proc/21186/task/21258/stat : 21258 (pb_cplex) R 21185 21185 20819 0 -1 4202560 658 0 0 0 350 3 0 0 15 0 8 0 146817491 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 136007327 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21186/tid=21259] ppid=21185 vsize=145100 CPUtime=3.58
/proc/21186/task/21259/stat : 21259 (pb_cplex) R 21185 21185 20819 0 -1 4202560 1173 0 0 0 355 3 0 0 15 0 8 0 146817491 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 136388073 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21186/tid=21260] ppid=21185 vsize=145100 CPUtime=3.15
/proc/21186/task/21260/stat : 21260 (pb_cplex) R 21185 21185 20819 0 -1 4202560 741 0 0 0 313 2 0 0 15 0 8 0 146817491 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21186/tid=21261] ppid=21185 vsize=145100 CPUtime=3.53
/proc/21186/task/21261/stat : 21261 (pb_cplex) R 21185 21185 20819 0 -1 4202560 662 0 0 0 350 3 0 0 15 0 8 0 146817491 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 136403129 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21186/tid=21262] ppid=21185 vsize=145100 CPUtime=3.5
/proc/21186/task/21262/stat : 21262 (pb_cplex) R 21185 21185 20819 0 -1 4202560 1197 0 0 0 348 2 0 0 15 0 8 0 146817491 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21186/tid=21263] ppid=21185 vsize=145100 CPUtime=3.65
/proc/21186/task/21263/stat : 21263 (pb_cplex) R 21185 21185 20819 0 -1 4202560 1356 0 0 0 361 4 0 0 15 0 8 0 146817491 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21186/tid=21264] ppid=21185 vsize=145100 CPUtime=3.17
/proc/21186/task/21264/stat : 21264 (pb_cplex) R 21185 21185 20819 0 -1 4202560 2817 0 0 0 315 2 0 0 15 0 8 0 146817491 148582400 14065 4089446400 134512640 146736320 4290474704 18446744073709551615 139044552 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 33.75
Current children cumulated vsize (KiB) 153800

[startup+25.5608 s]
/proc/loadavg: 9.70 6.81 3.16 7/199 21307
/proc/meminfo: memFree=31260204/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=158624 CPUtime=89.79
/proc/21186/stat : 21186 (pb_cplex) R 21185 21185 20819 0 -1 4202496 72989 0 0 0 8895 84 0 0 21 0 8 0 146817062 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 139042885 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21186/statm: 39656 17359 1088 2985 0 36198 0
[pid=21186/tid=21258] ppid=21185 vsize=158624 CPUtime=10.73
/proc/21186/task/21258/stat : 21258 (pb_cplex) S 21185 21185 20819 0 -1 4202560 4087 0 0 0 1065 8 0 0 20 0 8 0 146817491 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 7 0 0 0
[pid=21186/tid=21259] ppid=21185 vsize=158624 CPUtime=11.33
/proc/21186/task/21259/stat : 21259 (pb_cplex) R 21185 21185 20819 0 -1 4202560 5639 0 0 0 1124 9 0 0 19 0 8 0 146817491 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 136004436 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21260] ppid=21185 vsize=158624 CPUtime=10.82
/proc/21186/task/21260/stat : 21260 (pb_cplex) R 21185 21185 20819 0 -1 4202560 1877 0 0 0 1075 7 0 0 21 0 8 0 146817491 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 136008455 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21186/tid=21261] ppid=21185 vsize=158624 CPUtime=11.16
/proc/21186/task/21261/stat : 21261 (pb_cplex) R 21185 21185 20819 0 -1 4202560 4483 0 0 0 1106 10 0 0 21 0 8 0 146817491 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 136008356 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21186/tid=21262] ppid=21185 vsize=158624 CPUtime=11.14
/proc/21186/task/21262/stat : 21262 (pb_cplex) S 21185 21185 20819 0 -1 4202560 9136 0 0 0 1105 9 0 0 22 0 8 0 146817491 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 3 0 0 0
[pid=21186/tid=21263] ppid=21185 vsize=158624 CPUtime=9.85
/proc/21186/task/21263/stat : 21263 (pb_cplex) S 21185 21185 20819 0 -1 4202560 8240 0 0 0 973 12 0 0 18 0 8 0 146817491 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 0 0 0 0
[pid=21186/tid=21264] ppid=21185 vsize=158624 CPUtime=7.26
/proc/21186/task/21264/stat : 21264 (pb_cplex) S 21185 21185 20819 0 -1 4202560 8349 0 0 0 717 9 0 0 16 0 8 0 146817491 162430976 17359 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 89.79
Current children cumulated vsize (KiB) 167324

[startup+51.1041 s]
/proc/loadavg: 9.12 6.91 3.29 12/199 21322
/proc/meminfo: memFree=31253140/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=146124 CPUtime=148.69
/proc/21186/stat : 21186 (pb_cplex) S 21185 21185 20819 0 -1 4202496 97864 0 0 0 14746 123 0 0 15 0 8 0 146817062 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21186/statm: 36531 14284 1092 2985 0 33073 0
[pid=21186/tid=21315] ppid=21185 vsize=146124 CPUtime=4.85
/proc/21186/task/21315/stat : 21315 (pb_cplex) R 21185 21185 20819 0 -1 4202560 3677 0 0 0 483 2 0 0 15 0 8 0 146820398 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 144785853 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21186/tid=21316] ppid=21185 vsize=146124 CPUtime=6.2
/proc/21186/task/21316/stat : 21316 (pb_cplex) R 21185 21185 20819 0 -1 4202560 3305 0 0 0 616 4 0 0 15 0 8 0 146820398 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 135443386 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21186/tid=21317] ppid=21185 vsize=146124 CPUtime=6.12
/proc/21186/task/21317/stat : 21317 (pb_cplex) R 21185 21185 20819 0 -1 4202560 884 0 0 0 609 3 0 0 15 0 8 0 146820398 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21186/tid=21318] ppid=21185 vsize=146124 CPUtime=4.49
/proc/21186/task/21318/stat : 21318 (pb_cplex) R 21185 21185 20819 0 -1 4202560 1479 0 0 0 446 3 0 0 15 0 8 0 146820398 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 137340985 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21186/tid=21319] ppid=21185 vsize=146124 CPUtime=4.74
/proc/21186/task/21319/stat : 21319 (pb_cplex) R 21185 21185 20819 0 -1 4202560 3677 0 0 0 471 3 0 0 15 0 8 0 146820398 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21186/tid=21320] ppid=21185 vsize=146124 CPUtime=6.76
/proc/21186/task/21320/stat : 21320 (pb_cplex) R 21185 21185 20819 0 -1 4202560 1418 0 0 0 671 5 0 0 15 0 8 0 146820398 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21186/tid=21321] ppid=21185 vsize=146124 CPUtime=5.49
/proc/21186/task/21321/stat : 21321 (pb_cplex) R 21185 21185 20819 0 -1 4202560 238 0 0 0 545 4 0 0 15 0 8 0 146820398 149630976 14284 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 148.69
Current children cumulated vsize (KiB) 154824

[startup+102.303 s]
/proc/loadavg: 9.28 7.29 3.60 11/199 21323
/proc/meminfo: memFree=31250904/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=151244 CPUtime=352.01
/proc/21186/stat : 21186 (pb_cplex) S 21185 21185 20819 0 -1 4202496 129750 0 0 0 34904 297 0 0 16 0 8 0 146817062 154873856 15367 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/21186/statm: 37811 15367 1092 2985 0 34353 0
[pid=21186/tid=21315] ppid=21185 vsize=151244 CPUtime=29.95
/proc/21186/task/21315/stat : 21315 (pb_cplex) S 21185 21185 20819 0 -1 4202560 4802 0 0 0 2974 21 0 0 15 0 8 0 146820398 154873856 15367 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 0 0 0 0
[pid=21186/tid=21316] ppid=21185 vsize=151244 CPUtime=31.28
/proc/21186/task/21316/stat : 21316 (pb_cplex) R 21185 21185 20819 0 -1 4202560 8644 0 0 0 3105 23 0 0 15 0 8 0 146820398 154873856 15367 4089446400 134512640 146736320 4290474704 18446744073709551615 135523311 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21186/tid=21317] ppid=21185 vsize=151244 CPUtime=31.37
/proc/21186/task/21317/stat : 21317 (pb_cplex) R 21185 21185 20819 0 -1 4202560 8535 0 0 0 3110 27 0 0 16 0 8 0 146820398 154873856 15367 4089446400 134512640 146736320 4290474704 18446744073709551615 136388555 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21318] ppid=21185 vsize=151244 CPUtime=29.57
/proc/21186/task/21318/stat : 21318 (pb_cplex) S 21185 21185 20819 0 -1 4202560 3284 0 0 0 2932 25 0 0 15 0 8 0 146820398 154873856 15367 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/21186/statm: 38305 15597 1108 2985 0 34847 0
[pid=21186/tid=21315] ppid=21185 vsize=153220 CPUtime=115.61
/proc/21186/task/21315/stat : 21315 (pb_cplex) S 21185 21185 20819 0 -1 4202560 11844 0 0 0 11464 97 0 0 15 0 8 0 146820398 156897280 15597 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 1 0 0 0
[pid=21186/tid=21316] ppid=21185 vsize=153220 CPUtime=117.31
/proc/21186/task/21316/stat : 21316 (pb_cplex) S 21185 21185 20819 0 -1 4202560 20359 0 0 0 11634 97 0 0 15 0 8 0 146820398 156897280 15597 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 1 0 0 0
[pid=21186/tid=21317] ppid=21185 vsize=153220 CPUtime=116.56
/proc/21186/task/21317/stat : 21317 (pb_cplex) S 21185 21185 20819 0 -1 4202560 17765 0 0 0 11550 106 0 0 15 0 8 0 146820398 156897280 15597 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 1 0 0 0
[pid=21186/tid=21318] ppid=21185 vsize=153220 CPUtime=114.93
/proc/21186/task/21318/stat : 21318 (pb_cplex) S 21185 21185 20819 0 -1 4202560 13553 0 0 0 11390 103 0 0 15 0 8 0 146820398 156897280 15597 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 1 0 0 0
[pid=21186/tid=21319] ppid=21185 vsize=153220 CPUtime=115.11
/proc/21186/task/21319/stat : 21319 (pb_cplex) S 21185 21185 20819 0 -1 4202560 18540 0 0 0 11403 108 0 0 15 0 8 0 146820398 156897280 15597 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 1 0 0 0
[pid=21186/tid=21320] ppid=21185 vsize=153220 CPUtime=117.9
/proc/21186/task/21320/stat : 21320 (pb_cplex) S 21185 21185 20819 0 -1 4202560 14629 0 0 0 11684 106 0 0 15 0 8 0 146820398 156897280 15597 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 1 0 0 0
[pid=21186/tid=21321] ppid=21185 vsize=153220 CPUtime=116.71
/proc/21186/task/21321/stat : 21321 (pb_cplex) S 21185 21185 20819 0 -1 4202560 8307 0 0 0 11565 106 0 0 15 0 8 0 146820398 156897280 15597 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 1042.88
Current children cumulated vsize (KiB) 161920

[startup+342.315 s]
/proc/loadavg: 9.54 8.58 4.99 13/199 21329
/proc/meminfo: memFree=31251724/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=153220 CPUtime=1268.73
/proc/21186/stat : 21186 (pb_cplex) R 21185 21185 20819 0 -1 4202496 219359 0 0 0 125734 1139 0 0 16 0 8 0 146817062 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 136008234 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21186/statm: 38305 15603 1108 2985 0 34847 0
[pid=21186/tid=21315] ppid=21185 vsize=153220 CPUtime=143.75
/proc/21186/task/21315/stat : 21315 (pb_cplex) R 21185 21185 20819 0 -1 4202560 14404 0 0 0 14253 122 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 135526711 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21186/tid=21316] ppid=21185 vsize=153220 CPUtime=145.36
/proc/21186/task/21316/stat : 21316 (pb_cplex) R 21185 21185 20819 0 -1 4202560 23352 0 0 0 14413 123 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21186/tid=21317] ppid=21185 vsize=153220 CPUtime=144.22
/proc/21186/task/21317/stat : 21317 (pb_cplex) R 21185 21185 20819 0 -1 4202560 18900 0 0 0 14292 130 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 137341776 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21186/tid=21318] ppid=21185 vsize=153220 CPUtime=142.94
/proc/21186/task/21318/stat : 21318 (pb_cplex) R 21185 21185 20819 0 -1 4202560 16836 0 0 0 14166 128 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21319] ppid=21185 vsize=153220 CPUtime=143.1
/proc/21186/task/21319/stat : 21319 (pb_cplex) R 21185 21185 20819 0 -1 4202560 21271 0 0 0 14171 139 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21186/tid=21320] ppid=21185 vsize=153220 CPUtime=145.76
/proc/21186/task/21320/stat : 21320 (pb_cplex) R 21185 21185 20819 0 -1 4202560 17958 0 0 0 14442 134 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21186/tid=21321] ppid=21185 vsize=153220 CPUtime=144.72
/proc/21186/task/21321/stat : 21321 (pb_cplex) S 21185 21185 20819 0 -1 4202560 8895 0 0 0 14337 135 0 0 16 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1268.73
Current children cumulated vsize (KiB) 161920

[startup+402.323 s]
/proc/loadavg: 9.18 8.64 5.21 10/199 21331
/proc/meminfo: memFree=31252288/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=153220 CPUtime=1494.75
/proc/21186/stat : 21186 (pb_cplex) R 21185 21185 20819 0 -1 4202496 239366 0 0 0 148118 1357 0 0 16 0 8 0 146817062 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 136475837 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/21186/statm: 38305 15603 1108 2985 0 34847 0
[pid=21186/tid=21315] ppid=21185 vsize=153220 CPUtime=171.71
/proc/21186/task/21315/stat : 21315 (pb_cplex) S 21185 21185 20819 0 -1 4202560 15097 0 0 0 17021 150 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 7 0 0 0
[pid=21186/tid=21316] ppid=21185 vsize=153220 CPUtime=173.25
/proc/21186/task/21316/stat : 21316 (pb_cplex) S 21185 21185 20819 0 -1 4202560 26675 0 0 0 17172 153 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 1 0 0 0
[pid=21186/tid=21317] ppid=21185 vsize=153220 CPUtime=172.26
/proc/21186/task/21317/stat : 21317 (pb_cplex) S 21185 21185 20819 0 -1 4202560 22104 0 0 0 17068 158 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 3 0 0 0
[pid=21186/tid=21318] ppid=21185 vsize=153220 CPUtime=170.81
/proc/21186/task/21318/stat : 21318 (pb_cplex) S 21185 21185 20819 0 -1 4202560 18867 0 0 0 16928 153 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 6 0 0 0
[pid=21186/tid=21319] ppid=21185 vsize=153220 CPUtime=171.21
/proc/21186/task/21319/stat : 21319 (pb_cplex) S 21185 21185 20819 0 -1 4202560 24187 0 0 0 16955 166 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 0 0 0 0
[pid=21186/tid=21320] ppid=21185 vsize=153220 CPUtime=173.8
/proc/21186/task/21320/stat : 21320 (pb_cplex) S 21185 21185 20819 0 -1 4202560 21550 0 0 0 17219 161 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 5 0 0 0
[pid=21186/tid=21321] ppid=21185 vsize=153220 CPUtime=172.87
/proc/21186/task/21321/stat : 21321 (pb_cplex) S 21185 21185 20819 0 -1 4202560 11131 0 0 0 17126 161 0 0 15 0 8 0 146820398 156897280 15603 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1494.75
Current children cumulated vsize (KiB) 161920

[startup+462.309 s]
/proc/loadavg: 9.43 8.77 5.48 9/199 21346
/proc/meminfo: memFree=31239152/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=155128 CPUtime=1720.03
/proc/21186/stat : 21186 (pb_cplex) S 21185 21185 20819 0 -1 4202496 255044 0 0 0 170433 1570 0 0 16 0 8 0 146817062 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21186/statm: 38782 16238 1108 2985 0 35324 0
[pid=21186/tid=21340] ppid=21185 vsize=155128 CPUtime=2.32
/proc/21186/task/21340/stat : 21340 (pb_cplex) S 21185 21185 20819 0 -1 4202560 293 0 0 0 230 2 0 0 15 0 8 0 146862699 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 0 0 0 0
[pid=21186/tid=21341] ppid=21185 vsize=155128 CPUtime=2.35
/proc/21186/task/21341/stat : 21341 (pb_cplex) S 21185 21185 20819 0 -1 4202560 3 0 0 0 233 2 0 0 16 0 8 0 146862699 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 3 0 0 0
[pid=21186/tid=21342] ppid=21185 vsize=155128 CPUtime=2.35
/proc/21186/task/21342/stat : 21342 (pb_cplex) S 21185 21185 20819 0 -1 4202560 122 0 0 0 234 1 0 0 15 0 8 0 146862699 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 2 0 0 0
[pid=21186/tid=21343] ppid=21185 vsize=155128 CPUtime=2.38
/proc/21186/task/21343/stat : 21343 (pb_cplex) R 21185 21185 20819 0 -1 4202560 28 0 0 0 236 2 0 0 15 0 8 0 146862699 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21344] ppid=21185 vsize=155128 CPUtime=2.34
/proc/21186/task/21344/stat : 21344 (pb_cplex) S 21185 21185 20819 0 -1 4202560 266 0 0 0 232 2 0 0 16 0 8 0 146862699 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 4 0 0 0
[pid=21186/tid=21345] ppid=21185 vsize=155128 CPUtime=2.39
/proc/21186/task/21345/stat : 21345 (pb_cplex) S 21185 21185 20819 0 -1 4202560 45 0 0 0 237 2 0 0 15 0 8 0 146862699 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 7 0 0 0
[pid=21186/tid=21346] ppid=21185 vsize=155128 CPUtime=2.39
/proc/21186/task/21346/stat : 21346 (pb_cplex) S 21185 21185 20819 0 -1 4202560 391 0 0 0 236 3 0 0 15 0 8 0 146862699 158851072 16238 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1720.03
Current children cumulated vsize (KiB) 163828



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+483.927 s]
/proc/loadavg: 9.37 8.80 5.56 10/199 21347
/proc/meminfo: memFree=31237432/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=155428 CPUtime=1800.13
/proc/21186/stat : 21186 (pb_cplex) S 21185 21185 20819 0 -1 4202496 259073 0 0 0 178363 1650 0 0 16 0 8 0 146817062 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21186/statm: 38857 16305 1108 2985 0 35399 0
[pid=21186/tid=21340] ppid=21185 vsize=155428 CPUtime=12.23
/proc/21186/task/21340/stat : 21340 (pb_cplex) S 21185 21185 20819 0 -1 4202560 885 0 0 0 1210 13 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 6 0 0 0
[pid=21186/tid=21341] ppid=21185 vsize=155428 CPUtime=12.26
/proc/21186/task/21341/stat : 21341 (pb_cplex) R 21185 21185 20819 0 -1 4202560 871 0 0 0 1214 12 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21342] ppid=21185 vsize=155428 CPUtime=12.16
/proc/21186/task/21342/stat : 21342 (pb_cplex) S 21185 21185 20819 0 -1 4202560 386 0 0 0 1205 11 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 3 0 0 0
[pid=21186/tid=21343] ppid=21185 vsize=155428 CPUtime=12.35
/proc/21186/task/21343/stat : 21343 (pb_cplex) S 21185 21185 20819 0 -1 4202560 370 0 0 0 1225 10 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 2 0 0 0
[pid=21186/tid=21344] ppid=21185 vsize=155428 CPUtime=12.21
/proc/21186/task/21344/stat : 21344 (pb_cplex) R 21185 21185 20819 0 -1 4202560 768 0 0 0 1209 12 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21186/tid=21345] ppid=21185 vsize=155428 CPUtime=12.31
/proc/21186/task/21345/stat : 21345 (pb_cplex) R 21185 21185 20819 0 -1 4202560 906 0 0 0 1221 10 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 135520933 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21346] ppid=21185 vsize=155428 CPUtime=12.28
/proc/21186/task/21346/stat : 21346 (pb_cplex) R 21185 21185 20819 0 -1 4202560 701 0 0 0 1215 13 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1800.13
Current children cumulated vsize (KiB) 164128

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+483.927 s]
/proc/loadavg: 9.37 8.80 5.56 10/199 21347
/proc/meminfo: memFree=31237432/32951124 swapFree=67111528/67111528
[pid=21185] ppid=21183 vsize=8700 CPUtime=0
/proc/21185/stat : 21185 (run) S 21183 21185 20819 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146817061 8908800 249 4089446400 4194304 4922060 140737399695792 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21185/statm: 2175 249 208 178 0 68 0
[pid=21186] ppid=21185 vsize=155428 CPUtime=1800.13
/proc/21186/stat : 21186 (pb_cplex) S 21185 21185 20819 0 -1 4202496 259073 0 0 0 178363 1650 0 0 16 0 8 0 146817062 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21186/statm: 38857 16305 1108 2985 0 35399 0
[pid=21186/tid=21340] ppid=21185 vsize=155428 CPUtime=12.23
/proc/21186/task/21340/stat : 21340 (pb_cplex) S 21185 21185 20819 0 -1 4202560 885 0 0 0 1210 13 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 6 0 0 0
[pid=21186/tid=21341] ppid=21185 vsize=155428 CPUtime=12.26
/proc/21186/task/21341/stat : 21341 (pb_cplex) R 21185 21185 20819 0 -1 4202560 871 0 0 0 1214 12 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21342] ppid=21185 vsize=155428 CPUtime=12.16
/proc/21186/task/21342/stat : 21342 (pb_cplex) S 21185 21185 20819 0 -1 4202560 386 0 0 0 1205 11 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 3 0 0 0
[pid=21186/tid=21343] ppid=21185 vsize=155428 CPUtime=12.35
/proc/21186/task/21343/stat : 21343 (pb_cplex) S 21185 21185 20819 0 -1 4202560 370 0 0 0 1225 10 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144580073 0 2 4096 16387 18446604470849962176 0 0 -1 2 0 0 0
[pid=21186/tid=21344] ppid=21185 vsize=155428 CPUtime=12.21
/proc/21186/task/21344/stat : 21344 (pb_cplex) R 21185 21185 20819 0 -1 4202560 768 0 0 0 1209 12 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21186/tid=21345] ppid=21185 vsize=155428 CPUtime=12.31
/proc/21186/task/21345/stat : 21345 (pb_cplex) R 21185 21185 20819 0 -1 4202560 906 0 0 0 1221 10 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 135520933 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21186/tid=21346] ppid=21185 vsize=155428 CPUtime=12.28
/proc/21186/task/21346/stat : 21346 (pb_cplex) R 21185 21185 20819 0 -1 4202560 701 0 0 0 1215 13 0 0 15 0 8 0 146862699 159158272 16305 4089446400 134512640 146736320 4290474704 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1800.13
Current children cumulated vsize (KiB) 164128

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21185 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=0
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=999
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1800.13
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 483.986
CPU time (s): 1800.13
CPU user time (s): 1783.63
CPU system time (s): 16.5
CPU usage (%): 371.938
Max. virtual memory (cumulated for all children) (KiB): 169372

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0
system time used= 0.000999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 364
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3
involuntary context switches= 11

runsolver used 0.906862 second user time and 1.9937 second system time

The end

Launcher Data 

Begin job on node133 at 2010-07-02 09:20:14
IDJOB=2697992
IDBENCH=78984
IDSOLVER=1210
FILE ID=node133/2697992-1278055214
PBS_JOBID= 11198916
Free space on /tmp= 74188 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900546417--soft-33-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697992-1278055214/watcher-2697992-1278055214 -o /tmp/evaluation-result-2697992-1278055214/solver-2697992-1278055214 -C 1800 -W 2000 -M 3800  run HOME/instance-2697992-1278055214.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 4d7369f2c7754df76e462e1a1e9c34c5
RANDOM SEED=1244874401

node133.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7296.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31314944 kB
Buffers:        217452 kB
Cached:        1155944 kB
SwapCached:          0 kB
Active:         707556 kB
Inactive:       779832 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31314944 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16756 kB
Writeback:           0 kB
AnonPages:      114324 kB
Mapped:          19020 kB
Slab:            82416 kB
PageTables:       5048 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   340032 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74172 MiB
End job on node133 at 2010-07-02 09:28:21