Trace number 2697423

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.73 490.415

General information on the benchmark

Name/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/
tsp/normalized-t3002.11tsp11.1900565534--soft-0-100-0.wbo
MD5SUMcf389047fd441928b15b38cce26251ed
Bench CategorySOFT-SMALLINT-LIN (only soft constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark273.451
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints2707
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 135280
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 135279
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 685
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697423-1278055701.wbo.
0.00/0.00	c Instance file HOME/instance-2697423-1278055701.wbo
0.00/0.00	c File size is 96759 bytes.
1.39/3.46	c Highest Coefficient sum: 685
1.39/3.46	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
1.39/3.46	c ============================[ Problem Statistics ]=============================
1.39/3.46	c |                                                                             |
1.39/3.46	c |  Parsing time:          1.44         s                                      |
1.39/3.46	c |  Number Variables:      2938                                                |
1.39/3.46	c |  Number Constraints:    2708                                                |
1.39/3.46	c ===============================================================================
1800.68/490.41	c Remaining time 1798.56 sec.
1800.68/490.41	c 
1800.68/490.41	c *** INTERRUPTED ***
1800.68/490.41	s UNKNOWN
1800.68/490.41	c Total CPU time              : 1784.6 s
1800.68/490.41	c Memory used           : 142.00 MB
1800.68/490.41	c CPU time              : 1784.6 s
1800.68/490.41	c 
1800.68/490.41	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697423-1278055701/watcher-2697423-1278055701 -o /tmp/evaluation-result-2697423-1278055701/solver-2697423-1278055701 -C 1800 -W 2000 -M 3800 run HOME/instance-2697423-1278055701.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.37 8.80 5.56 9/190 21374
/proc/meminfo: memFree=31298956/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0

[startup+0.144274 s]
/proc/loadavg: 9.37 8.80 5.56 9/190 21374
/proc/meminfo: memFree=31298956/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.223093 s]
/proc/loadavg: 9.37 8.80 5.56 9/190 21374
/proc/meminfo: memFree=31298956/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300276 s]
/proc/loadavg: 9.37 8.80 5.56 9/190 21374
/proc/meminfo: memFree=31298956/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.700169 s]
/proc/loadavg: 9.37 8.80 5.56 9/190 21374
/proc/meminfo: memFree=31298956/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50202 s]
/proc/loadavg: 9.34 8.80 5.58 3/192 21376
/proc/meminfo: memFree=31297824/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=13368 CPUtime=0.32
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 544 0 0 0 32 0 0 0 20 0 1 0 146865732 13688832 497 4089446400 134512640 146736320 4293025024 18446744073709551615 140526619 0 0 4096 16387 0 0 0 17 4 0 0 0
/proc/21375/statm: 3342 497 297 2985 0 252 0
Current children cumulated CPU time (s) 0.32
Current children cumulated vsize (KiB) 22068

[startup+3.10073 s]
/proc/loadavg: 9.34 8.80 5.58 10/192 21376
/proc/meminfo: memFree=31298136/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=13808 CPUtime=1.13
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 669 0 0 0 113 0 0 0 22 0 1 0 146865732 14139392 622 4089446400 134512640 146736320 4293025024 18446744073709551615 140681180 0 0 4096 16387 0 0 0 17 2 0 0 0
/proc/21375/statm: 3452 622 307 2985 0 362 0
Current children cumulated CPU time (s) 1.13
Current children cumulated vsize (KiB) 22508
heavy processes:

[startup+6.30024 s]
/proc/loadavg: 9.47 8.84 5.61 10/192 21390
/proc/meminfo: memFree=31268748/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=128628 CPUtime=6.3
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 30003 0 0 0 621 9 0 0 17 0 8 0 146865732 131715072 9640 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21375/statm: 32157 9640 1080 2985 0 28699 0
Current children cumulated CPU time (s) 6.3
Current children cumulated vsize (KiB) 137328

[startup+12.713 s]
/proc/loadavg: 8.87 8.72 5.59 11/199 21454
/proc/meminfo: memFree=31269516/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=143904 CPUtime=36.08
/proc/21375/stat : 21375 (pb_cplex) S 21374 21374 20819 0 -1 4202496 55004 0 0 0 3567 41 0 0 16 0 8 0 146865732 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21375/statm: 35976 13602 1081 2985 0 32518 0
[pid=21375/tid=21405] ppid=21374 vsize=143904 CPUtime=3.56
/proc/21375/task/21405/stat : 21405 (pb_cplex) R 21374 21374 20819 0 -1 4202560 299 0 0 0 353 3 0 0 15 0 8 0 146866325 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21375/tid=21406] ppid=21374 vsize=143904 CPUtime=3.64
/proc/21375/task/21406/stat : 21406 (pb_cplex) R 21374 21374 20819 0 -1 4202560 1185 0 0 0 361 3 0 0 15 0 8 0 146866325 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21375/tid=21407] ppid=21374 vsize=143904 CPUtime=3.65
/proc/21375/task/21407/stat : 21407 (pb_cplex) R 21374 21374 20819 0 -1 4202560 4400 0 0 0 362 3 0 0 15 0 8 0 146866325 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21375/tid=21408] ppid=21374 vsize=143904 CPUtime=3.87
/proc/21375/task/21408/stat : 21408 (pb_cplex) R 21374 21374 20819 0 -1 4202560 2479 0 0 0 383 4 0 0 15 0 8 0 146866325 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21409] ppid=21374 vsize=143904 CPUtime=3.57
/proc/21375/task/21409/stat : 21409 (pb_cplex) R 21374 21374 20819 0 -1 4202560 792 0 0 0 354 3 0 0 15 0 8 0 146866325 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 136017748 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21375/tid=21410] ppid=21374 vsize=143904 CPUtime=4.01
/proc/21375/task/21410/stat : 21410 (pb_cplex) R 21374 21374 20819 0 -1 4202560 3239 0 0 0 398 3 0 0 15 0 8 0 146866325 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21375/tid=21411] ppid=21374 vsize=143904 CPUtime=3.57
/proc/21375/task/21411/stat : 21411 (pb_cplex) R 21374 21374 20819 0 -1 4202560 5080 0 0 0 353 4 0 0 16 0 8 0 146866325 147357696 13602 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 36.08
Current children cumulated vsize (KiB) 152604

[startup+25.5007 s]
/proc/loadavg: 9.31 8.83 5.68 9/199 21476
/proc/meminfo: memFree=31239968/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=155344 CPUtime=85.81
/proc/21375/stat : 21375 (pb_cplex) S 21374 21374 20819 0 -1 4202496 69587 0 0 0 8491 90 0 0 22 0 8 0 146865732 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21375/statm: 38836 16062 1088 2985 0 35378 0
[pid=21375/tid=21405] ppid=21374 vsize=155344 CPUtime=8.17
/proc/21375/task/21405/stat : 21405 (pb_cplex) S 21374 21374 20819 0 -1 4202560 1741 0 0 0 807 10 0 0 15 0 8 0 146866325 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 6 0 0 0
[pid=21375/tid=21406] ppid=21374 vsize=155344 CPUtime=10.26
/proc/21375/task/21406/stat : 21406 (pb_cplex) R 21374 21374 20819 0 -1 4202560 2138 0 0 0 1018 8 0 0 18 0 8 0 146866325 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 144785853 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21375/tid=21407] ppid=21374 vsize=155344 CPUtime=7.58
/proc/21375/task/21407/stat : 21407 (pb_cplex) S 21374 21374 20819 0 -1 4202560 7648 0 0 0 748 10 0 0 16 0 8 0 146866325 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 7 0 0 0
[pid=21375/tid=21408] ppid=21374 vsize=155344 CPUtime=11.02
/proc/21375/task/21408/stat : 21408 (pb_cplex) R 21374 21374 20819 0 -1 4202560 4655 0 0 0 1093 9 0 0 19 0 8 0 146866325 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 135453309 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21375/tid=21409] ppid=21374 vsize=155344 CPUtime=10.09
/proc/21375/task/21409/stat : 21409 (pb_cplex) S 21374 21374 20819 0 -1 4202560 3385 0 0 0 999 10 0 0 20 0 8 0 146866325 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 1 0 0 0
[pid=21375/tid=21410] ppid=21374 vsize=155344 CPUtime=9.9
/proc/21375/task/21410/stat : 21410 (pb_cplex) S 21374 21374 20819 0 -1 4202560 5201 0 0 0 980 10 0 0 19 0 8 0 146866325 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 2 0 0 0
[pid=21375/tid=21411] ppid=21374 vsize=155344 CPUtime=10.67
/proc/21375/task/21411/stat : 21411 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6312 0 0 0 1057 10 0 0 21 0 8 0 146866325 159072256 16062 4089446400 134512640 146736320 4293025024 18446744073709551615 136008163 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 85.81
Current children cumulated vsize (KiB) 164044

[startup+51.101 s]
/proc/loadavg: 9.96 9.01 5.82 12/199 21504
/proc/meminfo: memFree=31253352/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=146432 CPUtime=157.94
/proc/21375/stat : 21375 (pb_cplex) S 21374 21374 20819 0 -1 4202496 81466 0 0 0 15656 138 0 0 15 0 8 0 146865732 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21375/statm: 36608 14445 1089 2985 0 33150 0
[pid=21375/tid=21484] ppid=21374 vsize=146432 CPUtime=7.98
/proc/21375/task/21484/stat : 21484 (pb_cplex) S 21374 21374 20819 0 -1 4202560 1021 0 0 0 793 5 0 0 15 0 8 0 146869011 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 3 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=146432 CPUtime=7.29
/proc/21375/task/21485/stat : 21485 (pb_cplex) S 21374 21374 20819 0 -1 4202560 105 0 0 0 724 5 0 0 15 0 8 0 146869011 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 2 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=146432 CPUtime=8.06
/proc/21375/task/21486/stat : 21486 (pb_cplex) R 21374 21374 20819 0 -1 4202560 1002 0 0 0 801 5 0 0 16 0 8 0 146869011 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=146432 CPUtime=8.02
/proc/21375/task/21487/stat : 21487 (pb_cplex) S 21374 21374 20819 0 -1 4202560 1578 0 0 0 798 4 0 0 15 0 8 0 146869011 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 7 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=146432 CPUtime=7.07
/proc/21375/task/21488/stat : 21488 (pb_cplex) S 21374 21374 20819 0 -1 4202560 4471 0 0 0 702 5 0 0 15 0 8 0 146869011 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 7 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=146432 CPUtime=7.19
/proc/21375/task/21489/stat : 21489 (pb_cplex) S 21374 21374 20819 0 -1 4202560 1074 0 0 0 715 4 0 0 15 0 8 0 146869011 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 2 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=146432 CPUtime=7.01
/proc/21375/task/21490/stat : 21490 (pb_cplex) S 21374 21374 20819 0 -1 4202560 71 0 0 0 697 4 0 0 16 0 8 0 146869011 149946368 14445 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 157.94
Current children cumulated vsize (KiB) 155132

[startup+102.301 s]
/proc/loadavg: 9.63 9.08 6.01 10/199 21506
/proc/meminfo: memFree=31253680/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=145408 CPUtime=341.83
/proc/21375/stat : 21375 (pb_cplex) S 21374 21374 20819 0 -1 4202496 93354 0 0 0 33890 293 0 0 15 0 8 0 146865732 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21375/statm: 36352 14208 1089 2985 0 32894 0
[pid=21375/tid=21484] ppid=21374 vsize=145408 CPUtime=30.58
/proc/21375/task/21484/stat : 21484 (pb_cplex) R 21374 21374 20819 0 -1 4202560 1525 0 0 0 3032 26 0 0 15 0 8 0 146869011 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=145408 CPUtime=29.89
/proc/21375/task/21485/stat : 21485 (pb_cplex) S 21374 21374 20819 0 -1 4202560 905 0 0 0 2964 25 0 0 15 0 8 0 146869011 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 4 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=145408 CPUtime=30.77
/proc/21375/task/21486/stat : 21486 (pb_cplex) S 21374 21374 20819 0 -1 4202560 2308 0 0 0 3053 24 0 0 15 0 8 0 146869011 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 4 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=145408 CPUtime=30.78
/proc/21375/task/21487/stat : 21487 (pb_cplex) S 21374 21374 20819 0 -1 4202560 3586 0 0 0 3056 22 0 0 15 0 8 0 146869011 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 5 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=145408 CPUtime=29.8
/proc/21375/task/21488/stat : 21488 (pb_cplex) S 21374 21374 20819 0 -1 4202560 7071 0 0 0 2955 25 0 0 15 0 8 0 146869011 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 7 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=145408 CPUtime=29.63
/proc/21375/task/21489/stat : 21489 (pb_cplex) S 21374 21374 20819 0 -1 4202560 2777 0 0 0 2940 23 0 0 15 0 8 0 146869011 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 0 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=29.8
/proc/21375/task/21490/stat : 21490 (pb_cplex) S 21374 21374 20819 0 -1 4202560 1111 0 0 0 2955 25 0 0 15 0 8 0 146869011 148897792 14208 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 341.83
Current children cumulated vsize (KiB) 154108

[startup+162.301 s]
/proc/loadavg: 9.53 9.13 6.21 10/199 21507
/proc/meminfo: memFree=31253604/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0

################
# More data... #
################

/proc/21375/task/21489/stat : 21489 (pb_cplex) S 21374 21374 20819 0 -1 4202560 6817 0 0 0 8087 72 0 0 15 0 8 0 146869011 148897792 14224 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 4 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=81.67
/proc/21375/task/21490/stat : 21490 (pb_cplex) S 21374 21374 20819 0 -1 4202560 3932 0 0 0 8093 74 0 0 15 0 8 0 146869011 148897792 14224 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 762.61
Current children cumulated vsize (KiB) 154108

[startup+282.301 s]
/proc/loadavg: 9.21 9.17 6.58 11/199 21513
/proc/meminfo: memFree=31255400/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=145408 CPUtime=981.33
/proc/21375/stat : 21375 (pb_cplex) S 21374 21374 20819 0 -1 4202496 121631 0 0 0 97283 850 0 0 16 0 8 0 146865732 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21375/statm: 36352 14399 1105 2985 0 32894 0
[pid=21375/tid=21484] ppid=21374 vsize=145408 CPUtime=109.36
/proc/21375/task/21484/stat : 21484 (pb_cplex) R 21374 21374 20819 0 -1 4202560 5322 0 0 0 10846 90 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 135982686 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=145408 CPUtime=108.33
/proc/21375/task/21485/stat : 21485 (pb_cplex) S 21374 21374 20819 0 -1 4202560 3542 0 0 0 10738 95 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 1 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=145408 CPUtime=109.39
/proc/21375/task/21486/stat : 21486 (pb_cplex) R 21374 21374 20819 0 -1 4202560 4527 0 0 0 10841 98 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=145408 CPUtime=109.69
/proc/21375/task/21487/stat : 21487 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6881 0 0 0 10878 91 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 136019045 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=145408 CPUtime=108.51
/proc/21375/task/21488/stat : 21488 (pb_cplex) R 21374 21374 20819 0 -1 4202560 11306 0 0 0 10758 93 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 137341798 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=145408 CPUtime=108.95
/proc/21375/task/21489/stat : 21489 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7231 0 0 0 10799 96 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=108.86
/proc/21375/task/21490/stat : 21490 (pb_cplex) R 21374 21374 20819 0 -1 4202560 4561 0 0 0 10791 95 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 139046505 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 981.33
Current children cumulated vsize (KiB) 154108

[startup+342.319 s]
/proc/loadavg: 9.55 9.27 6.78 11/199 21515
/proc/meminfo: memFree=31252508/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=145408 CPUtime=1201.17
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 127394 0 0 0 119074 1043 0 0 15 0 8 0 146865732 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 137412881 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21375/statm: 36352 14399 1105 2985 0 32894 0
[pid=21375/tid=21484] ppid=21374 vsize=145408 CPUtime=136.41
/proc/21375/task/21484/stat : 21484 (pb_cplex) R 21374 21374 20819 0 -1 4202560 5919 0 0 0 13528 113 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 135526417 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=145408 CPUtime=135.39
/proc/21375/task/21485/stat : 21485 (pb_cplex) R 21374 21374 20819 0 -1 4202560 4156 0 0 0 13419 120 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=145408 CPUtime=136.49
/proc/21375/task/21486/stat : 21486 (pb_cplex) R 21374 21374 20819 0 -1 4202560 5440 0 0 0 13528 121 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=145408 CPUtime=136.95
/proc/21375/task/21487/stat : 21487 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7669 0 0 0 13579 116 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=145408 CPUtime=135.8
/proc/21375/task/21488/stat : 21488 (pb_cplex) R 21374 21374 20819 0 -1 4202560 12117 0 0 0 13466 114 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=145408 CPUtime=136.07
/proc/21375/task/21489/stat : 21489 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7797 0 0 0 13486 121 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 136017748 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=136.31
/proc/21375/task/21490/stat : 21490 (pb_cplex) R 21374 21374 20819 0 -1 4202560 5430 0 0 0 13510 121 0 0 15 0 8 0 146869011 148897792 14399 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1201.17
Current children cumulated vsize (KiB) 154108

[startup+402.334 s]
/proc/loadavg: 9.09 9.18 6.90 6/199 21517
/proc/meminfo: memFree=31251820/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=145408 CPUtime=1426.65
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 131179 0 0 0 141420 1245 0 0 15 0 8 0 146865732 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 135447033 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21375/statm: 36352 14400 1105 2985 0 32894 0
[pid=21375/tid=21484] ppid=21374 vsize=145408 CPUtime=164.4
/proc/21375/task/21484/stat : 21484 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6543 0 0 0 16300 140 0 0 15 0 8 0 146869011 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=145408 CPUtime=163.3
/proc/21375/task/21485/stat : 21485 (pb_cplex) S 21374 21374 20819 0 -1 4202560 4319 0 0 0 16185 145 0 0 16 0 8 0 146869011 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 2 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=145408 CPUtime=164.33
/proc/21375/task/21486/stat : 21486 (pb_cplex) R 21374 21374 20819 0 -1 4202560 5728 0 0 0 16288 145 0 0 15 0 8 0 146869011 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 136388105 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=145408 CPUtime=164.67
/proc/21375/task/21487/stat : 21487 (pb_cplex) S 21374 21374 20819 0 -1 4202560 8178 0 0 0 16328 139 0 0 15 0 8 0 146869011 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 144580073 0 2 4096 16387 18446604453947693248 0 0 -1 7 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=145408 CPUtime=163.87
/proc/21375/task/21488/stat : 21488 (pb_cplex) R 21374 21374 20819 0 -1 4202560 12488 0 0 0 16249 138 0 0 15 0 8 0 146869011 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 136403116 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=145408 CPUtime=163.66
/proc/21375/task/21489/stat : 21489 (pb_cplex) R 21374 21374 20819 0 -1 4202560 8650 0 0 0 16220 146 0 0 15 0 8 0 146869011 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=164.32
/proc/21375/task/21490/stat : 21490 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6111 0 0 0 16282 150 0 0 15 0 8 0 146869011 148897792 14400 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1426.65
Current children cumulated vsize (KiB) 154108

[startup+462.348 s]
/proc/loadavg: 9.54 9.27 7.07 8/199 21525
/proc/meminfo: memFree=31257024/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=145408 CPUtime=1651.28
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 143190 0 0 0 163664 1464 0 0 16 0 8 0 146865732 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21375/statm: 36352 14303 1105 2985 0 32894 0
[pid=21375/tid=21484] ppid=21374 vsize=145408 CPUtime=192.14
/proc/21375/task/21484/stat : 21484 (pb_cplex) R 21374 21374 20819 0 -1 4202560 8522 0 0 0 19048 166 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=145408 CPUtime=191.32
/proc/21375/task/21485/stat : 21485 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6099 0 0 0 18960 172 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=145408 CPUtime=192.17
/proc/21375/task/21486/stat : 21486 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6667 0 0 0 19043 174 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 136384861 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=145408 CPUtime=192.08
/proc/21375/task/21487/stat : 21487 (pb_cplex) R 21374 21374 20819 0 -1 4202560 9014 0 0 0 19043 165 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=145408 CPUtime=191.75
/proc/21375/task/21488/stat : 21488 (pb_cplex) R 21374 21374 20819 0 -1 4202560 15077 0 0 0 19008 167 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 136001985 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=145408 CPUtime=191.56
/proc/21375/task/21489/stat : 21489 (pb_cplex) R 21374 21374 20819 0 -1 4202560 9653 0 0 0 18983 173 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 135525914 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=192.2
/proc/21375/task/21490/stat : 21490 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7420 0 0 0 19040 180 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1651.28
Current children cumulated vsize (KiB) 154108



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+490.401 s]
/proc/loadavg: 8.99 9.16 7.10 10/183 21547
/proc/meminfo: memFree=31329244/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=145408 CPUtime=1800.68
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 148494 0 0 0 178455 1613 0 0 16 0 8 0 146865732 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 135443527 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21375/statm: 36352 14303 1105 2985 0 32894 0
[pid=21375/tid=21484] ppid=21374 vsize=145408 CPUtime=210.59
/proc/21375/task/21484/stat : 21484 (pb_cplex) R 21374 21374 20819 0 -1 4202560 8647 0 0 0 20873 186 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 135983228 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=145408 CPUtime=209.75
/proc/21375/task/21485/stat : 21485 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6629 0 0 0 20787 188 0 0 16 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=145408 CPUtime=210.79
/proc/21375/task/21486/stat : 21486 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7584 0 0 0 20887 192 0 0 16 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 136016700 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=145408 CPUtime=210.8
/proc/21375/task/21487/stat : 21487 (pb_cplex) R 21374 21374 20819 0 -1 4202560 9940 0 0 0 20896 184 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=145408 CPUtime=210.49
/proc/21375/task/21488/stat : 21488 (pb_cplex) R 21374 21374 20819 0 -1 4202560 15501 0 0 0 20862 187 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 137334235 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=145408 CPUtime=210.19
/proc/21375/task/21489/stat : 21489 (pb_cplex) R 21374 21374 20819 0 -1 4202560 10535 0 0 0 20827 192 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=210.94
/proc/21375/task/21490/stat : 21490 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7685 0 0 0 20895 199 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 136017757 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1800.68
Current children cumulated vsize (KiB) 154108

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+490.401 s]
/proc/loadavg: 8.99 9.16 7.10 10/183 21547
/proc/meminfo: memFree=31329244/32951124 swapFree=67111528/67111528
[pid=21374] ppid=21372 vsize=8700 CPUtime=0
/proc/21374/stat : 21374 (run) S 21372 21374 20819 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146865732 8908800 249 4089446400 4194304 4922060 140734357665472 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21374/statm: 2175 249 208 178 0 68 0
[pid=21375] ppid=21374 vsize=145408 CPUtime=1800.68
/proc/21375/stat : 21375 (pb_cplex) R 21374 21374 20819 0 -1 4202496 148494 0 0 0 178455 1613 0 0 16 0 8 0 146865732 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 135443527 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21375/statm: 36352 14303 1105 2985 0 32894 0
[pid=21375/tid=21484] ppid=21374 vsize=145408 CPUtime=210.59
/proc/21375/task/21484/stat : 21484 (pb_cplex) R 21374 21374 20819 0 -1 4202560 8647 0 0 0 20873 186 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 135983228 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21375/tid=21485] ppid=21374 vsize=145408 CPUtime=209.75
/proc/21375/task/21485/stat : 21485 (pb_cplex) R 21374 21374 20819 0 -1 4202560 6629 0 0 0 20787 188 0 0 16 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21375/tid=21486] ppid=21374 vsize=145408 CPUtime=210.79
/proc/21375/task/21486/stat : 21486 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7584 0 0 0 20887 192 0 0 16 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 136016700 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21375/tid=21487] ppid=21374 vsize=145408 CPUtime=210.8
/proc/21375/task/21487/stat : 21487 (pb_cplex) R 21374 21374 20819 0 -1 4202560 9940 0 0 0 20896 184 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21375/tid=21488] ppid=21374 vsize=145408 CPUtime=210.49
/proc/21375/task/21488/stat : 21488 (pb_cplex) R 21374 21374 20819 0 -1 4202560 15501 0 0 0 20862 187 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 137334235 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21375/tid=21489] ppid=21374 vsize=145408 CPUtime=210.19
/proc/21375/task/21489/stat : 21489 (pb_cplex) R 21374 21374 20819 0 -1 4202560 10535 0 0 0 20827 192 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21375/tid=21490] ppid=21374 vsize=145408 CPUtime=210.94
/proc/21375/task/21490/stat : 21490 (pb_cplex) R 21374 21374 20819 0 -1 4202560 7685 0 0 0 20895 199 0 0 15 0 8 0 146869011 148897792 14303 4089446400 134512640 146736320 4293025024 18446744073709551615 136017757 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1800.68
Current children cumulated vsize (KiB) 154108

Child status: 0
Real time (s): 490.415
CPU time (s): 1800.73
CPU user time (s): 1784.6
CPU system time (s): 16.1365
CPU usage (%): 367.185
Max. virtual memory (cumulated for all children) (KiB): 167280

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1784.6
system time used= 16.1365
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 148868
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 346626
involuntary context switches= 37059

runsolver used 0.899863 second user time and 2.03869 second system time

The end

Launcher Data 

Begin job on node133 at 2010-07-02 09:28:21
IDJOB=2697423
IDBENCH=79285
IDSOLVER=1210
FILE ID=node133/2697423-1278055701
PBS_JOBID= 11198916
Free space on /tmp= 74188 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900565534--soft-0-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697423-1278055701/watcher-2697423-1278055701 -o /tmp/evaluation-result-2697423-1278055701/solver-2697423-1278055701 -C 1800 -W 2000 -M 3800  run HOME/instance-2697423-1278055701.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= cf389047fd441928b15b38cce26251ed
RANDOM SEED=65852077

node133.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7296.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31299608 kB
Buffers:        217744 kB
Cached:        1155972 kB
SwapCached:          0 kB
Active:         724276 kB
Inactive:       779972 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31299608 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16720 kB
Writeback:           0 kB
AnonPages:      130508 kB
Mapped:          18932 kB
Slab:            82560 kB
PageTables:       5040 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   353056 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74188 MiB
End job on node133 at 2010-07-02 09:36:32