Trace number 2697354

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.31 491.167

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900667195--soft-66-100-0.wbo
MD5SUMc3c12ca4f6a34dfd4eee1807e0cef251
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark355.567
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints911
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 46278
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 46277
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 655
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697354-1278055218.wbo.
0.00/0.00	c Instance file HOME/instance-2697354-1278055218.wbo
0.00/0.00	c File size is 87915 bytes.
0.00/0.17	c Highest Coefficient sum: 655
0.00/0.17	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.18	c ============================[ Problem Statistics ]=============================
0.00/0.18	c |                                                                             |
0.00/0.18	c |  Parsing time:          0.18         s                                      |
0.00/0.18	c |  Number Variables:      1142                                                |
0.00/0.18	c |  Number Constraints:    2708                                                |
0.00/0.18	c ===============================================================================
1800.24/491.16	c Remaining time 1799.82 sec.
1800.24/491.16	c 
1800.24/491.16	c *** INTERRUPTED ***
1800.24/491.16	s UNKNOWN
1800.24/491.16	c Total CPU time              : 1783.5 s
1800.24/491.16	c Memory used           : 154.88 MB
1800.24/491.16	c CPU time              : 1783.5 s
1800.24/491.16	c 
1800.24/491.16	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697354-1278055218/watcher-2697354-1278055218 -o /tmp/evaluation-result-2697354-1278055218/solver-2697354-1278055218 -C 1800 -W 2000 -M 3800 run HOME/instance-2697354-1278055218.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.39 6.57 3.00 4/183 21227
/proc/meminfo: memFree=31335968/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0

[startup+0.0952369 s]
/proc/loadavg: 9.39 6.57 3.00 4/183 21227
/proc/meminfo: memFree=31335968/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.100246 s]
/proc/loadavg: 9.39 6.57 3.00 4/183 21227
/proc/meminfo: memFree=31335968/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300221 s]
/proc/loadavg: 9.39 6.57 3.00 4/183 21227
/proc/meminfo: memFree=31335968/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.719066 s]
/proc/loadavg: 9.39 6.57 3.00 4/183 21227
/proc/meminfo: memFree=31335968/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50105 s]
/proc/loadavg: 9.39 6.57 3.00 3/192 21271
/proc/meminfo: memFree=31290256/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=123176 CPUtime=2.84
/proc/21228/stat : 21228 (pb_cplex) R 21227 21227 20807 0 -1 4202496 23369 0 0 0 276 8 0 0 18 0 8 0 146817443 126132224 9109 4089446400 134512640 146736320 4288666032 18446744073709551615 139042981 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21228/statm: 30794 9109 1079 2985 0 27336 0
Current children cumulated CPU time (s) 2.84
Current children cumulated vsize (KiB) 131876

[startup+3.10063 s]
/proc/loadavg: 9.39 6.57 3.00 7/199 21292
/proc/meminfo: memFree=31283504/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=133624 CPUtime=5.13
/proc/21228/stat : 21228 (pb_cplex) R 21227 21227 20807 0 -1 4202496 26386 0 0 0 504 9 0 0 17 0 8 0 146817443 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 136400730 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21228/statm: 33406 11242 1080 2985 0 29948 0
[pid=21228/tid=21286] ppid=21227 vsize=133624 CPUtime=0.11
/proc/21228/task/21286/stat : 21286 (pb_cplex) R 21227 21227 20807 0 -1 4202560 16 0 0 0 11 0 0 0 17 0 8 0 146817586 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 136007313 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21228/tid=21287] ppid=21227 vsize=133624 CPUtime=0.2
/proc/21228/task/21287/stat : 21287 (pb_cplex) R 21227 21227 20807 0 -1 4202560 280 0 0 0 20 0 0 0 18 0 8 0 146817586 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 139044542 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21288] ppid=21227 vsize=133624 CPUtime=0.18
/proc/21228/task/21288/stat : 21288 (pb_cplex) R 21227 21227 20807 0 -1 4202560 235 0 0 0 18 0 0 0 18 0 8 0 146817586 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 135846783 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21228/tid=21289] ppid=21227 vsize=133624 CPUtime=0.17
/proc/21228/task/21289/stat : 21289 (pb_cplex) R 21227 21227 20807 0 -1 4202560 744 0 0 0 17 0 0 0 17 0 8 0 146817586 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 136007284 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21228/tid=21290] ppid=21227 vsize=133624 CPUtime=0.11
/proc/21228/task/21290/stat : 21290 (pb_cplex) R 21227 21227 20807 0 -1 4202560 49 0 0 0 11 0 0 0 17 0 8 0 146817586 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 136007371 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21228/tid=21291] ppid=21227 vsize=133624 CPUtime=0.09
/proc/21228/task/21291/stat : 21291 (pb_cplex) R 21227 21227 20807 0 -1 4202560 664 0 0 0 9 0 0 0 17 0 8 0 146817586 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21228/tid=21292] ppid=21227 vsize=133624 CPUtime=0.09
/proc/21228/task/21292/stat : 21292 (pb_cplex) R 21227 21227 20807 0 -1 4202560 610 0 0 0 9 0 0 0 16 0 8 0 146817586 136830976 11242 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 5.13
Current children cumulated vsize (KiB) 142324

[startup+6.30508 s]
/proc/loadavg: 9.36 6.61 3.04 9/199 21292
/proc/meminfo: memFree=31269616/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=134380 CPUtime=18.16
/proc/21228/stat : 21228 (pb_cplex) S 21227 21227 20807 0 -1 4202496 29759 0 0 0 1795 21 0 0 16 0 8 0 146817443 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21228/statm: 33595 11907 1080 2985 0 30137 0
[pid=21228/tid=21286] ppid=21227 vsize=134380 CPUtime=1.72
/proc/21228/task/21286/stat : 21286 (pb_cplex) S 21227 21227 20807 0 -1 4202560 96 0 0 0 171 1 0 0 15 0 8 0 146817586 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
[pid=21228/tid=21287] ppid=21227 vsize=134380 CPUtime=1.88
/proc/21228/task/21287/stat : 21287 (pb_cplex) S 21227 21227 20807 0 -1 4202560 869 0 0 0 187 1 0 0 15 0 8 0 146817586 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
[pid=21228/tid=21288] ppid=21227 vsize=134380 CPUtime=1.85
/proc/21228/task/21288/stat : 21288 (pb_cplex) S 21227 21227 20807 0 -1 4202560 289 0 0 0 184 1 0 0 15 0 8 0 146817586 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 0 0 0 0
[pid=21228/tid=21289] ppid=21227 vsize=134380 CPUtime=1.59
/proc/21228/task/21289/stat : 21289 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1368 0 0 0 158 1 0 0 15 0 8 0 146817586 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 5 0 0 0
[pid=21228/tid=21290] ppid=21227 vsize=134380 CPUtime=1.81
/proc/21228/task/21290/stat : 21290 (pb_cplex) S 21227 21227 20807 0 -1 4202560 410 0 0 0 180 1 0 0 16 0 8 0 146817586 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 0 0 0 0
[pid=21228/tid=21291] ppid=21227 vsize=134380 CPUtime=1.65
/proc/21228/task/21291/stat : 21291 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1495 0 0 0 164 1 0 0 15 0 8 0 146817586 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 3 0 0 0
[pid=21228/tid=21292] ppid=21227 vsize=134380 CPUtime=1.69
/proc/21228/task/21292/stat : 21292 (pb_cplex) R 21227 21227 20807 0 -1 4202560 754 0 0 0 168 1 0 0 15 0 8 0 146817586 137605120 11907 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 18.16
Current children cumulated vsize (KiB) 143080

[startup+12.7009 s]
/proc/loadavg: 9.25 6.63 3.06 10/199 21292
/proc/meminfo: memFree=31260696/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=139500 CPUtime=42.37
/proc/21228/stat : 21228 (pb_cplex) R 21227 21227 20807 0 -1 4202496 39770 0 0 0 4195 42 0 0 16 0 8 0 146817443 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 135846439 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21228/statm: 34875 12731 1080 2985 0 31417 0
[pid=21228/tid=21286] ppid=21227 vsize=139500 CPUtime=4.77
/proc/21228/task/21286/stat : 21286 (pb_cplex) R 21227 21227 20807 0 -1 4202560 723 0 0 0 474 3 0 0 15 0 8 0 146817586 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21228/tid=21287] ppid=21227 vsize=139500 CPUtime=4.85
/proc/21228/task/21287/stat : 21287 (pb_cplex) R 21227 21227 20807 0 -1 4202560 2019 0 0 0 482 3 0 0 15 0 8 0 146817586 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 136403117 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21228/tid=21288] ppid=21227 vsize=139500 CPUtime=4.88
/proc/21228/task/21288/stat : 21288 (pb_cplex) R 21227 21227 20807 0 -1 4202560 2535 0 0 0 484 4 0 0 15 0 8 0 146817586 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21289] ppid=21227 vsize=139500 CPUtime=4.69
/proc/21228/task/21289/stat : 21289 (pb_cplex) R 21227 21227 20807 0 -1 4202560 3186 0 0 0 465 4 0 0 15 0 8 0 146817586 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21228/tid=21290] ppid=21227 vsize=139500 CPUtime=4.71
/proc/21228/task/21290/stat : 21290 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1531 0 0 0 468 3 0 0 15 0 8 0 146817586 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21228/tid=21291] ppid=21227 vsize=139500 CPUtime=4.54
/proc/21228/task/21291/stat : 21291 (pb_cplex) R 21227 21227 20807 0 -1 4202560 2113 0 0 0 449 5 0 0 15 0 8 0 146817586 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21228/tid=21292] ppid=21227 vsize=139500 CPUtime=4.66
/proc/21228/task/21292/stat : 21292 (pb_cplex) R 21227 21227 20807 0 -1 4202560 2121 0 0 0 462 4 0 0 15 0 8 0 146817586 142848000 12731 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 42.37
Current children cumulated vsize (KiB) 148200

[startup+25.5006 s]
/proc/loadavg: 9.72 6.86 3.19 4/199 21307
/proc/meminfo: memFree=31251404/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=139132 CPUtime=81.54
/proc/21228/stat : 21228 (pb_cplex) R 21227 21227 20807 0 -1 4202496 55411 0 0 0 8078 76 0 0 22 0 8 0 146817443 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 139042897 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21228/statm: 34783 12756 1086 2985 0 31325 0
[pid=21228/tid=21301] ppid=21227 vsize=139132 CPUtime=3.18
/proc/21228/task/21301/stat : 21301 (pb_cplex) S 21227 21227 20807 0 -1 4202560 852 0 0 0 318 0 0 0 20 0 8 0 146819429 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 6 0 0 0
[pid=21228/tid=21302] ppid=21227 vsize=139132 CPUtime=1.91
/proc/21228/task/21302/stat : 21302 (pb_cplex) S 21227 21227 20807 0 -1 4202560 192 0 0 0 191 0 0 0 18 0 8 0 146819429 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 2 0 0 0
[pid=21228/tid=21303] ppid=21227 vsize=139132 CPUtime=2.43
/proc/21228/task/21303/stat : 21303 (pb_cplex) S 21227 21227 20807 0 -1 4202560 750 0 0 0 243 0 0 0 18 0 8 0 146819429 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 7 0 0 0
[pid=21228/tid=21304] ppid=21227 vsize=139132 CPUtime=1.32
/proc/21228/task/21304/stat : 21304 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1282 0 0 0 132 0 0 0 17 0 8 0 146819429 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 3 0 0 0
[pid=21228/tid=21305] ppid=21227 vsize=139132 CPUtime=2.37
/proc/21228/task/21305/stat : 21305 (pb_cplex) S 21227 21227 20807 0 -1 4202560 288 0 0 0 237 0 0 0 18 0 8 0 146819429 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
[pid=21228/tid=21306] ppid=21227 vsize=139132 CPUtime=0.79
/proc/21228/task/21306/stat : 21306 (pb_cplex) S 21227 21227 20807 0 -1 4202560 837 0 0 0 79 0 0 0 16 0 8 0 146819429 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 4 0 0 0
[pid=21228/tid=21307] ppid=21227 vsize=139132 CPUtime=1.22
/proc/21228/task/21307/stat : 21307 (pb_cplex) S 21227 21227 20807 0 -1 4202560 636 0 0 0 122 0 0 0 17 0 8 0 146819429 142471168 12756 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 81.54
Current children cumulated vsize (KiB) 147832

[startup+51.1016 s]
/proc/loadavg: 8.95 6.91 3.31 9/199 21322
/proc/meminfo: memFree=31251024/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=147324 CPUtime=192.65
/proc/21228/stat : 21228 (pb_cplex) R 21227 21227 20807 0 -1 4202496 97678 0 0 0 19079 186 0 0 15 0 8 0 146817443 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21228/statm: 36831 14538 1086 2985 0 33373 0
[pid=21228/tid=21301] ppid=21227 vsize=147324 CPUtime=16.76
/proc/21228/task/21301/stat : 21301 (pb_cplex) S 21227 21227 20807 0 -1 4202560 7397 0 0 0 1662 14 0 0 16 0 8 0 146819429 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 3 0 0 0
[pid=21228/tid=21302] ppid=21227 vsize=147324 CPUtime=15.68
/proc/21228/task/21302/stat : 21302 (pb_cplex) S 21227 21227 20807 0 -1 4202560 3923 0 0 0 1554 14 0 0 15 0 8 0 146819429 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 7 0 0 0
[pid=21228/tid=21303] ppid=21227 vsize=147324 CPUtime=16.02
/proc/21228/task/21303/stat : 21303 (pb_cplex) S 21227 21227 20807 0 -1 4202560 3027 0 0 0 1589 13 0 0 15 0 8 0 146819429 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 7 0 0 0
[pid=21228/tid=21304] ppid=21227 vsize=147324 CPUtime=14.99
/proc/21228/task/21304/stat : 21304 (pb_cplex) S 21227 21227 20807 0 -1 4202560 6539 0 0 0 1487 12 0 0 15 0 8 0 146819429 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 2 0 0 0
[pid=21228/tid=21305] ppid=21227 vsize=147324 CPUtime=16.03
/proc/21228/task/21305/stat : 21305 (pb_cplex) S 21227 21227 20807 0 -1 4202560 2879 0 0 0 1589 14 0 0 15 0 8 0 146819429 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 6 0 0 0
[pid=21228/tid=21306] ppid=21227 vsize=147324 CPUtime=14.27
/proc/21228/task/21306/stat : 21306 (pb_cplex) S 21227 21227 20807 0 -1 4202560 6098 0 0 0 1414 13 0 0 15 0 8 0 146819429 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 5 0 0 0
[pid=21228/tid=21307] ppid=21227 vsize=147324 CPUtime=15.07
/proc/21228/task/21307/stat : 21307 (pb_cplex) S 21227 21227 20807 0 -1 4202560 4460 0 0 0 1493 14 0 0 15 0 8 0 146819429 150859776 14538 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 192.65
Current children cumulated vsize (KiB) 156024

[startup+102.301 s]
/proc/loadavg: 9.42 7.35 3.64 8/199 21323
/proc/meminfo: memFree=31249188/32951124 swapFree=67111528/67111528

################
# More data... #
################

/proc/21228/task/21306/stat : 21306 (pb_cplex) S 21227 21227 20807 0 -1 4202560 16492 0 0 0 8903 82 0 0 15 0 8 0 146819429 145616896 13926 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 0 0 0 0
[pid=21228/tid=21307] ppid=21227 vsize=142204 CPUtime=90.95
/proc/21228/task/21307/stat : 21307 (pb_cplex) R 21227 21227 20807 0 -1 4202560 18081 0 0 0 9007 88 0 0 15 0 8 0 146819429 145616896 13926 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 804.84
Current children cumulated vsize (KiB) 150904

[startup+282.301 s]
/proc/loadavg: 9.58 8.37 4.69 12/199 21328
/proc/meminfo: memFree=31251640/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=143228 CPUtime=1020.99
/proc/21228/stat : 21228 (pb_cplex) S 21227 21227 20807 0 -1 4202496 206772 0 0 0 101158 941 0 0 15 0 8 0 146817443 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21228/statm: 35807 14099 1104 2985 0 32349 0
[pid=21228/tid=21301] ppid=21227 vsize=143228 CPUtime=119.01
/proc/21228/task/21301/stat : 21301 (pb_cplex) S 21227 21227 20807 0 -1 4202560 20863 0 0 0 11793 108 0 0 15 0 8 0 146819429 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 4 0 0 0
[pid=21228/tid=21302] ppid=21227 vsize=143228 CPUtime=117.92
/proc/21228/task/21302/stat : 21302 (pb_cplex) S 21227 21227 20807 0 -1 4202560 12354 0 0 0 11685 107 0 0 15 0 8 0 146819429 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 4 0 0 0
[pid=21228/tid=21303] ppid=21227 vsize=143228 CPUtime=117.98
/proc/21228/task/21303/stat : 21303 (pb_cplex) S 21227 21227 20807 0 -1 4202560 21971 0 0 0 11694 104 0 0 15 0 8 0 146819429 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 4 0 0 0
[pid=21228/tid=21304] ppid=21227 vsize=143228 CPUtime=117.41
/proc/21228/task/21304/stat : 21304 (pb_cplex) R 21227 21227 20807 0 -1 4202560 19776 0 0 0 11633 108 0 0 15 0 8 0 146819429 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21228/tid=21305] ppid=21227 vsize=143228 CPUtime=118.14
/proc/21228/task/21305/stat : 21305 (pb_cplex) S 21227 21227 20807 0 -1 4202560 15950 0 0 0 11705 109 0 0 15 0 8 0 146819429 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 4 0 0 0
[pid=21228/tid=21306] ppid=21227 vsize=143228 CPUtime=116.7
/proc/21228/task/21306/stat : 21306 (pb_cplex) S 21227 21227 20807 0 -1 4202560 18778 0 0 0 11563 107 0 0 15 0 8 0 146819429 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 0 0 0 0
[pid=21228/tid=21307] ppid=21227 vsize=143228 CPUtime=117.75
/proc/21228/task/21307/stat : 21307 (pb_cplex) S 21227 21227 20807 0 -1 4202560 19620 0 0 0 11665 110 0 0 16 0 8 0 146819429 146665472 14099 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1020.99
Current children cumulated vsize (KiB) 151928

[startup+342.33 s]
/proc/loadavg: 9.54 8.58 4.99 11/199 21329
/proc/meminfo: memFree=31251652/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=143228 CPUtime=1240.26
/proc/21228/stat : 21228 (pb_cplex) R 21227 21227 20807 0 -1 4202496 219877 0 0 0 122879 1147 0 0 16 0 8 0 146817443 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 136403109 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21228/statm: 35807 14103 1104 2985 0 32349 0
[pid=21228/tid=21301] ppid=21227 vsize=143228 CPUtime=146.45
/proc/21228/task/21301/stat : 21301 (pb_cplex) R 21227 21227 20807 0 -1 4202560 22603 0 0 0 14510 135 0 0 15 0 8 0 146819429 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21228/tid=21302] ppid=21227 vsize=143228 CPUtime=145.03
/proc/21228/task/21302/stat : 21302 (pb_cplex) R 21227 21227 20807 0 -1 4202560 14477 0 0 0 14369 134 0 0 15 0 8 0 146819429 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 136388102 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21228/tid=21303] ppid=21227 vsize=143228 CPUtime=145.22
/proc/21228/task/21303/stat : 21303 (pb_cplex) R 21227 21227 20807 0 -1 4202560 23943 0 0 0 14393 129 0 0 15 0 8 0 146819429 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21228/tid=21304] ppid=21227 vsize=143228 CPUtime=144.34
/proc/21228/task/21304/stat : 21304 (pb_cplex) S 21227 21227 20807 0 -1 4202560 20789 0 0 0 14301 133 0 0 15 0 8 0 146819429 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 3 0 0 0
[pid=21228/tid=21305] ppid=21227 vsize=143228 CPUtime=145.1
/proc/21228/task/21305/stat : 21305 (pb_cplex) R 21227 21227 20807 0 -1 4202560 17288 0 0 0 14377 133 0 0 15 0 8 0 146819429 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21306] ppid=21227 vsize=143228 CPUtime=143.77
/proc/21228/task/21306/stat : 21306 (pb_cplex) R 21227 21227 20807 0 -1 4202560 21236 0 0 0 14245 132 0 0 15 0 8 0 146819429 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21228/tid=21307] ppid=21227 vsize=143228 CPUtime=144.77
/proc/21228/task/21307/stat : 21307 (pb_cplex) R 21227 21227 20807 0 -1 4202560 20075 0 0 0 14340 137 0 0 15 0 8 0 146819429 146665472 14103 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1240.26
Current children cumulated vsize (KiB) 151928

[startup+402.301 s]
/proc/loadavg: 9.24 8.67 5.24 9/199 21331
/proc/meminfo: memFree=31252356/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=143228 CPUtime=1458.46
/proc/21228/stat : 21228 (pb_cplex) R 21227 21227 20807 0 -1 4202496 232606 0 0 0 144492 1354 0 0 16 0 8 0 146817443 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 136017474 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21228/statm: 35807 14127 1104 2985 0 32349 0
[pid=21228/tid=21301] ppid=21227 vsize=143228 CPUtime=173.59
/proc/21228/task/21301/stat : 21301 (pb_cplex) R 21227 21227 20807 0 -1 4202560 23695 0 0 0 17197 162 0 0 15 0 8 0 146819429 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21228/tid=21302] ppid=21227 vsize=143228 CPUtime=172.04
/proc/21228/task/21302/stat : 21302 (pb_cplex) R 21227 21227 20807 0 -1 4202560 16822 0 0 0 17042 162 0 0 15 0 8 0 146819429 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21228/tid=21303] ppid=21227 vsize=143228 CPUtime=171.98
/proc/21228/task/21303/stat : 21303 (pb_cplex) R 21227 21227 20807 0 -1 4202560 24876 0 0 0 17044 154 0 0 15 0 8 0 146819429 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21304] ppid=21227 vsize=143228 CPUtime=171.18
/proc/21228/task/21304/stat : 21304 (pb_cplex) R 21227 21227 20807 0 -1 4202560 22663 0 0 0 16958 160 0 0 15 0 8 0 146819429 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21228/tid=21305] ppid=21227 vsize=143228 CPUtime=172.11
/proc/21228/task/21305/stat : 21305 (pb_cplex) S 21227 21227 20807 0 -1 4202560 18945 0 0 0 17055 156 0 0 15 0 8 0 146819429 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
[pid=21228/tid=21306] ppid=21227 vsize=143228 CPUtime=170.74
/proc/21228/task/21306/stat : 21306 (pb_cplex) S 21227 21227 20807 0 -1 4202560 23294 0 0 0 16917 157 0 0 15 0 8 0 146819429 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
[pid=21228/tid=21307] ppid=21227 vsize=143228 CPUtime=171.84
/proc/21228/task/21307/stat : 21307 (pb_cplex) S 21227 21227 20807 0 -1 4202560 20653 0 0 0 17023 161 0 0 15 0 8 0 146819429 146665472 14127 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1458.46
Current children cumulated vsize (KiB) 151928

[startup+462.301 s]
/proc/loadavg: 9.43 8.77 5.48 10/199 21346
/proc/meminfo: memFree=31238240/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=157564 CPUtime=1677.09
/proc/21228/stat : 21228 (pb_cplex) S 21227 21227 20807 0 -1 4202496 245697 0 0 0 166141 1568 0 0 16 0 8 0 146817443 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21228/statm: 39391 17207 1104 2985 0 35933 0
[pid=21228/tid=21333] ppid=21227 vsize=157564 CPUtime=16.51
/proc/21228/task/21333/stat : 21333 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1632 0 0 0 1636 15 0 0 15 0 8 0 146860088 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 3 0 0 0
[pid=21228/tid=21334] ppid=21227 vsize=157564 CPUtime=16.33
/proc/21228/task/21334/stat : 21334 (pb_cplex) S 21227 21227 20807 0 -1 4202560 229 0 0 0 1618 15 0 0 15 0 8 0 146860088 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 1 0 0 0
[pid=21228/tid=21335] ppid=21227 vsize=157564 CPUtime=16.31
/proc/21228/task/21335/stat : 21335 (pb_cplex) S 21227 21227 20807 0 -1 4202560 746 0 0 0 1617 14 0 0 15 0 8 0 146860088 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 3 0 0 0
[pid=21228/tid=21336] ppid=21227 vsize=157564 CPUtime=16.31
/proc/21228/task/21336/stat : 21336 (pb_cplex) R 21227 21227 20807 0 -1 4202560 523 0 0 0 1616 15 0 0 15 0 8 0 146860088 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21228/tid=21337] ppid=21227 vsize=157564 CPUtime=16.3
/proc/21228/task/21337/stat : 21337 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1196 0 0 0 1614 16 0 0 15 0 8 0 146860088 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 2 0 0 0
[pid=21228/tid=21338] ppid=21227 vsize=157564 CPUtime=16.35
/proc/21228/task/21338/stat : 21338 (pb_cplex) S 21227 21227 20807 0 -1 4202560 812 0 0 0 1620 15 0 0 15 0 8 0 146860088 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 6 0 0 0
[pid=21228/tid=21339] ppid=21227 vsize=157564 CPUtime=16.39
/proc/21228/task/21339/stat : 21339 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1358 0 0 0 1623 16 0 0 15 0 8 0 146860088 161345536 17207 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1677.09
Current children cumulated vsize (KiB) 166264



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+491.147 s]
/proc/loadavg: 9.47 8.84 5.61 11/199 21411
/proc/meminfo: memFree=31246836/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=158588 CPUtime=1800.24
/proc/21228/stat : 21228 (pb_cplex) S 21227 21227 20807 0 -1 4202496 247722 0 0 0 178344 1680 0 0 16 0 8 0 146817443 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21228/statm: 39647 17380 1104 2985 0 36189 0
[pid=21228/tid=21333] ppid=21227 vsize=158588 CPUtime=31.84
/proc/21228/task/21333/stat : 21333 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1823 0 0 0 3155 29 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 138462792 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21334] ppid=21227 vsize=158588 CPUtime=31.48
/proc/21228/task/21334/stat : 21334 (pb_cplex) R 21227 21227 20807 0 -1 4202560 422 0 0 0 3120 28 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21228/tid=21335] ppid=21227 vsize=158588 CPUtime=31.45
/proc/21228/task/21335/stat : 21335 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1146 0 0 0 3121 24 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 135453913 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21336] ppid=21227 vsize=158588 CPUtime=31.67
/proc/21228/task/21336/stat : 21336 (pb_cplex) R 21227 21227 20807 0 -1 4202560 697 0 0 0 3138 29 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21228/tid=21337] ppid=21227 vsize=158588 CPUtime=31.7
/proc/21228/task/21337/stat : 21337 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1746 0 0 0 3136 34 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21228/tid=21338] ppid=21227 vsize=158588 CPUtime=31.67
/proc/21228/task/21338/stat : 21338 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1026 0 0 0 3138 29 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21339] ppid=21227 vsize=158588 CPUtime=31.69
/proc/21228/task/21339/stat : 21339 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1390 0 0 0 3139 30 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1800.24
Current children cumulated vsize (KiB) 167288

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+491.147 s]
/proc/loadavg: 9.47 8.84 5.61 11/199 21411
/proc/meminfo: memFree=31246836/32951124 swapFree=67111528/67111528
[pid=21227] ppid=21225 vsize=8700 CPUtime=0
/proc/21227/stat : 21227 (run) S 21225 21227 20807 0 -1 4202496 366 0 0 0 0 0 0 0 25 0 1 0 146817443 8908800 249 4089446400 4194304 4922060 140735241561312 18446744073709551615 271550356613 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/21227/statm: 2175 249 208 178 0 68 0
[pid=21228] ppid=21227 vsize=158588 CPUtime=1800.24
/proc/21228/stat : 21228 (pb_cplex) S 21227 21227 20807 0 -1 4202496 247722 0 0 0 178344 1680 0 0 16 0 8 0 146817443 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21228/statm: 39647 17380 1104 2985 0 36189 0
[pid=21228/tid=21333] ppid=21227 vsize=158588 CPUtime=31.84
/proc/21228/task/21333/stat : 21333 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1823 0 0 0 3155 29 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 138462792 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21334] ppid=21227 vsize=158588 CPUtime=31.48
/proc/21228/task/21334/stat : 21334 (pb_cplex) R 21227 21227 20807 0 -1 4202560 422 0 0 0 3120 28 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21228/tid=21335] ppid=21227 vsize=158588 CPUtime=31.45
/proc/21228/task/21335/stat : 21335 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1146 0 0 0 3121 24 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 135453913 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21336] ppid=21227 vsize=158588 CPUtime=31.67
/proc/21228/task/21336/stat : 21336 (pb_cplex) R 21227 21227 20807 0 -1 4202560 697 0 0 0 3138 29 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21228/tid=21337] ppid=21227 vsize=158588 CPUtime=31.7
/proc/21228/task/21337/stat : 21337 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1746 0 0 0 3136 34 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21228/tid=21338] ppid=21227 vsize=158588 CPUtime=31.67
/proc/21228/task/21338/stat : 21338 (pb_cplex) R 21227 21227 20807 0 -1 4202560 1026 0 0 0 3138 29 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21228/tid=21339] ppid=21227 vsize=158588 CPUtime=31.69
/proc/21228/task/21339/stat : 21339 (pb_cplex) S 21227 21227 20807 0 -1 4202560 1390 0 0 0 3139 30 0 0 15 0 8 0 146860088 162394112 17380 4089446400 134512640 146736320 4288666032 18446744073709551615 144580073 0 2 4096 16387 18446604471082982400 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1800.24
Current children cumulated vsize (KiB) 167288

Child status: 0
Real time (s): 491.167
CPU time (s): 1800.31
CPU user time (s): 1783.5
CPU system time (s): 16.8074
CPU usage (%): 366.537
Max. virtual memory (cumulated for all children) (KiB): 168312

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1783.5
system time used= 16.8074
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 248095
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 381479
involuntary context switches= 33005

runsolver used 0.972852 second user time and 1.93471 second system time

The end

Launcher Data 

Begin job on node133 at 2010-07-02 09:20:18
IDJOB=2697354
IDBENCH=78985
IDSOLVER=1210
FILE ID=node133/2697354-1278055218
PBS_JOBID= 11198915
Free space on /tmp= 74188 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900667195--soft-66-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697354-1278055218/watcher-2697354-1278055218 -o /tmp/evaluation-result-2697354-1278055218/solver-2697354-1278055218 -C 1800 -W 2000 -M 3800  run HOME/instance-2697354-1278055218.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= c3c12ca4f6a34dfd4eee1807e0cef251
RANDOM SEED=65852077

node133.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7296.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.821
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31336620 kB
Buffers:        217504 kB
Cached:        1155936 kB
SwapCached:          0 kB
Active:         684864 kB
Inactive:       779748 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31336620 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           33420 kB
Writeback:           0 kB
AnonPages:       91432 kB
Mapped:          18764 kB
Slab:            83076 kB
PageTables:       4892 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   282876 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74172 MiB
End job on node133 at 2010-07-02 09:28:30