Trace number 2697806

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.02 444.119

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900670675--soft-66-100-0.wbo
MD5SUM763c02f50955b910cbe48f7840773e4a
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark56.3114
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints906
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 44801
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 44800
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 671
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697806-1278054863.wbo.
0.00/0.00	c Instance file HOME/instance-2697806-1278054863.wbo
0.00/0.00	c File size is 87887 bytes.
0.00/0.17	c Highest Coefficient sum: 671
0.00/0.17	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.18	c ============================[ Problem Statistics ]=============================
0.00/0.18	c |                                                                             |
0.00/0.18	c |  Parsing time:          0.18         s                                      |
0.00/0.18	c |  Number Variables:      1137                                                |
0.00/0.18	c |  Number Constraints:    2708                                                |
0.00/0.18	c ===============================================================================
1800.02/444.11	c Remaining time 1799.82 sec.
1800.02/444.11	c 
1800.02/444.11	c *** INTERRUPTED ***
1800.02/444.11	s UNKNOWN
1800.02/444.11	c Total CPU time              : 1785.48 s
1800.02/444.11	c Memory used           : 146.82 MB
1800.02/444.11	c CPU time              : 1785.48 s
1800.02/444.11	c 
1800.02/444.11	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697806-1278054863/watcher-2697806-1278054863 -o /tmp/evaluation-result-2697806-1278054863/solver-2697806-1278054863 -C 1800 -W 2000 -M 3800 run HOME/instance-2697806-1278054863.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.15 0.05 0.01 2/177 22389
/proc/meminfo: memFree=31226732/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0

[startup+0.0922721 s]
/proc/loadavg: 0.15 0.05 0.01 2/177 22389
/proc/meminfo: memFree=31226732/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.100269 s]
/proc/loadavg: 0.15 0.05 0.01 2/177 22389
/proc/meminfo: memFree=31226732/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300254 s]
/proc/loadavg: 0.15 0.05 0.01 2/177 22389
/proc/meminfo: memFree=31226732/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.701176 s]
/proc/loadavg: 0.15 0.05 0.01 2/177 22389
/proc/meminfo: memFree=31226732/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50103 s]
/proc/loadavg: 0.15 0.05 0.01 2/179 22405
/proc/meminfo: memFree=31197296/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=133804 CPUtime=2.77
/proc/22390/stat : 22390 (pb_cplex) R 22389 22389 22164 0 -1 4202496 21394 0 0 0 269 8 0 0 18 0 8 0 146782453 137015296 10685 4089446400 134512640 146736320 4289645664 18446744073709551615 135982866 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/22390/statm: 33451 10685 1083 2985 0 29993 0
Current children cumulated CPU time (s) 2.77
Current children cumulated vsize (KiB) 142504

[startup+3.1007 s]
/proc/loadavg: 0.15 0.05 0.01 2/186 22419
/proc/meminfo: memFree=31186452/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=143020 CPUtime=6.51
/proc/22390/stat : 22390 (pb_cplex) R 22389 22389 22164 0 -1 4202496 26607 0 0 0 642 9 0 0 18 0 8 0 146782453 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 136007059 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/22390/statm: 35755 13150 1078 2985 0 32297 0
[pid=22390/tid=22413] ppid=22389 vsize=143020 CPUtime=0.22
/proc/22390/task/22413/stat : 22413 (pb_cplex) S 22389 22389 22164 0 -1 4202560 992 0 0 0 22 0 0 0 16 0 8 0 146782585 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22414] ppid=22389 vsize=143020 CPUtime=0.26
/proc/22390/task/22414/stat : 22414 (pb_cplex) S 22389 22389 22164 0 -1 4202560 653 0 0 0 26 0 0 0 18 0 8 0 146782585 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 6 0 0 0
[pid=22390/tid=22415] ppid=22389 vsize=143020 CPUtime=0.32
/proc/22390/task/22415/stat : 22415 (pb_cplex) R 22389 22389 22164 0 -1 4202560 658 0 0 0 32 0 0 0 18 0 8 0 146782585 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 136388106 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=22390/tid=22416] ppid=22389 vsize=143020 CPUtime=0.33
/proc/22390/task/22416/stat : 22416 (pb_cplex) R 22389 22389 22164 0 -1 4202560 42 0 0 0 33 0 0 0 16 0 8 0 146782585 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=22390/tid=22417] ppid=22389 vsize=143020 CPUtime=0.33
/proc/22390/task/22417/stat : 22417 (pb_cplex) R 22389 22389 22164 0 -1 4202560 56 0 0 0 33 0 0 0 16 0 8 0 146782585 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 136008157 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=22390/tid=22418] ppid=22389 vsize=143020 CPUtime=0.32
/proc/22390/task/22418/stat : 22418 (pb_cplex) R 22389 22389 22164 0 -1 4202560 921 0 0 0 32 0 0 0 16 0 8 0 146782585 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 136006331 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=22390/tid=22419] ppid=22389 vsize=143020 CPUtime=0.31
/proc/22390/task/22419/stat : 22419 (pb_cplex) S 22389 22389 22164 0 -1 4202560 44 0 0 0 31 0 0 0 18 0 8 0 146782585 146452480 13150 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 6.51
Current children cumulated vsize (KiB) 151720

[startup+6.32112 s]
/proc/loadavg: 0.78 0.18 0.05 3/192 22430
/proc/meminfo: memFree=31154168/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=144044 CPUtime=25.56
/proc/22390/stat : 22390 (pb_cplex) R 22389 22389 22164 0 -1 4202496 32028 0 0 0 2525 31 0 0 15 0 8 0 146782453 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/22390/statm: 36011 13427 1078 2985 0 32553 0
[pid=22390/tid=22413] ppid=22389 vsize=144044 CPUtime=2.44
/proc/22390/task/22413/stat : 22413 (pb_cplex) R 22389 22389 22164 0 -1 4202560 1221 0 0 0 242 2 0 0 15 0 8 0 146782585 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 135465109 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=22390/tid=22414] ppid=22389 vsize=144044 CPUtime=2.58
/proc/22390/task/22414/stat : 22414 (pb_cplex) R 22389 22389 22164 0 -1 4202560 1412 0 0 0 257 1 0 0 16 0 8 0 146782585 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=22390/tid=22415] ppid=22389 vsize=144044 CPUtime=2.81
/proc/22390/task/22415/stat : 22415 (pb_cplex) R 22389 22389 22164 0 -1 4202560 2246 0 0 0 278 3 0 0 16 0 8 0 146782585 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=22390/tid=22416] ppid=22389 vsize=144044 CPUtime=2.83
/proc/22390/task/22416/stat : 22416 (pb_cplex) R 22389 22389 22164 0 -1 4202560 341 0 0 0 281 2 0 0 15 0 8 0 146782585 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 135450684 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=22390/tid=22417] ppid=22389 vsize=144044 CPUtime=2.72
/proc/22390/task/22417/stat : 22417 (pb_cplex) R 22389 22389 22164 0 -1 4202560 882 0 0 0 268 4 0 0 15 0 8 0 146782585 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=22390/tid=22418] ppid=22389 vsize=144044 CPUtime=2.74
/proc/22390/task/22418/stat : 22418 (pb_cplex) R 22389 22389 22164 0 -1 4202560 2122 0 0 0 271 3 0 0 16 0 8 0 146782585 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=22390/tid=22419] ppid=22389 vsize=144044 CPUtime=2.62
/proc/22390/task/22419/stat : 22419 (pb_cplex) R 22389 22389 22164 0 -1 4202560 51 0 0 0 259 3 0 0 15 0 8 0 146782585 147501056 13427 4089446400 134512640 146736320 4289645664 18446744073709551615 136019061 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 25.56
Current children cumulated vsize (KiB) 152744

[startup+12.7011 s]
/proc/loadavg: 1.59 0.36 0.11 11/199 22473
/proc/meminfo: memFree=31102284/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=144044 CPUtime=53.68
/proc/22390/stat : 22390 (pb_cplex) S 22389 22389 22164 0 -1 4202496 37584 0 0 0 5304 64 0 0 16 0 8 0 146782453 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/22390/statm: 36011 13500 1098 2985 0 32553 0
[pid=22390/tid=22413] ppid=22389 vsize=144044 CPUtime=5.92
/proc/22390/task/22413/stat : 22413 (pb_cplex) S 22389 22389 22164 0 -1 4202560 1727 0 0 0 586 6 0 0 16 0 8 0 146782585 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22414] ppid=22389 vsize=144044 CPUtime=6
/proc/22390/task/22414/stat : 22414 (pb_cplex) S 22389 22389 22164 0 -1 4202560 1946 0 0 0 594 6 0 0 15 0 8 0 146782585 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 4 0 0 0
[pid=22390/tid=22415] ppid=22389 vsize=144044 CPUtime=6.39
/proc/22390/task/22415/stat : 22415 (pb_cplex) S 22389 22389 22164 0 -1 4202560 2724 0 0 0 633 6 0 0 15 0 8 0 146782585 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 4 0 0 0
[pid=22390/tid=22416] ppid=22389 vsize=144044 CPUtime=6.35
/proc/22390/task/22416/stat : 22416 (pb_cplex) S 22389 22389 22164 0 -1 4202560 476 0 0 0 629 6 0 0 15 0 8 0 146782585 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 2 0 0 0
[pid=22390/tid=22417] ppid=22389 vsize=144044 CPUtime=6.14
/proc/22390/task/22417/stat : 22417 (pb_cplex) S 22389 22389 22164 0 -1 4202560 1955 0 0 0 607 7 0 0 15 0 8 0 146782585 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 6 0 0 0
[pid=22390/tid=22418] ppid=22389 vsize=144044 CPUtime=6.24
/proc/22390/task/22418/stat : 22418 (pb_cplex) S 22389 22389 22164 0 -1 4202560 3116 0 0 0 618 6 0 0 15 0 8 0 146782585 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 0 0 0 0
[pid=22390/tid=22419] ppid=22389 vsize=144044 CPUtime=6.15
/proc/22390/task/22419/stat : 22419 (pb_cplex) R 22389 22389 22164 0 -1 4202560 57 0 0 0 608 7 0 0 15 0 8 0 146782585 147501056 13500 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 53.68
Current children cumulated vsize (KiB) 152744

[startup+25.5007 s]
/proc/loadavg: 3.41 0.82 0.26 20/199 22487
/proc/meminfo: memFree=31090348/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=139176 CPUtime=78.87
/proc/22390/stat : 22390 (pb_cplex) R 22389 22389 22164 0 -1 4202496 47706 0 0 0 7799 88 0 0 22 0 8 0 146782453 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 136007059 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/22390/statm: 34794 12184 1104 2985 0 31336 0
[pid=22390/tid=22481] ppid=22389 vsize=139176 CPUtime=1.31
/proc/22390/task/22481/stat : 22481 (pb_cplex) R 22389 22389 22164 0 -1 4202560 139 0 0 0 131 0 0 0 19 0 8 0 146784248 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 139044550 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=139176 CPUtime=1.19
/proc/22390/task/22482/stat : 22482 (pb_cplex) R 22389 22389 22164 0 -1 4202560 39 0 0 0 119 0 0 0 19 0 8 0 146784248 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 136007612 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=139176 CPUtime=0.97
/proc/22390/task/22483/stat : 22483 (pb_cplex) S 22389 22389 22164 0 -1 4202560 79 0 0 0 97 0 0 0 18 0 8 0 146784248 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 1 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=139176 CPUtime=1.28
/proc/22390/task/22484/stat : 22484 (pb_cplex) R 22389 22389 22164 0 -1 4202560 846 0 0 0 128 0 0 0 19 0 8 0 146784248 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 136008017 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=139176 CPUtime=1.26
/proc/22390/task/22485/stat : 22485 (pb_cplex) R 22389 22389 22164 0 -1 4202560 2233 0 0 0 126 0 0 0 19 0 8 0 146784248 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 136007647 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=139176 CPUtime=1.22
/proc/22390/task/22486/stat : 22486 (pb_cplex) R 22389 22389 22164 0 -1 4202560 751 0 0 0 121 1 0 0 19 0 8 0 146784248 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 137460830 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=139176 CPUtime=1.23
/proc/22390/task/22487/stat : 22487 (pb_cplex) R 22389 22389 22164 0 -1 4202560 439 0 0 0 123 0 0 0 18 0 8 0 146784248 142516224 12184 4089446400 134512640 146736320 4289645664 18446744073709551615 136008047 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 78.87
Current children cumulated vsize (KiB) 147876

[startup+51.1094 s]
/proc/loadavg: 5.86 1.59 0.53 11/199 22509
/proc/meminfo: memFree=31096548/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=142148 CPUtime=191.57
/proc/22390/stat : 22390 (pb_cplex) S 22389 22389 22164 0 -1 4202496 92703 0 0 0 18976 181 0 0 16 0 8 0 146782453 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/22390/statm: 35537 13250 1104 2985 0 32079 0
[pid=22390/tid=22481] ppid=22389 vsize=142148 CPUtime=14.82
/proc/22390/task/22481/stat : 22481 (pb_cplex) S 22389 22389 22164 0 -1 4202560 5973 0 0 0 1469 13 0 0 15 0 8 0 146784248 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 6 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=142148 CPUtime=14.56
/proc/22390/task/22482/stat : 22482 (pb_cplex) R 22389 22389 22164 0 -1 4202560 7465 0 0 0 1444 12 0 0 15 0 8 0 146784248 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=142148 CPUtime=14.03
/proc/22390/task/22483/stat : 22483 (pb_cplex) S 22389 22389 22164 0 -1 4202560 7668 0 0 0 1391 12 0 0 15 0 8 0 146784248 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 0 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=142148 CPUtime=15.89
/proc/22390/task/22484/stat : 22484 (pb_cplex) S 22389 22389 22164 0 -1 4202560 7207 0 0 0 1578 11 0 0 16 0 8 0 146784248 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 1 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=142148 CPUtime=14.64
/proc/22390/task/22485/stat : 22485 (pb_cplex) R 22389 22389 22164 0 -1 4202560 4437 0 0 0 1454 10 0 0 15 0 8 0 146784248 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=142148 CPUtime=16.82
/proc/22390/task/22486/stat : 22486 (pb_cplex) S 22389 22389 22164 0 -1 4202560 2681 0 0 0 1670 12 0 0 15 0 8 0 146784248 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=142148 CPUtime=16.06
/proc/22390/task/22487/stat : 22487 (pb_cplex) S 22389 22389 22164 0 -1 4202560 8888 0 0 0 1596 10 0 0 15 0 8 0 146784248 145559552 13250 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 191.57
Current children cumulated vsize (KiB) 150848

[startup+102.306 s]
/proc/loadavg: 7.67 2.77 0.99 11/199 22510
/proc/meminfo: memFree=31089228/32951124 swapFree=67111528/67111528

################
# More data... #
################

/proc/22390/statm: 36561 13775 1104 2985 0 33103 0
[pid=22390/tid=22481] ppid=22389 vsize=146244 CPUtime=71.32
/proc/22390/task/22481/stat : 22481 (pb_cplex) S 22389 22389 22164 0 -1 4202560 13648 0 0 0 7073 59 0 0 16 0 8 0 146784248 149753856 13775 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=146244 CPUtime=70.86
/proc/22390/task/22482/stat : 22482 (pb_cplex) S 22389 22389 22164 0 -1 4202560 17019 0 0 0 7027 59 0 0 16 0 8 0 146784248 149753856 13775 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 1 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=146244 CPUtime=70.3
/proc/22390/task/22483/stat : 22483 (pb_cplex) S 22389 22389 22164 0 -1 4202560 20928 0 0 0 6973 57 0 0 15 0 8 0 146784248 149753856 13775 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 3 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=146244 CPUtime=72.42
/proc/22390/task/22484/stat : 22484 (pb_cplex) S 22389 22389 22164 0 -1 4202560 20745 0 0 0 7185 57 0 0 15 0 8 0 146784248 149753856 13775 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 7 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=146244 CPUtime=71.01
/proc/22390/task/22485/stat : 22485 (pb_cplex) R 22389 22389 22164 0 -1 4202560 16299 0 0 0 7046 55 0 0 15 0 8 0 146784248 149753856 13775 4089446400 134512640 146736320 4289645664 18446744073709551615 136017481 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=146244 CPUtime=73.51
/proc/22390/task/22486/stat : 22486 (pb_cplex) S 22389 22389 22164 0 -1 4202560 14574 0 0 0 7293 58 0 0 15 0 8 0 146784248 149753856 13775 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 2 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=146244 CPUtime=72.6
/proc/22390/task/22487/stat : 22487 (pb_cplex) R 22389 22389 22164 0 -1 4202560 24924 0 0 0 7201 59 0 0 15 0 8 0 146784248 149753856 13775 4089446400 134512640 146736320 4289645664 18446744073709551615 137334161 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 647.18
Current children cumulated vsize (KiB) 154944

[startup+222.371 s]
/proc/loadavg: 9.08 4.93 1.99 11/199 22513
/proc/meminfo: memFree=31089656/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=148292 CPUtime=893.55
/proc/22390/stat : 22390 (pb_cplex) S 22389 22389 22164 0 -1 4202496 215632 0 0 0 88609 746 0 0 15 0 8 0 146782453 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/22390/statm: 37073 14482 1104 2985 0 33615 0
[pid=22390/tid=22481] ppid=22389 vsize=148292 CPUtime=101.77
/proc/22390/task/22481/stat : 22481 (pb_cplex) S 22389 22389 22164 0 -1 4202560 17503 0 0 0 10094 83 0 0 15 0 8 0 146784248 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=148292 CPUtime=101.39
/proc/22390/task/22482/stat : 22482 (pb_cplex) S 22389 22389 22164 0 -1 4202560 21304 0 0 0 10055 84 0 0 15 0 8 0 146784248 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=148292 CPUtime=100.82
/proc/22390/task/22483/stat : 22483 (pb_cplex) S 22389 22389 22164 0 -1 4202560 23729 0 0 0 9999 83 0 0 15 0 8 0 146784248 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=148292 CPUtime=102.79
/proc/22390/task/22484/stat : 22484 (pb_cplex) S 22389 22389 22164 0 -1 4202560 22511 0 0 0 10197 82 0 0 15 0 8 0 146784248 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=148292 CPUtime=101.43
/proc/22390/task/22485/stat : 22485 (pb_cplex) S 22389 22389 22164 0 -1 4202560 19737 0 0 0 10067 76 0 0 15 0 8 0 146784248 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 5 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=148292 CPUtime=104.13
/proc/22390/task/22486/stat : 22486 (pb_cplex) R 22389 22389 22164 0 -1 4202560 19460 0 0 0 10328 85 0 0 15 0 8 0 146784248 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=148292 CPUtime=103.31
/proc/22390/task/22487/stat : 22487 (pb_cplex) S 22389 22389 22164 0 -1 4202560 27727 0 0 0 10247 84 0 0 15 0 8 0 146784248 151851008 14482 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 893.55
Current children cumulated vsize (KiB) 156992

[startup+282.319 s]
/proc/loadavg: 9.43 5.79 2.47 11/199 22515
/proc/meminfo: memFree=31087944/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=150340 CPUtime=1137.61
/proc/22390/stat : 22390 (pb_cplex) R 22389 22389 22164 0 -1 4202496 233027 0 0 0 112821 940 0 0 15 0 8 0 146782453 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/22390/statm: 37585 14935 1104 2985 0 34127 0
[pid=22390/tid=22481] ppid=22389 vsize=150340 CPUtime=132.32
/proc/22390/task/22481/stat : 22481 (pb_cplex) R 22389 22389 22164 0 -1 4202560 20326 0 0 0 13123 109 0 0 15 0 8 0 146784248 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=150340 CPUtime=131.37
/proc/22390/task/22482/stat : 22482 (pb_cplex) R 22389 22389 22164 0 -1 4202560 23700 0 0 0 13030 107 0 0 15 0 8 0 146784248 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 135850298 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=150340 CPUtime=130.92
/proc/22390/task/22483/stat : 22483 (pb_cplex) R 22389 22389 22164 0 -1 4202560 26243 0 0 0 12985 107 0 0 15 0 8 0 146784248 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=150340 CPUtime=133.07
/proc/22390/task/22484/stat : 22484 (pb_cplex) R 22389 22389 22164 0 -1 4202560 25062 0 0 0 13201 106 0 0 15 0 8 0 146784248 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 139044580 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=150340 CPUtime=131.56
/proc/22390/task/22485/stat : 22485 (pb_cplex) R 22389 22389 22164 0 -1 4202560 20244 0 0 0 13057 99 0 0 15 0 8 0 146784248 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=150340 CPUtime=134.55
/proc/22390/task/22486/stat : 22486 (pb_cplex) R 22389 22389 22164 0 -1 4202560 22529 0 0 0 13344 111 0 0 15 0 8 0 146784248 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=150340 CPUtime=133.45
/proc/22390/task/22487/stat : 22487 (pb_cplex) R 22389 22389 22164 0 -1 4202560 28834 0 0 0 13239 106 0 0 15 0 8 0 146784248 153948160 14935 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1137.61
Current children cumulated vsize (KiB) 159040

[startup+342.301 s]
/proc/loadavg: 9.54 6.48 2.91 13/199 22517
/proc/meminfo: memFree=31087980/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=150340 CPUtime=1383.1
/proc/22390/stat : 22390 (pb_cplex) R 22389 22389 22164 0 -1 4202496 247900 0 0 0 137178 1132 0 0 16 0 8 0 146782453 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 136403109 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/22390/statm: 37585 14887 1104 2985 0 34127 0
[pid=22390/tid=22481] ppid=22389 vsize=150340 CPUtime=162.98
/proc/22390/task/22481/stat : 22481 (pb_cplex) R 22389 22389 22164 0 -1 4202560 23802 0 0 0 16165 133 0 0 15 0 8 0 146784248 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 135854527 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=150340 CPUtime=161.68
/proc/22390/task/22482/stat : 22482 (pb_cplex) R 22389 22389 22164 0 -1 4202560 24648 0 0 0 16037 131 0 0 15 0 8 0 146784248 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=150340 CPUtime=161.46
/proc/22390/task/22483/stat : 22483 (pb_cplex) R 22389 22389 22164 0 -1 4202560 27166 0 0 0 16017 129 0 0 15 0 8 0 146784248 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=150340 CPUtime=163.25
/proc/22390/task/22484/stat : 22484 (pb_cplex) R 22389 22389 22164 0 -1 4202560 27865 0 0 0 16195 130 0 0 15 0 8 0 146784248 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 135451622 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=150340 CPUtime=161.89
/proc/22390/task/22485/stat : 22485 (pb_cplex) R 22389 22389 22164 0 -1 4202560 20919 0 0 0 16068 121 0 0 15 0 8 0 146784248 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=150340 CPUtime=164.78
/proc/22390/task/22486/stat : 22486 (pb_cplex) R 22389 22389 22164 0 -1 4202560 23853 0 0 0 16343 135 0 0 15 0 8 0 146784248 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=150340 CPUtime=164
/proc/22390/task/22487/stat : 22487 (pb_cplex) R 22389 22389 22164 0 -1 4202560 31926 0 0 0 16267 133 0 0 15 0 8 0 146784248 153948160 14887 4089446400 134512640 146736320 4289645664 18446744073709551615 136017496 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1383.1
Current children cumulated vsize (KiB) 159040

[startup+402.301 s]
/proc/loadavg: 9.47 7.02 3.32 9/199 22518
/proc/meminfo: memFree=31086604/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=151364 CPUtime=1628.07
/proc/22390/stat : 22390 (pb_cplex) S 22389 22389 22164 0 -1 4202496 262625 0 0 0 161485 1322 0 0 16 0 8 0 146782453 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/22390/statm: 37841 15202 1106 2985 0 34383 0
[pid=22390/tid=22481] ppid=22389 vsize=151364 CPUtime=193.57
/proc/22390/task/22481/stat : 22481 (pb_cplex) S 22389 22389 22164 0 -1 4202560 24562 0 0 0 19199 158 0 0 15 0 8 0 146784248 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 4 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=151364 CPUtime=192.21
/proc/22390/task/22482/stat : 22482 (pb_cplex) S 22389 22389 22164 0 -1 4202560 25327 0 0 0 19066 155 0 0 16 0 8 0 146784248 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 2 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=151364 CPUtime=191.58
/proc/22390/task/22483/stat : 22483 (pb_cplex) S 22389 22389 22164 0 -1 4202560 28268 0 0 0 19005 153 0 0 16 0 8 0 146784248 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 6 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=151364 CPUtime=193.38
/proc/22390/task/22484/stat : 22484 (pb_cplex) S 22389 22389 22164 0 -1 4202560 31823 0 0 0 19185 153 0 0 15 0 8 0 146784248 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 1 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=151364 CPUtime=192.13
/proc/22390/task/22485/stat : 22485 (pb_cplex) S 22389 22389 22164 0 -1 4202560 23517 0 0 0 19070 143 0 0 15 0 8 0 146784248 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 3 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=151364 CPUtime=195.05
/proc/22390/task/22486/stat : 22486 (pb_cplex) S 22389 22389 22164 0 -1 4202560 25809 0 0 0 19344 161 0 0 15 0 8 0 146784248 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 0 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=151364 CPUtime=194.51
/proc/22390/task/22487/stat : 22487 (pb_cplex) R 22389 22389 22164 0 -1 4202560 32839 0 0 0 19294 157 0 0 16 0 8 0 146784248 154996736 15202 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1628.07
Current children cumulated vsize (KiB) 160064



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+444.101 s]
/proc/loadavg: 9.94 7.50 3.66 11/199 22519
/proc/meminfo: memFree=31086604/32951124 swapFree=67111528/67111528
[pid=22389] ppid=22387 vsize=8700 CPUtime=0
/proc/22389/stat : 22389 (run) S 22387 22389 22164 0 -1 4202496 363 0 0 0 0 0 0 0 21 0 1 0 146782453 8908800 248 4089446400 4194304 4922060 140736258084704 18446744073709551615 232335711365 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/22389/statm: 2175 248 208 178 0 68 0
[pid=22390] ppid=22389 vsize=150340 CPUtime=1800.02
/proc/22390/stat : 22390 (pb_cplex) S 22389 22389 22164 0 -1 4202496 269869 0 0 0 178548 1454 0 0 15 0 8 0 146782453 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/22390/statm: 37585 14999 1104 2985 0 34127 0
[pid=22390/tid=22481] ppid=22389 vsize=150340 CPUtime=215.06
/proc/22390/task/22481/stat : 22481 (pb_cplex) R 22389 22389 22164 0 -1 4202560 25644 0 0 0 21331 175 0 0 15 0 8 0 146784248 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=22390/tid=22482] ppid=22389 vsize=150340 CPUtime=213.57
/proc/22390/task/22482/stat : 22482 (pb_cplex) S 22389 22389 22164 0 -1 4202560 26132 0 0 0 21188 169 0 0 15 0 8 0 146784248 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 3 0 0 0
[pid=22390/tid=22483] ppid=22389 vsize=150340 CPUtime=212.71
/proc/22390/task/22483/stat : 22483 (pb_cplex) S 22389 22389 22164 0 -1 4202560 28901 0 0 0 21102 169 0 0 15 0 8 0 146784248 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 1 0 0 0
[pid=22390/tid=22484] ppid=22389 vsize=150340 CPUtime=214.67
/proc/22390/task/22484/stat : 22484 (pb_cplex) S 22389 22389 22164 0 -1 4202560 32777 0 0 0 21297 170 0 0 16 0 8 0 146784248 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 6 0 0 0
[pid=22390/tid=22485] ppid=22389 vsize=150340 CPUtime=213.31
/proc/22390/task/22485/stat : 22485 (pb_cplex) S 22389 22389 22164 0 -1 4202560 25187 0 0 0 21171 160 0 0 15 0 8 0 146784248 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 0 0 0 0
[pid=22390/tid=22486] ppid=22389 vsize=150340 CPUtime=216.28
/proc/22390/task/22486/stat : 22486 (pb_cplex) S 22389 22389 22164 0 -1 4202560 26550 0 0 0 21452 176 0 0 15 0 8 0 146784248 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 4 0 0 0
[pid=22390/tid=22487] ppid=22389 vsize=150340 CPUtime=215.87
/proc/22390/task/22487/stat : 22487 (pb_cplex) S 22389 22389 22164 0 -1 4202560 33276 0 0 0 21412 175 0 0 15 0 8 0 146784248 153948160 14999 4089446400 134512640 146736320 4289645664 18446744073709551615 144580073 0 2 4096 16387 18446604453950416000 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1800.02
Current children cumulated vsize (KiB) 159040

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 22389 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=0
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1800.02
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 444.119
CPU time (s): 1800.02
CPU user time (s): 1785.48
CPU system time (s): 14.54
CPU usage (%): 405.301
Max. virtual memory (cumulated for all children) (KiB): 164160

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.000999
system time used= 0
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 363
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3
involuntary context switches= 2

runsolver used 0.856869 second user time and 1.86672 second system time

The end

Launcher Data 

Begin job on node132 at 2010-07-02 09:14:23
IDJOB=2697806
IDBENCH=78880
IDSOLVER=1210
FILE ID=node132/2697806-1278054863
PBS_JOBID= 11198917
Free space on /tmp= 74188 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900670675--soft-66-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697806-1278054863/watcher-2697806-1278054863 -o /tmp/evaluation-result-2697806-1278054863/solver-2697806-1278054863 -C 1800 -W 2000 -M 3800  run HOME/instance-2697806-1278054863.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 763c02f50955b910cbe48f7840773e4a
RANDOM SEED=1244874401

node132.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.62
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.810
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      31227384 kB
Buffers:        218796 kB
Cached:        1293808 kB
SwapCached:          0 kB
Active:         584304 kB
Inactive:       991772 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      31227384 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16980 kB
Writeback:           0 kB
AnonPages:       63036 kB
Mapped:          14440 kB
Slab:            84580 kB
PageTables:       4244 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   181380 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74156 MiB
End job on node132 at 2010-07-02 09:21:49