Trace number 2697819

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.14 451.574

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900556231--soft-33-100-0.wbo
MD5SUM32bdb0ca9ee10d9e7c56deff56e319ba
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark188.053
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints1777
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 90281
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 90280
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 723
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697819-1278054864.wbo.
0.00/0.00	c Instance file HOME/instance-2697819-1278054864.wbo
0.00/0.00	c File size is 92211 bytes.
0.00/0.63	c Highest Coefficient sum: 723
0.00/0.63	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.63	c ============================[ Problem Statistics ]=============================
0.00/0.63	c |                                                                             |
0.00/0.63	c |  Parsing time:          0.63         s                                      |
0.00/0.63	c |  Number Variables:      2008                                                |
0.00/0.63	c |  Number Constraints:    2708                                                |
0.00/0.63	c ===============================================================================

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697819-1278054864/watcher-2697819-1278054864 -o /tmp/evaluation-result-2697819-1278054864/solver-2697819-1278054864 -C 1800 -W 2000 -M 3800 run HOME/instance-2697819-1278054864.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 21122
/proc/meminfo: memFree=31396740/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0

[startup+0.0726349 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 21122
/proc/meminfo: memFree=31396740/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.10063 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 21122
/proc/meminfo: memFree=31396740/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300598 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 21122
/proc/meminfo: memFree=31396740/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.700456 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 21122
/proc/meminfo: memFree=31396740/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50037 s]
/proc/loadavg: 0.00 0.00 0.00 2/175 21132
/proc/meminfo: memFree=31379184/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=141232 CPUtime=3.33
/proc/21123/stat : 21123 (pb_cplex) S 21122 21122 20925 0 -1 4202496 18082 0 0 0 328 5 0 0 16 0 8 0 146783340 144621568 12134 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21123/statm: 35308 12134 1095 2985 0 31850 0
Current children cumulated CPU time (s) 3.33
Current children cumulated vsize (KiB) 149932

[startup+3.10103 s]
/proc/loadavg: 0.00 0.00 0.00 2/175 21139
/proc/meminfo: memFree=31370296/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=143988 CPUtime=6.45
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 31232 0 0 0 636 9 0 0 18 0 8 0 146783340 147443712 13213 4089446400 134512640 146736320 4291304160 18446744073709551615 139044507 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21123/statm: 35997 13213 1094 2985 0 32539 0
Current children cumulated CPU time (s) 6.45
Current children cumulated vsize (KiB) 152688

[startup+6.33045 s]
/proc/loadavg: 0.08 0.02 0.01 11/188 21185
/proc/meminfo: memFree=31283144/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=151156 CPUtime=24.05
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 41735 0 0 0 2378 27 0 0 15 0 8 0 146783340 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21123/statm: 37789 14834 1079 2985 0 34331 0
[pid=21123/tid=21154] ppid=21122 vsize=151156 CPUtime=2.49
/proc/21123/task/21154/stat : 21154 (pb_cplex) R 21122 21122 20925 0 -1 4202560 1282 0 0 0 248 1 0 0 15 0 8 0 146783563 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 135408930 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21123/tid=21155] ppid=21122 vsize=151156 CPUtime=2.44
/proc/21123/task/21155/stat : 21155 (pb_cplex) R 21122 21122 20925 0 -1 4202560 53 0 0 0 243 1 0 0 15 0 8 0 146783563 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21123/tid=21156] ppid=21122 vsize=151156 CPUtime=2.19
/proc/21123/task/21156/stat : 21156 (pb_cplex) R 21122 21122 20925 0 -1 4202560 2453 0 0 0 217 2 0 0 15 0 8 0 146783563 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 137084981 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21123/tid=21157] ppid=21122 vsize=151156 CPUtime=2.4
/proc/21123/task/21157/stat : 21157 (pb_cplex) R 21122 21122 20925 0 -1 4202560 2040 0 0 0 238 2 0 0 16 0 8 0 146783563 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21123/tid=21158] ppid=21122 vsize=151156 CPUtime=2.15
/proc/21123/task/21158/stat : 21158 (pb_cplex) R 21122 21122 20925 0 -1 4202560 822 0 0 0 214 1 0 0 16 0 8 0 146783563 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 135854475 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21159] ppid=21122 vsize=151156 CPUtime=2.37
/proc/21123/task/21159/stat : 21159 (pb_cplex) S 21122 21122 20925 0 -1 4202560 4647 0 0 0 235 2 0 0 15 0 8 0 146783563 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 2 0 0 0
[pid=21123/tid=21160] ppid=21122 vsize=151156 CPUtime=2.6
/proc/21123/task/21160/stat : 21160 (pb_cplex) R 21122 21122 20925 0 -1 4202560 1160 0 0 0 258 2 0 0 15 0 8 0 146783563 154783744 14834 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.05
Current children cumulated vsize (KiB) 159856

[startup+12.7083 s]
/proc/loadavg: 0.23 0.05 0.02 11/195 21206
/proc/meminfo: memFree=31270280/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=152180 CPUtime=48.58
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 46379 0 0 0 4799 59 0 0 16 0 8 0 146783340 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21123/statm: 38045 15137 1079 2985 0 34587 0
[pid=21123/tid=21154] ppid=21122 vsize=152180 CPUtime=5.48
/proc/21123/task/21154/stat : 21154 (pb_cplex) R 21122 21122 20925 0 -1 4202560 1397 0 0 0 543 5 0 0 15 0 8 0 146783563 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21123/tid=21155] ppid=21122 vsize=152180 CPUtime=5.45
/proc/21123/task/21155/stat : 21155 (pb_cplex) R 21122 21122 20925 0 -1 4202560 60 0 0 0 541 4 0 0 15 0 8 0 146783563 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21156] ppid=21122 vsize=152180 CPUtime=5.19
/proc/21123/task/21156/stat : 21156 (pb_cplex) R 21122 21122 20925 0 -1 4202560 2811 0 0 0 512 7 0 0 15 0 8 0 146783563 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21123/tid=21157] ppid=21122 vsize=152180 CPUtime=5.39
/proc/21123/task/21157/stat : 21157 (pb_cplex) S 21122 21122 20925 0 -1 4202560 2906 0 0 0 532 7 0 0 15 0 8 0 146783563 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 1 0 0 0
[pid=21123/tid=21158] ppid=21122 vsize=152180 CPUtime=5.33
/proc/21123/task/21158/stat : 21158 (pb_cplex) S 21122 21122 20925 0 -1 4202560 889 0 0 0 527 6 0 0 15 0 8 0 146783563 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 0 0 0 0
[pid=21123/tid=21159] ppid=21122 vsize=152180 CPUtime=5.34
/proc/21123/task/21159/stat : 21159 (pb_cplex) S 21122 21122 20925 0 -1 4202560 7401 0 0 0 528 6 0 0 15 0 8 0 146783563 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 1 0 0 0
[pid=21123/tid=21160] ppid=21122 vsize=152180 CPUtime=5.66
/proc/21123/task/21160/stat : 21160 (pb_cplex) S 21122 21122 20925 0 -1 4202560 1195 0 0 0 561 5 0 0 16 0 8 0 146783563 155832320 15137 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 48.58
Current children cumulated vsize (KiB) 160880

[startup+25.5011 s]
/proc/loadavg: 3.62 0.80 0.26 18/195 21206
/proc/meminfo: memFree=31234940/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=172700 CPUtime=83.42
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 71739 0 0 0 8258 84 0 0 24 0 8 0 146783340 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 136008245 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21123/statm: 43175 19669 1092 2985 0 39717 0
[pid=21123/tid=21154] ppid=21122 vsize=172700 CPUtime=9.22
/proc/21123/task/21154/stat : 21154 (pb_cplex) R 21122 21122 20925 0 -1 4202560 6206 0 0 0 915 7 0 0 19 0 8 0 146783563 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 136403114 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21123/tid=21155] ppid=21122 vsize=172700 CPUtime=10.84
/proc/21123/task/21155/stat : 21155 (pb_cplex) R 21122 21122 20925 0 -1 4202560 3495 0 0 0 1076 8 0 0 25 0 8 0 146783563 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 139044557 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21123/tid=21156] ppid=21122 vsize=172700 CPUtime=9.2
/proc/21123/task/21156/stat : 21156 (pb_cplex) R 21122 21122 20925 0 -1 4202560 7885 0 0 0 910 10 0 0 19 0 8 0 146783563 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 135196708 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21123/tid=21157] ppid=21122 vsize=172700 CPUtime=9.69
/proc/21123/task/21157/stat : 21157 (pb_cplex) R 21122 21122 20925 0 -1 4202560 3166 0 0 0 958 11 0 0 20 0 8 0 146783563 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 136007605 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21123/tid=21158] ppid=21122 vsize=172700 CPUtime=9.11
/proc/21123/task/21158/stat : 21158 (pb_cplex) R 21122 21122 20925 0 -1 4202560 3408 0 0 0 902 9 0 0 19 0 8 0 146783563 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 136222369 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21159] ppid=21122 vsize=172700 CPUtime=9.59
/proc/21123/task/21159/stat : 21159 (pb_cplex) R 21122 21122 20925 0 -1 4202560 10856 0 0 0 949 10 0 0 20 0 8 0 146783563 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 144714257 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21123/tid=21160] ppid=21122 vsize=172700 CPUtime=9.67
/proc/21123/task/21160/stat : 21160 (pb_cplex) R 21122 21122 20925 0 -1 4202560 4755 0 0 0 959 8 0 0 19 0 8 0 146783563 176844800 19669 4089446400 134512640 146736320 4291304160 18446744073709551615 135455257 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 83.42
Current children cumulated vsize (KiB) 181400

[startup+51.1125 s]
/proc/loadavg: 5.50 1.46 0.49 6/195 21242
/proc/meminfo: memFree=31281316/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=135792 CPUtime=171.01
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 133333 0 0 0 16955 146 0 0 15 0 8 0 146783340 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 135231478 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21123/statm: 33948 12306 1089 2985 0 30490 0
[pid=21123/tid=21221] ppid=21122 vsize=135792 CPUtime=8.61
/proc/21123/task/21221/stat : 21221 (pb_cplex) R 21122 21122 20925 0 -1 4202560 11316 0 0 0 855 6 0 0 15 0 8 0 146786536 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=135792 CPUtime=7.6
/proc/21123/task/21222/stat : 21222 (pb_cplex) R 21122 21122 20925 0 -1 4202560 9831 0 0 0 754 6 0 0 15 0 8 0 146786536 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=135792 CPUtime=7.55
/proc/21123/task/21223/stat : 21223 (pb_cplex) R 21122 21122 20925 0 -1 4202560 5888 0 0 0 751 4 0 0 15 0 8 0 146786536 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=135792 CPUtime=9.18
/proc/21123/task/21224/stat : 21224 (pb_cplex) S 21122 21122 20925 0 -1 4202560 8960 0 0 0 911 7 0 0 15 0 8 0 146786536 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 3 0 0 0
[pid=21123/tid=21225] ppid=21122 vsize=135792 CPUtime=8.79
/proc/21123/task/21225/stat : 21225 (pb_cplex) S 21122 21122 20925 0 -1 4202560 6308 0 0 0 874 5 0 0 15 0 8 0 146786536 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 3 0 0 0
[pid=21123/tid=21226] ppid=21122 vsize=135792 CPUtime=9.61
/proc/21123/task/21226/stat : 21226 (pb_cplex) S 21122 21122 20925 0 -1 4202560 4255 0 0 0 954 7 0 0 15 0 8 0 146786536 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 3 0 0 0
[pid=21123/tid=21227] ppid=21122 vsize=135792 CPUtime=7.68
/proc/21123/task/21227/stat : 21227 (pb_cplex) S 21122 21122 20925 0 -1 4202560 4296 0 0 0 762 6 0 0 15 0 8 0 146786536 139051008 12306 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 171.01
Current children cumulated vsize (KiB) 144492

[startup+102.308 s]
/proc/loadavg: 7.63 2.68 0.96 8/195 21243
/proc/meminfo: memFree=31277304/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=140628 CPUtime=378.48
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 192014 0 0 0 37542 306 0 0 16 0 8 0 146783340 144003072 13268 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21123/statm: 35157 13268 1107 2985 0 31699 0
[pid=21123/tid=21221] ppid=21122 vsize=140628 CPUtime=34.24
/proc/21123/task/21221/stat : 21221 (pb_cplex) R 21122 21122 20925 0 -1 4202560 22842 0 0 0 3398 26 0 0 15 0 8 0 146786536 144003072 13268 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=140628 CPUtime=33.43
/proc/21123/task/21222/stat : 21222 (pb_cplex) R 21122 21122 20925 0 -1 4202560 16965 0 0 0 3316 27 0 0 15 0 8 0 146786536 144003072 13268 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=140628 CPUtime=33.39
/proc/21123/task/21223/stat : 21223 (pb_cplex) R 21122 21122 20925 0 -1 4202560 11950 0 0 0 3315 24 0 0 15 0 8 0 146786536 144003072 13268 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=140628 CPUtime=35.18
/proc/21123/task/21224/stat : 21224 (pb_cplex) S 21122 21122 20925 0 -1 4202560 17203 0 0 0 3491 27 0 0 15 0 8 0 146786536 144003072 13268 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/21123/statm: 35113 13319 1107 2985 0 31655 0
[pid=21123/tid=21221] ppid=21122 vsize=140452 CPUtime=65.27
/proc/21123/task/21221/stat : 21221 (pb_cplex) R 21122 21122 20925 0 -1 4202560 30944 0 0 0 6474 53 0 0 15 0 8 0 146786536 143822848 13319 4089446400 134512640 146736320 4291304160 18446744073709551615 135231516 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=140452 CPUtime=64.36
/proc/21123/task/21222/stat : 21222 (pb_cplex) S 21122 21122 20925 0 -1 4202560 23406 0 0 0 6387 49 0 0 15 0 8 0 146786536 143822848 13319 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 1 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=140452 CPUtime=64.29
/proc/21123/task/21223/stat : 21223 (pb_cplex) R 21122 21122 20925 0 -1 4202560 15798 0 0 0 6381 48 0 0 15 0 8 0 146786536 143822848 13319 4089446400 134512640 146736320 4291304160 18446744073709551615 136017757 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=140452 CPUtime=65.86
/proc/21123/task/21224/stat : 21224 (pb_cplex) R 21122 21122 20925 0 -1 4202560 20118 0 0 0 6536 50 0 0 15 0 8 0 146786536 143822848 13319 4089446400 134512640 146736320 4291304160 18446744073709551615 135526130 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21123/tid=21225] ppid=21122 vsize=140452 CPUtime=65.37
/proc/21123/task/21225/stat : 21225 (pb_cplex) R 21122 21122 20925 0 -1 4202560 17719 0 0 0 6484 53 0 0 15 0 8 0 146786536 143822848 13319 4089446400 134512640 146736320 4291304160 18446744073709551615 136475798 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21123/tid=21226] ppid=21122 vsize=140452 CPUtime=65.72
/proc/21123/task/21226/stat : 21226 (pb_cplex) R 21122 21122 20925 0 -1 4202560 13224 0 0 0 6522 50 0 0 15 0 8 0 146786536 143822848 13319 4089446400 134512640 146736320 4291304160 18446744073709551615 135526785 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21123/tid=21227] ppid=21122 vsize=140452 CPUtime=64.21
/proc/21123/task/21227/stat : 21227 (pb_cplex) R 21122 21122 20925 0 -1 4202560 11829 0 0 0 6369 52 0 0 15 0 8 0 146786536 143822848 13319 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 627.4
Current children cumulated vsize (KiB) 149152

[startup+222.303 s]
/proc/loadavg: 9.45 4.92 1.98 9/195 21246
/proc/meminfo: memFree=31274860/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=144180 CPUtime=867.86
/proc/21123/stat : 21123 (pb_cplex) S 21122 21122 20925 0 -1 4202496 251463 0 0 0 86077 709 0 0 16 0 8 0 146783340 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21123/statm: 36045 14028 1107 2985 0 32587 0
[pid=21123/tid=21221] ppid=21122 vsize=144180 CPUtime=95.02
/proc/21123/task/21221/stat : 21221 (pb_cplex) S 21122 21122 20925 0 -1 4202560 33192 0 0 0 9426 76 0 0 15 0 8 0 146786536 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 0 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=144180 CPUtime=94.18
/proc/21123/task/21222/stat : 21222 (pb_cplex) R 21122 21122 20925 0 -1 4202560 25887 0 0 0 9343 75 0 0 15 0 8 0 146786536 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 136403117 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=144180 CPUtime=94.15
/proc/21123/task/21223/stat : 21223 (pb_cplex) R 21122 21122 20925 0 -1 4202560 20100 0 0 0 9339 76 0 0 15 0 8 0 146786536 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=144180 CPUtime=95.43
/proc/21123/task/21224/stat : 21224 (pb_cplex) R 21122 21122 20925 0 -1 4202560 22427 0 0 0 9467 76 0 0 15 0 8 0 146786536 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21123/tid=21225] ppid=21122 vsize=144180 CPUtime=95.04
/proc/21123/task/21225/stat : 21225 (pb_cplex) R 21122 21122 20925 0 -1 4202560 22023 0 0 0 9427 77 0 0 15 0 8 0 146786536 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 137071142 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21123/tid=21226] ppid=21122 vsize=144180 CPUtime=95.45
/proc/21123/task/21226/stat : 21226 (pb_cplex) R 21122 21122 20925 0 -1 4202560 15571 0 0 0 9469 76 0 0 15 0 8 0 146786536 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21227] ppid=21122 vsize=144180 CPUtime=94.27
/proc/21123/task/21227/stat : 21227 (pb_cplex) R 21122 21122 20925 0 -1 4202560 16363 0 0 0 9349 78 0 0 15 0 8 0 146786536 147640320 14028 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 867.86
Current children cumulated vsize (KiB) 152880

[startup+282.33 s]
/proc/loadavg: 9.55 5.79 2.46 9/184 21260
/proc/meminfo: memFree=31323172/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=144180 CPUtime=1104.95
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 266651 0 0 0 109572 923 0 0 15 0 8 0 146783340 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21123/statm: 36045 14045 1107 2985 0 32587 0
[pid=21123/tid=21221] ppid=21122 vsize=144180 CPUtime=124.2
/proc/21123/task/21221/stat : 21221 (pb_cplex) R 21122 21122 20925 0 -1 4202560 35136 0 0 0 12320 100 0 0 15 0 8 0 146786536 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 137343484 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=144180 CPUtime=123.37
/proc/21123/task/21222/stat : 21222 (pb_cplex) R 21122 21122 20925 0 -1 4202560 29360 0 0 0 12236 101 0 0 15 0 8 0 146786536 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 136400469 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=144180 CPUtime=123.58
/proc/21123/task/21223/stat : 21223 (pb_cplex) R 21122 21122 20925 0 -1 4202560 21506 0 0 0 12255 103 0 0 15 0 8 0 146786536 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=144180 CPUtime=125.15
/proc/21123/task/21224/stat : 21224 (pb_cplex) R 21122 21122 20925 0 -1 4202560 23954 0 0 0 12412 103 0 0 15 0 8 0 146786536 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21123/tid=21225] ppid=21122 vsize=144180 CPUtime=124.34
/proc/21123/task/21225/stat : 21225 (pb_cplex) R 21122 21122 20925 0 -1 4202560 23528 0 0 0 12332 102 0 0 15 0 8 0 146786536 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21226] ppid=21122 vsize=144180 CPUtime=124.87
/proc/21123/task/21226/stat : 21226 (pb_cplex) R 21122 21122 20925 0 -1 4202560 18332 0 0 0 12383 104 0 0 15 0 8 0 146786536 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21123/tid=21227] ppid=21122 vsize=144180 CPUtime=123.72
/proc/21123/task/21227/stat : 21227 (pb_cplex) R 21122 21122 20925 0 -1 4202560 17161 0 0 0 12265 107 0 0 16 0 8 0 146786536 147640320 14045 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1104.95
Current children cumulated vsize (KiB) 152880

[startup+342.326 s]
/proc/loadavg: 10.18 6.58 2.93 9/195 21326
/proc/meminfo: memFree=31271868/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=144180 CPUtime=1383
/proc/21123/stat : 21123 (pb_cplex) R 21122 21122 20925 0 -1 4202496 286676 0 0 0 137123 1177 0 0 16 0 8 0 146783340 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21123/statm: 36045 14058 1107 2985 0 32587 0
[pid=21123/tid=21221] ppid=21122 vsize=144180 CPUtime=158.63
/proc/21123/task/21221/stat : 21221 (pb_cplex) S 21122 21122 20925 0 -1 4202560 38254 0 0 0 15734 129 0 0 15 0 8 0 146786536 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 5 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=144180 CPUtime=157.94
/proc/21123/task/21222/stat : 21222 (pb_cplex) S 21122 21122 20925 0 -1 4202560 32321 0 0 0 15661 133 0 0 15 0 8 0 146786536 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 1 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=144180 CPUtime=158.12
/proc/21123/task/21223/stat : 21223 (pb_cplex) S 21122 21122 20925 0 -1 4202560 24649 0 0 0 15674 138 0 0 15 0 8 0 146786536 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 3 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=144180 CPUtime=159.59
/proc/21123/task/21224/stat : 21224 (pb_cplex) S 21122 21122 20925 0 -1 4202560 27834 0 0 0 15824 135 0 0 15 0 8 0 146786536 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 0 0 0 0
[pid=21123/tid=21225] ppid=21122 vsize=144180 CPUtime=159.08
/proc/21123/task/21225/stat : 21225 (pb_cplex) S 21122 21122 20925 0 -1 4202560 23928 0 0 0 15776 132 0 0 15 0 8 0 146786536 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 6 0 0 0
[pid=21123/tid=21226] ppid=21122 vsize=144180 CPUtime=159.21
/proc/21123/task/21226/stat : 21226 (pb_cplex) S 21122 21122 20925 0 -1 4202560 22010 0 0 0 15782 139 0 0 15 0 8 0 146786536 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 5 0 0 0
[pid=21123/tid=21227] ppid=21122 vsize=144180 CPUtime=158.71
/proc/21123/task/21227/stat : 21227 (pb_cplex) S 21122 21122 20925 0 -1 4202560 18855 0 0 0 15732 139 0 0 15 0 8 0 146786536 147640320 14058 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1383
Current children cumulated vsize (KiB) 152880

[startup+402.327 s]
/proc/loadavg: 9.64 7.08 3.33 8/195 21328
/proc/meminfo: memFree=31271224/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=144180 CPUtime=1607.83
/proc/21123/stat : 21123 (pb_cplex) S 21122 21122 20925 0 -1 4202496 303072 0 0 0 159405 1378 0 0 16 0 8 0 146783340 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21123/statm: 36045 13987 1107 2985 0 32587 0
[pid=21123/tid=21221] ppid=21122 vsize=144180 CPUtime=186.41
/proc/21123/task/21221/stat : 21221 (pb_cplex) S 21122 21122 20925 0 -1 4202560 39942 0 0 0 18486 155 0 0 15 0 8 0 146786536 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 0 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=144180 CPUtime=185.81
/proc/21123/task/21222/stat : 21222 (pb_cplex) S 21122 21122 20925 0 -1 4202560 35304 0 0 0 18421 160 0 0 16 0 8 0 146786536 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 1 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=144180 CPUtime=185.91
/proc/21123/task/21223/stat : 21223 (pb_cplex) S 21122 21122 20925 0 -1 4202560 26825 0 0 0 18426 165 0 0 15 0 8 0 146786536 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 3 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=144180 CPUtime=187.62
/proc/21123/task/21224/stat : 21224 (pb_cplex) S 21122 21122 20925 0 -1 4202560 29864 0 0 0 18606 156 0 0 15 0 8 0 146786536 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 7 0 0 0
[pid=21123/tid=21225] ppid=21122 vsize=144180 CPUtime=186.85
/proc/21123/task/21225/stat : 21225 (pb_cplex) S 21122 21122 20925 0 -1 4202560 25834 0 0 0 18530 155 0 0 15 0 8 0 146786536 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 5 0 0 0
[pid=21123/tid=21226] ppid=21122 vsize=144180 CPUtime=186.95
/proc/21123/task/21226/stat : 21226 (pb_cplex) R 21122 21122 20925 0 -1 4202560 22660 0 0 0 18531 164 0 0 15 0 8 0 146786536 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21123/tid=21227] ppid=21122 vsize=144180 CPUtime=186.7
/proc/21123/task/21227/stat : 21227 (pb_cplex) S 21122 21122 20925 0 -1 4202560 21044 0 0 0 18504 166 0 0 15 0 8 0 146786536 147640320 13987 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1607.83
Current children cumulated vsize (KiB) 152880



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+451.501 s]
/proc/loadavg: 9.82 7.50 3.67 7/195 21330
/proc/meminfo: memFree=31270452/32950928 swapFree=67111528/67111528
[pid=21122] ppid=21120 vsize=8700 CPUtime=0
/proc/21122/stat : 21122 (run) S 21120 21122 20925 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 146783340 8908800 249 4089446400 4194304 4922060 140733745005648 18446744073709551615 235594685573 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21122/statm: 2175 249 208 178 0 68 0
[pid=21123] ppid=21122 vsize=143632 CPUtime=1800.14
/proc/21123/stat : 21123 (pb_cplex) S 21122 21122 20925 0 -1 4202496 319903 0 0 0 178455 1559 0 0 16 0 8 0 146783340 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21123/statm: 35908 13898 1107 2985 0 32450 0
[pid=21123/tid=21221] ppid=21122 vsize=143632 CPUtime=210.23
/proc/21123/task/21221/stat : 21221 (pb_cplex) R 21122 21122 20925 0 -1 4202560 41287 0 0 0 20848 175 0 0 15 0 8 0 146786536 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21123/tid=21222] ppid=21122 vsize=143632 CPUtime=209.61
/proc/21123/task/21222/stat : 21222 (pb_cplex) S 21122 21122 20925 0 -1 4202560 37401 0 0 0 20780 181 0 0 15 0 8 0 146786536 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 4 0 0 0
[pid=21123/tid=21223] ppid=21122 vsize=143632 CPUtime=209.63
/proc/21123/task/21223/stat : 21223 (pb_cplex) S 21122 21122 20925 0 -1 4202560 28441 0 0 0 20772 191 0 0 15 0 8 0 146786536 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 5 0 0 0
[pid=21123/tid=21224] ppid=21122 vsize=143632 CPUtime=211.48
/proc/21123/task/21224/stat : 21224 (pb_cplex) S 21122 21122 20925 0 -1 4202560 33134 0 0 0 20971 177 0 0 15 0 8 0 146786536 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144580073 0 2 4096 16387 18446604471104116928 0 0 -1 7 0 0 0
[pid=21123/tid=21225] ppid=21122 vsize=143632 CPUtime=210.87
/proc/21123/task/21225/stat : 21225 (pb_cplex) R 21122 21122 20925 0 -1 4202560 28015 0 0 0 20910 177 0 0 15 0 8 0 146786536 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21123/tid=21226] ppid=21122 vsize=143632 CPUtime=210.9
/proc/21123/task/21226/stat : 21226 (pb_cplex) R 21122 21122 20925 0 -1 4202560 23640 0 0 0 20903 187 0 0 15 0 8 0 146786536 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21123/tid=21227] ppid=21122 vsize=143632 CPUtime=210.48
/proc/21123/task/21227/stat : 21227 (pb_cplex) R 21122 21122 20925 0 -1 4202560 23113 0 0 0 20858 190 0 0 15 0 8 0 146786536 147079168 13898 4089446400 134512640 146736320 4291304160 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1800.14
Current children cumulated vsize (KiB) 152332

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21122 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=0
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1800.14
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 451.574
CPU time (s): 1800.14
CPU user time (s): 1784.55
CPU system time (s): 15.59
CPU usage (%): 398.637
Max. virtual memory (cumulated for all children) (KiB): 185952

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.000999
system time used= 0
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 365
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 2

runsolver used 0.874867 second user time and 1.83872 second system time

The end

Launcher Data 

Begin job on node120 at 2010-07-02 09:14:24
IDJOB=2697819
IDBENCH=78893
IDSOLVER=1210
FILE ID=node120/2697819-1278054864
PBS_JOBID= 11198941
Free space on /tmp= 74208 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900556231--soft-33-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697819-1278054864/watcher-2697819-1278054864 -o /tmp/evaluation-result-2697819-1278054864/solver-2697819-1278054864 -C 1800 -W 2000 -M 3800  run HOME/instance-2697819-1278054864.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 32bdb0ca9ee10d9e7c56deff56e319ba
RANDOM SEED=893539541

node120.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.820
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31397392 kB
Buffers:        220284 kB
Cached:        1124672 kB
SwapCached:          0 kB
Active:         645932 kB
Inactive:       762164 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31397392 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16964 kB
Writeback:           0 kB
AnonPages:       63124 kB
Mapped:          14624 kB
Slab:            82492 kB
PageTables:       4496 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   185052 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74176 MiB
End job on node120 at 2010-07-02 09:21:58