Trace number 2697374

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.66 502.216

General information on the benchmark

Name/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/
tsp/normalized-t3002.11tsp11.1900553647--soft-0-100-0.wbo
MD5SUM1693c6ddcdda9c3a56f7a1957868c0a9
Bench CategorySOFT-SMALLINT-LIN (only soft constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark118.75
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints2707
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 134314
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 134313
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 715
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697374-1278055498.wbo.
0.00/0.00	c Instance file HOME/instance-2697374-1278055498.wbo
0.00/0.00	c File size is 96739 bytes.
1.35/3.49	c Highest Coefficient sum: 715
1.35/3.49	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
1.35/3.49	c ============================[ Problem Statistics ]=============================
1.35/3.49	c |                                                                             |
1.35/3.49	c |  Parsing time:          1.45         s                                      |
1.35/3.49	c |  Number Variables:      2938                                                |
1.35/3.49	c |  Number Constraints:    2708                                                |
1.35/3.49	c ===============================================================================
1800.58/502.21	c Remaining time 1798.55 sec.
1800.58/502.21	c 
1800.58/502.21	c *** INTERRUPTED ***
1800.58/502.21	s UNKNOWN
1800.58/502.21	c Total CPU time              : 1784.88 s
1800.58/502.21	c Memory used           : 140.55 MB
1800.58/502.21	c CPU time              : 1784.88 s
1800.58/502.21	c 
1800.58/502.21	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697374-1278055498/watcher-2697374-1278055498 -o /tmp/evaluation-result-2697374-1278055498/solver-2697374-1278055498 -C 1800 -W 2000 -M 3800 run HOME/instance-2697374-1278055498.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.56 7.49 4.41 6/186 1108
/proc/meminfo: memFree=30902272/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0

[startup+0.279732 s]
/proc/loadavg: 7.56 7.49 4.41 6/186 1108
/proc/meminfo: memFree=30902272/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300781 s]
/proc/loadavg: 7.56 7.49 4.41 6/186 1108
/proc/meminfo: memFree=30902272/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.400777 s]
/proc/loadavg: 7.56 7.49 4.41 6/186 1108
/proc/meminfo: memFree=30902272/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.70065 s]
/proc/loadavg: 7.56 7.49 4.41 6/186 1108
/proc/meminfo: memFree=30902272/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50189 s]
/proc/loadavg: 7.56 7.49 4.41 10/188 1110
/proc/meminfo: memFree=30903124/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=13676 CPUtime=0.51
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 600 0 0 0 51 0 0 0 25 0 1 0 181152902 14004224 553 4089446400 134512640 146736320 4290120224 18446744073709551615 140683488 0 0 4096 16387 0 0 0 17 1 0 0 0
/proc/1109/statm: 3419 553 300 2985 0 329 0
Current children cumulated CPU time (s) 0.51
Current children cumulated vsize (KiB) 22376

[startup+3.10019 s]
/proc/loadavg: 7.56 7.49 4.41 3/188 1110
/proc/meminfo: memFree=30902380/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=13808 CPUtime=1.06
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 664 0 0 0 106 0 0 0 25 0 1 0 181152902 14139392 617 4089446400 134512640 146736320 4290120224 18446744073709551615 140526592 0 0 4096 16387 0 0 0 17 4 0 0 0
/proc/1109/statm: 3452 617 306 2985 0 362 0
Current children cumulated CPU time (s) 1.06
Current children cumulated vsize (KiB) 22508
heavy processes:

[startup+6.30058 s]
/proc/loadavg: 7.11 7.40 4.40 12/195 1124
/proc/meminfo: memFree=30855336/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=97400 CPUtime=5.25
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 18408 0 0 0 519 6 0 0 16 0 1 0 181152902 99737600 9916 4089446400 134512640 146736320 4290120224 18446744073709551615 136403112 0 0 4096 16387 0 0 0 17 5 0 0 0
/proc/1109/statm: 24350 9916 1070 2985 0 20892 0
Current children cumulated CPU time (s) 5.25
Current children cumulated vsize (KiB) 106100

[startup+12.7003 s]
/proc/loadavg: 7.26 7.42 4.42 9/195 1145
/proc/meminfo: memFree=30867504/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=139080 CPUtime=21.68
/proc/1109/stat : 1109 (pb_cplex) S 1108 1108 32648 0 -1 4202496 24080 0 0 0 2151 17 0 0 15 0 8 0 181152902 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/1109/statm: 34770 12943 1079 2985 0 31312 0
[pid=1109/tid=1139] ppid=1108 vsize=139080 CPUtime=1.55
/proc/1109/task/1139/stat : 1139 (pb_cplex) S 1108 1108 32648 0 -1 4202560 1654 0 0 0 154 1 0 0 15 0 8 0 181153610 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 7 0 0 0
[pid=1109/tid=1140] ppid=1108 vsize=139080 CPUtime=1.67
/proc/1109/task/1140/stat : 1140 (pb_cplex) S 1108 1108 32648 0 -1 4202560 918 0 0 0 166 1 0 0 15 0 8 0 181153610 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
[pid=1109/tid=1141] ppid=1108 vsize=139080 CPUtime=1.83
/proc/1109/task/1141/stat : 1141 (pb_cplex) R 1108 1108 32648 0 -1 4202560 303 0 0 0 182 1 0 0 15 0 8 0 181153610 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=1109/tid=1142] ppid=1108 vsize=139080 CPUtime=1.72
/proc/1109/task/1142/stat : 1142 (pb_cplex) S 1108 1108 32648 0 -1 4202560 289 0 0 0 172 0 0 0 15 0 8 0 181153610 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 5 0 0 0
[pid=1109/tid=1143] ppid=1108 vsize=139080 CPUtime=1.9
/proc/1109/task/1143/stat : 1143 (pb_cplex) S 1108 1108 32648 0 -1 4202560 142 0 0 0 189 1 0 0 16 0 8 0 181153610 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 7 0 0 0
[pid=1109/tid=1144] ppid=1108 vsize=139080 CPUtime=1.7
/proc/1109/task/1144/stat : 1144 (pb_cplex) R 1108 1108 32648 0 -1 4202560 372 0 0 0 169 1 0 0 16 0 8 0 181153610 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 136015710 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=1109/tid=1145] ppid=1108 vsize=139080 CPUtime=1.74
/proc/1109/task/1145/stat : 1145 (pb_cplex) R 1108 1108 32648 0 -1 4202560 1277 0 0 0 173 1 0 0 15 0 8 0 181153610 142417920 12943 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 21.68
Current children cumulated vsize (KiB) 147780

[startup+25.5239 s]
/proc/loadavg: 8.50 7.69 4.56 12/195 1146
/proc/meminfo: memFree=30857716/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=141128 CPUtime=57.74
/proc/1109/stat : 1109 (pb_cplex) S 1108 1108 32648 0 -1 4202496 27337 0 0 0 5721 53 0 0 15 0 8 0 181152902 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/1109/statm: 35282 13504 1079 2985 0 31824 0
[pid=1109/tid=1139] ppid=1108 vsize=141128 CPUtime=6.05
/proc/1109/task/1139/stat : 1139 (pb_cplex) S 1108 1108 32648 0 -1 4202560 2067 0 0 0 599 6 0 0 15 0 8 0 181153610 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 1 0 0 0
[pid=1109/tid=1140] ppid=1108 vsize=141128 CPUtime=6.12
/proc/1109/task/1140/stat : 1140 (pb_cplex) S 1108 1108 32648 0 -1 4202560 1278 0 0 0 607 5 0 0 15 0 8 0 181153610 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 4 0 0 0
[pid=1109/tid=1141] ppid=1108 vsize=141128 CPUtime=6.2
/proc/1109/task/1141/stat : 1141 (pb_cplex) S 1108 1108 32648 0 -1 4202560 644 0 0 0 616 4 0 0 15 0 8 0 181153610 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 4 0 0 0
[pid=1109/tid=1142] ppid=1108 vsize=141128 CPUtime=6.04
/proc/1109/task/1142/stat : 1142 (pb_cplex) S 1108 1108 32648 0 -1 4202560 837 0 0 0 600 4 0 0 15 0 8 0 181153610 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
[pid=1109/tid=1143] ppid=1108 vsize=141128 CPUtime=6.27
/proc/1109/task/1143/stat : 1143 (pb_cplex) S 1108 1108 32648 0 -1 4202560 674 0 0 0 621 6 0 0 15 0 8 0 181153610 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 5 0 0 0
[pid=1109/tid=1144] ppid=1108 vsize=141128 CPUtime=5.96
/proc/1109/task/1144/stat : 1144 (pb_cplex) S 1108 1108 32648 0 -1 4202560 921 0 0 0 590 6 0 0 15 0 8 0 181153610 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 5 0 0 0
[pid=1109/tid=1145] ppid=1108 vsize=141128 CPUtime=6.15
/proc/1109/task/1145/stat : 1145 (pb_cplex) S 1108 1108 32648 0 -1 4202560 1633 0 0 0 609 6 0 0 15 0 8 0 181153610 144515072 13504 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 57.74
Current children cumulated vsize (KiB) 149828

[startup+51.115 s]
/proc/loadavg: 8.12 7.68 4.64 11/195 1160
/proc/meminfo: memFree=30858584/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=140848 CPUtime=135.6
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 61258 0 0 0 13456 104 0 0 16 0 8 0 181152902 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 137343495 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/1109/statm: 35212 13735 1092 2985 0 31754 0
[pid=1109/tid=1154] ppid=1108 vsize=140848 CPUtime=4.22
/proc/1109/task/1154/stat : 1154 (pb_cplex) R 1108 1108 32648 0 -1 4202560 1278 0 0 0 420 2 0 0 15 0 8 0 181156857 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=140848 CPUtime=3.71
/proc/1109/task/1155/stat : 1155 (pb_cplex) R 1108 1108 32648 0 -1 4202560 1356 0 0 0 369 2 0 0 15 0 8 0 181156857 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 136017644 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=140848 CPUtime=4.49
/proc/1109/task/1156/stat : 1156 (pb_cplex) R 1108 1108 32648 0 -1 4202560 1725 0 0 0 447 2 0 0 15 0 8 0 181156857 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=140848 CPUtime=3.82
/proc/1109/task/1157/stat : 1157 (pb_cplex) R 1108 1108 32648 0 -1 4202560 2573 0 0 0 380 2 0 0 15 0 8 0 181156857 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=140848 CPUtime=3.4
/proc/1109/task/1158/stat : 1158 (pb_cplex) R 1108 1108 32648 0 -1 4202560 583 0 0 0 338 2 0 0 15 0 8 0 181156857 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=140848 CPUtime=3.94
/proc/1109/task/1159/stat : 1159 (pb_cplex) R 1108 1108 32648 0 -1 4202560 1877 0 0 0 391 3 0 0 15 0 8 0 181156857 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=140848 CPUtime=4.55
/proc/1109/task/1160/stat : 1160 (pb_cplex) R 1108 1108 32648 0 -1 4202560 1696 0 0 0 453 2 0 0 16 0 8 0 181156857 144228352 13735 4089446400 134512640 146736320 4290120224 18446744073709551615 136015689 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 135.6
Current children cumulated vsize (KiB) 149548

[startup+102.302 s]
/proc/loadavg: 9.55 8.11 4.94 10/195 1169
/proc/meminfo: memFree=30856972/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=141872 CPUtime=297.8
/proc/1109/stat : 1109 (pb_cplex) S 1108 1108 32648 0 -1 4202496 87358 0 0 0 29527 253 0 0 16 0 8 0 181152902 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/1109/statm: 35468 13912 1092 2985 0 32010 0
[pid=1109/tid=1154] ppid=1108 vsize=141872 CPUtime=23.61
/proc/1109/task/1154/stat : 1154 (pb_cplex) S 1108 1108 32648 0 -1 4202560 7629 0 0 0 2340 21 0 0 15 0 8 0 181156857 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 0 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=141872 CPUtime=23.11
/proc/1109/task/1155/stat : 1155 (pb_cplex) S 1108 1108 32648 0 -1 4202560 3650 0 0 0 2292 19 0 0 15 0 8 0 181156857 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 0 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=141872 CPUtime=24.13
/proc/1109/task/1156/stat : 1156 (pb_cplex) S 1108 1108 32648 0 -1 4202560 4189 0 0 0 2391 22 0 0 15 0 8 0 181156857 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 0 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=141872 CPUtime=23.31
/proc/1109/task/1157/stat : 1157 (pb_cplex) S 1108 1108 32648 0 -1 4202560 6648 0 0 0 2310 21 0 0 15 0 8 0 181156857 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 5 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=141872 CPUtime=23.17
/proc/1109/task/1158/stat : 1158 (pb_cplex) R 1108 1108 32648 0 -1 4202560 2554 0 0 0 2297 20 0 0 15 0 8 0 181156857 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=141872 CPUtime=23.42
/proc/1109/task/1159/stat : 1159 (pb_cplex) S 1108 1108 32648 0 -1 4202560 4864 0 0 0 2321 21 0 0 15 0 8 0 181156857 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 4 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=141872 CPUtime=24.32
/proc/1109/task/1160/stat : 1160 (pb_cplex) S 1108 1108 32648 0 -1 4202560 6365 0 0 0 2411 21 0 0 15 0 8 0 181156857 145276928 13912 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 297.8
Current children cumulated vsize (KiB) 150572

[startup+162.302 s]
/proc/loadavg: 10.28 8.55 5.29 12/195 1170
/proc/meminfo: memFree=30851640/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0

################
# More data... #
################

/proc/1109/task/1159/stat : 1159 (pb_cplex) S 1108 1108 32648 0 -1 4202560 7262 0 0 0 4559 42 0 0 15 0 8 0 181156857 144228352 13445 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 2 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=140848 CPUtime=46.98
/proc/1109/task/1160/stat : 1160 (pb_cplex) S 1108 1108 32648 0 -1 4202560 13142 0 0 0 4654 44 0 0 15 0 8 0 181156857 144228352 13445 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 483.2
Current children cumulated vsize (KiB) 149548

[startup+222.301 s]
/proc/loadavg: 9.56 8.64 5.52 14/195 1172
/proc/meminfo: memFree=30850648/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=143920 CPUtime=658.65
/proc/1109/stat : 1109 (pb_cplex) S 1108 1108 32648 0 -1 4202496 126528 0 0 0 65279 586 0 0 16 0 8 0 181152902 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/1109/statm: 35980 14215 1108 2985 0 32522 0
[pid=1109/tid=1154] ppid=1108 vsize=143920 CPUtime=67.31
/proc/1109/task/1154/stat : 1154 (pb_cplex) S 1108 1108 32648 0 -1 4202560 11664 0 0 0 6669 62 0 0 16 0 8 0 181156857 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 5 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=143920 CPUtime=66.75
/proc/1109/task/1155/stat : 1155 (pb_cplex) R 1108 1108 32648 0 -1 4202560 6824 0 0 0 6616 59 0 0 15 0 8 0 181156857 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=143920 CPUtime=67.71
/proc/1109/task/1156/stat : 1156 (pb_cplex) S 1108 1108 32648 0 -1 4202560 8487 0 0 0 6705 66 0 0 16 0 8 0 181156857 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 0 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=143920 CPUtime=66.82
/proc/1109/task/1157/stat : 1157 (pb_cplex) S 1108 1108 32648 0 -1 4202560 13719 0 0 0 6616 66 0 0 16 0 8 0 181156857 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 7 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=143920 CPUtime=67.03
/proc/1109/task/1158/stat : 1158 (pb_cplex) S 1108 1108 32648 0 -1 4202560 7278 0 0 0 6645 58 0 0 15 0 8 0 181156857 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 1 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=143920 CPUtime=67.32
/proc/1109/task/1159/stat : 1159 (pb_cplex) S 1108 1108 32648 0 -1 4202560 8166 0 0 0 6670 62 0 0 15 0 8 0 181156857 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 2 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=143920 CPUtime=68.23
/proc/1109/task/1160/stat : 1160 (pb_cplex) S 1108 1108 32648 0 -1 4202560 14673 0 0 0 6760 63 0 0 16 0 8 0 181156857 147374080 14215 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 658.65
Current children cumulated vsize (KiB) 152620

[startup+282.314 s]
/proc/loadavg: 8.69 8.57 5.69 10/195 1258
/proc/meminfo: memFree=31096712/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=143920 CPUtime=931.39
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 141023 0 0 0 92302 837 0 0 15 0 8 0 181152902 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/1109/statm: 35980 14274 1108 2985 0 32522 0
[pid=1109/tid=1154] ppid=1108 vsize=143920 CPUtime=101.28
/proc/1109/task/1154/stat : 1154 (pb_cplex) R 1108 1108 32648 0 -1 4202560 13686 0 0 0 10036 92 0 0 16 0 8 0 181156857 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 144580332 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=143920 CPUtime=100.23
/proc/1109/task/1155/stat : 1155 (pb_cplex) R 1108 1108 32648 0 -1 4202560 9003 0 0 0 9933 90 0 0 15 0 8 0 181156857 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=143920 CPUtime=101.71
/proc/1109/task/1156/stat : 1156 (pb_cplex) R 1108 1108 32648 0 -1 4202560 10121 0 0 0 10075 96 0 0 15 0 8 0 181156857 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=143920 CPUtime=100.85
/proc/1109/task/1157/stat : 1157 (pb_cplex) R 1108 1108 32648 0 -1 4202560 15109 0 0 0 9988 97 0 0 15 0 8 0 181156857 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 135468528 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=143920 CPUtime=100.88
/proc/1109/task/1158/stat : 1158 (pb_cplex) R 1108 1108 32648 0 -1 4202560 8572 0 0 0 9998 90 0 0 15 0 8 0 181156857 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 136017519 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=143920 CPUtime=101.5
/proc/1109/task/1159/stat : 1159 (pb_cplex) R 1108 1108 32648 0 -1 4202560 10483 0 0 0 10057 93 0 0 15 0 8 0 181156857 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=143920 CPUtime=102.27
/proc/1109/task/1160/stat : 1160 (pb_cplex) R 1108 1108 32648 0 -1 4202560 17733 0 0 0 10130 97 0 0 15 0 8 0 181156857 147374080 14274 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 931.39
Current children cumulated vsize (KiB) 152620

[startup+342.351 s]
/proc/loadavg: 9.10 8.74 5.93 10/195 1260
/proc/meminfo: memFree=31082876/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=143920 CPUtime=1155.41
/proc/1109/stat : 1109 (pb_cplex) S 1108 1108 32648 0 -1 4202496 150811 0 0 0 114517 1024 0 0 15 0 8 0 181152902 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/1109/statm: 35980 14276 1108 2985 0 32522 0
[pid=1109/tid=1154] ppid=1108 vsize=143920 CPUtime=128.79
/proc/1109/task/1154/stat : 1154 (pb_cplex) R 1108 1108 32648 0 -1 4202560 14370 0 0 0 12764 115 0 0 15 0 8 0 181156857 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=143920 CPUtime=127.89
/proc/1109/task/1155/stat : 1155 (pb_cplex) S 1108 1108 32648 0 -1 4202560 10514 0 0 0 12674 115 0 0 16 0 8 0 181156857 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 2 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=143920 CPUtime=129.4
/proc/1109/task/1156/stat : 1156 (pb_cplex) S 1108 1108 32648 0 -1 4202560 11591 0 0 0 12820 120 0 0 15 0 8 0 181156857 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 4 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=143920 CPUtime=128.66
/proc/1109/task/1157/stat : 1157 (pb_cplex) S 1108 1108 32648 0 -1 4202560 16935 0 0 0 12744 122 0 0 15 0 8 0 181156857 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 7 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=143920 CPUtime=128.63
/proc/1109/task/1158/stat : 1158 (pb_cplex) S 1108 1108 32648 0 -1 4202560 9697 0 0 0 12750 113 0 0 15 0 8 0 181156857 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=143920 CPUtime=129.39
/proc/1109/task/1159/stat : 1159 (pb_cplex) S 1108 1108 32648 0 -1 4202560 11874 0 0 0 12823 116 0 0 15 0 8 0 181156857 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=143920 CPUtime=129.9
/proc/1109/task/1160/stat : 1160 (pb_cplex) S 1108 1108 32648 0 -1 4202560 18480 0 0 0 12872 118 0 0 15 0 8 0 181156857 147374080 14276 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1155.41
Current children cumulated vsize (KiB) 152620

[startup+402.301 s]
/proc/loadavg: 9.39 8.87 6.14 8/195 1262
/proc/meminfo: memFree=31082864/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=143920 CPUtime=1392.04
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 163080 0 0 0 137978 1226 0 0 15 0 8 0 181152902 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/1109/statm: 35980 14277 1108 2985 0 32522 0
[pid=1109/tid=1154] ppid=1108 vsize=143920 CPUtime=158.37
/proc/1109/task/1154/stat : 1154 (pb_cplex) R 1108 1108 32648 0 -1 4202560 16393 0 0 0 15695 142 0 0 15 0 8 0 181156857 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=143920 CPUtime=157.09
/proc/1109/task/1155/stat : 1155 (pb_cplex) R 1108 1108 32648 0 -1 4202560 12150 0 0 0 15572 137 0 0 15 0 8 0 181156857 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=143920 CPUtime=158.57
/proc/1109/task/1156/stat : 1156 (pb_cplex) R 1108 1108 32648 0 -1 4202560 12722 0 0 0 15711 146 0 0 15 0 8 0 181156857 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=143920 CPUtime=157.93
/proc/1109/task/1157/stat : 1157 (pb_cplex) S 1108 1108 32648 0 -1 4202560 18968 0 0 0 15645 148 0 0 15 0 8 0 181156857 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=143920 CPUtime=157.66
/proc/1109/task/1158/stat : 1158 (pb_cplex) S 1108 1108 32648 0 -1 4202560 10945 0 0 0 15627 139 0 0 15 0 8 0 181156857 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=143920 CPUtime=158.66
/proc/1109/task/1159/stat : 1159 (pb_cplex) S 1108 1108 32648 0 -1 4202560 13678 0 0 0 15725 141 0 0 15 0 8 0 181156857 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=143920 CPUtime=159.35
/proc/1109/task/1160/stat : 1160 (pb_cplex) S 1108 1108 32648 0 -1 4202560 19866 0 0 0 15791 144 0 0 15 0 8 0 181156857 147374080 14277 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1392.04
Current children cumulated vsize (KiB) 152620

[startup+462.302 s]
/proc/loadavg: 8.79 8.79 6.29 10/195 1263
/proc/meminfo: memFree=31085832/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=143920 CPUtime=1629.02
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 175343 0 0 0 161477 1425 0 0 16 0 8 0 181152902 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 136366866 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/1109/statm: 35980 14279 1110 2985 0 32522 0
[pid=1109/tid=1154] ppid=1108 vsize=143920 CPUtime=187.7
/proc/1109/task/1154/stat : 1154 (pb_cplex) R 1108 1108 32648 0 -1 4202560 18258 0 0 0 18604 166 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 135982317 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=143920 CPUtime=186.49
/proc/1109/task/1155/stat : 1155 (pb_cplex) R 1108 1108 32648 0 -1 4202560 13277 0 0 0 18486 163 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 135454073 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=143920 CPUtime=187.65
/proc/1109/task/1156/stat : 1156 (pb_cplex) R 1108 1108 32648 0 -1 4202560 13908 0 0 0 18594 171 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=143920 CPUtime=187.13
/proc/1109/task/1157/stat : 1157 (pb_cplex) R 1108 1108 32648 0 -1 4202560 20642 0 0 0 18537 176 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=143920 CPUtime=187.15
/proc/1109/task/1158/stat : 1158 (pb_cplex) R 1108 1108 32648 0 -1 4202560 12032 0 0 0 18552 163 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=143920 CPUtime=188.09
/proc/1109/task/1159/stat : 1159 (pb_cplex) R 1108 1108 32648 0 -1 4202560 15636 0 0 0 18644 165 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=143920 CPUtime=188.89
/proc/1109/task/1160/stat : 1160 (pb_cplex) S 1108 1108 32648 0 -1 4202560 22365 0 0 0 18720 169 0 0 16 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144580073 0 2 4096 16387 18446604453956971328 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1629.02
Current children cumulated vsize (KiB) 152620



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+502.201 s]
/proc/loadavg: 9.08 8.87 6.42 13/188 1292
/proc/meminfo: memFree=31134540/32950928 swapFree=67111528/67111528
[pid=1108] ppid=1106 vsize=8700 CPUtime=0
/proc/1108/stat : 1108 (run) S 1106 1108 32648 0 -1 4202496 365 0 0 0 0 0 0 0 19 0 1 0 181152902 8908800 249 4089446400 4194304 4922060 140736161339152 18446744073709551615 256415210629 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/1108/statm: 2175 249 208 178 0 68 0
[pid=1109] ppid=1108 vsize=143920 CPUtime=1800.58
/proc/1109/stat : 1109 (pb_cplex) R 1108 1108 32648 0 -1 4202496 183415 0 0 0 178481 1577 0 0 15 0 8 0 181152902 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/1109/statm: 35980 14279 1110 2985 0 32522 0
[pid=1109/tid=1154] ppid=1108 vsize=143920 CPUtime=209.06
/proc/1109/task/1154/stat : 1154 (pb_cplex) R 1108 1108 32648 0 -1 4202560 19150 0 0 0 20722 184 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 136384846 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=1109/tid=1155] ppid=1108 vsize=143920 CPUtime=207.73
/proc/1109/task/1155/stat : 1155 (pb_cplex) R 1108 1108 32648 0 -1 4202560 14210 0 0 0 20592 181 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=1109/tid=1156] ppid=1108 vsize=143920 CPUtime=208.97
/proc/1109/task/1156/stat : 1156 (pb_cplex) R 1108 1108 32648 0 -1 4202560 15556 0 0 0 20706 191 0 0 16 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 135454073 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=1109/tid=1157] ppid=1108 vsize=143920 CPUtime=208.26
/proc/1109/task/1157/stat : 1157 (pb_cplex) R 1108 1108 32648 0 -1 4202560 21301 0 0 0 20628 198 0 0 16 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 136017763 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=1109/tid=1158] ppid=1108 vsize=143920 CPUtime=208.4
/proc/1109/task/1158/stat : 1158 (pb_cplex) R 1108 1108 32648 0 -1 4202560 12711 0 0 0 20659 181 0 0 16 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 139044557 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=1109/tid=1159] ppid=1108 vsize=143920 CPUtime=209.6
/proc/1109/task/1159/stat : 1159 (pb_cplex) R 1108 1108 32648 0 -1 4202560 16647 0 0 0 20776 184 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 135453244 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=1109/tid=1160] ppid=1108 vsize=143920 CPUtime=210.32
/proc/1109/task/1160/stat : 1160 (pb_cplex) R 1108 1108 32648 0 -1 4202560 23634 0 0 0 20843 189 0 0 15 0 8 0 181156857 147374080 14279 4089446400 134512640 146736320 4290120224 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1800.58
Current children cumulated vsize (KiB) 152620

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 502.216
CPU time (s): 1800.66
CPU user time (s): 1784.88
CPU system time (s): 15.7816
CPU usage (%): 358.543
Max. virtual memory (cumulated for all children) (KiB): 178156

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1784.88
system time used= 15.7816
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 183789
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 317324
involuntary context switches= 24733

runsolver used 1.01484 second user time and 1.93371 second system time

The end

Launcher Data 

Begin job on node115 at 2010-07-02 09:24:58
IDJOB=2697374
IDBENCH=79262
IDSOLVER=1210
FILE ID=node115/2697374-1278055498
PBS_JOBID= 11198952
Free space on /tmp= 74160 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900553647--soft-0-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697374-1278055498/watcher-2697374-1278055498 -o /tmp/evaluation-result-2697374-1278055498/solver-2697374-1278055498 -C 1800 -W 2000 -M 3800  run HOME/instance-2697374-1278055498.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 1693c6ddcdda9c3a56f7a1957868c0a9
RANDOM SEED=855406558

node115.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.08
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.796
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30902428 kB
Buffers:        221352 kB
Cached:        1322896 kB
SwapCached:          0 kB
Active:         899912 kB
Inactive:       995960 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30902428 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16844 kB
Writeback:           0 kB
AnonPages:      351600 kB
Mapped:          18876 kB
Slab:            84156 kB
PageTables:       5744 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   586720 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74144 MiB
End job on node115 at 2010-07-02 09:33:20