Trace number 2697824

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.56 488.546

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900670234--soft-33-100-0.wbo
MD5SUM74a38f74cac3b14e8d36cdbf48853bbd
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark525.533
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints1830
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 92327
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 92326
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 641
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697824-1278054864.wbo.
0.00/0.00	c Instance file HOME/instance-2697824-1278054864.wbo
0.00/0.00	c File size is 92432 bytes.
0.00/0.67	c Highest Coefficient sum: 641
0.00/0.67	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.67	c ============================[ Problem Statistics ]=============================
0.00/0.67	c |                                                                             |
0.00/0.67	c |  Parsing time:          0.67         s                                      |
0.00/0.67	c |  Number Variables:      2061                                                |
0.00/0.67	c |  Number Constraints:    2708                                                |
0.00/0.67	c ===============================================================================
1800.54/488.54	c Remaining time 1799.33 sec.
1800.54/488.54	c 
1800.54/488.54	c *** INTERRUPTED ***
1800.54/488.54	s UNKNOWN
1800.54/488.54	c Total CPU time              : 1784.47 s
1800.54/488.54	c Memory used           : 164.49 MB
1800.54/488.54	c CPU time              : 1784.47 s
1800.54/488.54	c 
1800.54/488.54	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697824-1278054864/watcher-2697824-1278054864 -o /tmp/evaluation-result-2697824-1278054864/solver-2697824-1278054864 -C 1800 -W 2000 -M 3800 run HOME/instance-2697824-1278054864.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.08 0.06 0.01 2/177 21842
/proc/meminfo: memFree=30773028/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0

[startup+0.076414 s]
/proc/loadavg: 0.08 0.06 0.01 2/177 21842
/proc/meminfo: memFree=30773028/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.100382 s]
/proc/loadavg: 0.08 0.06 0.01 2/177 21842
/proc/meminfo: memFree=30773028/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300373 s]
/proc/loadavg: 0.08 0.06 0.01 2/177 21842
/proc/meminfo: memFree=30773028/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.700226 s]
/proc/loadavg: 0.08 0.06 0.01 2/177 21842
/proc/meminfo: memFree=30773028/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50105 s]
/proc/loadavg: 0.08 0.06 0.01 3/185 21862
/proc/meminfo: memFree=30731176/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=141808 CPUtime=2.79
/proc/21843/stat : 21843 (pb_cplex) R 21842 21842 21606 0 -1 4202496 20597 0 0 0 273 6 0 0 16 0 8 0 146782052 145211392 12478 4089446400 134512640 146736320 4291308016 18446744073709551615 139043577 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21843/statm: 35452 12478 1074 2985 0 31994 0
Current children cumulated CPU time (s) 2.79
Current children cumulated vsize (KiB) 150508

[startup+3.10084 s]
/proc/loadavg: 0.08 0.06 0.01 3/185 21890
/proc/meminfo: memFree=30689264/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=137712 CPUtime=5.21
/proc/21843/stat : 21843 (pb_cplex) R 21842 21842 21606 0 -1 4202496 28547 0 0 0 513 8 0 0 18 0 8 0 146782052 141017088 11622 4089446400 134512640 146736320 4291308016 18446744073709551615 136403109 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21843/statm: 34428 11622 1078 2985 0 30970 0
Current children cumulated CPU time (s) 5.21
Current children cumulated vsize (KiB) 146412

[startup+6.30116 s]
/proc/loadavg: 0.24 0.10 0.02 12/199 21932
/proc/meminfo: memFree=30653316/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146596 CPUtime=15.47
/proc/21843/stat : 21843 (pb_cplex) S 21842 21842 21606 0 -1 4202496 37795 0 0 0 1531 16 0 0 15 0 8 0 146782052 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21843/statm: 36649 14282 1078 2985 0 33191 0
[pid=21843/tid=21905] ppid=21842 vsize=146596 CPUtime=1.59
/proc/21843/task/21905/stat : 21905 (pb_cplex) R 21842 21842 21606 0 -1 4202560 689 0 0 0 159 0 0 0 16 0 8 0 146782307 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21843/tid=21906] ppid=21842 vsize=146596 CPUtime=1.12
/proc/21843/task/21906/stat : 21906 (pb_cplex) S 21842 21842 21606 0 -1 4202560 691 0 0 0 112 0 0 0 16 0 8 0 146782307 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 4 0 0 0
[pid=21843/tid=21907] ppid=21842 vsize=146596 CPUtime=1.36
/proc/21843/task/21907/stat : 21907 (pb_cplex) S 21842 21842 21606 0 -1 4202560 2127 0 0 0 135 1 0 0 15 0 8 0 146782307 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 0 0 0 0
[pid=21843/tid=21908] ppid=21842 vsize=146596 CPUtime=1.27
/proc/21843/task/21908/stat : 21908 (pb_cplex) S 21842 21842 21606 0 -1 4202560 51 0 0 0 127 0 0 0 15 0 8 0 146782307 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 0 0 0 0
[pid=21843/tid=21909] ppid=21842 vsize=146596 CPUtime=1.32
/proc/21843/task/21909/stat : 21909 (pb_cplex) S 21842 21842 21606 0 -1 4202560 1885 0 0 0 131 1 0 0 15 0 8 0 146782307 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 0 0 0 0
[pid=21843/tid=21910] ppid=21842 vsize=146596 CPUtime=0.89
/proc/21843/task/21910/stat : 21910 (pb_cplex) S 21842 21842 21606 0 -1 4202560 289 0 0 0 89 0 0 0 15 0 8 0 146782307 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 6 0 0 0
[pid=21843/tid=21911] ppid=21842 vsize=146596 CPUtime=1.2
/proc/21843/task/21911/stat : 21911 (pb_cplex) S 21842 21842 21606 0 -1 4202560 639 0 0 0 120 0 0 0 15 0 8 0 146782307 150114304 14282 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 15.47
Current children cumulated vsize (KiB) 155296

[startup+12.702 s]
/proc/loadavg: 0.94 0.24 0.07 9/199 21932
/proc/meminfo: memFree=30651340/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146596 CPUtime=37.47
/proc/21843/stat : 21843 (pb_cplex) S 21842 21842 21606 0 -1 4202496 38603 0 0 0 3711 36 0 0 16 0 8 0 146782052 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21843/statm: 36649 14349 1078 2985 0 33191 0
[pid=21843/tid=21905] ppid=21842 vsize=146596 CPUtime=4.22
/proc/21843/task/21905/stat : 21905 (pb_cplex) S 21842 21842 21606 0 -1 4202560 815 0 0 0 419 3 0 0 15 0 8 0 146782307 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 6 0 0 0
[pid=21843/tid=21906] ppid=21842 vsize=146596 CPUtime=3.98
/proc/21843/task/21906/stat : 21906 (pb_cplex) S 21842 21842 21606 0 -1 4202560 988 0 0 0 395 3 0 0 15 0 8 0 146782307 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 2 0 0 0
[pid=21843/tid=21907] ppid=21842 vsize=146596 CPUtime=4.2
/proc/21843/task/21907/stat : 21907 (pb_cplex) S 21842 21842 21606 0 -1 4202560 2150 0 0 0 417 3 0 0 15 0 8 0 146782307 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 2 0 0 0
[pid=21843/tid=21908] ppid=21842 vsize=146596 CPUtime=4.06
/proc/21843/task/21908/stat : 21908 (pb_cplex) S 21842 21842 21606 0 -1 4202560 62 0 0 0 403 3 0 0 15 0 8 0 146782307 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 7 0 0 0
[pid=21843/tid=21909] ppid=21842 vsize=146596 CPUtime=4.01
/proc/21843/task/21909/stat : 21909 (pb_cplex) R 21842 21842 21606 0 -1 4202560 2151 0 0 0 398 3 0 0 15 0 8 0 146782307 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 136017496 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21843/tid=21910] ppid=21842 vsize=146596 CPUtime=3.49
/proc/21843/task/21910/stat : 21910 (pb_cplex) S 21842 21842 21606 0 -1 4202560 352 0 0 0 346 3 0 0 15 0 8 0 146782307 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 4 0 0 0
[pid=21843/tid=21911] ppid=21842 vsize=146596 CPUtime=3.91
/proc/21843/task/21911/stat : 21911 (pb_cplex) R 21842 21842 21606 0 -1 4202560 645 0 0 0 388 3 0 0 15 0 8 0 146782307 150114304 14349 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37.47
Current children cumulated vsize (KiB) 155296

[startup+25.5006 s]
/proc/loadavg: 3.03 0.74 0.23 9/199 21947
/proc/meminfo: memFree=30642532/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=159532 CPUtime=94.43
/proc/21843/stat : 21843 (pb_cplex) S 21842 21842 21606 0 -1 4202496 57419 0 0 0 9362 81 0 0 16 0 8 0 146782052 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/21843/statm: 39883 17134 1091 2985 0 36425 0
[pid=21843/tid=21905] ppid=21842 vsize=159532 CPUtime=12.92
/proc/21843/task/21905/stat : 21905 (pb_cplex) S 21842 21842 21606 0 -1 4202560 4438 0 0 0 1283 9 0 0 25 0 8 0 146782307 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 6 0 0 0
[pid=21843/tid=21906] ppid=21842 vsize=159532 CPUtime=10.89
/proc/21843/task/21906/stat : 21906 (pb_cplex) S 21842 21842 21606 0 -1 4202560 2379 0 0 0 1081 8 0 0 18 0 8 0 146782307 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 3 0 0 0
[pid=21843/tid=21907] ppid=21842 vsize=159532 CPUtime=11.06
/proc/21843/task/21907/stat : 21907 (pb_cplex) R 21842 21842 21606 0 -1 4202560 4441 0 0 0 1097 9 0 0 20 0 8 0 146782307 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 136475800 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21843/tid=21908] ppid=21842 vsize=159532 CPUtime=11.98
/proc/21843/task/21908/stat : 21908 (pb_cplex) R 21842 21842 21606 0 -1 4202560 3503 0 0 0 1190 8 0 0 21 0 8 0 146782307 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 144785853 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21843/tid=21909] ppid=21842 vsize=159532 CPUtime=9.79
/proc/21843/task/21909/stat : 21909 (pb_cplex) S 21842 21842 21606 0 -1 4202560 4600 0 0 0 970 9 0 0 17 0 8 0 146782307 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 4 0 0 0
[pid=21843/tid=21910] ppid=21842 vsize=159532 CPUtime=12.1
/proc/21843/task/21910/stat : 21910 (pb_cplex) R 21842 21842 21606 0 -1 4202560 2831 0 0 0 1201 9 0 0 25 0 8 0 146782307 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 135468524 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21843/tid=21911] ppid=21842 vsize=159532 CPUtime=11.71
/proc/21843/task/21911/stat : 21911 (pb_cplex) R 21842 21842 21606 0 -1 4202560 2683 0 0 0 1163 8 0 0 21 0 8 0 146782307 163360768 17134 4089446400 134512640 146736320 4291308016 18446744073709551615 136400428 0 2 4096 16387 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 94.43
Current children cumulated vsize (KiB) 168232

[startup+51.1009 s]
/proc/loadavg: 4.66 1.28 0.42 9/199 21962
/proc/meminfo: memFree=30654888/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=143856 CPUtime=157.56
/proc/21843/stat : 21843 (pb_cplex) R 21842 21842 21606 0 -1 4202496 76640 0 0 0 15627 129 0 0 15 0 8 0 146782052 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21843/statm: 35964 13539 1092 2985 0 32506 0
[pid=21843/tid=21955] ppid=21842 vsize=143856 CPUtime=8.57
/proc/21843/task/21955/stat : 21955 (pb_cplex) S 21842 21842 21606 0 -1 4202560 1735 0 0 0 852 5 0 0 16 0 8 0 146784891 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 0 0 0 0
[pid=21843/tid=21956] ppid=21842 vsize=143856 CPUtime=7.19
/proc/21843/task/21956/stat : 21956 (pb_cplex) S 21842 21842 21606 0 -1 4202560 964 0 0 0 715 4 0 0 15 0 8 0 146784891 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 5 0 0 0
[pid=21843/tid=21957] ppid=21842 vsize=143856 CPUtime=5.48
/proc/21843/task/21957/stat : 21957 (pb_cplex) S 21842 21842 21606 0 -1 4202560 4459 0 0 0 543 5 0 0 16 0 8 0 146784891 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 2 0 0 0
[pid=21843/tid=21958] ppid=21842 vsize=143856 CPUtime=5.62
/proc/21843/task/21958/stat : 21958 (pb_cplex) S 21842 21842 21606 0 -1 4202560 2592 0 0 0 558 4 0 0 16 0 8 0 146784891 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 4 0 0 0
[pid=21843/tid=21959] ppid=21842 vsize=143856 CPUtime=5.75
/proc/21843/task/21959/stat : 21959 (pb_cplex) S 21842 21842 21606 0 -1 4202560 97 0 0 0 570 5 0 0 15 0 8 0 146784891 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
[pid=21843/tid=21960] ppid=21842 vsize=143856 CPUtime=6.08
/proc/21843/task/21960/stat : 21960 (pb_cplex) S 21842 21842 21606 0 -1 4202560 340 0 0 0 603 5 0 0 15 0 8 0 146784891 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 3 0 0 0
[pid=21843/tid=21961] ppid=21842 vsize=143856 CPUtime=6.91
/proc/21843/task/21961/stat : 21961 (pb_cplex) S 21842 21842 21606 0 -1 4202560 1318 0 0 0 687 4 0 0 15 0 8 0 146784891 147308544 13539 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 157.56
Current children cumulated vsize (KiB) 152556

[startup+102.339 s]
/proc/loadavg: 7.05 2.45 0.86 11/199 21963
/proc/meminfo: memFree=30651392/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146928 CPUtime=345.88
/proc/21843/stat : 21843 (pb_cplex) S 21842 21842 21606 0 -1 4202496 93973 0 0 0 34302 286 0 0 16 0 8 0 146782052 150454272 13897 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21843/statm: 36732 13897 1110 2985 0 33274 0
[pid=21843/tid=21955] ppid=21842 vsize=146928 CPUtime=31.78
/proc/21843/task/21955/stat : 21955 (pb_cplex) R 21842 21842 21606 0 -1 4202560 3969 0 0 0 3153 25 0 0 15 0 8 0 146784891 150454272 13897 4089446400 134512640 146736320 4291308016 18446744073709551615 135526863 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21843/tid=21956] ppid=21842 vsize=146928 CPUtime=30.18
/proc/21843/task/21956/stat : 21956 (pb_cplex) S 21842 21842 21606 0 -1 4202560 3554 0 0 0 2994 24 0 0 15 0 8 0 146784891 150454272 13897 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 0 0 0 0
[pid=21843/tid=21957] ppid=21842 vsize=146928 CPUtime=28.69
/proc/21843/task/21957/stat : 21957 (pb_cplex) R 21842 21842 21606 0 -1 4202560 7122 0 0 0 2846 23 0 0 15 0 8 0 146784891 150454272 13897 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21843/tid=21958] ppid=21842 vsize=146928 CPUtime=28.87
/proc/21843/task/21958/stat : 21958 (pb_cplex) R 21842 21842 21606 0 -1 4202560 5889 0 0 0 2863 24 0 0 15 0 8 0 146784891 150454272 13897 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/21843/task/21959/stat : 21959 (pb_cplex) R 21842 21842 21606 0 -1 4202560 6743 0 0 0 5640 47 0 0 15 0 8 0 146784891 150454272 14037 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21843/tid=21960] ppid=21842 vsize=146928 CPUtime=57.08
/proc/21843/task/21960/stat : 21960 (pb_cplex) R 21842 21842 21606 0 -1 4202560 4456 0 0 0 5659 49 0 0 15 0 8 0 146784891 150454272 14037 4089446400 134512640 146736320 4291308016 18446744073709551615 137361201 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21843/tid=21961] ppid=21842 vsize=146928 CPUtime=57.94
/proc/21843/task/21961/stat : 21961 (pb_cplex) R 21842 21842 21606 0 -1 4202560 7139 0 0 0 5745 49 0 0 15 0 8 0 146784891 150454272 14037 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 571.09
Current children cumulated vsize (KiB) 155628

[startup+222.316 s]
/proc/loadavg: 9.12 4.75 1.90 12/199 21966
/proc/meminfo: memFree=30650004/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146928 CPUtime=798.73
/proc/21843/stat : 21843 (pb_cplex) R 21842 21842 21606 0 -1 4202496 159950 0 0 0 79197 676 0 0 16 0 8 0 146782052 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21843/statm: 36732 14079 1110 2985 0 33274 0
[pid=21843/tid=21955] ppid=21842 vsize=146928 CPUtime=88.66
/proc/21843/task/21955/stat : 21955 (pb_cplex) R 21842 21842 21606 0 -1 4202560 12444 0 0 0 8791 75 0 0 15 0 8 0 146784891 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21843/tid=21956] ppid=21842 vsize=146928 CPUtime=85.92
/proc/21843/task/21956/stat : 21956 (pb_cplex) R 21842 21842 21606 0 -1 4202560 14800 0 0 0 8519 73 0 0 15 0 8 0 146784891 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 138423679 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21843/tid=21957] ppid=21842 vsize=146928 CPUtime=84.66
/proc/21843/task/21957/stat : 21957 (pb_cplex) S 21842 21842 21606 0 -1 4202560 13585 0 0 0 8394 72 0 0 15 0 8 0 146784891 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 7 0 0 0
[pid=21843/tid=21958] ppid=21842 vsize=146928 CPUtime=84.52
/proc/21843/task/21958/stat : 21958 (pb_cplex) R 21842 21842 21606 0 -1 4202560 12546 0 0 0 8383 69 0 0 15 0 8 0 146784891 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 139044554 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21843/tid=21959] ppid=21842 vsize=146928 CPUtime=85.09
/proc/21843/task/21959/stat : 21959 (pb_cplex) R 21842 21842 21606 0 -1 4202560 9667 0 0 0 8436 73 0 0 15 0 8 0 146784891 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 135454070 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21843/tid=21960] ppid=21842 vsize=146928 CPUtime=85.22
/proc/21843/task/21960/stat : 21960 (pb_cplex) R 21842 21842 21606 0 -1 4202560 10350 0 0 0 8446 76 0 0 15 0 8 0 146784891 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21843/tid=21961] ppid=21842 vsize=146928 CPUtime=86.06
/proc/21843/task/21961/stat : 21961 (pb_cplex) R 21842 21842 21606 0 -1 4202560 11774 0 0 0 8530 76 0 0 16 0 8 0 146784891 150454272 14079 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 798.73
Current children cumulated vsize (KiB) 155628

[startup+282.301 s]
/proc/loadavg: 8.86 5.47 2.32 10/199 21968
/proc/meminfo: memFree=30648116/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146928 CPUtime=1026.43
/proc/21843/stat : 21843 (pb_cplex) R 21842 21842 21606 0 -1 4202496 181844 0 0 0 101749 894 0 0 15 0 8 0 146782052 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 137345110 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21843/statm: 36732 14066 1110 2985 0 33274 0
[pid=21843/tid=21955] ppid=21842 vsize=146928 CPUtime=117.02
/proc/21843/task/21955/stat : 21955 (pb_cplex) R 21842 21842 21606 0 -1 4202560 14624 0 0 0 11599 103 0 0 15 0 8 0 146784891 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21843/tid=21956] ppid=21842 vsize=146928 CPUtime=114.02
/proc/21843/task/21956/stat : 21956 (pb_cplex) R 21842 21842 21606 0 -1 4202560 16653 0 0 0 11304 98 0 0 15 0 8 0 146784891 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21843/tid=21957] ppid=21842 vsize=146928 CPUtime=112.71
/proc/21843/task/21957/stat : 21957 (pb_cplex) R 21842 21842 21606 0 -1 4202560 19743 0 0 0 11171 100 0 0 15 0 8 0 146784891 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21843/tid=21958] ppid=21842 vsize=146928 CPUtime=112.26
/proc/21843/task/21958/stat : 21958 (pb_cplex) S 21842 21842 21606 0 -1 4202560 14410 0 0 0 11131 95 0 0 15 0 8 0 146784891 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
[pid=21843/tid=21959] ppid=21842 vsize=146928 CPUtime=113.38
/proc/21843/task/21959/stat : 21959 (pb_cplex) S 21842 21842 21606 0 -1 4202560 11562 0 0 0 11239 99 0 0 15 0 8 0 146784891 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
[pid=21843/tid=21960] ppid=21842 vsize=146928 CPUtime=113.72
/proc/21843/task/21960/stat : 21960 (pb_cplex) S 21842 21842 21606 0 -1 4202560 14025 0 0 0 11269 103 0 0 15 0 8 0 146784891 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
[pid=21843/tid=21961] ppid=21842 vsize=146928 CPUtime=114.23
/proc/21843/task/21961/stat : 21961 (pb_cplex) S 21842 21842 21606 0 -1 4202560 13669 0 0 0 11320 103 0 0 16 0 8 0 146784891 150454272 14066 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1026.43
Current children cumulated vsize (KiB) 155628

[startup+342.302 s]
/proc/loadavg: 9.99 6.40 2.84 11/199 21969
/proc/meminfo: memFree=30651108/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146928 CPUtime=1252.86
/proc/21843/stat : 21843 (pb_cplex) R 21842 21842 21606 0 -1 4202496 206082 0 0 0 124193 1093 0 0 15 0 8 0 146782052 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 136403112 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21843/statm: 36732 14072 1110 2985 0 33274 0
[pid=21843/tid=21955] ppid=21842 vsize=146928 CPUtime=145.26
/proc/21843/task/21955/stat : 21955 (pb_cplex) S 21842 21842 21606 0 -1 4202560 17097 0 0 0 14398 128 0 0 15 0 8 0 146784891 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 5 0 0 0
[pid=21843/tid=21956] ppid=21842 vsize=146928 CPUtime=141.81
/proc/21843/task/21956/stat : 21956 (pb_cplex) S 21842 21842 21606 0 -1 4202560 19446 0 0 0 14061 120 0 0 15 0 8 0 146784891 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
[pid=21843/tid=21957] ppid=21842 vsize=146928 CPUtime=140.63
/proc/21843/task/21957/stat : 21957 (pb_cplex) R 21842 21842 21606 0 -1 4202560 22737 0 0 0 13938 125 0 0 15 0 8 0 146784891 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21843/tid=21958] ppid=21842 vsize=146928 CPUtime=139.98
/proc/21843/task/21958/stat : 21958 (pb_cplex) R 21842 21842 21606 0 -1 4202560 17284 0 0 0 13878 120 0 0 15 0 8 0 146784891 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21843/tid=21959] ppid=21842 vsize=146928 CPUtime=141.44
/proc/21843/task/21959/stat : 21959 (pb_cplex) R 21842 21842 21606 0 -1 4202560 14826 0 0 0 14020 124 0 0 15 0 8 0 146784891 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 136017519 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21843/tid=21960] ppid=21842 vsize=146928 CPUtime=141.92
/proc/21843/task/21960/stat : 21960 (pb_cplex) R 21842 21842 21606 0 -1 4202560 19297 0 0 0 14062 130 0 0 15 0 8 0 146784891 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21843/tid=21961] ppid=21842 vsize=146928 CPUtime=142.52
/proc/21843/task/21961/stat : 21961 (pb_cplex) R 21842 21842 21606 0 -1 4202560 16207 0 0 0 14122 130 0 0 15 0 8 0 146784891 150454272 14072 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1252.86
Current children cumulated vsize (KiB) 155628

[startup+402.302 s]
/proc/loadavg: 9.71 6.97 3.25 10/199 21971
/proc/meminfo: memFree=30649780/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146928 CPUtime=1480.26
/proc/21843/stat : 21843 (pb_cplex) R 21842 21842 21606 0 -1 4202496 227511 0 0 0 146730 1296 0 0 16 0 8 0 146782052 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/21843/statm: 36732 14125 1110 2985 0 33274 0
[pid=21843/tid=21955] ppid=21842 vsize=146928 CPUtime=173.69
/proc/21843/task/21955/stat : 21955 (pb_cplex) R 21842 21842 21606 0 -1 4202560 19638 0 0 0 17218 151 0 0 15 0 8 0 146784891 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21843/tid=21956] ppid=21842 vsize=146928 CPUtime=169.8
/proc/21843/task/21956/stat : 21956 (pb_cplex) R 21842 21842 21606 0 -1 4202560 21595 0 0 0 16836 144 0 0 15 0 8 0 146784891 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 136017505 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21843/tid=21957] ppid=21842 vsize=146928 CPUtime=168.54
/proc/21843/task/21957/stat : 21957 (pb_cplex) R 21842 21842 21606 0 -1 4202560 25648 0 0 0 16703 151 0 0 15 0 8 0 146784891 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21843/tid=21958] ppid=21842 vsize=146928 CPUtime=167.93
/proc/21843/task/21958/stat : 21958 (pb_cplex) S 21842 21842 21606 0 -1 4202560 19261 0 0 0 16649 144 0 0 15 0 8 0 146784891 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 0 0 0 0
[pid=21843/tid=21959] ppid=21842 vsize=146928 CPUtime=169.44
/proc/21843/task/21959/stat : 21959 (pb_cplex) S 21842 21842 21606 0 -1 4202560 17225 0 0 0 16795 149 0 0 15 0 8 0 146784891 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 3 0 0 0
[pid=21843/tid=21960] ppid=21842 vsize=146928 CPUtime=170.35
/proc/21843/task/21960/stat : 21960 (pb_cplex) S 21842 21842 21606 0 -1 4202560 22196 0 0 0 16881 154 0 0 15 0 8 0 146784891 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 0 0 0 0
[pid=21843/tid=21961] ppid=21842 vsize=146928 CPUtime=170.59
/proc/21843/task/21961/stat : 21961 (pb_cplex) S 21842 21842 21606 0 -1 4202560 19674 0 0 0 16899 160 0 0 15 0 8 0 146784891 150454272 14125 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 1480.26
Current children cumulated vsize (KiB) 155628

[startup+462.35 s]
/proc/loadavg: 9.37 7.39 3.63 12/199 21972
/proc/meminfo: memFree=30650172/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=146928 CPUtime=1700.05
/proc/21843/stat : 21843 (pb_cplex) S 21842 21842 21606 0 -1 4202496 243266 0 0 0 168498 1507 0 0 16 0 8 0 146782052 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21843/statm: 36732 14126 1110 2985 0 33274 0
[pid=21843/tid=21955] ppid=21842 vsize=146928 CPUtime=200.91
/proc/21843/task/21955/stat : 21955 (pb_cplex) R 21842 21842 21606 0 -1 4202560 20966 0 0 0 19913 178 0 0 15 0 8 0 146784891 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21843/tid=21956] ppid=21842 vsize=146928 CPUtime=197.07
/proc/21843/task/21956/stat : 21956 (pb_cplex) S 21842 21842 21606 0 -1 4202560 24123 0 0 0 19536 171 0 0 15 0 8 0 146784891 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
[pid=21843/tid=21957] ppid=21842 vsize=146928 CPUtime=195.58
/proc/21843/task/21957/stat : 21957 (pb_cplex) S 21842 21842 21606 0 -1 4202560 26785 0 0 0 19380 178 0 0 15 0 8 0 146784891 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 5 0 0 0
[pid=21843/tid=21958] ppid=21842 vsize=146928 CPUtime=195.1
/proc/21843/task/21958/stat : 21958 (pb_cplex) R 21842 21842 21606 0 -1 4202560 21093 0 0 0 19340 170 0 0 15 0 8 0 146784891 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21843/tid=21959] ppid=21842 vsize=146928 CPUtime=196.62
/proc/21843/task/21959/stat : 21959 (pb_cplex) R 21842 21842 21606 0 -1 4202560 20314 0 0 0 19486 176 0 0 15 0 8 0 146784891 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21843/tid=21960] ppid=21842 vsize=146928 CPUtime=197.66
/proc/21843/task/21960/stat : 21960 (pb_cplex) R 21842 21842 21606 0 -1 4202560 24027 0 0 0 19585 181 0 0 15 0 8 0 146784891 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21843/tid=21961] ppid=21842 vsize=146928 CPUtime=197.83
/proc/21843/task/21961/stat : 21961 (pb_cplex) R 21842 21842 21606 0 -1 4202560 21388 0 0 0 19597 186 0 0 15 0 8 0 146784891 150454272 14126 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 1700.05
Current children cumulated vsize (KiB) 155628



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+488.53 s]
/proc/loadavg: 9.36 7.57 3.81 6/192 22009
/proc/meminfo: memFree=30681956/32951124 swapFree=67111528/67111528
[pid=21842] ppid=21840 vsize=8700 CPUtime=0
/proc/21842/stat : 21842 (run) S 21840 21842 21606 0 -1 4202496 366 0 0 0 0 0 0 0 18 0 1 0 146782052 8908800 249 4089446400 4194304 4922060 140735966053760 18446744073709551615 255377120389 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/21842/statm: 2175 249 208 178 0 68 0
[pid=21843] ppid=21842 vsize=168432 CPUtime=1800.54
/proc/21843/stat : 21843 (pb_cplex) S 21842 21842 21606 0 -1 4202496 252925 0 0 0 178446 1608 0 0 15 0 8 0 146782052 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21843/statm: 42108 18303 1111 2985 0 38650 0
[pid=21843/tid=21973] ppid=21842 vsize=168432 CPUtime=10.53
/proc/21843/task/21973/stat : 21973 (pb_cplex) S 21842 21842 21606 0 -1 4202560 89 0 0 0 1044 9 0 0 16 0 8 0 146828691 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 3 0 0 0
[pid=21843/tid=21974] ppid=21842 vsize=168432 CPUtime=10.68
/proc/21843/task/21974/stat : 21974 (pb_cplex) S 21842 21842 21606 0 -1 4202560 1694 0 0 0 1057 11 0 0 15 0 8 0 146828691 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 1 0 0 0
[pid=21843/tid=21975] ppid=21842 vsize=168432 CPUtime=10.5
/proc/21843/task/21975/stat : 21975 (pb_cplex) S 21842 21842 21606 0 -1 4202560 645 0 0 0 1041 9 0 0 16 0 8 0 146828691 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 6 0 0 0
[pid=21843/tid=21976] ppid=21842 vsize=168432 CPUtime=10.61
/proc/21843/task/21976/stat : 21976 (pb_cplex) R 21842 21842 21606 0 -1 4202560 998 0 0 0 1052 9 0 0 16 0 8 0 146828691 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21843/tid=21977] ppid=21842 vsize=168432 CPUtime=10.7
/proc/21843/task/21977/stat : 21977 (pb_cplex) S 21842 21842 21606 0 -1 4202560 1770 0 0 0 1060 10 0 0 15 0 8 0 146828691 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 5 0 0 0
[pid=21843/tid=21978] ppid=21842 vsize=168432 CPUtime=10.68
/proc/21843/task/21978/stat : 21978 (pb_cplex) R 21842 21842 21606 0 -1 4202560 487 0 0 0 1058 10 0 0 15 0 8 0 146828691 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21843/tid=21979] ppid=21842 vsize=168432 CPUtime=10.71
/proc/21843/task/21979/stat : 21979 (pb_cplex) S 21842 21842 21606 0 -1 4202560 283 0 0 0 1060 11 0 0 16 0 8 0 146828691 172474368 18303 4089446400 134512640 146736320 4291308016 18446744073709551615 144580073 0 2 4096 16387 18446604471129771072 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1800.54
Current children cumulated vsize (KiB) 177132

Sending SIGTERM to process tree (bottom up)
Solver jusSolver just ended. Dumping a history of the last processes samples

Child status: 0
Real time (s): 488.546
CPU time (s): 1800.56
CPU user time (s): 1784.47
CPU system time (s): 16.0866
CPU usage (%): 368.555
Max. virtual memory (cumulated for all children) (KiB): 179180

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1784.47
system time used= 16.0866
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 253298
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 338159
involuntary context switches= 36024

runsolver used 0.955854 second user time and 1.9677 second system time

The end

Launcher Data 

Begin job on node135 at 2010-07-02 09:14:24
IDJOB=2697824
IDBENCH=78898
IDSOLVER=1210
FILE ID=node135/2697824-1278054864
PBS_JOBID= 11198912
Free space on /tmp= 74132 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900670234--soft-33-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697824-1278054864/watcher-2697824-1278054864 -o /tmp/evaluation-result-2697824-1278054864/solver-2697824-1278054864 -C 1800 -W 2000 -M 3800  run HOME/instance-2697824-1278054864.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 74a38f74cac3b14e8d36cdbf48853bbd
RANDOM SEED=1244874401

node135.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7186.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.857
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30773556 kB
Buffers:        218484 kB
Cached:        1746228 kB
SwapCached:          0 kB
Active:         603804 kB
Inactive:      1423712 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30773556 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16924 kB
Writeback:           0 kB
AnonPages:       62328 kB
Mapped:          14428 kB
Slab:            86436 kB
PageTables:       4588 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   180276 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74100 MiB
End job on node135 at 2010-07-02 09:22:33