Trace number 2697362

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.45 507.148

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900577940--soft-66-100-0.wbo
MD5SUM95aa4a8f0430d7fc4dd41a3f578cfb3e
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark267.837
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints937
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 46894
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 46893
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 661
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697362-1278055256.wbo.
0.00/0.00	c Instance file HOME/instance-2697362-1278055256.wbo
0.00/0.00	c File size is 88056 bytes.
0.00/0.18	c Highest Coefficient sum: 661
0.00/0.18	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.19	c ============================[ Problem Statistics ]=============================
0.00/0.19	c |                                                                             |
0.00/0.19	c |  Parsing time:          0.19         s                                      |
0.00/0.19	c |  Number Variables:      1168                                                |
0.00/0.19	c |  Number Constraints:    2708                                                |
0.00/0.19	c ===============================================================================
1800.43/507.14	c Remaining time 1799.81 sec.
1800.43/507.14	c 
1800.43/507.14	c *** INTERRUPTED ***
1800.43/507.14	s UNKNOWN
1800.43/507.14	c Total CPU time              : 1784.43 s
1800.43/507.14	c Memory used           : 155.58 MB
1800.43/507.14	c CPU time              : 1784.43 s
1800.43/507.14	c 
1800.43/507.14	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697362-1278055256/watcher-2697362-1278055256 -o /tmp/evaluation-result-2697362-1278055256/solver-2697362-1278055256 -C 1800 -W 2000 -M 3800 run HOME/instance-2697362-1278055256.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.92 6.78 3.18 4/186 24462
/proc/meminfo: memFree=30950532/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0

[startup+0.0746729 s]
/proc/loadavg: 8.92 6.78 3.18 4/186 24462
/proc/meminfo: memFree=30950532/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.100683 s]
/proc/loadavg: 8.92 6.78 3.18 4/186 24462
/proc/meminfo: memFree=30950532/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.30056 s]
/proc/loadavg: 8.92 6.78 3.18 4/186 24462
/proc/meminfo: memFree=30950532/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.700464 s]
/proc/loadavg: 8.92 6.78 3.18 4/186 24462
/proc/meminfo: memFree=30950532/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50037 s]
/proc/loadavg: 8.92 6.78 3.18 3/181 24485
/proc/meminfo: memFree=30936932/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=95632 CPUtime=2.69
/proc/24463/stat : 24463 (pb_cplex) R 24462 24462 23735 0 -1 4202496 20831 0 0 0 263 6 0 0 17 0 1 0 146820823 97927168 10006 4089446400 134512640 146736320 4290591312 18446744073709551615 135526831 0 0 4096 16387 0 0 0 17 6 0 0 0
/proc/24463/statm: 23908 10006 1071 2985 0 20450 0
Current children cumulated CPU time (s) 2.69
Current children cumulated vsize (KiB) 104332

[startup+3.10098 s]
/proc/loadavg: 8.92 6.78 3.18 3/195 24534
/proc/meminfo: memFree=30944756/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=130448 CPUtime=4.84
/proc/24463/stat : 24463 (pb_cplex) R 24462 24462 23735 0 -1 4202496 29352 0 0 0 477 7 0 0 23 0 8 0 146820823 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 144797937 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/24463/statm: 32612 10610 1079 2985 0 29154 0
[pid=24463/tid=24507] ppid=24462 vsize=130448 CPUtime=0.12
/proc/24463/task/24507/stat : 24507 (pb_cplex) S 24462 24462 23735 0 -1 4202560 3 0 0 0 12 0 0 0 16 0 8 0 146820997 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 1 0 0 0
[pid=24463/tid=24508] ppid=24462 vsize=130448 CPUtime=0.1
/proc/24463/task/24508/stat : 24508 (pb_cplex) R 24462 24462 23735 0 -1 4202560 148 0 0 0 10 0 0 0 18 0 8 0 146820997 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=24463/tid=24509] ppid=24462 vsize=130448 CPUtime=0.1
/proc/24463/task/24509/stat : 24509 (pb_cplex) R 24462 24462 23735 0 -1 4202560 10 0 0 0 10 0 0 0 17 0 8 0 146820997 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 136007030 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=24463/tid=24510] ppid=24462 vsize=130448 CPUtime=0
/proc/24463/task/24510/stat : 24510 (pb_cplex) R 24462 24462 23735 0 -1 4202560 36 0 0 0 0 0 0 0 17 0 8 0 146820997 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=24463/tid=24511] ppid=24462 vsize=130448 CPUtime=0.01
/proc/24463/task/24511/stat : 24511 (pb_cplex) R 24462 24462 23735 0 -1 4202560 579 0 0 0 1 0 0 0 16 0 8 0 146820997 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 136403116 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=24463/tid=24512] ppid=24462 vsize=130448 CPUtime=0
/proc/24463/task/24512/stat : 24512 (pb_cplex) R 24462 24462 23735 0 -1 4202560 33 0 0 0 0 0 0 0 16 0 8 0 146820997 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=24463/tid=24513] ppid=24462 vsize=130448 CPUtime=0
/proc/24463/task/24513/stat : 24513 (pb_cplex) R 24462 24462 23735 0 -1 4202560 34 0 0 0 0 0 0 0 16 0 8 0 146820997 133578752 10610 4089446400 134512640 146736320 4290591312 18446744073709551615 139028308 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4.84
Current children cumulated vsize (KiB) 139148

[startup+6.30038 s]
/proc/loadavg: 8.53 6.74 3.18 3/195 24534
/proc/meminfo: memFree=30927272/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=132088 CPUtime=16.29
/proc/24463/stat : 24463 (pb_cplex) S 24462 24462 23735 0 -1 4202496 35538 0 0 0 1610 19 0 0 15 0 8 0 146820823 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/24463/statm: 33022 11140 1079 2985 0 29564 0
[pid=24463/tid=24507] ppid=24462 vsize=132088 CPUtime=1.4
/proc/24463/task/24507/stat : 24507 (pb_cplex) S 24462 24462 23735 0 -1 4202560 685 0 0 0 139 1 0 0 15 0 8 0 146820997 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 5 0 0 0
[pid=24463/tid=24508] ppid=24462 vsize=132088 CPUtime=1.58
/proc/24463/task/24508/stat : 24508 (pb_cplex) S 24462 24462 23735 0 -1 4202560 525 0 0 0 157 1 0 0 15 0 8 0 146820997 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 5 0 0 0
[pid=24463/tid=24509] ppid=24462 vsize=132088 CPUtime=1.7
/proc/24463/task/24509/stat : 24509 (pb_cplex) S 24462 24462 23735 0 -1 4202560 304 0 0 0 169 1 0 0 15 0 8 0 146820997 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 4 0 0 0
[pid=24463/tid=24510] ppid=24462 vsize=132088 CPUtime=1.22
/proc/24463/task/24510/stat : 24510 (pb_cplex) S 24462 24462 23735 0 -1 4202560 782 0 0 0 121 1 0 0 15 0 8 0 146820997 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 2 0 0 0
[pid=24463/tid=24511] ppid=24462 vsize=132088 CPUtime=1.52
/proc/24463/task/24511/stat : 24511 (pb_cplex) S 24462 24462 23735 0 -1 4202560 1851 0 0 0 150 2 0 0 15 0 8 0 146820997 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 7 0 0 0
[pid=24463/tid=24512] ppid=24462 vsize=132088 CPUtime=1.44
/proc/24463/task/24512/stat : 24512 (pb_cplex) S 24462 24462 23735 0 -1 4202560 280 0 0 0 143 1 0 0 15 0 8 0 146820997 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24513] ppid=24462 vsize=132088 CPUtime=1.6
/proc/24463/task/24513/stat : 24513 (pb_cplex) R 24462 24462 23735 0 -1 4202560 285 0 0 0 159 1 0 0 15 0 8 0 146820997 135258112 11140 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 16.29
Current children cumulated vsize (KiB) 140788

[startup+12.7012 s]
/proc/loadavg: 8.08 6.67 3.18 9/195 24534
/proc/meminfo: memFree=30911904/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=134252 CPUtime=33.74
/proc/24463/stat : 24463 (pb_cplex) S 24462 24462 23735 0 -1 4202496 45652 0 0 0 3338 36 0 0 15 0 8 0 146820823 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/24463/statm: 33563 11724 1079 2985 0 30105 0
[pid=24463/tid=24507] ppid=24462 vsize=134252 CPUtime=3.53
/proc/24463/task/24507/stat : 24507 (pb_cplex) S 24462 24462 23735 0 -1 4202560 797 0 0 0 349 4 0 0 15 0 8 0 146820997 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 2 0 0 0
[pid=24463/tid=24508] ppid=24462 vsize=134252 CPUtime=3.69
/proc/24463/task/24508/stat : 24508 (pb_cplex) S 24462 24462 23735 0 -1 4202560 1128 0 0 0 366 3 0 0 15 0 8 0 146820997 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24509] ppid=24462 vsize=134252 CPUtime=3.86
/proc/24463/task/24509/stat : 24509 (pb_cplex) S 24462 24462 23735 0 -1 4202560 2887 0 0 0 382 4 0 0 15 0 8 0 146820997 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24510] ppid=24462 vsize=134252 CPUtime=3.39
/proc/24463/task/24510/stat : 24510 (pb_cplex) S 24462 24462 23735 0 -1 4202560 1230 0 0 0 336 3 0 0 15 0 8 0 146820997 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 1 0 0 0
[pid=24463/tid=24511] ppid=24462 vsize=134252 CPUtime=3.66
/proc/24463/task/24511/stat : 24511 (pb_cplex) S 24462 24462 23735 0 -1 4202560 2160 0 0 0 363 3 0 0 15 0 8 0 146820997 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 5 0 0 0
[pid=24463/tid=24512] ppid=24462 vsize=134252 CPUtime=3.5
/proc/24463/task/24512/stat : 24512 (pb_cplex) S 24462 24462 23735 0 -1 4202560 2660 0 0 0 347 3 0 0 15 0 8 0 146820997 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 5 0 0 0
[pid=24463/tid=24513] ppid=24462 vsize=134252 CPUtime=3.79
/proc/24463/task/24513/stat : 24513 (pb_cplex) R 24462 24462 23735 0 -1 4202560 1231 0 0 0 375 4 0 0 15 0 8 0 146820997 137474048 11724 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 33.74
Current children cumulated vsize (KiB) 142952

[startup+25.5007 s]
/proc/loadavg: 8.22 6.77 3.27 11/195 24555
/proc/meminfo: memFree=30911656/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=135692 CPUtime=68.75
/proc/24463/stat : 24463 (pb_cplex) R 24462 24462 23735 0 -1 4202496 67038 0 0 0 6805 70 0 0 22 0 8 0 146820823 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 139043558 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/24463/statm: 33923 12358 1082 2985 0 30465 0
[pid=24463/tid=24549] ppid=24462 vsize=135692 CPUtime=0.99
/proc/24463/task/24549/stat : 24549 (pb_cplex) S 24462 24462 23735 0 -1 4202560 50 0 0 0 99 0 0 0 19 0 8 0 146823085 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 2 0 0 0
[pid=24463/tid=24550] ppid=24462 vsize=135692 CPUtime=0.99
/proc/24463/task/24550/stat : 24550 (pb_cplex) R 24462 24462 23735 0 -1 4202560 57 0 0 0 99 0 0 0 18 0 8 0 146823085 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 136382629 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=24463/tid=24551] ppid=24462 vsize=135692 CPUtime=1.06
/proc/24463/task/24551/stat : 24551 (pb_cplex) R 24462 24462 23735 0 -1 4202560 560 0 0 0 106 0 0 0 19 0 8 0 146823085 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 136382490 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=24463/tid=24552] ppid=24462 vsize=135692 CPUtime=1.4
/proc/24463/task/24552/stat : 24552 (pb_cplex) R 24462 24462 23735 0 -1 4202560 2687 0 0 0 140 0 0 0 19 0 8 0 146823085 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=24463/tid=24553] ppid=24462 vsize=135692 CPUtime=1.05
/proc/24463/task/24553/stat : 24553 (pb_cplex) R 24462 24462 23735 0 -1 4202560 276 0 0 0 105 0 0 0 18 0 8 0 146823085 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 136004875 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=24463/tid=24554] ppid=24462 vsize=135692 CPUtime=0.89
/proc/24463/task/24554/stat : 24554 (pb_cplex) R 24462 24462 23735 0 -1 4202560 360 0 0 0 89 0 0 0 18 0 8 0 146823085 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 136388081 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=24463/tid=24555] ppid=24462 vsize=135692 CPUtime=0.67
/proc/24463/task/24555/stat : 24555 (pb_cplex) R 24462 24462 23735 0 -1 4202560 451 0 0 0 67 0 0 0 18 0 8 0 146823085 138948608 12358 4089446400 134512640 146736320 4290591312 18446744073709551615 136006079 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 68.75
Current children cumulated vsize (KiB) 144392

[startup+51.1133 s]
/proc/loadavg: 8.56 6.96 3.43 11/195 24556
/proc/meminfo: memFree=30898636/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=145024 CPUtime=164.12
/proc/24463/stat : 24463 (pb_cplex) R 24462 24462 23735 0 -1 4202496 118120 0 0 0 16260 152 0 0 15 0 8 0 146820823 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/24463/statm: 36256 14261 1082 2985 0 32798 0
[pid=24463/tid=24549] ppid=24462 vsize=145024 CPUtime=12.3
/proc/24463/task/24549/stat : 24549 (pb_cplex) R 24462 24462 23735 0 -1 4202560 5809 0 0 0 1219 11 0 0 15 0 8 0 146823085 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=24463/tid=24550] ppid=24462 vsize=145024 CPUtime=13.16
/proc/24463/task/24550/stat : 24550 (pb_cplex) R 24462 24462 23735 0 -1 4202560 6692 0 0 0 1307 9 0 0 15 0 8 0 146823085 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=24463/tid=24551] ppid=24462 vsize=145024 CPUtime=13.25
/proc/24463/task/24551/stat : 24551 (pb_cplex) R 24462 24462 23735 0 -1 4202560 5126 0 0 0 1315 10 0 0 15 0 8 0 146823085 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=24463/tid=24552] ppid=24462 vsize=145024 CPUtime=13.11
/proc/24463/task/24552/stat : 24552 (pb_cplex) S 24462 24462 23735 0 -1 4202560 7002 0 0 0 1302 9 0 0 15 0 8 0 146823085 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24553] ppid=24462 vsize=145024 CPUtime=12.44
/proc/24463/task/24553/stat : 24553 (pb_cplex) S 24462 24462 23735 0 -1 4202560 5868 0 0 0 1235 9 0 0 15 0 8 0 146823085 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24554] ppid=24462 vsize=145024 CPUtime=13.44
/proc/24463/task/24554/stat : 24554 (pb_cplex) S 24462 24462 23735 0 -1 4202560 4960 0 0 0 1334 10 0 0 15 0 8 0 146823085 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24555] ppid=24462 vsize=145024 CPUtime=12.17
/proc/24463/task/24555/stat : 24555 (pb_cplex) S 24462 24462 23735 0 -1 4202560 7499 0 0 0 1205 12 0 0 16 0 8 0 146823085 148504576 14261 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 164.12
Current children cumulated vsize (KiB) 153724

[startup+102.315 s]
/proc/loadavg: 8.96 7.35 3.74 9/195 24557
/proc/meminfo: memFree=30897064/32951124 swapFree=67111528/67111528

################
# More data... #
################

/proc/24463/task/24554/stat : 24554 (pb_cplex) S 24462 24462 23735 0 -1 4202560 12405 0 0 0 8900 81 0 0 15 0 8 0 146823085 148504576 14692 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24555] ppid=24462 vsize=145024 CPUtime=89.43
/proc/24463/task/24555/stat : 24555 (pb_cplex) S 24462 24462 23735 0 -1 4202560 14625 0 0 0 8861 82 0 0 15 0 8 0 146823085 148504576 14692 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 782.88
Current children cumulated vsize (KiB) 153724

[startup+282.318 s]
/proc/loadavg: 9.02 8.23 4.73 7/195 24626
/proc/meminfo: memFree=30898792/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=145024 CPUtime=1014.03
/proc/24463/stat : 24463 (pb_cplex) S 24462 24462 23735 0 -1 4202496 173135 0 0 0 100482 921 0 0 16 0 8 0 146820823 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/24463/statm: 36256 14652 1101 2985 0 32798 0
[pid=24463/tid=24549] ppid=24462 vsize=145024 CPUtime=117.14
/proc/24463/task/24549/stat : 24549 (pb_cplex) S 24462 24462 23735 0 -1 4202560 11189 0 0 0 11607 107 0 0 16 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 0 0 0 0
[pid=24463/tid=24550] ppid=24462 vsize=145024 CPUtime=118.04
/proc/24463/task/24550/stat : 24550 (pb_cplex) S 24462 24462 23735 0 -1 4202560 12806 0 0 0 11699 105 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24551] ppid=24462 vsize=145024 CPUtime=118.16
/proc/24463/task/24551/stat : 24551 (pb_cplex) S 24462 24462 23735 0 -1 4202560 11578 0 0 0 11713 103 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 7 0 0 0
[pid=24463/tid=24552] ppid=24462 vsize=145024 CPUtime=118.24
/proc/24463/task/24552/stat : 24552 (pb_cplex) S 24462 24462 23735 0 -1 4202560 13543 0 0 0 11721 103 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 2 0 0 0
[pid=24463/tid=24553] ppid=24462 vsize=145024 CPUtime=117.2
/proc/24463/task/24553/stat : 24553 (pb_cplex) R 24462 24462 23735 0 -1 4202560 13381 0 0 0 11615 105 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=24463/tid=24554] ppid=24462 vsize=145024 CPUtime=118.43
/proc/24463/task/24554/stat : 24554 (pb_cplex) S 24462 24462 23735 0 -1 4202560 13615 0 0 0 11733 110 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 4 0 0 0
[pid=24463/tid=24555] ppid=24462 vsize=145024 CPUtime=118.3
/proc/24463/task/24555/stat : 24555 (pb_cplex) S 24462 24462 23735 0 -1 4202560 15301 0 0 0 11719 111 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1014.03
Current children cumulated vsize (KiB) 153724

[startup+342.309 s]
/proc/loadavg: 8.73 8.24 4.95 12/195 24641
/proc/meminfo: memFree=30899020/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=145024 CPUtime=1218.77
/proc/24463/stat : 24463 (pb_cplex) S 24462 24462 23735 0 -1 4202496 175714 0 0 0 120767 1110 0 0 16 0 8 0 146820823 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/24463/statm: 36256 14652 1101 2985 0 32798 0
[pid=24463/tid=24549] ppid=24462 vsize=145024 CPUtime=142.63
/proc/24463/task/24549/stat : 24549 (pb_cplex) S 24462 24462 23735 0 -1 4202560 11462 0 0 0 14133 130 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 7 0 0 0
[pid=24463/tid=24550] ppid=24462 vsize=145024 CPUtime=143.23
/proc/24463/task/24550/stat : 24550 (pb_cplex) R 24462 24462 23735 0 -1 4202560 13020 0 0 0 14195 128 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 135450791 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=24463/tid=24551] ppid=24462 vsize=145024 CPUtime=143.54
/proc/24463/task/24551/stat : 24551 (pb_cplex) R 24462 24462 23735 0 -1 4202560 11946 0 0 0 14226 128 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=24463/tid=24552] ppid=24462 vsize=145024 CPUtime=143.44
/proc/24463/task/24552/stat : 24552 (pb_cplex) R 24462 24462 23735 0 -1 4202560 13840 0 0 0 14217 127 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 135450630 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=24463/tid=24553] ppid=24462 vsize=145024 CPUtime=142.23
/proc/24463/task/24553/stat : 24553 (pb_cplex) R 24462 24462 23735 0 -1 4202560 13515 0 0 0 14096 127 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=24463/tid=24554] ppid=24462 vsize=145024 CPUtime=143.37
/proc/24463/task/24554/stat : 24554 (pb_cplex) R 24462 24462 23735 0 -1 4202560 14350 0 0 0 14204 133 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=24463/tid=24555] ppid=24462 vsize=145024 CPUtime=143.61
/proc/24463/task/24555/stat : 24555 (pb_cplex) R 24462 24462 23735 0 -1 4202560 15530 0 0 0 14226 135 0 0 15 0 8 0 146823085 148504576 14652 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1218.77
Current children cumulated vsize (KiB) 153724

[startup+402.399 s]
/proc/loadavg: 9.29 8.47 5.23 16/195 24656
/proc/meminfo: memFree=30882000/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=160336 CPUtime=1400.82
/proc/24463/stat : 24463 (pb_cplex) R 24462 24462 23735 0 -1 4202496 206038 0 0 0 138811 1271 0 0 16 0 8 0 146820823 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/24463/statm: 40084 17729 1101 2985 0 36626 0
[pid=24463/tid=24650] ppid=24462 vsize=160336 CPUtime=2.63
/proc/24463/task/24650/stat : 24650 (pb_cplex) R 24462 24462 23735 0 -1 4202560 1749 0 0 0 263 0 0 0 16 0 8 0 146859788 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 137380888 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=24463/tid=24651] ppid=24462 vsize=160336 CPUtime=3.11
/proc/24463/task/24651/stat : 24651 (pb_cplex) R 24462 24462 23735 0 -1 4202560 1715 0 0 0 311 0 0 0 16 0 8 0 146859788 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=24463/tid=24652] ppid=24462 vsize=160336 CPUtime=2.56
/proc/24463/task/24652/stat : 24652 (pb_cplex) R 24462 24462 23735 0 -1 4202560 706 0 0 0 255 1 0 0 15 0 8 0 146859788 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=24463/tid=24653] ppid=24462 vsize=160336 CPUtime=2.44
/proc/24463/task/24653/stat : 24653 (pb_cplex) R 24462 24462 23735 0 -1 4202560 1737 0 0 0 243 1 0 0 15 0 8 0 146859788 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 136017519 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=24463/tid=24654] ppid=24462 vsize=160336 CPUtime=3.75
/proc/24463/task/24654/stat : 24654 (pb_cplex) R 24462 24462 23735 0 -1 4202560 3028 0 0 0 373 2 0 0 15 0 8 0 146859788 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=24463/tid=24655] ppid=24462 vsize=160336 CPUtime=6.31
/proc/24463/task/24655/stat : 24655 (pb_cplex) R 24462 24462 23735 0 -1 4202560 1146 0 0 0 630 1 0 0 16 0 8 0 146859788 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 135450545 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=24463/tid=24656] ppid=24462 vsize=160336 CPUtime=5.61
/proc/24463/task/24656/stat : 24656 (pb_cplex) R 24462 24462 23735 0 -1 4202560 2765 0 0 0 560 1 0 0 15 0 8 0 146859788 164184064 17729 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1400.82
Current children cumulated vsize (KiB) 169036

[startup+462.341 s]
/proc/loadavg: 9.42 8.66 5.49 9/195 24658
/proc/meminfo: memFree=30882544/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=159312 CPUtime=1638.68
/proc/24463/stat : 24463 (pb_cplex) R 24462 24462 23735 0 -1 4202496 246945 0 0 0 162403 1465 0 0 16 0 8 0 146820823 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/24463/statm: 39828 17511 1101 2985 0 36370 0
[pid=24463/tid=24650] ppid=24462 vsize=159312 CPUtime=32.25
/proc/24463/task/24650/stat : 24650 (pb_cplex) R 24462 24462 23735 0 -1 4202560 5790 0 0 0 3201 24 0 0 15 0 8 0 146859788 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=24463/tid=24651] ppid=24462 vsize=159312 CPUtime=32.42
/proc/24463/task/24651/stat : 24651 (pb_cplex) R 24462 24462 23735 0 -1 4202560 7887 0 0 0 3220 22 0 0 15 0 8 0 146859788 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=24463/tid=24652] ppid=24462 vsize=159312 CPUtime=31.8
/proc/24463/task/24652/stat : 24652 (pb_cplex) R 24462 24462 23735 0 -1 4202560 4427 0 0 0 3155 25 0 0 15 0 8 0 146859788 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=24463/tid=24653] ppid=24462 vsize=159312 CPUtime=31.79
/proc/24463/task/24653/stat : 24653 (pb_cplex) R 24462 24462 23735 0 -1 4202560 5131 0 0 0 3151 28 0 0 15 0 8 0 146859788 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 135526863 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=24463/tid=24654] ppid=24462 vsize=159312 CPUtime=33.16
/proc/24463/task/24654/stat : 24654 (pb_cplex) R 24462 24462 23735 0 -1 4202560 8871 0 0 0 3291 25 0 0 15 0 8 0 146859788 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=24463/tid=24655] ppid=24462 vsize=159312 CPUtime=35.81
/proc/24463/task/24655/stat : 24655 (pb_cplex) R 24462 24462 23735 0 -1 4202560 6003 0 0 0 3554 27 0 0 15 0 8 0 146859788 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=24463/tid=24656] ppid=24462 vsize=159312 CPUtime=35.09
/proc/24463/task/24656/stat : 24656 (pb_cplex) R 24462 24462 23735 0 -1 4202560 11458 0 0 0 3481 28 0 0 15 0 8 0 146859788 163135488 17511 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1638.68
Current children cumulated vsize (KiB) 168012



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+507.133 s]
/proc/loadavg: 8.99 8.65 5.64 10/195 24660
/proc/meminfo: memFree=30883568/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=159312 CPUtime=1800.43
/proc/24463/stat : 24463 (pb_cplex) S 24462 24462 23735 0 -1 4202496 269804 0 0 0 178443 1600 0 0 16 0 8 0 146820823 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/24463/statm: 39828 17649 1101 2985 0 36370 0
[pid=24463/tid=24650] ppid=24462 vsize=159312 CPUtime=52.19
/proc/24463/task/24650/stat : 24650 (pb_cplex) S 24462 24462 23735 0 -1 4202560 7527 0 0 0 5179 40 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24651] ppid=24462 vsize=159312 CPUtime=52.55
/proc/24463/task/24651/stat : 24651 (pb_cplex) S 24462 24462 23735 0 -1 4202560 10502 0 0 0 5215 40 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24652] ppid=24462 vsize=159312 CPUtime=51.82
/proc/24463/task/24652/stat : 24652 (pb_cplex) S 24462 24462 23735 0 -1 4202560 7375 0 0 0 5139 43 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24653] ppid=24462 vsize=159312 CPUtime=51.76
/proc/24463/task/24653/stat : 24653 (pb_cplex) R 24462 24462 23735 0 -1 4202560 8149 0 0 0 5130 46 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=24463/tid=24654] ppid=24462 vsize=159312 CPUtime=53.06
/proc/24463/task/24654/stat : 24654 (pb_cplex) S 24462 24462 23735 0 -1 4202560 10891 0 0 0 5264 42 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24655] ppid=24462 vsize=159312 CPUtime=55.67
/proc/24463/task/24655/stat : 24655 (pb_cplex) S 24462 24462 23735 0 -1 4202560 8363 0 0 0 5524 43 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24656] ppid=24462 vsize=159312 CPUtime=55.04
/proc/24463/task/24656/stat : 24656 (pb_cplex) S 24462 24462 23735 0 -1 4202560 14972 0 0 0 5457 47 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1800.43
Current children cumulated vsize (KiB) 168012

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+507.133 s]
/proc/loadavg: 8.99 8.65 5.64 10/195 24660
/proc/meminfo: memFree=30883568/32951124 swapFree=67111528/67111528
[pid=24462] ppid=24460 vsize=8700 CPUtime=0
/proc/24462/stat : 24462 (run) S 24460 24462 23735 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146820823 8908800 248 4089446400 4194304 4922060 140733801467600 18446744073709551615 264726224005 0 65536 4100 65538 18446744071562232103 0 0 17 4 0 0 0
/proc/24462/statm: 2175 248 208 178 0 68 0
[pid=24463] ppid=24462 vsize=159312 CPUtime=1800.43
/proc/24463/stat : 24463 (pb_cplex) S 24462 24462 23735 0 -1 4202496 269804 0 0 0 178443 1600 0 0 16 0 8 0 146820823 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/24463/statm: 39828 17649 1101 2985 0 36370 0
[pid=24463/tid=24650] ppid=24462 vsize=159312 CPUtime=52.19
/proc/24463/task/24650/stat : 24650 (pb_cplex) S 24462 24462 23735 0 -1 4202560 7527 0 0 0 5179 40 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24651] ppid=24462 vsize=159312 CPUtime=52.55
/proc/24463/task/24651/stat : 24651 (pb_cplex) S 24462 24462 23735 0 -1 4202560 10502 0 0 0 5215 40 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24652] ppid=24462 vsize=159312 CPUtime=51.82
/proc/24463/task/24652/stat : 24652 (pb_cplex) S 24462 24462 23735 0 -1 4202560 7375 0 0 0 5139 43 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24653] ppid=24462 vsize=159312 CPUtime=51.76
/proc/24463/task/24653/stat : 24653 (pb_cplex) R 24462 24462 23735 0 -1 4202560 8149 0 0 0 5130 46 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=24463/tid=24654] ppid=24462 vsize=159312 CPUtime=53.06
/proc/24463/task/24654/stat : 24654 (pb_cplex) S 24462 24462 23735 0 -1 4202560 10891 0 0 0 5264 42 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 6 0 0 0
[pid=24463/tid=24655] ppid=24462 vsize=159312 CPUtime=55.67
/proc/24463/task/24655/stat : 24655 (pb_cplex) S 24462 24462 23735 0 -1 4202560 8363 0 0 0 5524 43 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 3 0 0 0
[pid=24463/tid=24656] ppid=24462 vsize=159312 CPUtime=55.04
/proc/24463/task/24656/stat : 24656 (pb_cplex) S 24462 24462 23735 0 -1 4202560 14972 0 0 0 5457 47 0 0 15 0 8 0 146859788 163135488 17649 4089446400 134512640 146736320 4290591312 18446744073709551615 144580073 0 2 4096 16387 18446604469812544448 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1800.43
Current children cumulated vsize (KiB) 168012

Child status: 0
Real time (s): 507.148
CPU time (s): 1800.45
CPU user time (s): 1784.43
CPU system time (s): 16.0126
CPU usage (%): 355.014
Max. virtual memory (cumulated for all children) (KiB): 174156

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1784.43
system time used= 16.0126
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 270176
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 355123
involuntary context switches= 29915

runsolver used 0.931858 second user time and 1.9487 second system time

The end

Launcher Data 

Begin job on node141 at 2010-07-02 09:20:56
IDJOB=2697362
IDBENCH=78990
IDSOLVER=1210
FILE ID=node141/2697362-1278055256
PBS_JOBID= 11198900
Free space on /tmp= 74164 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900577940--soft-66-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697362-1278055256/watcher-2697362-1278055256 -o /tmp/evaluation-result-2697362-1278055256/solver-2697362-1278055256 -C 1800 -W 2000 -M 3800  run HOME/instance-2697362-1278055256.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 95aa4a8f0430d7fc4dd41a3f578cfb3e
RANDOM SEED=1593565720

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.57
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.788
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30951184 kB
Buffers:        224432 kB
Cached:        1499736 kB
SwapCached:          0 kB
Active:         763308 kB
Inactive:      1086848 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30951184 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           29276 kB
Writeback:           0 kB
AnonPages:      125992 kB
Mapped:          18856 kB
Slab:            84676 kB
PageTables:       4936 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   348500 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74148 MiB
End job on node141 at 2010-07-02 09:29:23