Trace number 2697811

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.44 481.841

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900584219--soft-33-100-0.wbo
MD5SUM8d817301a4b1c2c9b4fea9d5d7fdc9d1
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark263.875
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints1868
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 94981
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 94980
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 689
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697811-1278054863.wbo.
0.00/0.00	c Instance file HOME/instance-2697811-1278054863.wbo
0.00/0.00	c File size is 92616 bytes.
0.00/0.69	c Highest Coefficient sum: 689
0.00/0.69	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.70	c ============================[ Problem Statistics ]=============================
0.00/0.70	c |                                                                             |
0.00/0.70	c |  Parsing time:          0.70         s                                      |
0.00/0.70	c |  Number Variables:      2099                                                |
0.00/0.70	c |  Number Constraints:    2708                                                |
0.00/0.70	c ===============================================================================
1800.44/481.82	c Remaining time 1799.30 sec.
1800.44/481.82	c 
1800.44/481.82	c *** INTERRUPTED ***
1800.44/481.82	s UNKNOWN
1800.44/481.82	c Total CPU time              : 1783.34 s
1800.44/481.82	c Memory used           : 150.09 MB
1800.44/481.82	c CPU time              : 1783.34 s
1800.44/481.82	c 
1800.44/481.82	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697811-1278054863/watcher-2697811-1278054863 -o /tmp/evaluation-result-2697811-1278054863/solver-2697811-1278054863 -C 1800 -W 2000 -M 3800 run HOME/instance-2697811-1278054863.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 832
/proc/meminfo: memFree=30955396/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0

[startup+0.0728011 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 832
/proc/meminfo: memFree=30955396/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.10076 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 832
/proc/meminfo: memFree=30955396/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300738 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 832
/proc/meminfo: memFree=30955396/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.700668 s]
/proc/loadavg: 0.00 0.00 0.00 2/173 832
/proc/meminfo: memFree=30955396/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50045 s]
/proc/loadavg: 0.16 0.03 0.01 3/181 867
/proc/meminfo: memFree=30905484/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=137416 CPUtime=2.56
/proc/833/stat : 833 (pb_cplex) R 832 832 685 0 -1 4202496 20615 0 0 0 250 6 0 0 16 0 8 0 179125889 140713984 11756 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/833/statm: 34354 11756 1082 2985 0 30896 0
Current children cumulated CPU time (s) 2.56
Current children cumulated vsize (KiB) 146116

[startup+3.10024 s]
/proc/loadavg: 0.16 0.03 0.01 3/181 910
/proc/meminfo: memFree=30867416/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=126020 CPUtime=4.7
/proc/833/stat : 833 (pb_cplex) R 832 832 685 0 -1 4202496 25193 0 0 0 462 8 0 0 21 0 8 0 179125889 129044480 9498 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/833/statm: 31505 9498 1083 2985 0 28047 0
Current children cumulated CPU time (s) 4.7
Current children cumulated vsize (KiB) 134720

[startup+6.30071 s]
/proc/loadavg: 0.95 0.20 0.06 3/195 961
/proc/meminfo: memFree=30848464/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=134808 CPUtime=19.26
/proc/833/stat : 833 (pb_cplex) S 832 832 685 0 -1 4202496 35323 0 0 0 1903 23 0 0 16 0 8 0 179125889 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/833/statm: 33702 11902 1083 2985 0 30244 0
[pid=833/tid=926] ppid=832 vsize=134808 CPUtime=1.65
/proc/833/task/926/stat : 926 (pb_cplex) S 832 832 685 0 -1 4202560 31 0 0 0 164 1 0 0 15 0 8 0 179126121 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 4 0 0 0
[pid=833/tid=927] ppid=832 vsize=134808 CPUtime=1.66
/proc/833/task/927/stat : 927 (pb_cplex) S 832 832 685 0 -1 4202560 243 0 0 0 164 2 0 0 16 0 8 0 179126121 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 7 0 0 0
[pid=833/tid=928] ppid=832 vsize=134808 CPUtime=1.79
/proc/833/task/928/stat : 928 (pb_cplex) S 832 832 685 0 -1 4202560 3703 0 0 0 177 2 0 0 15 0 8 0 179126121 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 3 0 0 0
[pid=833/tid=929] ppid=832 vsize=134808 CPUtime=1.89
/proc/833/task/929/stat : 929 (pb_cplex) S 832 832 685 0 -1 4202560 1006 0 0 0 188 1 0 0 15 0 8 0 179126121 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 6 0 0 0
[pid=833/tid=930] ppid=832 vsize=134808 CPUtime=1.68
/proc/833/task/930/stat : 930 (pb_cplex) S 832 832 685 0 -1 4202560 137 0 0 0 166 2 0 0 15 0 8 0 179126121 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=931] ppid=832 vsize=134808 CPUtime=1.82
/proc/833/task/931/stat : 931 (pb_cplex) S 832 832 685 0 -1 4202560 416 0 0 0 181 1 0 0 15 0 8 0 179126121 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 2 0 0 0
[pid=833/tid=932] ppid=832 vsize=134808 CPUtime=2.16
/proc/833/task/932/stat : 932 (pb_cplex) R 832 832 685 0 -1 4202560 1382 0 0 0 214 2 0 0 15 0 8 0 179126121 138043392 11902 4089446400 134512640 146736320 4288283536 18446744073709551615 139043591 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 19.26
Current children cumulated vsize (KiB) 143508

[startup+12.7016 s]
/proc/loadavg: 1.43 0.31 0.10 10/195 961
/proc/meminfo: memFree=30834648/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=137672 CPUtime=41.86
/proc/833/stat : 833 (pb_cplex) S 832 832 685 0 -1 4202496 42966 0 0 0 4140 46 0 0 16 0 8 0 179125889 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/833/statm: 34418 12606 1083 2985 0 30960 0
[pid=833/tid=926] ppid=832 vsize=137672 CPUtime=4.44
/proc/833/task/926/stat : 926 (pb_cplex) S 832 832 685 0 -1 4202560 414 0 0 0 440 4 0 0 16 0 8 0 179126121 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 2 0 0 0
[pid=833/tid=927] ppid=832 vsize=137672 CPUtime=4.44
/proc/833/task/927/stat : 927 (pb_cplex) S 832 832 685 0 -1 4202560 1245 0 0 0 439 5 0 0 15 0 8 0 179126121 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 7 0 0 0
[pid=833/tid=928] ppid=832 vsize=137672 CPUtime=4.45
/proc/833/task/928/stat : 928 (pb_cplex) S 832 832 685 0 -1 4202560 5104 0 0 0 441 4 0 0 15 0 8 0 179126121 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=929] ppid=832 vsize=137672 CPUtime=4.55
/proc/833/task/929/stat : 929 (pb_cplex) S 832 832 685 0 -1 4202560 1060 0 0 0 453 2 0 0 15 0 8 0 179126121 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=930] ppid=832 vsize=137672 CPUtime=4.54
/proc/833/task/930/stat : 930 (pb_cplex) R 832 832 685 0 -1 4202560 1498 0 0 0 449 5 0 0 15 0 8 0 179126121 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=833/tid=931] ppid=832 vsize=137672 CPUtime=4.6
/proc/833/task/931/stat : 931 (pb_cplex) S 832 832 685 0 -1 4202560 1243 0 0 0 457 3 0 0 15 0 8 0 179126121 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=932] ppid=832 vsize=137672 CPUtime=4.94
/proc/833/task/932/stat : 932 (pb_cplex) S 832 832 685 0 -1 4202560 3169 0 0 0 489 5 0 0 15 0 8 0 179126121 140976128 12606 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 41.86
Current children cumulated vsize (KiB) 146372

[startup+25.5004 s]
/proc/loadavg: 2.60 0.60 0.20 11/195 975
/proc/meminfo: memFree=30814064/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=138172 CPUtime=75.58
/proc/833/stat : 833 (pb_cplex) S 832 832 685 0 -1 4202496 56148 0 0 0 7474 84 0 0 22 0 8 0 179125889 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/833/statm: 34543 12063 1086 2985 0 31085 0
[pid=833/tid=969] ppid=832 vsize=138172 CPUtime=1.02
/proc/833/task/969/stat : 969 (pb_cplex) S 832 832 685 0 -1 4202560 45 0 0 0 102 0 0 0 19 0 8 0 179127922 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=970] ppid=832 vsize=138172 CPUtime=0.74
/proc/833/task/970/stat : 970 (pb_cplex) R 832 832 685 0 -1 4202560 330 0 0 0 74 0 0 0 18 0 8 0 179127922 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 137345137 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=833/tid=971] ppid=832 vsize=138172 CPUtime=1.11
/proc/833/task/971/stat : 971 (pb_cplex) S 832 832 685 0 -1 4202560 130 0 0 0 111 0 0 0 18 0 8 0 179127922 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=972] ppid=832 vsize=138172 CPUtime=0.02
/proc/833/task/972/stat : 972 (pb_cplex) S 832 832 685 0 -1 4202560 316 0 0 0 2 0 0 0 16 0 8 0 179127922 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 2 0 0 0
[pid=833/tid=973] ppid=832 vsize=138172 CPUtime=1.26
/proc/833/task/973/stat : 973 (pb_cplex) S 832 832 685 0 -1 4202560 743 0 0 0 126 0 0 0 17 0 8 0 179127922 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=974] ppid=832 vsize=138172 CPUtime=1.99
/proc/833/task/974/stat : 974 (pb_cplex) R 832 832 685 0 -1 4202560 182 0 0 0 199 0 0 0 18 0 8 0 179127922 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 136017757 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=833/tid=975] ppid=832 vsize=138172 CPUtime=0.89
/proc/833/task/975/stat : 975 (pb_cplex) S 832 832 685 0 -1 4202560 732 0 0 0 89 0 0 0 17 0 8 0 179127922 141488128 12063 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 75.58
Current children cumulated vsize (KiB) 146872

[startup+51.1061 s]
/proc/loadavg: 4.45 1.24 0.42 10/195 997
/proc/meminfo: memFree=30824852/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=144456 CPUtime=195.03
/proc/833/stat : 833 (pb_cplex) S 832 832 685 0 -1 4202496 102709 0 0 0 19306 197 0 0 15 0 8 0 179125889 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/833/statm: 36114 13490 1086 2985 0 32656 0
[pid=833/tid=969] ppid=832 vsize=144456 CPUtime=15.75
/proc/833/task/969/stat : 969 (pb_cplex) R 832 832 685 0 -1 4202560 5181 0 0 0 1562 13 0 0 15 0 8 0 179127922 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=833/tid=970] ppid=832 vsize=144456 CPUtime=15.97
/proc/833/task/970/stat : 970 (pb_cplex) S 832 832 685 0 -1 4202560 1818 0 0 0 1586 11 0 0 15 0 8 0 179127922 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 3 0 0 0
[pid=833/tid=971] ppid=832 vsize=144456 CPUtime=15.77
/proc/833/task/971/stat : 971 (pb_cplex) S 832 832 685 0 -1 4202560 11757 0 0 0 1564 13 0 0 15 0 8 0 179127922 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=972] ppid=832 vsize=144456 CPUtime=15.1
/proc/833/task/972/stat : 972 (pb_cplex) S 832 832 685 0 -1 4202560 5159 0 0 0 1496 14 0 0 15 0 8 0 179127922 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=973] ppid=832 vsize=144456 CPUtime=15.95
/proc/833/task/973/stat : 973 (pb_cplex) S 832 832 685 0 -1 4202560 5439 0 0 0 1582 13 0 0 15 0 8 0 179127922 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 2 0 0 0
[pid=833/tid=974] ppid=832 vsize=144456 CPUtime=16.79
/proc/833/task/974/stat : 974 (pb_cplex) S 832 832 685 0 -1 4202560 7946 0 0 0 1666 13 0 0 15 0 8 0 179127922 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 5 0 0 0
[pid=833/tid=975] ppid=832 vsize=144456 CPUtime=15.82
/proc/833/task/975/stat : 975 (pb_cplex) S 832 832 685 0 -1 4202560 6838 0 0 0 1569 13 0 0 15 0 8 0 179127922 147922944 13490 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 195.03
Current children cumulated vsize (KiB) 153156

[startup+102.307 s]
/proc/loadavg: 7.23 2.49 0.89 9/195 999
/proc/meminfo: memFree=30823256/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=146504 CPUtime=380.68
/proc/833/stat : 833 (pb_cplex) S 832 832 685 0 -1 4202496 151569 0 0 0 37700 368 0 0 15 0 8 0 179125889 150020096 13877 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/833/statm: 36626 13877 1086 2985 0 33168 0
[pid=833/tid=969] ppid=832 vsize=146504 CPUtime=38.59
/proc/833/task/969/stat : 969 (pb_cplex) S 832 832 685 0 -1 4202560 9566 0 0 0 3825 34 0 0 15 0 8 0 179127922 150020096 13877 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=970] ppid=832 vsize=146504 CPUtime=39.04
/proc/833/task/970/stat : 970 (pb_cplex) S 832 832 685 0 -1 4202560 5220 0 0 0 3874 30 0 0 15 0 8 0 179127922 150020096 13877 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=971] ppid=832 vsize=146504 CPUtime=38.79
/proc/833/task/971/stat : 971 (pb_cplex) S 832 832 685 0 -1 4202560 20934 0 0 0 3844 35 0 0 15 0 8 0 179127922 150020096 13877 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=972] ppid=832 vsize=146504 CPUtime=38.1
/proc/833/task/972/stat : 972 (pb_cplex) S 832 832 685 0 -1 4202560 10973 0 0 0 3774 36 0 0 15 0 8 0 179127922 150020096 13877 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/833/statm: 37394 14620 1104 2985 0 33936 0
[pid=833/tid=969] ppid=832 vsize=149576 CPUtime=124.86
/proc/833/task/969/stat : 969 (pb_cplex) R 832 832 685 0 -1 4202560 21911 0 0 0 12375 111 0 0 15 0 8 0 179127922 153165824 14620 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=833/tid=970] ppid=832 vsize=149576 CPUtime=124.37
/proc/833/task/970/stat : 970 (pb_cplex) R 832 832 685 0 -1 4202560 21375 0 0 0 12325 112 0 0 15 0 8 0 179127922 153165824 14620 4089446400 134512640 146736320 4288283536 18446744073709551615 137334509 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=833/tid=971] ppid=832 vsize=149576 CPUtime=124.49
/proc/833/task/971/stat : 971 (pb_cplex) R 832 832 685 0 -1 4202560 32271 0 0 0 12334 115 0 0 15 0 8 0 179127922 153165824 14620 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=833/tid=972] ppid=832 vsize=149576 CPUtime=124.23
/proc/833/task/972/stat : 972 (pb_cplex) S 832 832 685 0 -1 4202560 27273 0 0 0 12308 115 0 0 15 0 8 0 179127922 153165824 14620 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 2 0 0 0
[pid=833/tid=973] ppid=832 vsize=149576 CPUtime=124.81
/proc/833/task/973/stat : 973 (pb_cplex) S 832 832 685 0 -1 4202560 22719 0 0 0 12366 115 0 0 15 0 8 0 179127922 153165824 14620 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=974] ppid=832 vsize=149576 CPUtime=125.69
/proc/833/task/974/stat : 974 (pb_cplex) S 832 832 685 0 -1 4202560 23912 0 0 0 12455 114 0 0 15 0 8 0 179127922 153165824 14620 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=975] ppid=832 vsize=149576 CPUtime=125.82
/proc/833/task/975/stat : 975 (pb_cplex) S 832 832 685 0 -1 4202560 29428 0 0 0 12462 120 0 0 15 0 8 0 179127922 153165824 14620 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 1074.91
Current children cumulated vsize (KiB) 158276

[startup+342.315 s]
/proc/loadavg: 9.04 6.24 2.80 9/195 1089
/proc/meminfo: memFree=30831448/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=149576 CPUtime=1293.42
/proc/833/stat : 833 (pb_cplex) R 832 832 685 0 -1 4202496 283034 0 0 0 128132 1210 0 0 16 0 8 0 179125889 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 135964004 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/833/statm: 37394 14622 1104 2985 0 33936 0
[pid=833/tid=969] ppid=832 vsize=149576 CPUtime=151.91
/proc/833/task/969/stat : 969 (pb_cplex) R 832 832 685 0 -1 4202560 26413 0 0 0 15057 134 0 0 15 0 8 0 179127922 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=833/tid=970] ppid=832 vsize=149576 CPUtime=151.2
/proc/833/task/970/stat : 970 (pb_cplex) R 832 832 685 0 -1 4202560 23386 0 0 0 14983 137 0 0 15 0 8 0 179127922 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=833/tid=971] ppid=832 vsize=149576 CPUtime=151.41
/proc/833/task/971/stat : 971 (pb_cplex) R 832 832 685 0 -1 4202560 34608 0 0 0 15001 140 0 0 15 0 8 0 179127922 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 135456843 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=833/tid=972] ppid=832 vsize=149576 CPUtime=151.11
/proc/833/task/972/stat : 972 (pb_cplex) R 832 832 685 0 -1 4202560 31584 0 0 0 14971 140 0 0 15 0 8 0 179127922 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=833/tid=973] ppid=832 vsize=149576 CPUtime=152.07
/proc/833/task/973/stat : 973 (pb_cplex) R 832 832 685 0 -1 4202560 24428 0 0 0 15068 139 0 0 15 0 8 0 179127922 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 135844960 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=833/tid=974] ppid=832 vsize=149576 CPUtime=152.73
/proc/833/task/974/stat : 974 (pb_cplex) R 832 832 685 0 -1 4202560 26418 0 0 0 15135 138 0 0 15 0 8 0 179127922 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=833/tid=975] ppid=832 vsize=149576 CPUtime=152.94
/proc/833/task/975/stat : 975 (pb_cplex) R 832 832 685 0 -1 4202560 34140 0 0 0 15145 149 0 0 15 0 8 0 179127922 153165824 14622 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1293.42
Current children cumulated vsize (KiB) 158276

[startup+402.305 s]
/proc/loadavg: 9.11 6.79 3.20 9/195 1091
/proc/meminfo: memFree=30833092/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=148256 CPUtime=1513.62
/proc/833/stat : 833 (pb_cplex) R 832 832 685 0 -1 4202496 304776 0 0 0 149945 1417 0 0 16 0 8 0 179125889 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/833/statm: 37064 14274 1104 2985 0 33606 0
[pid=833/tid=969] ppid=832 vsize=148256 CPUtime=179.39
/proc/833/task/969/stat : 969 (pb_cplex) S 832 832 685 0 -1 4202560 29116 0 0 0 17779 160 0 0 15 0 8 0 179127922 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 7 0 0 0
[pid=833/tid=970] ppid=832 vsize=148256 CPUtime=178.24
/proc/833/task/970/stat : 970 (pb_cplex) S 832 832 685 0 -1 4202560 26204 0 0 0 17661 163 0 0 15 0 8 0 179127922 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=971] ppid=832 vsize=148256 CPUtime=178.49
/proc/833/task/971/stat : 971 (pb_cplex) R 832 832 685 0 -1 4202560 36676 0 0 0 17684 165 0 0 15 0 8 0 179127922 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=833/tid=972] ppid=832 vsize=148256 CPUtime=178.36
/proc/833/task/972/stat : 972 (pb_cplex) R 832 832 685 0 -1 4202560 34545 0 0 0 17669 167 0 0 15 0 8 0 179127922 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=833/tid=973] ppid=832 vsize=148256 CPUtime=179.2
/proc/833/task/973/stat : 973 (pb_cplex) R 832 832 685 0 -1 4202560 28829 0 0 0 17754 166 0 0 15 0 8 0 179127922 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=833/tid=974] ppid=832 vsize=148256 CPUtime=180.07
/proc/833/task/974/stat : 974 (pb_cplex) R 832 832 685 0 -1 4202560 28788 0 0 0 17845 162 0 0 15 0 8 0 179127922 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=833/tid=975] ppid=832 vsize=148256 CPUtime=180.43
/proc/833/task/975/stat : 975 (pb_cplex) R 832 832 685 0 -1 4202560 36932 0 0 0 17870 173 0 0 15 0 8 0 179127922 151814144 14274 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 1513.62
Current children cumulated vsize (KiB) 156956

[startup+462.315 s]
/proc/loadavg: 8.62 7.04 3.51 9/195 1100
/proc/meminfo: memFree=30825548/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=153632 CPUtime=1730.89
/proc/833/stat : 833 (pb_cplex) R 832 832 685 0 -1 4202496 330942 0 0 0 171450 1639 0 0 16 0 8 0 179125889 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/833/statm: 38408 15762 1104 2985 0 34950 0
[pid=833/tid=1092] ppid=832 vsize=153632 CPUtime=15.63
/proc/833/task/1092/stat : 1092 (pb_cplex) R 832 832 685 0 -1 4202560 1026 0 0 0 1550 13 0 0 15 0 8 0 179168615 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=833/tid=1093] ppid=832 vsize=153632 CPUtime=15.56
/proc/833/task/1093/stat : 1093 (pb_cplex) R 832 832 685 0 -1 4202560 895 0 0 0 1542 14 0 0 15 0 8 0 179168615 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=833/tid=1094] ppid=832 vsize=153632 CPUtime=15.6
/proc/833/task/1094/stat : 1094 (pb_cplex) R 832 832 685 0 -1 4202560 1564 0 0 0 1544 16 0 0 15 0 8 0 179168615 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=833/tid=1095] ppid=832 vsize=153632 CPUtime=15.4
/proc/833/task/1095/stat : 1095 (pb_cplex) R 832 832 685 0 -1 4202560 420 0 0 0 1526 14 0 0 15 0 8 0 179168615 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=833/tid=1096] ppid=832 vsize=153632 CPUtime=15.62
/proc/833/task/1096/stat : 1096 (pb_cplex) R 832 832 685 0 -1 4202560 687 0 0 0 1546 16 0 0 15 0 8 0 179168615 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=833/tid=1097] ppid=832 vsize=153632 CPUtime=15.49
/proc/833/task/1097/stat : 1097 (pb_cplex) R 832 832 685 0 -1 4202560 1244 0 0 0 1532 17 0 0 15 0 8 0 179168615 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 137374790 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=833/tid=1098] ppid=832 vsize=153632 CPUtime=15.57
/proc/833/task/1098/stat : 1098 (pb_cplex) R 832 832 685 0 -1 4202560 631 0 0 0 1543 14 0 0 15 0 8 0 179168615 157319168 15762 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1730.89
Current children cumulated vsize (KiB) 162332



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+481.801 s]
/proc/loadavg: 8.52 7.12 3.61 8/195 1100
/proc/meminfo: memFree=30825688/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=153688 CPUtime=1800.44
/proc/833/stat : 833 (pb_cplex) S 832 832 685 0 -1 4202496 335425 0 0 0 178332 1712 0 0 16 0 8 0 179125889 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/833/statm: 38422 15793 1104 2985 0 34964 0
[pid=833/tid=1092] ppid=832 vsize=153688 CPUtime=24.36
/proc/833/task/1092/stat : 1092 (pb_cplex) S 832 832 685 0 -1 4202560 1891 0 0 0 2415 21 0 0 16 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=1093] ppid=832 vsize=153688 CPUtime=24.05
/proc/833/task/1093/stat : 1093 (pb_cplex) S 832 832 685 0 -1 4202560 1181 0 0 0 2380 25 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 4 0 0 0
[pid=833/tid=1094] ppid=832 vsize=153688 CPUtime=24.19
/proc/833/task/1094/stat : 1094 (pb_cplex) S 832 832 685 0 -1 4202560 2223 0 0 0 2392 27 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 6 0 0 0
[pid=833/tid=1095] ppid=832 vsize=153688 CPUtime=23.96
/proc/833/task/1095/stat : 1095 (pb_cplex) S 832 832 685 0 -1 4202560 1207 0 0 0 2372 24 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 7 0 0 0
[pid=833/tid=1096] ppid=832 vsize=153688 CPUtime=24.22
/proc/833/task/1096/stat : 1096 (pb_cplex) S 832 832 685 0 -1 4202560 1079 0 0 0 2397 25 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 3 0 0 0
[pid=833/tid=1097] ppid=832 vsize=153688 CPUtime=24.12
/proc/833/task/1097/stat : 1097 (pb_cplex) S 832 832 685 0 -1 4202560 1676 0 0 0 2387 25 0 0 16 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 1 0 0 0
[pid=833/tid=1098] ppid=832 vsize=153688 CPUtime=24.08
/proc/833/task/1098/stat : 1098 (pb_cplex) R 832 832 685 0 -1 4202560 1563 0 0 0 2385 23 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1800.44
Current children cumulated vsize (KiB) 162388

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+481.801 s]
/proc/loadavg: 8.52 7.12 3.61 8/195 1100
/proc/meminfo: memFree=30825688/32951124 swapFree=67111528/67111528
[pid=832] ppid=830 vsize=8700 CPUtime=0
/proc/832/stat : 832 (run) S 830 832 685 0 -1 4202496 364 0 0 0 0 0 0 0 18 0 1 0 179125889 8908800 248 4089446400 4194304 4922060 140735966485760 18446744073709551615 224284744837 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/832/statm: 2175 248 208 178 0 68 0
[pid=833] ppid=832 vsize=153688 CPUtime=1800.44
/proc/833/stat : 833 (pb_cplex) S 832 832 685 0 -1 4202496 335425 0 0 0 178332 1712 0 0 16 0 8 0 179125889 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/833/statm: 38422 15793 1104 2985 0 34964 0
[pid=833/tid=1092] ppid=832 vsize=153688 CPUtime=24.36
/proc/833/task/1092/stat : 1092 (pb_cplex) S 832 832 685 0 -1 4202560 1891 0 0 0 2415 21 0 0 16 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 0 0 0 0
[pid=833/tid=1093] ppid=832 vsize=153688 CPUtime=24.05
/proc/833/task/1093/stat : 1093 (pb_cplex) S 832 832 685 0 -1 4202560 1181 0 0 0 2380 25 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 4 0 0 0
[pid=833/tid=1094] ppid=832 vsize=153688 CPUtime=24.19
/proc/833/task/1094/stat : 1094 (pb_cplex) S 832 832 685 0 -1 4202560 2223 0 0 0 2392 27 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 6 0 0 0
[pid=833/tid=1095] ppid=832 vsize=153688 CPUtime=23.96
/proc/833/task/1095/stat : 1095 (pb_cplex) S 832 832 685 0 -1 4202560 1207 0 0 0 2372 24 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 7 0 0 0
[pid=833/tid=1096] ppid=832 vsize=153688 CPUtime=24.22
/proc/833/task/1096/stat : 1096 (pb_cplex) S 832 832 685 0 -1 4202560 1079 0 0 0 2397 25 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 3 0 0 0
[pid=833/tid=1097] ppid=832 vsize=153688 CPUtime=24.12
/proc/833/task/1097/stat : 1097 (pb_cplex) S 832 832 685 0 -1 4202560 1676 0 0 0 2387 25 0 0 16 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144580073 0 2 4096 16387 18446604471103650688 0 0 -1 1 0 0 0
[pid=833/tid=1098] ppid=832 vsize=153688 CPUtime=24.08
/proc/833/task/1098/stat : 1098 (pb_cplex) R 832 832 685 0 -1 4202560 1563 0 0 0 2385 23 0 0 15 0 8 0 179168615 157376512 15793 4089446400 134512640 146736320 4288283536 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1800.44
Current children cumulated vsize (KiB) 162388

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 832 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=0
# CPU time returned by wait4() is 0.000999
# while last known CPU time is 1800.44
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 481.841
CPU time (s): 1800.44
CPU user time (s): 1783.32
CPU system time (s): 17.12
CPU usage (%): 373.659
Max. virtual memory (cumulated for all children) (KiB): 164420

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.000999
system time used= 0
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 364
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 3
involuntary context switches= 2

runsolver used 0.960853 second user time and 1.89471 second system time

The end

Launcher Data 

Begin job on node118 at 2010-07-02 09:14:23
IDJOB=2697811
IDBENCH=78885
IDSOLVER=1210
FILE ID=node118/2697811-1278054863
PBS_JOBID= 11198946
Free space on /tmp= 74208 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900584219--soft-33-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697811-1278054863/watcher-2697811-1278054863 -o /tmp/evaluation-result-2697811-1278054863/solver-2697811-1278054863 -C 1800 -W 2000 -M 3800  run HOME/instance-2697811-1278054863.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 8d817301a4b1c2c9b4fea9d5d7fdc9d1
RANDOM SEED=881519741

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.58
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.791
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5448.09
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30955924 kB
Buffers:        217652 kB
Cached:        1567348 kB
SwapCached:          0 kB
Active:         624080 kB
Inactive:      1224628 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30955924 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16812 kB
Writeback:           0 kB
AnonPages:       63668 kB
Mapped:          14628 kB
Slab:            83400 kB
PageTables:       4516 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   184664 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74176 MiB
End job on node118 at 2010-07-02 09:22:27