Trace number 2697917

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.18 466.518

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/
normalized-t3002.11tsp11.1900576927--soft-33-100-0.wbo
MD5SUMb0bcd9e760e7f0ffa9653d41066a148b
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark278.087
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints1785
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 88246
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 88245
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 671
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697917-1278054868.wbo.
0.00/0.00	c Instance file HOME/instance-2697917-1278054868.wbo
0.00/0.00	c File size is 92233 bytes.
0.00/0.63	c Highest Coefficient sum: 671
0.00/0.63	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
0.00/0.64	c ============================[ Problem Statistics ]=============================
0.00/0.64	c |                                                                             |
0.00/0.64	c |  Parsing time:          0.64         s                                      |
0.00/0.64	c |  Number Variables:      2016                                                |
0.00/0.64	c |  Number Constraints:    2708                                                |
0.00/0.64	c ===============================================================================
1800.16/466.51	c Remaining time 1799.36 sec.
1800.16/466.51	c 
1800.16/466.51	c *** INTERRUPTED ***
1800.16/466.51	s UNKNOWN
1800.16/466.51	c Total CPU time              : 1784.99 s
1800.16/466.51	c Memory used           : 151.06 MB
1800.16/466.51	c CPU time              : 1784.99 s
1800.16/466.51	c 
1800.16/466.51	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697917-1278054868/watcher-2697917-1278054868 -o /tmp/evaluation-result-2697917-1278054868/solver-2697917-1278054868 -C 1800 -W 2000 -M 3800 run HOME/instance-2697917-1278054868.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.15 0.03 0.01 8/186 20998
/proc/meminfo: memFree=31190284/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0

[startup+0.071537 s]
/proc/loadavg: 0.15 0.03 0.01 8/186 20998
/proc/meminfo: memFree=31190284/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.100577 s]
/proc/loadavg: 0.15 0.03 0.01 8/186 20998
/proc/meminfo: memFree=31190284/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300544 s]
/proc/loadavg: 0.15 0.03 0.01 8/186 20998
/proc/meminfo: memFree=31190284/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.700421 s]
/proc/loadavg: 0.15 0.03 0.01 8/186 20998
/proc/meminfo: memFree=31190284/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50039 s]
/proc/loadavg: 0.15 0.03 0.01 9/195 21035
/proc/meminfo: memFree=31153144/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=82348 CPUtime=2.5
/proc/20999/stat : 20999 (pb_cplex) R 20998 20998 20685 0 -1 4202496 17003 0 0 0 244 6 0 0 16 0 1 0 146783529 84324352 6512 4089446400 134512640 146736320 4291758800 18446744073709551615 137703359 0 0 4096 16387 0 0 0 17 2 0 0 0
/proc/20999/statm: 20587 6512 1069 2985 0 17129 0
Current children cumulated CPU time (s) 2.5
Current children cumulated vsize (KiB) 91048

[startup+3.10107 s]
/proc/loadavg: 0.15 0.03 0.01 3/189 21049
/proc/meminfo: memFree=31147488/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=125356 CPUtime=4.44
/proc/20999/stat : 20999 (pb_cplex) R 20998 20998 20685 0 -1 4202496 26526 0 0 0 436 8 0 0 21 0 8 0 146783529 128364544 9003 4089446400 134512640 146736320 4291758800 18446744073709551615 136400430 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/20999/statm: 31339 9003 1077 2985 0 27881 0
Current children cumulated CPU time (s) 4.44
Current children cumulated vsize (KiB) 134056

[startup+6.31942 s]
/proc/loadavg: 1.10 0.23 0.07 10/195 21056
/proc/meminfo: memFree=31124136/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=136788 CPUtime=14.25
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 37919 0 0 0 1406 19 0 0 16 0 8 0 146783529 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/20999/statm: 34197 11807 1077 2985 0 30739 0
[pid=20999/tid=21050] ppid=20998 vsize=136788 CPUtime=1.13
/proc/20999/task/21050/stat : 21050 (pb_cplex) S 20998 20998 20685 0 -1 4202560 1128 0 0 0 112 1 0 0 15 0 8 0 146783740 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 3 0 0 0
[pid=20999/tid=21051] ppid=20998 vsize=136788 CPUtime=1.25
/proc/20999/task/21051/stat : 21051 (pb_cplex) S 20998 20998 20685 0 -1 4202560 178 0 0 0 124 1 0 0 15 0 8 0 146783740 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
[pid=20999/tid=21052] ppid=20998 vsize=136788 CPUtime=1.03
/proc/20999/task/21052/stat : 21052 (pb_cplex) S 20998 20998 20685 0 -1 4202560 413 0 0 0 103 0 0 0 15 0 8 0 146783740 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
[pid=20999/tid=21053] ppid=20998 vsize=136788 CPUtime=1.17
/proc/20999/task/21053/stat : 21053 (pb_cplex) S 20998 20998 20685 0 -1 4202560 284 0 0 0 117 0 0 0 15 0 8 0 146783740 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
[pid=20999/tid=21054] ppid=20998 vsize=136788 CPUtime=1.16
/proc/20999/task/21054/stat : 21054 (pb_cplex) S 20998 20998 20685 0 -1 4202560 1719 0 0 0 115 1 0 0 15 0 8 0 146783740 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 4 0 0 0
[pid=20999/tid=21055] ppid=20998 vsize=136788 CPUtime=1.31
/proc/20999/task/21055/stat : 21055 (pb_cplex) S 20998 20998 20685 0 -1 4202560 1207 0 0 0 130 1 0 0 15 0 8 0 146783740 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 4 0 0 0
[pid=20999/tid=21056] ppid=20998 vsize=136788 CPUtime=1.12
/proc/20999/task/21056/stat : 21056 (pb_cplex) R 20998 20998 20685 0 -1 4202560 2290 0 0 0 111 1 0 0 15 0 8 0 146783740 140070912 11807 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 14.25
Current children cumulated vsize (KiB) 145488

[startup+12.7186 s]
/proc/loadavg: 1.89 0.41 0.13 10/195 21056
/proc/meminfo: memFree=31120912/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=137812 CPUtime=36.97
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 43463 0 0 0 3658 39 0 0 16 0 8 0 146783529 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/20999/statm: 34453 12234 1077 2985 0 30995 0
[pid=20999/tid=21050] ppid=20998 vsize=137812 CPUtime=4.02
/proc/20999/task/21050/stat : 21050 (pb_cplex) S 20998 20998 20685 0 -1 4202560 1842 0 0 0 398 4 0 0 15 0 8 0 146783740 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 5 0 0 0
[pid=20999/tid=21051] ppid=20998 vsize=137812 CPUtime=4.1
/proc/20999/task/21051/stat : 21051 (pb_cplex) S 20998 20998 20685 0 -1 4202560 488 0 0 0 407 3 0 0 16 0 8 0 146783740 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
[pid=20999/tid=21052] ppid=20998 vsize=137812 CPUtime=3.7
/proc/20999/task/21052/stat : 21052 (pb_cplex) S 20998 20998 20685 0 -1 4202560 862 0 0 0 367 3 0 0 15 0 8 0 146783740 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 3 0 0 0
[pid=20999/tid=21053] ppid=20998 vsize=137812 CPUtime=3.96
/proc/20999/task/21053/stat : 21053 (pb_cplex) R 20998 20998 20685 0 -1 4202560 1062 0 0 0 393 3 0 0 15 0 8 0 146783740 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 136017461 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=20999/tid=21054] ppid=20998 vsize=137812 CPUtime=3.92
/proc/20999/task/21054/stat : 21054 (pb_cplex) S 20998 20998 20685 0 -1 4202560 3207 0 0 0 388 4 0 0 15 0 8 0 146783740 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 4 0 0 0
[pid=20999/tid=21055] ppid=20998 vsize=137812 CPUtime=4.16
/proc/20999/task/21055/stat : 21055 (pb_cplex) R 20998 20998 20685 0 -1 4202560 2100 0 0 0 413 3 0 0 15 0 8 0 146783740 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 139044510 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=20999/tid=21056] ppid=20998 vsize=137812 CPUtime=3.99
/proc/20999/task/21056/stat : 21056 (pb_cplex) R 20998 20998 20685 0 -1 4202560 3054 0 0 0 395 4 0 0 15 0 8 0 146783740 141119488 12234 4089446400 134512640 146736320 4291758800 18446744073709551615 136388106 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 36.97
Current children cumulated vsize (KiB) 146512

[startup+25.501 s]
/proc/loadavg: 3.75 0.89 0.30 12/195 21085
/proc/meminfo: memFree=31125252/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=140864 CPUtime=80.54
/proc/20999/stat : 20999 (pb_cplex) R 20998 20998 20685 0 -1 4202496 69517 0 0 0 7984 70 0 0 19 0 8 0 146783529 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 135921205 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/20999/statm: 35216 12906 1080 2985 0 31758 0
[pid=20999/tid=21065] ppid=20998 vsize=140864 CPUtime=2.14
/proc/20999/task/21065/stat : 21065 (pb_cplex) R 20998 20998 20685 0 -1 4202560 732 0 0 0 214 0 0 0 15 0 8 0 146785550 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=140864 CPUtime=2.28
/proc/20999/task/21066/stat : 21066 (pb_cplex) R 20998 20998 20685 0 -1 4202560 1203 0 0 0 227 1 0 0 15 0 8 0 146785550 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 138462928 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=140864 CPUtime=2.53
/proc/20999/task/21067/stat : 21067 (pb_cplex) R 20998 20998 20685 0 -1 4202560 1987 0 0 0 252 1 0 0 15 0 8 0 146785550 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=140864 CPUtime=2.39
/proc/20999/task/21068/stat : 21068 (pb_cplex) R 20998 20998 20685 0 -1 4202560 2031 0 0 0 238 1 0 0 15 0 8 0 146785550 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=140864 CPUtime=2.43
/proc/20999/task/21069/stat : 21069 (pb_cplex) R 20998 20998 20685 0 -1 4202560 304 0 0 0 243 0 0 0 15 0 8 0 146785550 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=140864 CPUtime=1.98
/proc/20999/task/21070/stat : 21070 (pb_cplex) R 20998 20998 20685 0 -1 4202560 1755 0 0 0 197 1 0 0 15 0 8 0 146785550 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=140864 CPUtime=2.14
/proc/20999/task/21071/stat : 21071 (pb_cplex) R 20998 20998 20685 0 -1 4202560 5718 0 0 0 213 1 0 0 15 0 8 0 146785550 144244736 12906 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 80.54
Current children cumulated vsize (KiB) 149564

[startup+51.1004 s]
/proc/loadavg: 5.25 1.48 0.50 11/195 21086
/proc/meminfo: memFree=31119548/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=145160 CPUtime=172.74
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 91276 0 0 0 17123 151 0 0 15 0 8 0 146783529 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/20999/statm: 36290 13981 1080 2985 0 32832 0
[pid=20999/tid=21065] ppid=20998 vsize=145160 CPUtime=13.26
/proc/20999/task/21065/stat : 21065 (pb_cplex) S 20998 20998 20685 0 -1 4202560 4313 0 0 0 1315 11 0 0 15 0 8 0 146785550 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 2 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=145160 CPUtime=13.53
/proc/20999/task/21066/stat : 21066 (pb_cplex) S 20998 20998 20685 0 -1 4202560 4516 0 0 0 1342 11 0 0 15 0 8 0 146785550 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 6 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=145160 CPUtime=14.03
/proc/20999/task/21067/stat : 21067 (pb_cplex) S 20998 20998 20685 0 -1 4202560 4013 0 0 0 1392 11 0 0 15 0 8 0 146785550 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 1 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=145160 CPUtime=13.9
/proc/20999/task/21068/stat : 21068 (pb_cplex) R 20998 20998 20685 0 -1 4202560 3018 0 0 0 1380 10 0 0 15 0 8 0 146785550 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=145160 CPUtime=13.74
/proc/20999/task/21069/stat : 21069 (pb_cplex) S 20998 20998 20685 0 -1 4202560 3042 0 0 0 1363 11 0 0 15 0 8 0 146785550 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 0 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=145160 CPUtime=13.45
/proc/20999/task/21070/stat : 21070 (pb_cplex) S 20998 20998 20685 0 -1 4202560 6062 0 0 0 1334 11 0 0 15 0 8 0 146785550 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 1 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=145160 CPUtime=13.61
/proc/20999/task/21071/stat : 21071 (pb_cplex) S 20998 20998 20685 0 -1 4202560 10460 0 0 0 1351 10 0 0 15 0 8 0 146785550 148643840 13981 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 172.74
Current children cumulated vsize (KiB) 153860

[startup+102.308 s]
/proc/loadavg: 7.63 2.69 0.96 10/195 21087
/proc/meminfo: memFree=31114804/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=150356 CPUtime=360.75
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 123465 0 0 0 35769 306 0 0 15 0 8 0 146783529 153964544 15075 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/20999/statm: 37589 15075 1080 2985 0 34131 0
[pid=20999/tid=21065] ppid=20998 vsize=150356 CPUtime=36.69
/proc/20999/task/21065/stat : 21065 (pb_cplex) S 20998 20998 20685 0 -1 4202560 7896 0 0 0 3640 29 0 0 16 0 8 0 146785550 153964544 15075 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 4 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=150356 CPUtime=36.83
/proc/20999/task/21066/stat : 21066 (pb_cplex) S 20998 20998 20685 0 -1 4202560 14414 0 0 0 3650 33 0 0 15 0 8 0 146785550 153964544 15075 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 3 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=150356 CPUtime=37.4
/proc/20999/task/21067/stat : 21067 (pb_cplex) S 20998 20998 20685 0 -1 4202560 12317 0 0 0 3709 31 0 0 15 0 8 0 146785550 153964544 15075 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 3 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=150356 CPUtime=36.97
/proc/20999/task/21068/stat : 21068 (pb_cplex) S 20998 20998 20685 0 -1 4202560 3409 0 0 0 3666 31 0 0 15 0 8 0 146785550 153964544 15075 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/20999/task/21070/stat : 21070 (pb_cplex) R 20998 20998 20685 0 -1 4202560 9133 0 0 0 9213 76 0 0 15 0 8 0 146785550 158453760 16191 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=154740 CPUtime=93.25
/proc/20999/task/21071/stat : 21071 (pb_cplex) S 20998 20998 20685 0 -1 4202560 15420 0 0 0 9251 74 0 0 15 0 8 0 146785550 158453760 16191 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 816.29
Current children cumulated vsize (KiB) 163440

[startup+282.301 s]
/proc/loadavg: 9.61 5.73 2.44 10/195 21092
/proc/meminfo: memFree=31109676/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=154740 CPUtime=1043.99
/proc/20999/stat : 20999 (pb_cplex) R 20998 20998 20685 0 -1 4202496 146274 0 0 0 103545 854 0 0 16 0 8 0 146783529 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 136403117 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/20999/statm: 38685 16219 1096 2985 0 35227 0
[pid=20999/tid=21065] ppid=20998 vsize=154740 CPUtime=120.74
/proc/20999/task/21065/stat : 21065 (pb_cplex) R 20998 20998 20685 0 -1 4202560 10901 0 0 0 11975 99 0 0 16 0 8 0 146785550 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=154740 CPUtime=121.71
/proc/20999/task/21066/stat : 21066 (pb_cplex) R 20998 20998 20685 0 -1 4202560 18572 0 0 0 12076 95 0 0 15 0 8 0 146785550 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=154740 CPUtime=122.59
/proc/20999/task/21067/stat : 21067 (pb_cplex) R 20998 20998 20685 0 -1 4202560 14948 0 0 0 12162 97 0 0 15 0 8 0 146785550 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=154740 CPUtime=121.7
/proc/20999/task/21068/stat : 21068 (pb_cplex) R 20998 20998 20685 0 -1 4202560 4916 0 0 0 12072 98 0 0 15 0 8 0 146785550 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=154740 CPUtime=121.19
/proc/20999/task/21069/stat : 21069 (pb_cplex) R 20998 20998 20685 0 -1 4202560 9604 0 0 0 12014 105 0 0 15 0 8 0 146785550 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=154740 CPUtime=121.03
/proc/20999/task/21070/stat : 21070 (pb_cplex) R 20998 20998 20685 0 -1 4202560 10203 0 0 0 12004 99 0 0 15 0 8 0 146785550 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=154740 CPUtime=121.28
/proc/20999/task/21071/stat : 21071 (pb_cplex) R 20998 20998 20685 0 -1 4202560 15729 0 0 0 12030 98 0 0 15 0 8 0 146785550 158453760 16219 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1043.99
Current children cumulated vsize (KiB) 163440

[startup+342.301 s]
/proc/loadavg: 9.70 6.43 2.88 9/195 21094
/proc/meminfo: memFree=31108592/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=154336 CPUtime=1268.75
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 150839 0 0 0 125832 1043 0 0 15 0 8 0 146783529 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/20999/statm: 38584 16220 1096 2985 0 35126 0
[pid=20999/tid=21065] ppid=20998 vsize=154336 CPUtime=148.38
/proc/20999/task/21065/stat : 21065 (pb_cplex) R 20998 20998 20685 0 -1 4202560 11501 0 0 0 14717 121 0 0 15 0 8 0 146785550 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=154336 CPUtime=149.36
/proc/20999/task/21066/stat : 21066 (pb_cplex) S 20998 20998 20685 0 -1 4202560 19007 0 0 0 14817 119 0 0 15 0 8 0 146785550 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=154336 CPUtime=150.34
/proc/20999/task/21067/stat : 21067 (pb_cplex) R 20998 20998 20685 0 -1 4202560 15048 0 0 0 14911 123 0 0 15 0 8 0 146785550 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 137334161 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=154336 CPUtime=149.61
/proc/20999/task/21068/stat : 21068 (pb_cplex) R 20998 20998 20685 0 -1 4202560 5594 0 0 0 14840 121 0 0 15 0 8 0 146785550 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=154336 CPUtime=149.13
/proc/20999/task/21069/stat : 21069 (pb_cplex) R 20998 20998 20685 0 -1 4202560 10736 0 0 0 14787 126 0 0 15 0 8 0 146785550 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 137081840 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=154336 CPUtime=148.83
/proc/20999/task/21070/stat : 21070 (pb_cplex) R 20998 20998 20685 0 -1 4202560 11089 0 0 0 14763 120 0 0 15 0 8 0 146785550 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 137341255 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=154336 CPUtime=149.17
/proc/20999/task/21071/stat : 21071 (pb_cplex) R 20998 20998 20685 0 -1 4202560 15858 0 0 0 14791 126 0 0 15 0 8 0 146785550 158040064 16220 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1268.75
Current children cumulated vsize (KiB) 163036

[startup+402.301 s]
/proc/loadavg: 9.92 7.09 3.33 5/195 21095
/proc/meminfo: memFree=31110096/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=154684 CPUtime=1496.54
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 155637 0 0 0 148418 1236 0 0 15 0 8 0 146783529 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/20999/statm: 38671 16285 1096 2985 0 35213 0
[pid=20999/tid=21065] ppid=20998 vsize=154684 CPUtime=176.62
/proc/20999/task/21065/stat : 21065 (pb_cplex) S 20998 20998 20685 0 -1 4202560 11521 0 0 0 17519 143 0 0 15 0 8 0 146785550 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=154684 CPUtime=177.6
/proc/20999/task/21066/stat : 21066 (pb_cplex) S 20998 20998 20685 0 -1 4202560 19093 0 0 0 17615 145 0 0 16 0 8 0 146785550 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 3 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=154684 CPUtime=178.87
/proc/20999/task/21067/stat : 21067 (pb_cplex) R 20998 20998 20685 0 -1 4202560 15533 0 0 0 17739 148 0 0 15 0 8 0 146785550 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 136016558 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=154684 CPUtime=177.7
/proc/20999/task/21068/stat : 21068 (pb_cplex) R 20998 20998 20685 0 -1 4202560 5956 0 0 0 17629 141 0 0 15 0 8 0 146785550 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=154684 CPUtime=177.2
/proc/20999/task/21069/stat : 21069 (pb_cplex) R 20998 20998 20685 0 -1 4202560 11772 0 0 0 17569 151 0 0 15 0 8 0 146785550 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 135857482 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=154684 CPUtime=176.83
/proc/20999/task/21070/stat : 21070 (pb_cplex) R 20998 20998 20685 0 -1 4202560 12825 0 0 0 17539 144 0 0 15 0 8 0 146785550 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 135450603 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=154684 CPUtime=177.3
/proc/20999/task/21071/stat : 21071 (pb_cplex) R 20998 20998 20685 0 -1 4202560 16472 0 0 0 17580 150 0 0 15 0 8 0 146785550 158396416 16285 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1496.54
Current children cumulated vsize (KiB) 163384

[startup+462.301 s]
/proc/loadavg: 9.24 7.44 3.68 3/195 21187
/proc/meminfo: memFree=31123776/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=154684 CPUtime=1776.48
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 161516 0 0 0 176151 1497 0 0 15 0 8 0 146783529 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/20999/statm: 38671 16287 1096 2985 0 35213 0
[pid=20999/tid=21065] ppid=20998 vsize=154684 CPUtime=211.15
/proc/20999/task/21065/stat : 21065 (pb_cplex) R 20998 20998 20685 0 -1 4202560 12815 0 0 0 20941 174 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 139044580 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=154684 CPUtime=212.15
/proc/20999/task/21066/stat : 21066 (pb_cplex) R 20998 20998 20685 0 -1 4202560 20398 0 0 0 21039 176 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=154684 CPUtime=213.75
/proc/20999/task/21067/stat : 21067 (pb_cplex) S 20998 20998 20685 0 -1 4202560 15851 0 0 0 21194 181 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 2 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=154684 CPUtime=212.44
/proc/20999/task/21068/stat : 21068 (pb_cplex) R 20998 20998 20685 0 -1 4202560 6897 0 0 0 21068 176 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 135450878 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=154684 CPUtime=212.09
/proc/20999/task/21069/stat : 21069 (pb_cplex) S 20998 20998 20685 0 -1 4202560 12339 0 0 0 21024 185 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 3 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=154684 CPUtime=211.82
/proc/20999/task/21070/stat : 21070 (pb_cplex) S 20998 20998 20685 0 -1 4202560 13223 0 0 0 21003 179 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 4 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=154684 CPUtime=212.33
/proc/20999/task/21071/stat : 21071 (pb_cplex) S 20998 20998 20685 0 -1 4202560 17184 0 0 0 21050 183 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1776.48
Current children cumulated vsize (KiB) 163384



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+466.503 s]
/proc/loadavg: 9.22 7.46 3.71 4/195 21188
/proc/meminfo: memFree=31110524/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=154684 CPUtime=1800.16
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 162113 0 0 0 178498 1518 0 0 15 0 8 0 146783529 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/20999/statm: 38671 16287 1096 2985 0 35213 0
[pid=20999/tid=21065] ppid=20998 vsize=154684 CPUtime=214.01
/proc/20999/task/21065/stat : 21065 (pb_cplex) S 20998 20998 20685 0 -1 4202560 12816 0 0 0 21225 176 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 1 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=154684 CPUtime=215.22
/proc/20999/task/21066/stat : 21066 (pb_cplex) S 20998 20998 20685 0 -1 4202560 20529 0 0 0 21342 180 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 2 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=154684 CPUtime=216.67
/proc/20999/task/21067/stat : 21067 (pb_cplex) S 20998 20998 20685 0 -1 4202560 15853 0 0 0 21484 183 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 5 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=154684 CPUtime=215.44
/proc/20999/task/21068/stat : 21068 (pb_cplex) S 20998 20998 20685 0 -1 4202560 6898 0 0 0 21365 179 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 6 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=154684 CPUtime=215.05
/proc/20999/task/21069/stat : 21069 (pb_cplex) R 20998 20998 20685 0 -1 4202560 12339 0 0 0 21317 188 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 139044576 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=154684 CPUtime=214.84
/proc/20999/task/21070/stat : 21070 (pb_cplex) S 20998 20998 20685 0 -1 4202560 13225 0 0 0 21304 180 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 0 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=154684 CPUtime=215.21
/proc/20999/task/21071/stat : 21071 (pb_cplex) S 20998 20998 20685 0 -1 4202560 17502 0 0 0 21335 186 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1800.16
Current children cumulated vsize (KiB) 163384

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+466.503 s]
/proc/loadavg: 9.22 7.46 3.71 4/195 21188
/proc/meminfo: memFree=31110524/32950904 swapFree=67111528/67111528
[pid=20998] ppid=20996 vsize=8700 CPUtime=0
/proc/20998/stat : 20998 (run) S 20996 20998 20685 0 -1 4202496 365 0 0 0 0 0 0 0 24 0 1 0 146783528 8908800 249 4089446400 4194304 4922060 140735478837392 18446744073709551615 237278698629 0 65536 4100 65538 18446744071562232103 0 0 17 0 0 0 0
/proc/20998/statm: 2175 249 208 178 0 68 0
[pid=20999] ppid=20998 vsize=154684 CPUtime=1800.16
/proc/20999/stat : 20999 (pb_cplex) S 20998 20998 20685 0 -1 4202496 162113 0 0 0 178498 1518 0 0 15 0 8 0 146783529 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/20999/statm: 38671 16287 1096 2985 0 35213 0
[pid=20999/tid=21065] ppid=20998 vsize=154684 CPUtime=214.01
/proc/20999/task/21065/stat : 21065 (pb_cplex) S 20998 20998 20685 0 -1 4202560 12816 0 0 0 21225 176 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 1 0 0 0
[pid=20999/tid=21066] ppid=20998 vsize=154684 CPUtime=215.22
/proc/20999/task/21066/stat : 21066 (pb_cplex) S 20998 20998 20685 0 -1 4202560 20529 0 0 0 21342 180 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 2 0 0 0
[pid=20999/tid=21067] ppid=20998 vsize=154684 CPUtime=216.67
/proc/20999/task/21067/stat : 21067 (pb_cplex) S 20998 20998 20685 0 -1 4202560 15853 0 0 0 21484 183 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 5 0 0 0
[pid=20999/tid=21068] ppid=20998 vsize=154684 CPUtime=215.44
/proc/20999/task/21068/stat : 21068 (pb_cplex) S 20998 20998 20685 0 -1 4202560 6898 0 0 0 21365 179 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 6 0 0 0
[pid=20999/tid=21069] ppid=20998 vsize=154684 CPUtime=215.05
/proc/20999/task/21069/stat : 21069 (pb_cplex) R 20998 20998 20685 0 -1 4202560 12339 0 0 0 21317 188 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 139044576 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=20999/tid=21070] ppid=20998 vsize=154684 CPUtime=214.84
/proc/20999/task/21070/stat : 21070 (pb_cplex) S 20998 20998 20685 0 -1 4202560 13225 0 0 0 21304 180 0 0 16 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 0 0 0 0
[pid=20999/tid=21071] ppid=20998 vsize=154684 CPUtime=215.21
/proc/20999/task/21071/stat : 21071 (pb_cplex) S 20998 20998 20685 0 -1 4202560 17502 0 0 0 21335 186 0 0 15 0 8 0 146785550 158396416 16287 4089446400 134512640 146736320 4291758800 18446744073709551615 144580073 0 2 4096 16387 18446604453937945536 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 1800.16
Current children cumulated vsize (KiB) 163384

Child status: 0
Real time (s): 466.518
CPU time (s): 1800.18
CPU user time (s): 1784.99
CPU system time (s): 15.1877
CPU usage (%): 385.876
Max. virtual memory (cumulated for all children) (KiB): 166512

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1784.99
system time used= 15.1877
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 162486
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 343339
involuntary context switches= 25775

runsolver used 0.903862 second user time and 1.79773 second system time

The end

Launcher Data 

Begin job on node121 at 2010-07-02 09:14:28
IDJOB=2697917
IDBENCH=78946
IDSOLVER=1210
FILE ID=node121/2697917-1278054868
PBS_JOBID= 11198939
Free space on /tmp= 74188 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900576927--soft-33-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697917-1278054868/watcher-2697917-1278054868 -o /tmp/evaluation-result-2697917-1278054868/solver-2697917-1278054868 -C 1800 -W 2000 -M 3800  run HOME/instance-2697917-1278054868.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= b0bcd9e760e7f0ffa9653d41066a148b
RANDOM SEED=1937513331

node121.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.23
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.876
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950904 kB
MemFree:      31192424 kB
Buffers:        216152 kB
Cached:        1303860 kB
SwapCached:          0 kB
Active:         669784 kB
Inactive:       938912 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950904 kB
LowFree:      31192424 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           33708 kB
Writeback:           0 kB
AnonPages:       88432 kB
Mapped:          18772 kB
Slab:            84108 kB
PageTables:       4868 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586980 kB
Committed_AS:   304244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74172 MiB
End job on node121 at 2010-07-02 09:22:15