Trace number 2697433

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.11 364.115

General information on the benchmark

Name/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/
tsp/normalized-t3002.11tsp11.1900581320--soft-0-100-0.wbo
MD5SUM85ba7bcc847457eeeb822d6fc8af8950
Bench CategorySOFT-SMALLINT-LIN (only soft constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark254.891
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints2707
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 138262
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 138261
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 641
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697433-1278055723.wbo.
0.00/0.00	c Instance file HOME/instance-2697433-1278055723.wbo
0.00/0.00	c File size is 96759 bytes.
1.39/1.43	c Highest Coefficient sum: 641
1.39/1.43	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
1.39/1.43	c ============================[ Problem Statistics ]=============================
1.39/1.43	c |                                                                             |
1.39/1.43	c |  Parsing time:          1.43         s                                      |
1.39/1.43	c |  Number Variables:      2938                                                |
1.39/1.44	c |  Number Constraints:    2708                                                |
1.39/1.44	c ===============================================================================
1800.06/364.11	c Remaining time 1798.56 sec.
1800.06/364.11	c 
1800.06/364.11	c *** INTERRUPTED ***
1800.06/364.11	s UNKNOWN
1800.06/364.11	c Total CPU time              : 1784.53 s
1800.06/364.11	c Memory used           : 139.99 MB
1800.06/364.11	c CPU time              : 1784.53 s
1800.06/364.11	c 
1800.06/364.11	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697433-1278055723/watcher-2697433-1278055723 -o /tmp/evaluation-result-2697433-1278055723/solver-2697433-1278055723 -C 1800 -W 2000 -M 3800 run HOME/instance-2697433-1278055723.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 10.00 8.87 5.44 7/186 23114
/proc/meminfo: memFree=31034936/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0

[startup+0.055208 s]
/proc/loadavg: 10.00 8.87 5.44 7/186 23114
/proc/meminfo: memFree=31034936/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.101202 s]
/proc/loadavg: 10.00 8.87 5.44 7/186 23114
/proc/meminfo: memFree=31034936/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300222 s]
/proc/loadavg: 10.00 8.87 5.44 7/186 23114
/proc/meminfo: memFree=31034936/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.70115 s]
/proc/loadavg: 10.00 8.87 5.44 7/186 23114
/proc/meminfo: memFree=31034936/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50101 s]
/proc/loadavg: 10.00 8.87 5.44 4/188 23116
/proc/meminfo: memFree=31032440/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=27328 CPUtime=1.49
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 3624 0 0 0 149 0 0 0 25 0 1 0 146868634 27983872 3156 4089446400 134512640 146736320 4293920176 18446744073709551615 139022260 0 0 4096 16387 0 0 0 17 2 0 0 0
/proc/23115/statm: 6832 3156 979 2985 0 3374 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 36028

[startup+3.10067 s]
/proc/loadavg: 10.00 8.87 5.44 8/195 23130
/proc/meminfo: memFree=30993324/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=100824 CPUtime=5.74
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 23113 0 0 0 565 9 0 0 16 0 1 0 146868634 103243776 10654 4089446400 134512640 146736320 4293920176 18446744073709551615 136385184 0 0 4096 16387 0 0 0 17 2 0 0 0
/proc/23115/statm: 25206 10654 1075 2985 0 21748 0
Current children cumulated CPU time (s) 5.74
Current children cumulated vsize (KiB) 109524

[startup+6.3011 s]
/proc/loadavg: 10.32 8.96 5.48 16/195 23151
/proc/meminfo: memFree=30973732/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=138668 CPUtime=13.46
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 44482 0 0 0 1331 15 0 0 20 0 8 0 146868634 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/23115/statm: 34667 12265 1086 2985 0 31209 0
[pid=23115/tid=23145] ppid=23114 vsize=138668 CPUtime=0.9
/proc/23115/task/23145/stat : 23145 (pb_cplex) R 23114 23114 22401 0 -1 4202560 56 0 0 0 90 0 0 0 16 0 8 0 146869039 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23146] ppid=23114 vsize=138668 CPUtime=0.92
/proc/23115/task/23146/stat : 23146 (pb_cplex) R 23114 23114 22401 0 -1 4202560 1510 0 0 0 92 0 0 0 16 0 8 0 146869039 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23115/tid=23147] ppid=23114 vsize=138668 CPUtime=0.79
/proc/23115/task/23147/stat : 23147 (pb_cplex) R 23114 23114 22401 0 -1 4202560 79 0 0 0 79 0 0 0 16 0 8 0 146869039 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23115/tid=23148] ppid=23114 vsize=138668 CPUtime=0.47
/proc/23115/task/23148/stat : 23148 (pb_cplex) R 23114 23114 22401 0 -1 4202560 458 0 0 0 47 0 0 0 16 0 8 0 146869039 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 138462755 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23115/tid=23149] ppid=23114 vsize=138668 CPUtime=0.34
/proc/23115/task/23149/stat : 23149 (pb_cplex) R 23114 23114 22401 0 -1 4202560 433 0 0 0 34 0 0 0 16 0 8 0 146869046 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 136008039 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23115/tid=23150] ppid=23114 vsize=138668 CPUtime=0.66
/proc/23115/task/23150/stat : 23150 (pb_cplex) R 23114 23114 22401 0 -1 4202560 1011 0 0 0 66 0 0 0 16 0 8 0 146869046 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23115/tid=23151] ppid=23114 vsize=138668 CPUtime=0.95
/proc/23115/task/23151/stat : 23151 (pb_cplex) R 23114 23114 22401 0 -1 4202560 2577 0 0 0 94 1 0 0 16 0 8 0 146869046 141996032 12265 4089446400 134512640 146736320 4293920176 18446744073709551615 136007150 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 13.46
Current children cumulated vsize (KiB) 147368

[startup+12.701 s]
/proc/loadavg: 10.30 8.97 5.51 13/195 23151
/proc/meminfo: memFree=30963228/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=141740 CPUtime=35.49
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 53651 0 0 0 3514 35 0 0 24 0 8 0 146868634 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/23115/statm: 35435 12988 1084 2985 0 31977 0
[pid=23115/tid=23145] ppid=23114 vsize=141740 CPUtime=3.67
/proc/23115/task/23145/stat : 23145 (pb_cplex) S 23114 23114 22401 0 -1 4202560 1979 0 0 0 365 2 0 0 15 0 8 0 146869039 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 0 0 0 0
[pid=23115/tid=23146] ppid=23114 vsize=141740 CPUtime=3.7
/proc/23115/task/23146/stat : 23146 (pb_cplex) S 23114 23114 22401 0 -1 4202560 2397 0 0 0 367 3 0 0 15 0 8 0 146869039 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 5 0 0 0
[pid=23115/tid=23147] ppid=23114 vsize=141740 CPUtime=3.56
/proc/23115/task/23147/stat : 23147 (pb_cplex) S 23114 23114 22401 0 -1 4202560 199 0 0 0 354 2 0 0 15 0 8 0 146869039 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 7 0 0 0
[pid=23115/tid=23148] ppid=23114 vsize=141740 CPUtime=3.18
/proc/23115/task/23148/stat : 23148 (pb_cplex) S 23114 23114 22401 0 -1 4202560 736 0 0 0 316 2 0 0 15 0 8 0 146869039 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 3 0 0 0
[pid=23115/tid=23149] ppid=23114 vsize=141740 CPUtime=3.14
/proc/23115/task/23149/stat : 23149 (pb_cplex) S 23114 23114 22401 0 -1 4202560 460 0 0 0 312 2 0 0 15 0 8 0 146869046 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 1 0 0 0
[pid=23115/tid=23150] ppid=23114 vsize=141740 CPUtime=3.37
/proc/23115/task/23150/stat : 23150 (pb_cplex) S 23114 23114 22401 0 -1 4202560 2804 0 0 0 334 3 0 0 15 0 8 0 146869046 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 4 0 0 0
[pid=23115/tid=23151] ppid=23114 vsize=141740 CPUtime=3.65
/proc/23115/task/23151/stat : 23151 (pb_cplex) S 23114 23114 22401 0 -1 4202560 3639 0 0 0 362 3 0 0 15 0 8 0 146869046 145141760 12988 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 35.49
Current children cumulated vsize (KiB) 150440

[startup+25.5007 s]
/proc/loadavg: 9.93 8.96 5.56 14/195 23152
/proc/meminfo: memFree=30961212/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=142764 CPUtime=66.21
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 66602 0 0 0 6560 61 0 0 18 0 8 0 146868634 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 139023034 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/23115/statm: 35691 13117 1084 2985 0 32233 0
[pid=23115/tid=23145] ppid=23114 vsize=142764 CPUtime=7.46
/proc/23115/task/23145/stat : 23145 (pb_cplex) R 23114 23114 22401 0 -1 4202560 2091 0 0 0 743 3 0 0 15 0 8 0 146869039 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23146] ppid=23114 vsize=142764 CPUtime=7.51
/proc/23115/task/23146/stat : 23146 (pb_cplex) R 23114 23114 22401 0 -1 4202560 4981 0 0 0 744 7 0 0 15 0 8 0 146869039 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23115/tid=23147] ppid=23114 vsize=142764 CPUtime=7.44
/proc/23115/task/23147/stat : 23147 (pb_cplex) R 23114 23114 22401 0 -1 4202560 235 0 0 0 738 6 0 0 15 0 8 0 146869039 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23115/tid=23148] ppid=23114 vsize=142764 CPUtime=6.93
/proc/23115/task/23148/stat : 23148 (pb_cplex) R 23114 23114 22401 0 -1 4202560 1669 0 0 0 688 5 0 0 15 0 8 0 146869039 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23115/tid=23149] ppid=23114 vsize=142764 CPUtime=7.08
/proc/23115/task/23149/stat : 23149 (pb_cplex) R 23114 23114 22401 0 -1 4202560 2909 0 0 0 702 6 0 0 15 0 8 0 146869046 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 138462824 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23115/tid=23150] ppid=23114 vsize=142764 CPUtime=7.21
/proc/23115/task/23150/stat : 23150 (pb_cplex) R 23114 23114 22401 0 -1 4202560 5187 0 0 0 715 6 0 0 15 0 8 0 146869046 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23115/tid=23151] ppid=23114 vsize=142764 CPUtime=7.33
/proc/23115/task/23151/stat : 23151 (pb_cplex) R 23114 23114 22401 0 -1 4202560 7475 0 0 0 727 6 0 0 15 0 8 0 146869046 146190336 13117 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 66.21
Current children cumulated vsize (KiB) 151464

[startup+51.1012 s]
/proc/loadavg: 10.22 9.08 5.69 10/195 23167
/proc/meminfo: memFree=30959792/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=142324 CPUtime=153.82
/proc/23115/stat : 23115 (pb_cplex) S 23114 23114 22401 0 -1 4202496 84679 0 0 0 15250 132 0 0 15 0 8 0 146868634 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/23115/statm: 35581 13870 1087 2985 0 32123 0
[pid=23115/tid=23160] ppid=23114 vsize=142324 CPUtime=8.13
/proc/23115/task/23160/stat : 23160 (pb_cplex) S 23114 23114 22401 0 -1 4202560 94 0 0 0 809 4 0 0 15 0 8 0 146871706 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 4 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=142324 CPUtime=7.21
/proc/23115/task/23161/stat : 23161 (pb_cplex) S 23114 23114 22401 0 -1 4202560 2112 0 0 0 716 5 0 0 15 0 8 0 146871706 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 1 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=142324 CPUtime=6.95
/proc/23115/task/23162/stat : 23162 (pb_cplex) S 23114 23114 22401 0 -1 4202560 309 0 0 0 690 5 0 0 15 0 8 0 146871706 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 3 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=142324 CPUtime=6.43
/proc/23115/task/23163/stat : 23163 (pb_cplex) S 23114 23114 22401 0 -1 4202560 652 0 0 0 638 5 0 0 15 0 8 0 146871706 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 7 0 0 0
[pid=23115/tid=23164] ppid=23114 vsize=142324 CPUtime=8.54
/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 448 0 0 0 848 6 0 0 25 0 8 0 146871706 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=142324 CPUtime=6.96
/proc/23115/task/23165/stat : 23165 (pb_cplex) S 23114 23114 22401 0 -1 4202560 2589 0 0 0 692 4 0 0 15 0 8 0 146871706 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 2 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=142324 CPUtime=7.7
/proc/23115/task/23166/stat : 23166 (pb_cplex) S 23114 23114 22401 0 -1 4202560 1932 0 0 0 768 2 0 0 15 0 8 0 146871706 145739776 13870 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 153.82
Current children cumulated vsize (KiB) 151024

[startup+102.301 s]
/proc/loadavg: 10.80 9.37 5.96 18/195 23168
/proc/meminfo: memFree=30956948/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=145396 CPUtime=381.02
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 105513 0 0 0 37794 308 0 0 16 0 8 0 146868634 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/23115/statm: 36349 14593 1087 2985 0 32891 0
[pid=23115/tid=23160] ppid=23114 vsize=145396 CPUtime=36.7
/proc/23115/task/23160/stat : 23160 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3291 0 0 0 3645 25 0 0 16 0 8 0 146871706 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=145396 CPUtime=35.35
/proc/23115/task/23161/stat : 23161 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3379 0 0 0 3508 27 0 0 15 0 8 0 146871706 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 136388081 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=145396 CPUtime=35.37
/proc/23115/task/23162/stat : 23162 (pb_cplex) R 23114 23114 22401 0 -1 4202560 889 0 0 0 3512 25 0 0 15 0 8 0 146871706 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=145396 CPUtime=34.76
/proc/23115/task/23163/stat : 23163 (pb_cplex) R 23114 23114 22401 0 -1 4202560 4142 0 0 0 3451 25 0 0 16 0 8 0 146871706 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 1400 0 0 0 3655 27 0 0 16 0 8 0 146871706 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=145396 CPUtime=35.25
/proc/23115/task/23165/stat : 23165 (pb_cplex) R 23114 23114 22401 0 -1 4202560 5839 0 0 0 3497 28 0 0 16 0 8 0 146871706 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 135854511 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=145396 CPUtime=36.32
/proc/23115/task/23166/stat : 23166 (pb_cplex) R 23114 23114 22401 0 -1 4202560 6934 0 0 0 3606 26 0 0 15 0 8 0 146871706 148885504 14593 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 381.02
Current children cumulated vsize (KiB) 154096

[startup+162.3 s]
/proc/loadavg: 12.75 10.29 6.49 18/195 23169
/proc/meminfo: memFree=30953144/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=143348 CPUtime=714.82
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 108810 0 0 0 70900 582 0 0 15 0 8 0 146868634 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 136008177 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/23115/statm: 35837 14115 1087 2985 0 32379 0
[pid=23115/tid=23160] ppid=23114 vsize=143348 CPUtime=78.18
/proc/23115/task/23160/stat : 23160 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3487 0 0 0 7756 62 0 0 15 0 8 0 146871706 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=143348 CPUtime=76.69
/proc/23115/task/23161/stat : 23161 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3656 0 0 0 7606 63 0 0 15 0 8 0 146871706 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=143348 CPUtime=77.34
/proc/23115/task/23162/stat : 23162 (pb_cplex) R 23114 23114 22401 0 -1 4202560 1591 0 0 0 7688 46 0 0 15 0 8 0 146871706 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=143348 CPUtime=76.16
/proc/23115/task/23163/stat : 23163 (pb_cplex) R 23114 23114 22401 0 -1 4202560 4213 0 0 0 7557 59 0 0 15 0 8 0 146871706 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23115/tid=23164] ppid=23114 vsize=143348 CPUtime=78.81
/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 1757 0 0 0 7815 66 0 0 17 0 8 0 146871706 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 144580332 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=143348 CPUtime=76.95
/proc/23115/task/23165/stat : 23165 (pb_cplex) R 23114 23114 22401 0 -1 4202560 6657 0 0 0 7632 63 0 0 15 0 8 0 146871706 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 135468509 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=143348 CPUtime=78.29
/proc/23115/task/23166/stat : 23166 (pb_cplex) R 23114 23114 22401 0 -1 4202560 7147 0 0 0 7769 60 0 0 15 0 8 0 146871706 146788352 14115 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 714.82
Current children cumulated vsize (KiB) 152048

[startup+222.301 s]
/proc/loadavg: 11.23 10.40 6.77 6/195 23174
/proc/meminfo: memFree=30929388/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=140276 CPUtime=967.53
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 126948 0 0 0 95947 806 0 0 18 0 8 0 146868634 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 137381182 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/23115/statm: 35069 13539 1087 2985 0 31611 0
[pid=23115/tid=23160] ppid=23114 vsize=140276 CPUtime=109.43
/proc/23115/task/23160/stat : 23160 (pb_cplex) R 23114 23114 22401 0 -1 4202560 6417 0 0 0 10851 92 0 0 15 0 8 0 146871706 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=140276 CPUtime=108.04
/proc/23115/task/23161/stat : 23161 (pb_cplex) R 23114 23114 22401 0 -1 4202560 5957 0 0 0 10715 89 0 0 15 0 8 0 146871706 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 137334241 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=140276 CPUtime=108.89
/proc/23115/task/23162/stat : 23162 (pb_cplex) R 23114 23114 22401 0 -1 4202560 4868 0 0 0 10821 68 0 0 16 0 8 0 146871706 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=140276 CPUtime=107.29
/proc/23115/task/23163/stat : 23163 (pb_cplex) R 23114 23114 22401 0 -1 4202560 7372 0 0 0 10637 92 0 0 15 0 8 0 146871706 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23115/tid=23164] ppid=23114 vsize=140276 CPUtime=110.66
/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 2680 0 0 0 10971 95 0 0 15 0 8 0 146871706 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=140276 CPUtime=108.68
/proc/23115/task/23165/stat : 23165 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8742 0 0 0 10780 88 0 0 15 0 8 0 146871706 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 138462752 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=140276 CPUtime=109.75
/proc/23115/task/23166/stat : 23166 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8506 0 0 0 10887 88 0 0 15 0 8 0 146871706 143642624 13539 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 967.53
Current children cumulated vsize (KiB) 148976

[startup+282.301 s]
/proc/loadavg: 12.52 10.91 7.17 18/195 23176
/proc/meminfo: memFree=30927076/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=143348 CPUtime=1310.42
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 136034 0 0 0 129937 1105 0 0 16 0 8 0 146868634 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 135467878 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/23115/statm: 35837 14156 1087 2985 0 32379 0
[pid=23115/tid=23160] ppid=23114 vsize=143348 CPUtime=152.17
/proc/23115/task/23160/stat : 23160 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8175 0 0 0 15085 132 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 136008163 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=143348 CPUtime=150.32
/proc/23115/task/23161/stat : 23161 (pb_cplex) R 23114 23114 22401 0 -1 4202560 7898 0 0 0 14907 125 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=143348 CPUtime=151.66
/proc/23115/task/23162/stat : 23162 (pb_cplex) R 23114 23114 22401 0 -1 4202560 5653 0 0 0 15056 110 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=143348 CPUtime=150.07
/proc/23115/task/23163/stat : 23163 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8021 0 0 0 14875 132 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 137412857 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23115/tid=23164] ppid=23114 vsize=143348 CPUtime=153.47
/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3097 0 0 0 15220 127 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=143348 CPUtime=151.57
/proc/23115/task/23165/stat : 23165 (pb_cplex) R 23114 23114 22401 0 -1 4202560 10538 0 0 0 15040 117 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=143348 CPUtime=152.92
/proc/23115/task/23166/stat : 23166 (pb_cplex) R 23114 23114 22401 0 -1 4202560 9675 0 0 0 15165 127 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1310.42
Current children cumulated vsize (KiB) 152048

[startup+342.301 s]
/proc/loadavg: 11.87 11.12 7.48 6/179 23199
/proc/meminfo: memFree=31217116/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=143348 CPUtime=1647.52
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 138605 0 0 0 163332 1420 0 0 16 0 8 0 146868634 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 137334510 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/23115/statm: 35837 14156 1087 2985 0 32379 0
[pid=23115/tid=23160] ppid=23114 vsize=143348 CPUtime=193.76
/proc/23115/task/23160/stat : 23160 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8653 0 0 0 19206 170 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=143348 CPUtime=192.1
/proc/23115/task/23161/stat : 23161 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8038 0 0 0 19045 165 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=143348 CPUtime=193.71
/proc/23115/task/23162/stat : 23162 (pb_cplex) R 23114 23114 22401 0 -1 4202560 5938 0 0 0 19221 150 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 139044583 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=143348 CPUtime=191.91
/proc/23115/task/23163/stat : 23163 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8218 0 0 0 19023 168 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23115/tid=23164] ppid=23114 vsize=143348 CPUtime=195.65
/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3478 0 0 0 19397 168 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 135450630 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=143348 CPUtime=193.7
/proc/23115/task/23165/stat : 23165 (pb_cplex) R 23114 23114 22401 0 -1 4202560 10619 0 0 0 19215 155 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=143348 CPUtime=195.27
/proc/23115/task/23166/stat : 23166 (pb_cplex) R 23114 23114 22401 0 -1 4202560 10315 0 0 0 19361 166 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1647.52
Current children cumulated vsize (KiB) 152048



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+364.101 s]
/proc/loadavg: 10.24 10.79 7.47 2/179 23200
/proc/meminfo: memFree=31219056/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=143348 CPUtime=1800.06
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 139096 0 0 0 178449 1557 0 0 16 0 8 0 146868634 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 136674344 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/23115/statm: 35837 14156 1087 2985 0 32379 0
[pid=23115/tid=23160] ppid=23114 vsize=143348 CPUtime=212.82
/proc/23115/task/23160/stat : 23160 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8699 0 0 0 21093 189 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=143348 CPUtime=211.06
/proc/23115/task/23161/stat : 23161 (pb_cplex) S 23114 23114 22401 0 -1 4202560 8055 0 0 0 20925 181 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 7 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=143348 CPUtime=212.88
/proc/23115/task/23162/stat : 23162 (pb_cplex) S 23114 23114 22401 0 -1 4202560 5950 0 0 0 21123 165 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 2 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=143348 CPUtime=210.88
/proc/23115/task/23163/stat : 23163 (pb_cplex) S 23114 23114 22401 0 -1 4202560 8469 0 0 0 20906 182 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 5 0 0 0
[pid=23115/tid=23164] ppid=23114 vsize=143348 CPUtime=214.74
/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3506 0 0 0 21286 188 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580332 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=143348 CPUtime=212.73
/proc/23115/task/23165/stat : 23165 (pb_cplex) S 23114 23114 22401 0 -1 4202560 10649 0 0 0 21103 170 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 0 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=143348 CPUtime=214.55
/proc/23115/task/23166/stat : 23166 (pb_cplex) S 23114 23114 22401 0 -1 4202560 10370 0 0 0 21272 183 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1800.06
Current children cumulated vsize (KiB) 152048

Sending SIGTERM to process tree (bottom up)
Solver just ended. DSolver just ended. Dumping a history of the last processes samples

[startup+364.101 s]
/proc/loadavg: 10.24 10.79 7.47 2/179 23200
/proc/meminfo: memFree=31219056/32950928 swapFree=4192956/4192956
[pid=23114] ppid=23112 vsize=8700 CPUtime=0
/proc/23114/stat : 23114 (run) S 23112 23114 22401 0 -1 4202496 364 0 0 0 0 0 0 0 25 0 1 0 146868633 8908800 248 4089446400 4194304 4922060 140733292507008 18446744073709551615 230565715077 0 65536 4100 65538 18446744071562232103 0 0 17 3 0 0 0
/proc/23114/statm: 2175 248 208 178 0 68 0
[pid=23115] ppid=23114 vsize=143348 CPUtime=1800.06
/proc/23115/stat : 23115 (pb_cplex) R 23114 23114 22401 0 -1 4202496 139096 0 0 0 178449 1557 0 0 16 0 8 0 146868634 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 136674344 0 0 4096 16387 18446744073709551615 0 0 17 3 0 0 0
/proc/23115/statm: 35837 14156 1087 2985 0 32379 0
[pid=23115/tid=23160] ppid=23114 vsize=143348 CPUtime=212.82
/proc/23115/task/23160/stat : 23160 (pb_cplex) R 23114 23114 22401 0 -1 4202560 8699 0 0 0 21093 189 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23115/tid=23161] ppid=23114 vsize=143348 CPUtime=211.06
/proc/23115/task/23161/stat : 23161 (pb_cplex) S 23114 23114 22401 0 -1 4202560 8055 0 0 0 20925 181 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 7 0 0 0
[pid=23115/tid=23162] ppid=23114 vsize=143348 CPUtime=212.88
/proc/23115/task/23162/stat : 23162 (pb_cplex) S 23114 23114 22401 0 -1 4202560 5950 0 0 0 21123 165 0 0 15 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 2 0 0 0
[pid=23115/tid=23163] ppid=23114 vsize=143348 CPUtime=210.88
/proc/23115/task/23163/stat : 23163 (pb_cplex) S 23114 23114 22401 0 -1 4202560 8469 0 0 0 20906 182 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 5 0 0 0
[pid=23115/tid=23164] ppid=23114 vsize=143348 CPUtime=214.74
/proc/23115/task/23164/stat : 23164 (pb_cplex) R 23114 23114 22401 0 -1 4202560 3506 0 0 0 21286 188 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580332 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23115/tid=23165] ppid=23114 vsize=143348 CPUtime=212.73
/proc/23115/task/23165/stat : 23165 (pb_cplex) S 23114 23114 22401 0 -1 4202560 10649 0 0 0 21103 170 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 0 0 0 0
[pid=23115/tid=23166] ppid=23114 vsize=143348 CPUtime=214.55
/proc/23115/task/23166/stat : 23166 (pb_cplex) S 23114 23114 22401 0 -1 4202560 10370 0 0 0 21272 183 0 0 16 0 8 0 146871706 146788352 14156 4089446400 134512640 146736320 4293920176 18446744073709551615 144580073 0 2 4096 16387 18446604453941539200 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1800.06
Current children cumulated vsize (KiB) 152048

Child status: 0
Real time (s): 364.115
CPU time (s): 1800.11
CPU user time (s): 1784.53
CPU system time (s): 15.5756
CPU usage (%): 494.379
Max. virtual memory (cumulated for all children) (KiB): 156304

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1784.53
system time used= 15.5756
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 139469
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 280848
involuntary context switches= 11135

runsolver used 0.660899 second user time and 1.38179 second system time

The end

Launcher Data 

Begin job on node124 at 2010-07-02 09:28:43
IDJOB=2697433
IDBENCH=79290
IDSOLVER=1210
FILE ID=node124/2697433-1278055723
PBS_JOBID= 11198934
Free space on /tmp= 130600 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900581320--soft-0-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697433-1278055723/watcher-2697433-1278055723 -o /tmp/evaluation-result-2697433-1278055723/solver-2697433-1278055723 -C 1800 -W 2000 -M 3800  run HOME/instance-2697433-1278055723.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 85ba7bcc847457eeeb822d6fc8af8950
RANDOM SEED=1631707519

node124.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31035464 kB
Buffers:        225760 kB
Cached:        1292152 kB
SwapCached:          0 kB
Active:         839760 kB
Inactive:       957064 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31035464 kB
SwapTotal:     4192956 kB
SwapFree:      4192956 kB
Dirty:           16716 kB
Writeback:           0 kB
AnonPages:      278888 kB
Mapped:          18840 kB
Slab:            84024 kB
PageTables:       5304 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  20668420 kB
Committed_AS:   522344 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    265212 kB
VmallocChunk: 34359471699 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 130600 MiB
End job on node124 at 2010-07-02 09:34:47