Trace number 2697425

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.19 378.717

General information on the benchmark

Name/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/
tsp/normalized-t3002.11tsp11.1900554420--soft-0-100-0.wbo
MD5SUMce7c3a365231565303f31d834d531ee5
Bench CategorySOFT-SMALLINT-LIN (only soft constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark38.4642
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables231
Total number of constraints2707
Number of soft constraints2707
Number of constraints which are clauses2684
Number of constraints which are cardinality constraints (but not clauses)22
Number of constraints which are nor clauses,nor cardinality constraints1
Minimum length of a constraint2
Maximum length of a constraint110
Top cost 134314
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 134313
Biggest number in a constraint 25
Number of bits of the biggest number in a constraint 5
Biggest sum of numbers in a constraint 689
Number of bits of the biggest sum of numbers10
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697425-1278055702.wbo.
0.00/0.00	c Instance file HOME/instance-2697425-1278055702.wbo
0.00/0.00	c File size is 96743 bytes.
1.39/1.44	c Highest Coefficient sum: 689
1.39/1.44	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
1.39/1.44	c ============================[ Problem Statistics ]=============================
1.39/1.44	c |                                                                             |
1.39/1.44	c |  Parsing time:          1.44         s                                      |
1.39/1.44	c |  Number Variables:      2938                                                |
1.39/1.44	c |  Number Constraints:    2708                                                |
1.39/1.44	c ===============================================================================
1800.15/378.71	c Remaining time 1798.56 sec.
1800.15/378.71	c 
1800.15/378.71	c *** INTERRUPTED ***
1800.15/378.71	s UNKNOWN
1800.15/378.71	c Total CPU time              : 1779.78 s
1800.15/378.71	c Memory used           : 166.27 MB
1800.15/378.71	c CPU time              : 1779.78 s
1800.15/378.71	c 
1800.15/378.71	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697425-1278055702/watcher-2697425-1278055702 -o /tmp/evaluation-result-2697425-1278055702/solver-2697425-1278055702 -C 1800 -W 2000 -M 3800 run HOME/instance-2697425-1278055702.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.27 6.87 4.67 4/179 21355
/proc/meminfo: memFree=31368380/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0

[startup+0.0229309 s]
/proc/loadavg: 6.27 6.87 4.67 4/179 21355
/proc/meminfo: memFree=31368380/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.100972 s]
/proc/loadavg: 6.27 6.87 4.67 4/179 21355
/proc/meminfo: memFree=31368380/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.300905 s]
/proc/loadavg: 6.27 6.87 4.67 4/179 21355
/proc/meminfo: memFree=31368380/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.700824 s]
/proc/loadavg: 6.27 6.87 4.67 4/179 21355
/proc/meminfo: memFree=31368380/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.50067 s]
/proc/loadavg: 5.93 6.79 4.66 3/181 21357
/proc/meminfo: memFree=31366008/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=25812 CPUtime=1.49
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 3306 0 0 0 149 0 0 0 25 0 1 0 146715855 26431488 2872 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 0 4096 16387 0 0 0 17 5 0 0 0
/proc/21356/statm: 6453 2872 959 2985 0 2995 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 34512

[startup+3.10036 s]
/proc/loadavg: 5.93 6.79 4.66 8/188 21378
/proc/meminfo: memFree=31330364/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=86624 CPUtime=5.6
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 21199 0 0 0 550 10 0 0 16 0 1 0 146715855 88702976 7542 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 0 4096 16387 0 0 0 17 5 0 0 0
/proc/21356/statm: 21656 7542 1071 2985 0 18198 0
Current children cumulated CPU time (s) 5.6
Current children cumulated vsize (KiB) 95324

[startup+6.30064 s]
/proc/loadavg: 6.17 6.82 4.68 7/188 21406
/proc/meminfo: memFree=31320196/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=145628 CPUtime=14.47
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 39356 0 0 0 1429 18 0 0 19 0 8 0 146715855 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21356/statm: 36407 13034 1082 2985 0 32949 0
[pid=21356/tid=21400] ppid=21355 vsize=145628 CPUtime=1.07
/proc/21356/task/21400/stat : 21400 (pb_cplex) R 21355 21355 20862 0 -1 4202560 189 0 0 0 107 0 0 0 19 0 8 0 146716251 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 139046546 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21356/tid=21401] ppid=21355 vsize=145628 CPUtime=0.99
/proc/21356/task/21401/stat : 21401 (pb_cplex) R 21355 21355 20862 0 -1 4202560 98 0 0 0 99 0 0 0 16 0 8 0 146716251 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 136007605 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21356/tid=21402] ppid=21355 vsize=145628 CPUtime=0.84
/proc/21356/task/21402/stat : 21402 (pb_cplex) R 21355 21355 20862 0 -1 4202560 1045 0 0 0 84 0 0 0 16 0 8 0 146716251 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21356/tid=21403] ppid=21355 vsize=145628 CPUtime=0.74
/proc/21356/task/21403/stat : 21403 (pb_cplex) R 21355 21355 20862 0 -1 4202560 251 0 0 0 74 0 0 0 16 0 8 0 146716251 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21404] ppid=21355 vsize=145628 CPUtime=0.95
/proc/21356/task/21404/stat : 21404 (pb_cplex) R 21355 21355 20862 0 -1 4202560 3331 0 0 0 95 0 0 0 16 0 8 0 146716251 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 136017485 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21356/tid=21405] ppid=21355 vsize=145628 CPUtime=0.58
/proc/21356/task/21405/stat : 21405 (pb_cplex) R 21355 21355 20862 0 -1 4202560 261 0 0 0 58 0 0 0 16 0 8 0 146716251 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21356/tid=21406] ppid=21355 vsize=145628 CPUtime=0.64
/proc/21356/task/21406/stat : 21406 (pb_cplex) R 21355 21355 20862 0 -1 4202560 1031 0 0 0 64 0 0 0 16 0 8 0 146716251 149123072 13034 4089446400 134512640 146736320 4288739920 18446744073709551615 136001604 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 14.47
Current children cumulated vsize (KiB) 154328

[startup+12.7004 s]
/proc/loadavg: 5.84 6.74 4.67 10/188 21406
/proc/meminfo: memFree=31314024/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=150748 CPUtime=42.95
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 46795 0 0 0 4247 48 0 0 16 0 8 0 146715855 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21356/statm: 37687 14214 1082 2985 0 34229 0
[pid=21356/tid=21400] ppid=21355 vsize=150748 CPUtime=4.55
/proc/21356/task/21400/stat : 21400 (pb_cplex) R 21355 21355 20862 0 -1 4202560 386 0 0 0 451 4 0 0 15 0 8 0 146716251 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 135441119 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21356/tid=21401] ppid=21355 vsize=150748 CPUtime=4.59
/proc/21356/task/21401/stat : 21401 (pb_cplex) S 21355 21355 20862 0 -1 4202560 216 0 0 0 455 4 0 0 16 0 8 0 146716251 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 5 0 0 0
[pid=21356/tid=21402] ppid=21355 vsize=150748 CPUtime=4.37
/proc/21356/task/21402/stat : 21402 (pb_cplex) R 21355 21355 20862 0 -1 4202560 1565 0 0 0 434 3 0 0 15 0 8 0 146716251 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21356/tid=21403] ppid=21355 vsize=150748 CPUtime=4.21
/proc/21356/task/21403/stat : 21403 (pb_cplex) S 21355 21355 20862 0 -1 4202560 2637 0 0 0 417 4 0 0 15 0 8 0 146716251 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 4 0 0 0
[pid=21356/tid=21404] ppid=21355 vsize=150748 CPUtime=4.46
/proc/21356/task/21404/stat : 21404 (pb_cplex) S 21355 21355 20862 0 -1 4202560 6326 0 0 0 442 4 0 0 15 0 8 0 146716251 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 0 0 0 0
[pid=21356/tid=21405] ppid=21355 vsize=150748 CPUtime=4.27
/proc/21356/task/21405/stat : 21405 (pb_cplex) S 21355 21355 20862 0 -1 4202560 545 0 0 0 423 4 0 0 16 0 8 0 146716251 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 7 0 0 0
[pid=21356/tid=21406] ppid=21355 vsize=150748 CPUtime=4.31
/proc/21356/task/21406/stat : 21406 (pb_cplex) S 21355 21355 20862 0 -1 4202560 1360 0 0 0 427 4 0 0 15 0 8 0 146716251 154365952 14214 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 42.95
Current children cumulated vsize (KiB) 159448

[startup+25.501 s]
/proc/loadavg: 5.79 6.70 4.67 4/188 21407
/proc/meminfo: memFree=31304064/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=160576 CPUtime=100.78
/proc/21356/stat : 21356 (pb_cplex) S 21355 21355 20862 0 -1 4202496 77715 0 0 0 9975 103 0 0 16 0 8 0 146715855 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21356/statm: 40144 16833 1092 2985 0 36686 0
[pid=21356/tid=21400] ppid=21355 vsize=160576 CPUtime=13.39
/proc/21356/task/21400/stat : 21400 (pb_cplex) S 21355 21355 20862 0 -1 4202560 10441 0 0 0 1326 13 0 0 25 0 8 0 146716251 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 2 0 0 0
[pid=21356/tid=21401] ppid=21355 vsize=160576 CPUtime=9.19
/proc/21356/task/21401/stat : 21401 (pb_cplex) S 21355 21355 20862 0 -1 4202560 1674 0 0 0 907 12 0 0 15 0 8 0 146716251 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 3 0 0 0
[pid=21356/tid=21402] ppid=21355 vsize=160576 CPUtime=12.79
/proc/21356/task/21402/stat : 21402 (pb_cplex) S 21355 21355 20862 0 -1 4202560 4830 0 0 0 1269 10 0 0 22 0 8 0 146716251 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 3 0 0 0
[pid=21356/tid=21403] ppid=21355 vsize=160576 CPUtime=7.96
/proc/21356/task/21403/stat : 21403 (pb_cplex) S 21355 21355 20862 0 -1 4202560 6069 0 0 0 786 10 0 0 15 0 8 0 146716251 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 4 0 0 0
[pid=21356/tid=21404] ppid=21355 vsize=160576 CPUtime=15.36
/proc/21356/task/21404/stat : 21404 (pb_cplex) R 21355 21355 20862 0 -1 4202560 12444 0 0 0 1524 12 0 0 25 0 8 0 146716251 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 136008157 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21405] ppid=21355 vsize=160576 CPUtime=15.06
/proc/21356/task/21405/stat : 21405 (pb_cplex) R 21355 21355 20862 0 -1 4202560 4194 0 0 0 1496 10 0 0 25 0 8 0 146716251 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 136008230 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21356/tid=21406] ppid=21355 vsize=160576 CPUtime=10.88
/proc/21356/task/21406/stat : 21406 (pb_cplex) S 21355 21355 20862 0 -1 4202560 3283 0 0 0 1078 10 0 0 19 0 8 0 146716251 164429824 16833 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.78
Current children cumulated vsize (KiB) 169276

[startup+51.167 s]
/proc/loadavg: 6.04 6.64 4.72 7/188 21421
/proc/meminfo: memFree=31304020/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=162012 CPUtime=194.57
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 148567 0 0 0 19261 196 0 0 15 0 8 0 146715855 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 139043575 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/21356/statm: 40503 16621 1110 2985 0 37045 0
[pid=21356/tid=21415] ppid=21355 vsize=162012 CPUtime=10.95
/proc/21356/task/21415/stat : 21415 (pb_cplex) R 21355 21355 20862 0 -1 4202560 650 0 0 0 1085 10 0 0 15 0 8 0 146718673 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 135454087 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21416] ppid=21355 vsize=162012 CPUtime=10.93
/proc/21356/task/21416/stat : 21416 (pb_cplex) R 21355 21355 20862 0 -1 4202560 4807 0 0 0 1083 10 0 0 15 0 8 0 146718673 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21356/tid=21417] ppid=21355 vsize=162012 CPUtime=10.49
/proc/21356/task/21417/stat : 21417 (pb_cplex) R 21355 21355 20862 0 -1 4202560 7390 0 0 0 1039 10 0 0 15 0 8 0 146718673 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21356/tid=21418] ppid=21355 vsize=162012 CPUtime=9.98
/proc/21356/task/21418/stat : 21418 (pb_cplex) R 21355 21355 20862 0 -1 4202560 2048 0 0 0 988 10 0 0 15 0 8 0 146718673 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 136403116 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21419] ppid=21355 vsize=162012 CPUtime=10.48
/proc/21356/task/21419/stat : 21419 (pb_cplex) R 21355 21355 20862 0 -1 4202560 13590 0 0 0 1034 14 0 0 15 0 8 0 146718673 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21356/tid=21420] ppid=21355 vsize=162012 CPUtime=11.36
/proc/21356/task/21420/stat : 21420 (pb_cplex) R 21355 21355 20862 0 -1 4202560 13627 0 0 0 1124 12 0 0 15 0 8 0 146718673 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21356/tid=21421] ppid=21355 vsize=162012 CPUtime=11.22
/proc/21356/task/21421/stat : 21421 (pb_cplex) R 21355 21355 20862 0 -1 4202560 11957 0 0 0 1114 8 0 0 15 0 8 0 146718673 165900288 16621 4089446400 134512640 146736320 4288739920 18446744073709551615 136017791 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 194.57
Current children cumulated vsize (KiB) 170712

[startup+102.307 s]
/proc/loadavg: 6.43 6.63 4.81 12/195 21437
/proc/meminfo: memFree=30880824/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=164060 CPUtime=426.2
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 222208 0 0 0 42165 455 0 0 16 0 8 0 146715855 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21356/statm: 41015 17153 1110 2985 0 37557 0
[pid=21356/tid=21415] ppid=21355 vsize=164060 CPUtime=40.2
/proc/21356/task/21415/stat : 21415 (pb_cplex) R 21355 21355 20862 0 -1 4202560 12414 0 0 0 3976 44 0 0 16 0 8 0 146718673 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 137371800 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21356/tid=21416] ppid=21355 vsize=164060 CPUtime=39.63
/proc/21356/task/21416/stat : 21416 (pb_cplex) R 21355 21355 20862 0 -1 4202560 12478 0 0 0 3923 40 0 0 16 0 8 0 146718673 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21356/tid=21417] ppid=21355 vsize=164060 CPUtime=39.21
/proc/21356/task/21417/stat : 21417 (pb_cplex) R 21355 21355 20862 0 -1 4202560 16078 0 0 0 3874 47 0 0 15 0 8 0 146718673 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21356/tid=21418] ppid=21355 vsize=164060 CPUtime=39.06
/proc/21356/task/21418/stat : 21418 (pb_cplex) R 21355 21355 20862 0 -1 4202560 12712 0 0 0 3865 41 0 0 15 0 8 0 146718673 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0

################
# More data... #
################

/proc/21356/task/21419/stat : 21419 (pb_cplex) R 21355 21355 20862 0 -1 4202560 18988 0 0 0 3899 48 0 0 15 0 8 0 146718673 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21420] ppid=21355 vsize=164060 CPUtime=40.46
/proc/21356/task/21420/stat : 21420 (pb_cplex) R 21355 21355 20862 0 -1 4202560 21487 0 0 0 4001 45 0 0 15 0 8 0 146718673 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21356/tid=21421] ppid=21355 vsize=164060 CPUtime=40.09
/proc/21356/task/21421/stat : 21421 (pb_cplex) R 21355 21355 20862 0 -1 4202560 25508 0 0 0 3970 39 0 0 15 0 8 0 146718673 167997440 17153 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 426.2
Current children cumulated vsize (KiB) 172760

[startup+162.301 s]
/proc/loadavg: 12.46 8.39 5.53 18/195 21438
/proc/meminfo: memFree=30831340/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=169180 CPUtime=831.65
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 269742 0 0 0 82254 911 0 0 15 0 8 0 146715855 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 0 0 0 0
/proc/21356/statm: 42295 18041 1110 2985 0 38837 0
[pid=21356/tid=21415] ppid=21355 vsize=169180 CPUtime=90.48
/proc/21356/task/21415/stat : 21415 (pb_cplex) R 21355 21355 20862 0 -1 4202560 16479 0 0 0 8946 102 0 0 15 0 8 0 146718673 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21356/tid=21416] ppid=21355 vsize=169180 CPUtime=90.12
/proc/21356/task/21416/stat : 21416 (pb_cplex) R 21355 21355 20862 0 -1 4202560 20127 0 0 0 8908 104 0 0 15 0 8 0 146718673 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21356/tid=21417] ppid=21355 vsize=169180 CPUtime=89.57
/proc/21356/task/21417/stat : 21417 (pb_cplex) R 21355 21355 20862 0 -1 4202560 21787 0 0 0 8852 105 0 0 15 0 8 0 146718673 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21356/tid=21418] ppid=21355 vsize=169180 CPUtime=89.84
/proc/21356/task/21418/stat : 21418 (pb_cplex) R 21355 21355 20862 0 -1 4202560 16824 0 0 0 8882 102 0 0 16 0 8 0 146718673 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21356/tid=21419] ppid=21355 vsize=169180 CPUtime=89.73
/proc/21356/task/21419/stat : 21419 (pb_cplex) R 21355 21355 20862 0 -1 4202560 25793 0 0 0 8865 108 0 0 16 0 8 0 146718673 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 135857490 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21420] ppid=21355 vsize=169180 CPUtime=91.27
/proc/21356/task/21420/stat : 21420 (pb_cplex) R 21355 21355 20862 0 -1 4202560 29345 0 0 0 9023 104 0 0 16 0 8 0 146718673 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 136017748 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21356/tid=21421] ppid=21355 vsize=169180 CPUtime=91.14
/proc/21356/task/21421/stat : 21421 (pb_cplex) R 21355 21355 20862 0 -1 4202560 31784 0 0 0 9014 100 0 0 15 0 8 0 146718673 173240320 18041 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 831.65
Current children cumulated vsize (KiB) 177880

[startup+222.301 s]
/proc/loadavg: 14.44 9.69 6.16 17/195 21440
/proc/meminfo: memFree=30830144/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=166108 CPUtime=1228.37
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 280789 0 0 0 121452 1385 0 0 15 0 8 0 146715855 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21356/statm: 41527 17576 1110 2985 0 38069 0
[pid=21356/tid=21415] ppid=21355 vsize=166108 CPUtime=139.66
/proc/21356/task/21415/stat : 21415 (pb_cplex) R 21355 21355 20862 0 -1 4202560 17536 0 0 0 13801 165 0 0 16 0 8 0 146718673 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 135468117 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21356/tid=21416] ppid=21355 vsize=166108 CPUtime=139.2
/proc/21356/task/21416/stat : 21416 (pb_cplex) R 21355 21355 20862 0 -1 4202560 20613 0 0 0 13757 163 0 0 16 0 8 0 146718673 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21356/tid=21417] ppid=21355 vsize=166108 CPUtime=139.08
/proc/21356/task/21417/stat : 21417 (pb_cplex) R 21355 21355 20862 0 -1 4202560 23997 0 0 0 13750 158 0 0 16 0 8 0 146718673 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 139044509 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=21356/tid=21418] ppid=21355 vsize=166108 CPUtime=139.49
/proc/21356/task/21418/stat : 21418 (pb_cplex) R 21355 21355 20862 0 -1 4202560 18785 0 0 0 13800 149 0 0 15 0 8 0 146718673 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=21356/tid=21419] ppid=21355 vsize=166108 CPUtime=139.33
/proc/21356/task/21419/stat : 21419 (pb_cplex) R 21355 21355 20862 0 -1 4202560 26275 0 0 0 13763 170 0 0 15 0 8 0 146718673 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 137341772 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=21356/tid=21420] ppid=21355 vsize=166108 CPUtime=141.07
/proc/21356/task/21420/stat : 21420 (pb_cplex) R 21355 21355 20862 0 -1 4202560 31252 0 0 0 13937 170 0 0 16 0 8 0 146718673 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21356/tid=21421] ppid=21355 vsize=166108 CPUtime=140.87
/proc/21356/task/21421/stat : 21421 (pb_cplex) R 21355 21355 20862 0 -1 4202560 33801 0 0 0 13923 164 0 0 16 0 8 0 146718673 170094592 17576 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 1228.37
Current children cumulated vsize (KiB) 174808

[startup+282.357 s]
/proc/loadavg: 14.10 10.51 6.66 10/195 21449
/proc/meminfo: memFree=30821276/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=170572 CPUtime=1606.22
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 290785 0 0 0 158791 1831 0 0 23 0 8 0 146715855 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 138377200 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/21356/statm: 42643 19019 1110 2985 0 39185 0
[pid=21356/tid=21443] ppid=21355 vsize=170572 CPUtime=2.7
/proc/21356/task/21443/stat : 21443 (pb_cplex) S 21355 21355 20862 0 -1 4202560 138 0 0 0 269 1 0 0 15 0 8 0 146743478 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 0 0 0 0
[pid=21356/tid=21444] ppid=21355 vsize=170572 CPUtime=2.61
/proc/21356/task/21444/stat : 21444 (pb_cplex) S 21355 21355 20862 0 -1 4202560 17 0 0 0 258 3 0 0 15 0 8 0 146743478 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 1 0 0 0
[pid=21356/tid=21445] ppid=21355 vsize=170572 CPUtime=2.68
/proc/21356/task/21445/stat : 21445 (pb_cplex) S 21355 21355 20862 0 -1 4202560 216 0 0 0 266 2 0 0 15 0 8 0 146743478 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 3 0 0 0
[pid=21356/tid=21446] ppid=21355 vsize=170572 CPUtime=2.65
/proc/21356/task/21446/stat : 21446 (pb_cplex) S 21355 21355 20862 0 -1 4202560 194 0 0 0 263 2 0 0 16 0 8 0 146743478 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 6 0 0 0
[pid=21356/tid=21447] ppid=21355 vsize=170572 CPUtime=2.69
/proc/21356/task/21447/stat : 21447 (pb_cplex) S 21355 21355 20862 0 -1 4202560 122 0 0 0 267 2 0 0 16 0 8 0 146743478 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 5 0 0 0
[pid=21356/tid=21448] ppid=21355 vsize=170572 CPUtime=2.67
/proc/21356/task/21448/stat : 21448 (pb_cplex) S 21355 21355 20862 0 -1 4202560 224 0 0 0 265 2 0 0 16 0 8 0 146743478 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 4 0 0 0
[pid=21356/tid=21449] ppid=21355 vsize=170572 CPUtime=2.69
/proc/21356/task/21449/stat : 21449 (pb_cplex) S 21355 21355 20862 0 -1 4202560 430 0 0 0 266 3 0 0 15 0 8 0 146743478 174665728 19019 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1606.22
Current children cumulated vsize (KiB) 179272

[startup+342.301 s]
/proc/loadavg: 12.26 10.66 6.95 15/195 21450
/proc/meminfo: memFree=30812292/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=170260 CPUtime=1695.08
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 291455 0 0 0 167589 1919 0 0 25 0 8 0 146715855 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 5 0 0 0
/proc/21356/statm: 42565 19163 1110 2985 0 39107 0
[pid=21356/tid=21443] ppid=21355 vsize=170260 CPUtime=13.74
/proc/21356/task/21443/stat : 21443 (pb_cplex) S 21355 21355 20862 0 -1 4202560 371 0 0 0 1366 8 0 0 15 0 8 0 146743478 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 0 0 0 0
[pid=21356/tid=21444] ppid=21355 vsize=170260 CPUtime=13.67
/proc/21356/task/21444/stat : 21444 (pb_cplex) S 21355 21355 20862 0 -1 4202560 111 0 0 0 1353 14 0 0 15 0 8 0 146743478 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 1 0 0 0
[pid=21356/tid=21445] ppid=21355 vsize=170260 CPUtime=13.75
/proc/21356/task/21445/stat : 21445 (pb_cplex) S 21355 21355 20862 0 -1 4202560 335 0 0 0 1362 13 0 0 15 0 8 0 146743478 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 3 0 0 0
[pid=21356/tid=21446] ppid=21355 vsize=170260 CPUtime=13.55
/proc/21356/task/21446/stat : 21446 (pb_cplex) S 21355 21355 20862 0 -1 4202560 321 0 0 0 1340 15 0 0 15 0 8 0 146743478 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 6 0 0 0
[pid=21356/tid=21447] ppid=21355 vsize=170260 CPUtime=13.78
/proc/21356/task/21447/stat : 21447 (pb_cplex) S 21355 21355 20862 0 -1 4202560 165 0 0 0 1363 15 0 0 15 0 8 0 146743478 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 4 0 0 0
[pid=21356/tid=21448] ppid=21355 vsize=170260 CPUtime=13.84
/proc/21356/task/21448/stat : 21448 (pb_cplex) S 21355 21355 20862 0 -1 4202560 229 0 0 0 1372 12 0 0 15 0 8 0 146743478 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 7 0 0 0
[pid=21356/tid=21449] ppid=21355 vsize=170260 CPUtime=13.7
/proc/21356/task/21449/stat : 21449 (pb_cplex) S 21355 21355 20862 0 -1 4202560 466 0 0 0 1357 13 0 0 15 0 8 0 146743478 174346240 19163 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1695.08
Current children cumulated vsize (KiB) 178960



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+378.701 s]
/proc/loadavg: 12.06 10.78 7.13 16/195 21451
/proc/meminfo: memFree=30806452/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=170260 CPUtime=1800.15
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 291466 0 0 0 177975 2040 0 0 25 0 8 0 146715855 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 135921175 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21356/statm: 42565 19174 1110 2985 0 39107 0
[pid=21356/tid=21443] ppid=21355 vsize=170260 CPUtime=26.83
/proc/21356/task/21443/stat : 21443 (pb_cplex) S 21355 21355 20862 0 -1 4202560 374 0 0 0 2668 15 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 0 0 0 0
[pid=21356/tid=21444] ppid=21355 vsize=170260 CPUtime=26.66
/proc/21356/task/21444/stat : 21444 (pb_cplex) R 21355 21355 20862 0 -1 4202560 117 0 0 0 2637 29 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21356/tid=21445] ppid=21355 vsize=170260 CPUtime=26.89
/proc/21356/task/21445/stat : 21445 (pb_cplex) R 21355 21355 20862 0 -1 4202560 335 0 0 0 2662 27 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21356/tid=21446] ppid=21355 vsize=170260 CPUtime=26.45
/proc/21356/task/21446/stat : 21446 (pb_cplex) R 21355 21355 20862 0 -1 4202560 321 0 0 0 2616 29 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21447] ppid=21355 vsize=170260 CPUtime=26.65
/proc/21356/task/21447/stat : 21447 (pb_cplex) R 21355 21355 20862 0 -1 4202560 165 0 0 0 2636 29 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21356/tid=21448] ppid=21355 vsize=170260 CPUtime=26.99
/proc/21356/task/21448/stat : 21448 (pb_cplex) R 21355 21355 20862 0 -1 4202560 229 0 0 0 2671 28 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 136017769 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21356/tid=21449] ppid=21355 vsize=170260 CPUtime=26.72
/proc/21356/task/21449/stat : 21449 (pb_cplex) R 21355 21355 20862 0 -1 4202560 467 0 0 0 2644 28 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1800.15
Current children cumulated vsize (KiB) 178960

Sending SIGTERM to process tree (bottom up)
SSolver just ended. Dumping a history of the last processes samples

[startup+378.701 s]
/proc/loadavg: 12.06 10.78 7.13 16/195 21451
/proc/meminfo: memFree=30806452/32950928 swapFree=67111528/67111528
[pid=21355] ppid=21353 vsize=8700 CPUtime=0
/proc/21355/stat : 21355 (run) S 21353 21355 20862 0 -1 4202496 365 0 0 0 0 0 0 0 25 0 1 0 146715855 8908800 249 4089446400 4194304 4922060 140735086672208 18446744073709551615 208080051333 0 65536 4100 65538 18446744071562232103 0 0 17 6 0 0 0
/proc/21355/statm: 2175 249 208 178 0 68 0
[pid=21356] ppid=21355 vsize=170260 CPUtime=1800.15
/proc/21356/stat : 21356 (pb_cplex) R 21355 21355 20862 0 -1 4202496 291466 0 0 0 177975 2040 0 0 25 0 8 0 146715855 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 135921175 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/21356/statm: 42565 19174 1110 2985 0 39107 0
[pid=21356/tid=21443] ppid=21355 vsize=170260 CPUtime=26.83
/proc/21356/task/21443/stat : 21443 (pb_cplex) S 21355 21355 20862 0 -1 4202560 374 0 0 0 2668 15 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144580073 0 2 4096 16387 18446604454224124608 0 0 -1 0 0 0 0
[pid=21356/tid=21444] ppid=21355 vsize=170260 CPUtime=26.66
/proc/21356/task/21444/stat : 21444 (pb_cplex) R 21355 21355 20862 0 -1 4202560 117 0 0 0 2637 29 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=21356/tid=21445] ppid=21355 vsize=170260 CPUtime=26.89
/proc/21356/task/21445/stat : 21445 (pb_cplex) R 21355 21355 20862 0 -1 4202560 335 0 0 0 2662 27 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=21356/tid=21446] ppid=21355 vsize=170260 CPUtime=26.45
/proc/21356/task/21446/stat : 21446 (pb_cplex) R 21355 21355 20862 0 -1 4202560 321 0 0 0 2616 29 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=21356/tid=21447] ppid=21355 vsize=170260 CPUtime=26.65
/proc/21356/task/21447/stat : 21447 (pb_cplex) R 21355 21355 20862 0 -1 4202560 165 0 0 0 2636 29 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=21356/tid=21448] ppid=21355 vsize=170260 CPUtime=26.99
/proc/21356/task/21448/stat : 21448 (pb_cplex) R 21355 21355 20862 0 -1 4202560 229 0 0 0 2671 28 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 136017769 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=21356/tid=21449] ppid=21355 vsize=170260 CPUtime=26.72
/proc/21356/task/21449/stat : 21449 (pb_cplex) R 21355 21355 20862 0 -1 4202560 467 0 0 0 2644 28 0 0 15 0 8 0 146743478 174346240 19174 4089446400 134512640 146736320 4288739920 18446744073709551615 144757825 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1800.15
Current children cumulated vsize (KiB) 178960

Child status: 0
Real time (s): 378.717
CPU time (s): 1800.19
CPU user time (s): 1779.79
CPU system time (s): 20.4089
CPU usage (%): 475.34
Max. virtual memory (cumulated for all children) (KiB): 181976

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1779.79
system time used= 20.4089
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 291840
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 380449
involuntary context switches= 9335

runsolver used 0.687895 second user time and 1.62475 second system time

The end

Launcher Data 

Begin job on node127 at 2010-07-02 09:28:22
IDJOB=2697425
IDBENCH=79286
IDSOLVER=1210
FILE ID=node127/2697425-1278055702
PBS_JOBID= 11198928
Free space on /tmp= 74172 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/SOFT-SMALLINT-LIN/PB06/submitted-PB06/namasivayam/tsp/normalized-t3002.11tsp11.1900554420--soft-0-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697425-1278055702/watcher-2697425-1278055702 -o /tmp/evaluation-result-2697425-1278055702/solver-2697425-1278055702 -C 1800 -W 2000 -M 3800  run HOME/instance-2697425-1278055702.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= ce7c3a365231565303f31d834d531ee5
RANDOM SEED=1466307901

node127.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.72
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      31368908 kB
Buffers:        221256 kB
Cached:        1033640 kB
SwapCached:          0 kB
Active:         743248 kB
Inactive:       689948 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      31368908 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16744 kB
Writeback:           0 kB
AnonPages:      178020 kB
Mapped:          18776 kB
Slab:            81608 kB
PageTables:       4784 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   386624 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74156 MiB
End job on node127 at 2010-07-02 09:34:41