Trace number 2697434

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
pb_cplex 2010-07-01? (TO) 1800.25 755.718

General information on the benchmark

Name/PARTIAL-SMALLINT-LIN/PB06/web/uclid_pb_benchmarks/
normalized-ooo.rf8.ucl--soft-66-100-0.wbo
MD5SUM321b555888f42ada31256d1d017f1ca8
Bench CategoryPARTIAL-SMALLINT-LIN (both soft and hard constraints, small integers, linear constraints)
Best result obtained on this benchmarkMOPT
Best cost obtained on this benchmark1
Best CPU time to get the best result obtained on this benchmark0.289955
Max-Satisfiable
Max-(Un)Satisfiability was proved
Best value of the cost
Optimality of the best cost was proved
Number of variables6832
Total number of constraints19477
Number of soft constraints6629
Number of constraints which are clauses18953
Number of constraints which are cardinality constraints (but not clauses)0
Number of constraints which are nor clauses,nor cardinality constraints524
Minimum length of a constraint1
Maximum length of a constraint11
Top cost 330532
Min constraint cost 1
Max constraint cost 100
Sum of constraints costs 330531
Biggest number in a constraint 33
Number of bits of the biggest number in a constraint 6
Biggest sum of numbers in a constraint 126
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.00/0.00	c pb_cplex 1.0 - 20100701
0.00/0.00	c Parsing wbo file format. File HOME/instance-2697434-1278055347.wbo.
0.00/0.00	c Instance file HOME/instance-2697434-1278055347.wbo
0.00/0.00	c File size is 610067 bytes.
9.18/11.65	c Highest Coefficient sum: 126
9.18/11.65	c  IBM ILOG License Manager: "IBM ILOG Optimization Suite for Academic Initiative" is accessing CPLEX 12 with option(s): "e m b q ".
9.18/11.66	c ============================[ Problem Statistics ]=============================
9.18/11.66	c |                                                                             |
9.18/11.66	c |  Parsing time:          9.25         s                                      |
9.18/11.66	c |  Number Variables:      13461                                               |
9.18/11.67	c |  Number Constraints:    19478                                               |
9.18/11.67	c ===============================================================================
1800.25/755.71	c Remaining time 1790.75 sec.
1800.25/755.71	c 
1800.25/755.71	c *** INTERRUPTED ***
1800.25/755.71	s UNKNOWN
1800.25/755.71	c Total CPU time              : 1797.29 s
1800.25/755.71	c Memory used           : 337.07 MB
1800.25/755.71	c CPU time              : 1797.29 s
1800.25/755.71	c 
1800.25/755.71	c *** INTERRUPTED ***

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver version 3.2.9 (svn:492) (c) roussel@cril.univ-artois.fr

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-2697434-1278055347/watcher-2697434-1278055347 -o /tmp/evaluation-result-2697434-1278055347/solver-2697434-1278055347 -C 1800 -W 2000 -M 3800 run HOME/instance-2697434-1278055347.wbo 

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 2000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 3891200 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 3942400 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 9.03 7.37 3.71 9/190 23251
/proc/meminfo: memFree=30848664/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0

[startup+0.107383 s]
/proc/loadavg: 9.03 7.37 3.71 9/190 23251
/proc/meminfo: memFree=30848664/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.252013 s]
/proc/loadavg: 9.03 7.37 3.71 9/190 23251
/proc/meminfo: memFree=30848664/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.301064 s]
/proc/loadavg: 9.03 7.37 3.71 9/190 23251
/proc/meminfo: memFree=30848664/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+0.74592 s]
/proc/loadavg: 9.03 7.37 3.71 9/190 23251
/proc/meminfo: memFree=30848664/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 8700

[startup+1.51478 s]
/proc/loadavg: 9.03 7.37 3.71 11/192 23254
/proc/meminfo: memFree=30843688/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=15624 CPUtime=1.09
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 1003 0 0 0 109 0 0 0 25 0 1 0 146830906 15998976 956 4089446400 134512640 146736320 4287828144 18446744073709551615 140679015 0 0 4096 16387 0 0 0 17 6 0 0 0
/proc/23252/statm: 3906 956 335 2985 0 691 0
Current children cumulated CPU time (s) 1.09
Current children cumulated vsize (KiB) 24324

[startup+3.10052 s]
/proc/loadavg: 9.03 7.37 3.71 3/192 23254
/proc/meminfo: memFree=30843936/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=16840 CPUtime=2.67
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 1317 0 0 0 267 0 0 0 25 0 1 0 146830906 17244160 1270 4089446400 134512640 146736320 4287828144 18446744073709551615 141014048 0 0 4096 16387 0 0 0 17 4 0 0 0
/proc/23252/statm: 4210 1270 361 2985 0 995 0
Current children cumulated CPU time (s) 2.67
Current children cumulated vsize (KiB) 25540

[startup+6.36967 s]
/proc/loadavg: 9.11 7.42 3.75 3/192 23254
/proc/meminfo: memFree=30841084/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=18400 CPUtime=5.27
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 1750 0 0 0 527 0 0 0 25 0 1 0 146830906 18841600 1670 4089446400 134512640 146736320 4287828144 18446744073709551615 140408977 0 0 4096 16387 0 0 0 17 2 0 0 0
/proc/23252/statm: 4600 1670 392 2985 0 1385 0
Current children cumulated CPU time (s) 5.27
Current children cumulated vsize (KiB) 27100

[startup+12.7007 s]
/proc/loadavg: 9.26 7.47 3.79 4/192 23254
/proc/meminfo: memFree=30823748/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=71980 CPUtime=10.27
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 19165 0 0 0 1024 3 0 0 25 0 1 0 146830906 73707520 10041 4089446400 134512640 146736320 4287828144 18446744073709551615 136007083 0 0 4096 16387 0 0 0 17 0 0 0 0
/proc/23252/statm: 17995 10041 1015 2985 0 14537 0
Current children cumulated CPU time (s) 10.27
Current children cumulated vsize (KiB) 80680

[startup+25.5114 s]
/proc/loadavg: 8.17 7.32 3.79 11/192 23254
/proc/meminfo: memFree=30804340/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=73432 CPUtime=18.92
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 54523 0 0 0 1884 8 0 0 25 0 1 0 146830906 75194368 11125 4089446400 134512640 146736320 4287828144 18446744073709551615 136388081 0 0 4096 16387 0 0 0 17 4 0 0 0
/proc/23252/statm: 18358 11125 1045 2985 0 14900 0
Current children cumulated CPU time (s) 18.92
Current children cumulated vsize (KiB) 82132
heavy processes:

[startup+51.1008 s]
/proc/loadavg: 9.56 7.70 4.01 9/199 23262
/proc/meminfo: memFree=30645204/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=361984 CPUtime=68.51
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 213483 0 0 0 6811 40 0 0 25 0 8 0 146830906 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 136007337 0 0 4096 16387 18446744073709551615 0 0 17 4 0 0 0
/proc/23252/statm: 90496 50251 1106 2985 0 87038 0
[pid=23252/tid=23255] ppid=23251 vsize=361984 CPUtime=4.87
/proc/23252/task/23255/stat : 23255 (pb_cplex) S 23251 23251 22816 0 -1 4202560 10147 0 0 0 485 2 0 0 25 0 8 0 146834324 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 2 0 0 0
[pid=23252/tid=23256] ppid=23251 vsize=361984 CPUtime=4.55
/proc/23252/task/23256/stat : 23256 (pb_cplex) R 23251 23251 22816 0 -1 4202560 12921 0 0 0 453 2 0 0 25 0 8 0 146834324 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 139044554 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23252/tid=23257] ppid=23251 vsize=361984 CPUtime=5.15
/proc/23252/task/23257/stat : 23257 (pb_cplex) R 23251 23251 22816 0 -1 4202560 14033 0 0 0 513 2 0 0 25 0 8 0 146834324 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 135454093 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=23252/tid=23258] ppid=23251 vsize=361984 CPUtime=5.45
/proc/23252/task/23258/stat : 23258 (pb_cplex) R 23251 23251 22816 0 -1 4202560 19477 0 0 0 543 2 0 0 25 0 8 0 146834324 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 136403109 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23252/tid=23259] ppid=23251 vsize=361984 CPUtime=5.77
/proc/23252/task/23259/stat : 23259 (pb_cplex) R 23251 23251 22816 0 -1 4202560 15201 0 0 0 573 4 0 0 25 0 8 0 146834324 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 136006886 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23252/tid=23260] ppid=23251 vsize=361984 CPUtime=5.65
/proc/23252/task/23260/stat : 23260 (pb_cplex) R 23251 23251 22816 0 -1 4202560 11214 0 0 0 563 2 0 0 25 0 8 0 146834324 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 136388127 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23252/tid=23261] ppid=23251 vsize=361984 CPUtime=6.21
/proc/23252/task/23261/stat : 23261 (pb_cplex) R 23251 23251 22816 0 -1 4202560 16271 0 0 0 618 3 0 0 25 0 8 0 146834324 370671616 50251 4089446400 134512640 146736320 4287828144 18446744073709551615 139044505 0 2 4096 16387 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 68.51
Current children cumulated vsize (KiB) 370684

[startup+102.3 s]
/proc/loadavg: 12.55 8.81 4.58 17/199 23263
/proc/meminfo: memFree=30620208/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=373844 CPUtime=170.18
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 373189 0 0 0 16939 79 0 0 18 0 8 0 146830906 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 136007319 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/23252/statm: 93461 56235 1095 2985 0 90003 0
[pid=23252/tid=23255] ppid=23251 vsize=373844 CPUtime=15.01
/proc/23252/task/23255/stat : 23255 (pb_cplex) R 23251 23251 22816 0 -1 4202560 27537 0 0 0 1494 7 0 0 18 0 8 0 146834324 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 139042916 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23252/tid=23256] ppid=23251 vsize=373844 CPUtime=17.33
/proc/23252/task/23256/stat : 23256 (pb_cplex) R 23251 23251 22816 0 -1 4202560 27604 0 0 0 1727 6 0 0 18 0 8 0 146834324 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 136403106 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23252/tid=23257] ppid=23251 vsize=373844 CPUtime=16.36
/proc/23252/task/23257/stat : 23257 (pb_cplex) R 23251 23251 22816 0 -1 4202560 31160 0 0 0 1630 6 0 0 18 0 8 0 146834324 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 136007313 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23252/tid=23258] ppid=23251 vsize=373844 CPUtime=16.27
/proc/23252/task/23258/stat : 23258 (pb_cplex) R 23251 23251 22816 0 -1 4202560 42027 0 0 0 1619 8 0 0 18 0 8 0 146834324 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 136400447 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23252/tid=23259] ppid=23251 vsize=373844 CPUtime=19.5
/proc/23252/task/23259/stat : 23259 (pb_cplex) R 23251 23251 22816 0 -1 4202560 32423 0 0 0 1942 8 0 0 25 0 8 0 146834324 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 144797937 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23252/tid=23260] ppid=23251 vsize=373844 CPUtime=20.96
/proc/23252/task/23260/stat : 23260 (pb_cplex) R 23251 23251 22816 0 -1 4202560 41177 0 0 0 2087 9 0 0 17 0 8 0 146834324 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 136007319 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23252/tid=23261] ppid=23251 vsize=373844 CPUtime=23.19
/proc/23252/task/23261/stat : 23261 (pb_cplex) R 23251 23251 22816 0 -1 4202560 40547 0 0 0 2310 9 0 0 17 0 8 0 146834324 382816256 56235 4089446400 134512640 146736320 4287828144 18446744073709551615 139044514 0 2 4096 16387 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 170.18
Current children cumulated vsize (KiB) 382544

[startup+162.352 s]
/proc/loadavg: 14.27 10.03 5.26 5/199 23265
/proc/meminfo: memFree=30583176/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=397852 CPUtime=284.41
/proc/23252/stat : 23252 (pb_cplex) S 23251 23251 22816 0 -1 4202496 551704 0 0 0 28310 131 0 0 22 0 8 0 146830906 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/23252/statm: 99463 65552 1100 2985 0 96005 0
[pid=23252/tid=23255] ppid=23251 vsize=397852 CPUtime=30.46
/proc/23252/task/23255/stat : 23255 (pb_cplex) S 23251 23251 22816 0 -1 4202560 44069 0 0 0 3035 11 0 0 18 0 8 0 146834324 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 1 0 0 0
[pid=23252/tid=23256] ppid=23251 vsize=397852 CPUtime=36.11
/proc/23252/task/23256/stat : 23256 (pb_cplex) R 23251 23251 22816 0 -1 4202560 63492 0 0 0 3594 17 0 0 25 0 8 0 146834324 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 136007065 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23252/tid=23257] ppid=23251 vsize=397852 CPUtime=29.77
/proc/23252/task/23257/stat : 23257 (pb_cplex) S 23251 23251 22816 0 -1 4202560 50810 0 0 0 2967 10 0 0 20 0 8 0 146834324 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 5 0 0 0
[pid=23252/tid=23258] ppid=23251 vsize=397852 CPUtime=30.23
/proc/23252/task/23258/stat : 23258 (pb_cplex) S 23251 23251 22816 0 -1 4202560 66819 0 0 0 3008 15 0 0 22 0 8 0 146834324 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 2 0 0 0
[pid=23252/tid=23259] ppid=23251 vsize=397852 CPUtime=30.34
/proc/23252/task/23259/stat : 23259 (pb_cplex) S 23251 23251 22816 0 -1 4202560 48666 0 0 0 3021 13 0 0 20 0 8 0 146834324 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 2 0 0 0
[pid=23252/tid=23260] ppid=23251 vsize=397852 CPUtime=35.62
/proc/23252/task/23260/stat : 23260 (pb_cplex) S 23251 23251 22816 0 -1 4202560 60206 0 0 0 3547 15 0 0 19 0 8 0 146834324 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 2 0 0 0
[pid=23252/tid=23261] ppid=23251 vsize=397852 CPUtime=38.25
/proc/23252/task/23261/stat : 23261 (pb_cplex) R 23251 23251 22816 0 -1 4202560 72013 0 0 0 3807 18 0 0 24 0 8 0 146834324 407400448 65552 4089446400 134512640 146736320 4287828144 18446744073709551615 136007677 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 284.41
Current children cumulated vsize (KiB) 406552

[startup+222.307 s]
/proc/loadavg: 9.78 9.48 5.37 10/192 23266
/proc/meminfo: memFree=30690640/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0

################
# More data... #
################

/proc/23252/statm: 83682 57907 1082 2985 0 80224 0
[pid=23252/tid=23332] ppid=23251 vsize=334728 CPUtime=31.8
/proc/23252/task/23332/stat : 23332 (pb_cplex) R 23251 23251 22816 0 -1 4202560 5114 0 0 0 3178 2 0 0 25 0 8 0 146859856 342761472 57907 4089446400 134512640 146736320 4287828144 18446744073709551615 139044499 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23252/tid=23333] ppid=23251 vsize=334728 CPUtime=33
/proc/23252/task/23333/stat : 23333 (pb_cplex) R 23251 23251 22816 0 -1 4202560 7412 0 0 0 3296 4 0 0 25 0 8 0 146859856 342761472 57907 4089446400 134512640 146736320 4287828144 18446744073709551615 136008065 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23252/tid=23334] ppid=23251 vsize=334728 CPUtime=27.24
/proc/23252/task/23334/stat : 23334 (pb_cplex) R 23251 23251 22816 0 -1 4202560 8017 0 0 0 2718 6 0 0 25 0 8 0 146859856 342761472 57907 4089446400 134512640 146736320 4287828144 18446744073709551615 136008245 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23252/tid=23335] ppid=23251 vsize=334728 CPUtime=44.81
/proc/23252/task/23335/stat : 23335 (pb_cplex) R 23251 23251 22816 0 -1 4202560 11332 0 0 0 4478 3 0 0 25 0 8 0 146859856 342761472 57907 4089446400 134512640 146736320 4287828144 18446744073709551615 136400560 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=23252/tid=23336] ppid=23251 vsize=334728 CPUtime=41.48
/proc/23252/task/23336/stat : 23336 (pb_cplex) R 23251 23251 22816 0 -1 4202560 14226 0 0 0 4141 7 0 0 25 0 8 0 146859856 342761472 57907 4089446400 134512640 146736320 4287828144 18446744073709551615 136008226 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23252/tid=23337] ppid=23251 vsize=334728 CPUtime=35.1
/proc/23252/task/23337/stat : 23337 (pb_cplex) R 23251 23251 22816 0 -1 4202560 14088 0 0 0 3504 6 0 0 25 0 8 0 146859857 342761472 57907 4089446400 134512640 146736320 4287828144 18446744073709551615 136008042 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23252/tid=23338] ppid=23251 vsize=334728 CPUtime=34.12
/proc/23252/task/23338/stat : 23338 (pb_cplex) R 23251 23251 22816 0 -1 4202560 5384 0 0 0 3409 3 0 0 25 0 8 0 146859857 342761472 57907 4089446400 134512640 146736320 4287828144 18446744073709551615 136019067 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 671.75
Current children cumulated vsize (KiB) 343428

[startup+522.301 s]
/proc/loadavg: 15.57 12.23 7.51 17/199 23358
/proc/meminfo: memFree=30619652/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=333776 CPUtime=757.09
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 665895 0 0 0 75514 195 0 0 25 0 8 0 146830906 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 136008441 0 0 4096 16387 18446744073709551615 0 0 17 1 0 0 0
/proc/23252/statm: 83444 57896 1080 2985 0 79986 0
[pid=23252/tid=23332] ppid=23251 vsize=333776 CPUtime=42.62
/proc/23252/task/23332/stat : 23332 (pb_cplex) R 23251 23251 22816 0 -1 4202560 5114 0 0 0 4259 3 0 0 25 0 8 0 146859856 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 136017763 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23252/tid=23333] ppid=23251 vsize=333776 CPUtime=43.73
/proc/23252/task/23333/stat : 23333 (pb_cplex) R 23251 23251 22816 0 -1 4202560 15502 0 0 0 4366 7 0 0 25 0 8 0 146859856 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 136388871 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=23252/tid=23334] ppid=23251 vsize=333776 CPUtime=37.89
/proc/23252/task/23334/stat : 23334 (pb_cplex) R 23251 23251 22816 0 -1 4202560 16235 0 0 0 3781 8 0 0 25 0 8 0 146859856 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 136403117 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23252/tid=23335] ppid=23251 vsize=333776 CPUtime=55.64
/proc/23252/task/23335/stat : 23335 (pb_cplex) R 23251 23251 22816 0 -1 4202560 16222 0 0 0 5559 5 0 0 25 0 8 0 146859856 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 136008689 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23252/tid=23336] ppid=23251 vsize=333776 CPUtime=51.35
/proc/23252/task/23336/stat : 23336 (pb_cplex) R 23251 23251 22816 0 -1 4202560 23838 0 0 0 5124 11 0 0 25 0 8 0 146859856 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 136008042 0 2 4096 16387 0 0 0 -1 3 0 0 0
[pid=23252/tid=23337] ppid=23251 vsize=333776 CPUtime=45.72
/proc/23252/task/23337/stat : 23337 (pb_cplex) R 23251 23251 22816 0 -1 4202560 14088 0 0 0 4565 7 0 0 25 0 8 0 146859857 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 136008339 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23252/tid=23338] ppid=23251 vsize=333776 CPUtime=44.9
/proc/23252/task/23338/stat : 23338 (pb_cplex) R 23251 23251 22816 0 -1 4202560 5385 0 0 0 4487 3 0 0 25 0 8 0 146859857 341786624 57896 4089446400 134512640 146736320 4287828144 18446744073709551615 138374191 0 2 4096 16387 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 757.09
Current children cumulated vsize (KiB) 342476

[startup+582.301 s]
/proc/loadavg: 12.62 11.99 7.72 10/199 23363
/proc/meminfo: memFree=30609104/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=342208 CPUtime=964.12
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 758088 0 0 0 96191 221 0 0 25 0 8 0 146830906 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 135468521 0 0 4096 16387 18446744073709551615 0 0 17 2 0 0 0
/proc/23252/statm: 85552 60409 1085 2985 0 82094 0
[pid=23252/tid=23332] ppid=23251 vsize=342208 CPUtime=68.06
/proc/23252/task/23332/stat : 23332 (pb_cplex) R 23251 23251 22816 0 -1 4202560 5174 0 0 0 6802 4 0 0 25 0 8 0 146859856 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 136388084 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23252/tid=23333] ppid=23251 vsize=342208 CPUtime=71.4
/proc/23252/task/23333/stat : 23333 (pb_cplex) R 23251 23251 22816 0 -1 4202560 41422 0 0 0 7129 11 0 0 25 0 8 0 146859856 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 135999385 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23252/tid=23334] ppid=23251 vsize=342208 CPUtime=61.51
/proc/23252/task/23334/stat : 23334 (pb_cplex) R 23251 23251 22816 0 -1 4202560 26965 0 0 0 6140 11 0 0 25 0 8 0 146859856 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 136008234 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23252/tid=23335] ppid=23251 vsize=342208 CPUtime=82.34
/proc/23252/task/23335/stat : 23335 (pb_cplex) R 23251 23251 22816 0 -1 4202560 16519 0 0 0 8226 8 0 0 25 0 8 0 146859856 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 136008157 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23252/tid=23336] ppid=23251 vsize=342208 CPUtime=78.87
/proc/23252/task/23336/stat : 23336 (pb_cplex) R 23251 23251 22816 0 -1 4202560 36831 0 0 0 7871 16 0 0 25 0 8 0 146859856 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 136008050 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23252/tid=23337] ppid=23251 vsize=342208 CPUtime=71.05
/proc/23252/task/23337/stat : 23337 (pb_cplex) R 23251 23251 22816 0 -1 4202560 28050 0 0 0 7094 11 0 0 25 0 8 0 146859857 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 136008042 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23252/tid=23338] ppid=23251 vsize=342208 CPUtime=71.39
/proc/23252/task/23338/stat : 23338 (pb_cplex) R 23251 23251 22816 0 -1 4202560 19776 0 0 0 7133 6 0 0 25 0 8 0 146859857 350420992 60409 4089446400 134512640 146736320 4287828144 18446744073709551615 136008050 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 964.12
Current children cumulated vsize (KiB) 350908

[startup+642.301 s]
/proc/loadavg: 9.53 11.25 7.73 10/192 23392
/proc/meminfo: memFree=30652292/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=342380 CPUtime=1309.63
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 885209 0 0 0 130708 255 0 0 25 0 8 0 146830906 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 136008047 0 0 4096 16387 18446744073709551615 0 0 17 6 0 0 0
/proc/23252/statm: 85595 60760 1082 2985 0 82137 0
[pid=23252/tid=23332] ppid=23251 vsize=342380 CPUtime=114.01
/proc/23252/task/23332/stat : 23332 (pb_cplex) R 23251 23251 22816 0 -1 4202560 5176 0 0 0 11395 6 0 0 25 0 8 0 146859856 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 136385265 0 2 4096 16387 0 0 0 -1 7 0 0 0
[pid=23252/tid=23333] ppid=23251 vsize=342380 CPUtime=108.21
/proc/23252/task/23333/stat : 23333 (pb_cplex) S 23251 23251 22816 0 -1 4202560 50853 0 0 0 10805 16 0 0 25 0 8 0 146859856 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 4 0 0 0
[pid=23252/tid=23334] ppid=23251 vsize=342380 CPUtime=108.08
/proc/23252/task/23334/stat : 23334 (pb_cplex) R 23251 23251 22816 0 -1 4202560 38092 0 0 0 10794 14 0 0 25 0 8 0 146859856 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 136008441 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23252/tid=23335] ppid=23251 vsize=342380 CPUtime=126.19
/proc/23252/task/23335/stat : 23335 (pb_cplex) R 23251 23251 22816 0 -1 4202560 46784 0 0 0 12604 15 0 0 25 0 8 0 146859856 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 136008065 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23252/tid=23336] ppid=23251 vsize=342380 CPUtime=106.83
/proc/23252/task/23336/stat : 23336 (pb_cplex) R 23251 23251 22816 0 -1 4202560 49991 0 0 0 10663 20 0 0 22 0 8 0 146859856 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 136008450 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23252/tid=23337] ppid=23251 vsize=342380 CPUtime=119.1
/proc/23252/task/23337/stat : 23337 (pb_cplex) R 23251 23251 22816 0 -1 4202560 61684 0 0 0 11892 18 0 0 22 0 8 0 146859857 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 135454079 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23252/tid=23338] ppid=23251 vsize=342380 CPUtime=122.35
/proc/23252/task/23338/stat : 23338 (pb_cplex) R 23251 23251 22816 0 -1 4202560 28617 0 0 0 12227 8 0 0 25 0 8 0 146859857 350597120 60760 4089446400 134512640 146736320 4287828144 18446744073709551615 136471573 0 2 4096 16387 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 1309.63
Current children cumulated vsize (KiB) 351080

[startup+702.301 s]
/proc/loadavg: 8.77 10.60 7.72 8/199 23401
/proc/meminfo: memFree=30361844/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=346964 CPUtime=1567.79
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 988815 0 0 0 156500 279 0 0 25 0 8 0 146830906 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 136400472 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/23252/statm: 86741 62000 1080 2985 0 83283 0
[pid=23252/tid=23332] ppid=23251 vsize=346964 CPUtime=136.79
/proc/23252/task/23332/stat : 23332 (pb_cplex) S 23251 23251 22816 0 -1 4202560 5176 0 0 0 13673 6 0 0 25 0 8 0 146859856 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 7 0 0 0
[pid=23252/tid=23333] ppid=23251 vsize=346964 CPUtime=130.82
/proc/23252/task/23333/stat : 23333 (pb_cplex) R 23251 23251 22816 0 -1 4202560 73658 0 0 0 13063 19 0 0 25 0 8 0 146859856 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 136008167 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23252/tid=23334] ppid=23251 vsize=346964 CPUtime=143.35
/proc/23252/task/23334/stat : 23334 (pb_cplex) R 23251 23251 22816 0 -1 4202560 58973 0 0 0 14317 18 0 0 25 0 8 0 146859856 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 136008256 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23252/tid=23335] ppid=23251 vsize=346964 CPUtime=153.44
/proc/23252/task/23335/stat : 23335 (pb_cplex) R 23251 23251 22816 0 -1 4202560 51987 0 0 0 15326 18 0 0 25 0 8 0 146859856 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 136008230 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23252/tid=23336] ppid=23251 vsize=346964 CPUtime=154.99
/proc/23252/task/23336/stat : 23336 (pb_cplex) S 23251 23251 22816 0 -1 4202560 52196 0 0 0 15478 21 0 0 25 0 8 0 146859856 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 1 0 0 0
[pid=23252/tid=23337] ppid=23251 vsize=346964 CPUtime=155.62
/proc/23252/task/23337/stat : 23337 (pb_cplex) S 23251 23251 22816 0 -1 4202560 89171 0 0 0 15539 23 0 0 25 0 8 0 146859857 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 6 0 0 0
[pid=23252/tid=23338] ppid=23251 vsize=346964 CPUtime=156.5
/proc/23252/task/23338/stat : 23338 (pb_cplex) S 23251 23251 22816 0 -1 4202560 28617 0 0 0 15642 8 0 0 25 0 8 0 146859857 355291136 62000 4089446400 134512640 146736320 4287828144 18446744073709551615 144580073 0 2 4096 16387 18446604471131474752 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1567.79
Current children cumulated vsize (KiB) 355664



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+755.701 s]
/proc/loadavg: 7.62 10.06 7.71 12/192 23403
/proc/meminfo: memFree=30483004/32951124 swapFree=67111528/67111528
[pid=23251] ppid=23249 vsize=8700 CPUtime=0
/proc/23251/stat : 23251 (run) S 23249 23251 22816 0 -1 4202496 363 0 0 0 0 0 0 0 25 0 1 0 146830906 8908800 248 4089446400 4194304 4922060 140734236732448 18446744073709551615 246516653189 0 65536 4100 65538 18446744071562232103 0 0 17 5 0 0 0
/proc/23251/statm: 2175 248 208 178 0 68 0
[pid=23252] ppid=23251 vsize=345152 CPUtime=1800.25
/proc/23252/stat : 23252 (pb_cplex) R 23251 23251 22816 0 -1 4202496 1080504 0 0 0 179724 301 0 0 18 0 8 0 146830906 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 135854535 0 0 4096 16387 18446744073709551615 0 0 17 7 0 0 0
/proc/23252/statm: 86288 62040 1083 2985 0 82830 0
[pid=23252/tid=23332] ppid=23251 vsize=345152 CPUtime=166.82
/proc/23252/task/23332/stat : 23332 (pb_cplex) R 23251 23251 22816 0 -1 4202560 5266 0 0 0 16675 7 0 0 18 0 8 0 146859856 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 136004958 0 2 4096 16387 0 0 0 -1 6 0 0 0
[pid=23252/tid=23333] ppid=23251 vsize=345152 CPUtime=168.9
/proc/23252/task/23333/stat : 23333 (pb_cplex) R 23251 23251 22816 0 -1 4202560 99286 0 0 0 16865 25 0 0 18 0 8 0 146859856 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 136008328 0 2 4096 16387 0 0 0 -1 5 0 0 0
[pid=23252/tid=23334] ppid=23251 vsize=345152 CPUtime=172
/proc/23252/task/23334/stat : 23334 (pb_cplex) R 23251 23251 22816 0 -1 4202560 63187 0 0 0 17180 20 0 0 18 0 8 0 146859856 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 136006068 0 2 4096 16387 0 0 0 -1 2 0 0 0
[pid=23252/tid=23335] ppid=23251 vsize=345152 CPUtime=185.08
/proc/23252/task/23335/stat : 23335 (pb_cplex) R 23251 23251 22816 0 -1 4202560 79463 0 0 0 18484 24 0 0 18 0 8 0 146859856 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 136403112 0 2 4096 16387 0 0 0 -1 0 0 0 0
[pid=23252/tid=23336] ppid=23251 vsize=345152 CPUtime=180.49
/proc/23252/task/23336/stat : 23336 (pb_cplex) R 23251 23251 22816 0 -1 4202560 60457 0 0 0 18025 24 0 0 18 0 8 0 146859856 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 135525914 0 2 4096 16387 0 0 0 -1 1 0 0 0
[pid=23252/tid=23337] ppid=23251 vsize=345152 CPUtime=179.23
/proc/23252/task/23337/stat : 23337 (pb_cplex) R 23251 23251 22816 0 -1 4202560 89964 0 0 0 17900 23 0 0 18 0 8 0 146859857 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 136388087 0 2 4096 16387 0 0 0 -1 4 0 0 0
[pid=23252/tid=23338] ppid=23251 vsize=345152 CPUtime=183.59
/proc/23252/task/23338/stat : 23338 (pb_cplex) R 23251 23251 22816 0 -1 4202560 29576 0 0 0 18350 9 0 0 18 0 8 0 146859857 353435648 62040 4089446400 134512640 146736320 4287828144 18446744073709551615 136008065 0 2 4096 16387 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 1800.25
Current children cumulated vsize (KiB) 353852

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 23251 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=999
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=999
# CPU time returned by wait4() is 0.001998
# while last known CPU time is 1800.25
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 755.718
CPU time (s): 1800.25
CPU user time (s): 1797.24
CPU system time (s): 3.01
CPU usage (%): 238.217
Max. virtual memory (cumulated for all children) (KiB): 444556

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.000999
system time used= 0.000999
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 363
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2
involuntary context switches= 3

runsolver used 1.3168 second user time and 3.00054 second system time

The end

Launcher Data 

Begin job on node134 at 2010-07-02 09:22:27
IDJOB=2697434
IDBENCH=79028
IDSOLVER=1210
FILE ID=node134/2697434-1278055347
PBS_JOBID= 11198914
Free space on /tmp= 74160 MiB

SOLVER NAME= pb_cplex 2010-07-01
BENCH NAME= PB10/normalized-PB10/PARTIAL-SMALLINT-LIN/PB06/web/uclid_pb_benchmarks/normalized-ooo.rf8.ucl--soft-66-100-0.wbo
COMMAND LINE= run BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-2697434-1278055347/watcher-2697434-1278055347 -o /tmp/evaluation-result-2697434-1278055347/solver-2697434-1278055347 -C 1800 -W 2000 -M 3800  run HOME/instance-2697434-1278055347.wbo

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 3800 MiB
MAX NB THREAD= 0

MD5SUM BENCH= 321b555888f42ada31256d1d017f1ca8
RANDOM SEED=1729222425

node134.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 7397.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.818
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30848820 kB
Buffers:        227704 kB
Cached:        1607312 kB
SwapCached:          0 kB
Active:         809780 kB
Inactive:      1141816 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30848820 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           16724 kB
Writeback:           0 kB
AnonPages:      116436 kB
Mapped:          19124 kB
Slab:            84900 kB
PageTables:       5464 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   342660 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 74144 MiB
End job on node134 at 2010-07-02 09:35:05