Trace number 3741163

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 422.44 557.669

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-70.opb
MD5SUM4f155d4f8b05885565510d4fbd89c993
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables12552
Total number of constraints283802
Number of constraints which are clauses254979
Number of constraints which are cardinality constraints (but not clauses)914
Number of constraints which are nor clauses,nor cardinality constraints27909
Minimum length of a constraint1
Maximum length of a constraint270
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 271
Number of bits of the biggest sum of numbers9
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.37/1.28	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.37/1.28	c INFO: solving HOME/instance-3741163-1338721435.opb
2.35/3.26	c INFO: parsing took 1.94 s
5.64/6.57	c INFO: features took 3.31 CPU seconds
5.74/6.61	c INFO: running wbo1.4b-fixed@0 for 45 with 1794 remaining (b = 0.96)
55.67/56.57	c INFO: running pbct-0.1.2-linear@0 for 45 with 1745 remaining (b = 0.96)
103.03/103.89	c INFO: running wbo1.4a@0 for 45 with 1698 remaining (b = 0.95)
152.57/153.32	c INFO: running scip-2.0.1-clp@0 for 45 with 1649 remaining (b = 0.83)
201.73/202.47	c INFO: running scip-2.0.1-spx@0 for 45 with 1601 remaining (b = 0.68)
268.40/269.04	c INFO: wrote minion input file to HOME/tmpRSxWtT.minion
268.40/269.04	c INFO: running minion-0.12@0 for 45 with 1552 remaining (b = 0.55)
277.56/413.50	c INFO: running bsolo_pb10-l2@0 for 45 with 1530 remaining (b = 0.29)
326.04/461.80	c INFO: running clasp-1.3.7@0 for 45 with 1482 remaining (b = 0.18)
373.19/508.97	c INFO: running clasp-1.3.7@47 for 45 with 1435 remaining (b = 0.02)
421.48/557.13	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1387 remaining (b = 0.05)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3741163-1338721435/watcher-3741163-1338721435 -o /tmp/evaluation-result-3741163-1338721435/solver-3741163-1338721435 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741163-1338721435.opb -seed 399512522 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.82 3.40 2.82 3/181 1501
/proc/meminfo: memFree=22409764/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/1501/stat : 1501 (python) D 1499 1501 27000 0 -1 4202496 632 0 0 0 0 0 0 0 20 0 1 0 338271108 15642624 526 33554432000 4194304 5683644 140734509508976 140734509490648 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 3819 526 311 364 0 261 0

[startup+0.0950919 s]
/proc/loadavg: 4.82 3.40 2.82 3/181 1501
/proc/meminfo: memFree=22409764/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=19120 CPUtime=0.01 cores=1,3,5,7
/proc/1501/stat : 1501 (python) R 1499 1501 27000 0 -1 4202496 1370 0 0 0 1 0 0 0 20 0 1 0 338271108 19578880 1126 33554432000 4194304 5683644 140734509508976 140734509439480 219513963904 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 4772 1126 430 364 0 696 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 19120

[startup+0.100233 s]
/proc/loadavg: 4.82 3.40 2.82 3/181 1501
/proc/meminfo: memFree=22409764/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=19088 CPUtime=0.02 cores=1,3,5,7
/proc/1501/stat : 1501 (python) R 1499 1501 27000 0 -1 4202496 1394 0 0 0 2 0 0 0 20 0 1 0 338271108 19546112 1149 33554432000 4194304 5683644 140734509508976 140734509439480 219513589652 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 4772 1149 430 364 0 696 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19088

[startup+0.300211 s]
/proc/loadavg: 4.82 3.40 2.82 3/181 1501
/proc/meminfo: memFree=22409764/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=96896 CPUtime=0.1 cores=1,3,5,7
/proc/1501/stat : 1501 (python) D 1499 1501 27000 0 -1 4202496 3946 1012 1 0 8 2 0 0 20 0 1 0 338271108 99221504 2648 33554432000 4194304 5683644 140734509508976 140734509395368 219513962165 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 24224 2648 823 364 0 1715 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 96896

[startup+0.700276 s]
/proc/loadavg: 4.82 3.40 2.82 3/181 1501
/proc/meminfo: memFree=22409764/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=125764 CPUtime=0.2 cores=1,3,5,7
/proc/1501/stat : 1501 (python) D 1499 1501 27000 0 -1 4202496 6057 1012 1 0 14 6 0 0 20 0 1 0 338271108 128782336 3918 33554432000 4194304 5683644 140734509508976 140734509413608 219504799015 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 31441 3918 1017 364 0 2863 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 125764

[startup+1.50019 s]
/proc/loadavg: 4.75 3.41 2.83 2/182 1506
/proc/meminfo: memFree=22383600/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=300156 CPUtime=0.64 cores=1,3,5,7
/proc/1501/stat : 1501 (python) R 1499 1501 27000 0 -1 4202496 12136 1012 1 0 52 12 0 0 20 0 1 0 338271108 307359744 11725 33554432000 4194304 5683644 140734509508976 140734509423048 219513634688 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 75039 11725 1811 364 0 10268 0
Current children cumulated CPU time (s) 0.64
Current children cumulated vsize (KiB) 300156

[startup+3.10029 s]
/proc/loadavg: 4.75 3.41 2.83 3/182 1506
/proc/meminfo: memFree=22330316/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=362056 CPUtime=2.24 cores=1,3,5,7
/proc/1501/stat : 1501 (python) R 1499 1501 27000 0 -1 4202496 25268 1012 1 0 209 15 0 0 20 0 1 0 338271108 370745344 26390 33554432000 4194304 5683644 140734509508976 140734509501592 140268705184453 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 90514 26391 1811 364 0 25743 0
Current children cumulated CPU time (s) 2.24
Current children cumulated vsize (KiB) 362056
heavy processes:

[startup+6.30019 s]
/proc/loadavg: 4.53 3.39 2.82 3/182 1506
/proc/meminfo: memFree=21929128/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=359784 CPUtime=5.44 cores=1,3,5,7
/proc/1501/stat : 1501 (python) R 1499 1501 27000 0 -1 4202496 31133 1012 1 0 528 16 0 0 20 0 1 0 338271108 368418816 26698 33554432000 4194304 5683644 140734509508976 140734509502152 140268705281105 0 0 16781312 33554434 0 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 89946 26698 1847 364 0 25175 0
Current children cumulated CPU time (s) 5.44
Current children cumulated vsize (KiB) 359784

[startup+12.7004 s]
/proc/loadavg: 4.33 3.37 2.82 3/185 1509
/proc/meminfo: memFree=20179052/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=440024 CPUtime=5.74 cores=1,3,5,7
/proc/1501/stat : 1501 (python) S 1499 1501 27000 0 -1 4202496 32521 1012 1 0 558 16 0 0 20 0 2 0 338271108 450584576 27860 33554432000 4194304 5683644 140734509508976 140734509503808 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 110006 27860 1894 364 0 45229 0
[pid=1507] ppid=1501 vsize=364244 CPUtime=0 cores=1,3,5,7
/proc/1507/stat : 1507 (python) S 1501 1501 27000 0 -1 4202560 883 0 0 0 0 0 0 0 20 0 1 0 338271769 372985856 26322 33554432000 4194304 5683644 140734509508976 140734509500064 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/1507/statm: 91061 26322 362 364 0 26284 0
[pid=1509] ppid=1507 vsize=37228 CPUtime=6.07 cores=1,3,5,7
/proc/1509/stat : 1509 (wbo1.4b-fixed) R 1507 1509 1509 0 -1 4202496 12290 0 0 0 586 21 0 0 20 0 1 0 338271770 38121472 8587 33554432000 134512640 135752315 4291081280 4291079356 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/1509/statm: 9307 8588 146 303 0 9002 0
[pid=1501/tid=1508] ppid=1499 vsize=440024 CPUtime=0 cores=1,3,5,7
/proc/1501/task/1508/stat : 1508 (python) S 1499 1501 27000 0 -1 4202560 27 1012 0 0 0 0 0 0 20 0 2 0 338271770 450584576 27860 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.81
Current children cumulated vsize (KiB) 841496

[startup+25.5004 s]
/proc/loadavg: 3.97 3.32 2.81 3/185 1509
/proc/meminfo: memFree=19950520/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=440024 CPUtime=5.74 cores=1,3,5,7
/proc/1501/stat : 1501 (python) S 1499 1501 27000 0 -1 4202496 32521 1012 1 0 558 16 0 0 20 0 2 0 338271108 450584576 27860 33554432000 4194304 5683644 140734509508976 140734509503808 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 110006 27860 1894 364 0 45229 0
[pid=1507] ppid=1501 vsize=364244 CPUtime=0.02 cores=1,3,5,7
/proc/1507/stat : 1507 (python) S 1501 1501 27000 0 -1 4202560 895 0 0 0 1 1 0 0 20 0 1 0 338271769 372985856 26323 33554432000 4194304 5683644 140734509508976 140734509500064 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/1507/statm: 91061 26323 363 364 0 26284 0
[pid=1509] ppid=1507 vsize=45580 CPUtime=18.87 cores=1,3,5,7
/proc/1509/stat : 1509 (wbo1.4b-fixed) R 1507 1509 1509 0 -1 4202496 14343 0 0 0 1851 36 0 0 20 0 1 0 338271770 46673920 10640 33554432000 134512640 135752315 4291081280 4291079356 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/1509/statm: 11395 10640 146 303 0 11090 0
[pid=1501/tid=1508] ppid=1499 vsize=440024 CPUtime=0 cores=1,3,5,7
/proc/1501/task/1508/stat : 1508 (python) S 1499 1501 27000 0 -1 4202560 27 1012 0 0 0 0 0 0 20 0 2 0 338271770 450584576 27860 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.63
Current children cumulated vsize (KiB) 849848

[startup+51.1005 s]
/proc/loadavg: 3.32 3.22 2.79 3/189 1513
/proc/meminfo: memFree=19879312/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=440024 CPUtime=5.74 cores=1,3,5,7
/proc/1501/stat : 1501 (python) S 1499 1501 27000 0 -1 4202496 32521 1012 1 0 558 16 0 0 20 0 2 0 338271108 450584576 27860 33554432000 4194304 5683644 140734509508976 140734509503808 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 110006 27860 1894 364 0 45229 0
[pid=1507] ppid=1501 vsize=364244 CPUtime=0.06 cores=1,3,5,7
/proc/1507/stat : 1507 (python) S 1501 1501 27000 0 -1 4202560 1005 0 0 0 4 2 0 0 20 0 1 0 338271769 372985856 26328 33554432000 4194304 5683644 140734509508976 140734509500064 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/1507/statm: 91061 26328 368 364 0 26284 0
[pid=1509] ppid=1507 vsize=63992 CPUtime=44.47 cores=1,3,5,7
/proc/1509/stat : 1509 (wbo1.4b-fixed) R 1507 1509 1509 0 -1 4202496 18898 0 0 0 4389 58 0 0 20 0 1 0 338271770 65527808 15195 33554432000 134512640 135752315 4291081280 4291079356 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/1509/statm: 15998 15195 146 303 0 15693 0
[pid=1501/tid=1508] ppid=1499 vsize=440024 CPUtime=0 cores=1,3,5,7
/proc/1501/task/1508/stat : 1508 (python) S 1499 1501 27000 0 -1 4202560 27 1012 0 0 0 0 0 0 20 0 2 0 338271770 450584576 27860 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.27
Current children cumulated vsize (KiB) 868260

[startup+102.307 s]
/proc/loadavg: 2.57 3.03 2.75 3/196 1521
/proc/meminfo: memFree=18711996/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=515816 CPUtime=5.77 cores=1,3,5,7
/proc/1501/stat : 1501 (python) S 1499 1501 27000 0 -1 4202496 33440 1012 1 0 560 17 0 0 20 0 3 0 338271108 528195584 27877 33554432000 4194304 5683644 140734509508976 140734509503808 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 128954 27877 1905 364 0 64174 0
[pid=1507] ppid=1501 vsize=440024 CPUtime=0.07 cores=1,3,5,7
/proc/1507/stat : 1507 (python) S 1501 1501 27000 0 -1 4202560 1138 0 0 0 4 3 0 0 20 0 2 0 338271769 450584576 26363 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1507/statm: 110006 26363 396 364 0 45229 0
[pid=1509] ppid=1507 vsize=66080 CPUtime=49.92 cores=1,3,5,7
/proc/1509/stat : 1509 (wbo1.4b-fixed) T 1507 1509 1509 0 -1 4202496 19400 0 0 0 4930 62 0 0 20 0 1 0 338271770 67665920 15697 33554432000 134512640 135752315 4291081280 4291079272 134517414 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1509/statm: 16520 15697 146 303 0 16215 0
[pid=1507/tid=1514] ppid=1501 vsize=440024 CPUtime=0 cores=1,3,5,7
/proc/1507/task/1514/stat : 1514 (python) S 1501 1501 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338276764 450584576 26363 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1515] ppid=1501 vsize=440036 CPUtime=0.07 cores=1,3,5,7
/proc/1515/stat : 1515 (python) S 1501 1501 27000 0 -1 4202560 924 0 0 0 4 3 0 0 20 0 1 0 338276766 450596864 26324 33554432000 4194304 5683644 140734509508976 140734509500064 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/1515/statm: 110009 26324 358 364 0 45229 0
[pid=1517] ppid=1515 vsize=1144612 CPUtime=45.71 cores=1,3,5,7
/proc/1517/stat : 1517 (pbct-0.1.2-linu) R 1515 1517 1517 0 -1 4202496 268534 0 0 0 4495 76 0 0 20 0 1 0 338276766 1172082688 273466 33554432000 134512640 136422188 4286996032 4286993408 134974186 0 0 16781312 16387 0 0 0 17 1 0 0 0 0 0
/proc/1517/statm: 286153 273466 215 467 0 285684 0
[pid=1501/tid=1508] ppid=1499 vsize=515816 CPUtime=0 cores=1,3,5,7
/proc/1501/task/1508/stat : 1508 (python) S 1499 1501 27000 0 -1 4202560 27 1012 0 0 0 0 0 0 20 0 3 0 338271770 528195584 27877 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1516] ppid=1499 vsize=515816 CPUtime=0 cores=1,3,5,7
/proc/1501/task/1516/stat : 1516 (python) S 1499 1501 27000 0 -1 4202560 21 1012 0 0 0 0 0 0 20 0 3 0 338276766 528195584 27877 33554432000 4194304 5683644 140734509508976 140268521398832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.54
Current children cumulated vsize (KiB) 2606568

[startup+162.301 s]
/proc/loadavg: 2.21 2.84 2.70 3/226 1555
/proc/meminfo: memFree=15975936/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=667400 CPUtime=5.82 cores=1,3,5,7
/proc/1501/stat : 1501 (python) S 1499 1501 27000 0 -1 4202496 35211 1012 1 0 565 17 0 0 20 0 5 0 338271108 683417600 27893 33554432000 4194304 5683644 140734509508976 140734509503808 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1501/statm: 166850 27893 1909 364 0 102064 0
[pid=1507] ppid=1501 vsize=440024 CPUtime=0.07 cores=1,3,5,7
/proc/1507/stat : 1507 (python) S 1501 1501 27000 0 -1 4202560 1138 0 0 0 4 3 0 0 20 0 2 0 338271769 450584576 26363 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1507/statm: 110006 26363 396 364 0 45229 0
[pid=1509] ppid=1507 vsize=66080 CPUtime=49.92 cores=1,3,5,7
/proc/1509/stat : 1509 (wbo1.4b-fixed) T 1507 1509 1509 0 -1 4202496 19400 0 0 0 4930 62 0 0 20 0 1 0 338271770 67665920 15697 33554432000 134512640 135752315 4291081280 4291079272 134517414 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1509/statm: 16520 15697 146 303 0 16215 0
[pid=1507/tid=1514] ppid=1501 vsize=440024 CPUtime=0 cores=1,3,5,7
/proc/1507/task/1514/stat : 1514 (python) S 1501 1501 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338276764 450584576 26363 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1515] ppid=1501 vsize=440036 CPUtime=0.07 cores=1,3,5,7
/proc/1515/stat : 1515 (python) S 1501 1501 27000 0 -1 4202560 1077 0 0 0 4 3 0 0 20 0 2 0 338276766 450596864 26350 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1515/statm: 110009 26350 383 364 0 45229 0
[pid=1517] ppid=1515 vsize=1144612 CPUtime=47.27 cores=1,3,5,7
/proc/1517/stat : 1517 (pbct-0.1.2-linu) T 1515 1517 1517 0 -1 4202496 268534 0 0 0 4650 77 0 0 20 0 1 0 338276766 1172082688 273466 33554432000 134512640 136422188 4286996032 4286993552 135006609 0 0 16781312 16387 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1517/statm: 286153 273466 215 467 0 285684 0
[pid=1515/tid=1522] ppid=1501 vsize=440036 CPUtime=0 cores=1,3,5,7
/proc/1515/task/1522/stat : 1522 (python) S 1501 1501 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338281495 450596864 26350 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1523] ppid=1501 vsize=515828 CPUtime=0.07 cores=1,3,5,7
/proc/1523/stat : 1523 (python) S 1501 1501 27000 0 -1 4202560 1098 0 0 0 4 3 0 0 20 0 2 0 338281498 528207872 26356 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1523/statm: 128957 26356 383 364 0 64174 0
[pid=1525] ppid=1523 vsize=60820 CPUtime=49.39 cores=1,3,5,7
/proc/1525/stat : 1525 (wbo1.4a) T 1523 1525 1525 0 -1 4202496 18093 0 0 0 4875 64 0 0 20 0 1 0 338281498 62279680 14389 33554432000 134512640 135751879 4294317664 4294315840 134616480 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1525/statm: 15205 14389 143 303 0 14900 0
[pid=1523/tid=1552] ppid=1501 vsize=515828 CPUtime=0 cores=1,3,5,7
/proc/1523/task/1552/stat : 1552 (python) S 1501 1501 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338286439 528207872 26356 33554432000 4194304 5683644 140734509508976 140268521398832 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0

################
# More data... #
################

/proc/1726/stat : 1726 (clasp-1.3.7-x86) R 1724 1726 1726 0 -1 4202496 15562 0 0 0 6040 4 0 0 20 0 1 0 338317289 57815040 13601 33554432000 134512640 136139019 4291376528 4291375392 134608331 0 0 16781312 16386 0 0 0 17 1 0 0 0 0 0
/proc/1726/statm: 14115 13601 183 398 0 13715 0
[pid=1724/tid=1729] ppid=1501 vsize=894788 CPUtime=0 cores=1,3,5,7
/proc/1724/task/1729/stat : 1729 (python) S 1501 1501 27000 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338322004 916262912 26392 33554432000 4194304 5683644 140734509508976 140268458451504 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1501/tid=1508] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1508/stat : 1508 (python) S 1499 1501 27000 0 -1 4202560 27 1039909 0 0 0 0 545 377 20 0 9 0 338271770 993861632 27925 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1516] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1516/stat : 1516 (python) S 1499 1501 27000 0 -1 4202560 21 1039909 0 0 0 0 545 377 20 0 9 0 338276766 993861632 27925 33554432000 4194304 5683644 140734509508976 140268521398832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1524] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1524/stat : 1524 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 9 0 338281498 993861632 27925 33554432000 4194304 5683644 140734509508976 140268510908976 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1554] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1554/stat : 1554 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 9 0 338286441 993861632 27925 33554432000 4194304 5683644 140734509508976 140268500419120 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1636] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1636/stat : 1636 (python) S 1499 1501 27000 0 -1 4202560 21 1039909 0 0 0 0 545 377 20 0 9 0 338291356 993861632 27925 33554432000 4194304 5683644 140734509508976 140268489929264 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1700] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1700/stat : 1700 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 9 0 338298013 993861632 27925 33554432000 4194304 5683644 140734509508976 140268479439408 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1717] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1717/stat : 1717 (python) S 1499 1501 27000 0 -1 4202560 21 1039909 0 0 0 0 545 377 20 0 9 0 338312459 993861632 27925 33554432000 4194304 5683644 140734509508976 140268458451504 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1501/tid=1725] ppid=1499 vsize=970568 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1725/stat : 1725 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 9 0 338317289 993861632 27925 33554432000 4194304 5683644 140734509508976 140268447949360 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
Current children cumulated CPU time (s) 386.62
Current children cumulated vsize (KiB) 8481804
heavy processes:



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+557.601 s]
/proc/loadavg: 2.26 2.91 2.80 5/267 1770
/proc/meminfo: memFree=18055724/32873648 swapFree=7120/7120
[pid=1501] ppid=1499 vsize=1046360 CPUtime=32.57 cores=1,3,5,7
/proc/1501/stat : 1501 (python) S 1499 1501 27000 0 -1 4202496 39952 1039909 1 0 2311 24 545 377 20 0 10 0 338271108 1071472640 27933 33554432000 4194304 5683644 140734509508976 140734509503808 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/1501/statm: 261590 27933 1919 364 0 196789 0
[pid=1507] ppid=1501 vsize=440024 CPUtime=0.07 cores=1,3,5,7
/proc/1507/stat : 1507 (python) S 1501 1501 27000 0 -1 4202560 1138 0 0 0 4 3 0 0 20 0 2 0 338271769 450584576 26363 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1507/statm: 110006 26363 396 364 0 45229 0
[pid=1509] ppid=1507 vsize=66080 CPUtime=49.92 cores=1,3,5,7
/proc/1509/stat : 1509 (wbo1.4b-fixed) T 1507 1509 1509 0 -1 4202496 19400 0 0 0 4930 62 0 0 20 0 1 0 338271770 67665920 15769 33554432000 134512640 135752315 4291081280 4291079272 134517414 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1509/statm: 16520 15769 146 303 0 16215 0
[pid=1507/tid=1514] ppid=1501 vsize=440024 CPUtime=0 cores=1,3,5,7
/proc/1507/task/1514/stat : 1514 (python) S 1501 1501 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338276764 450584576 26363 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1515] ppid=1501 vsize=440036 CPUtime=0.07 cores=1,3,5,7
/proc/1515/stat : 1515 (python) S 1501 1501 27000 0 -1 4202560 1077 0 0 0 4 3 0 0 20 0 2 0 338276766 450596864 26350 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1515/statm: 110009 26350 383 364 0 45229 0
[pid=1517] ppid=1515 vsize=1144612 CPUtime=47.27 cores=1,3,5,7
/proc/1517/stat : 1517 (pbct-0.1.2-linu) T 1515 1517 1517 0 -1 4202496 268534 0 0 0 4650 77 0 0 20 0 1 0 338276766 1172082688 273469 33554432000 134512640 136422188 4286996032 4286993552 135006609 0 0 16781312 16387 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1517/statm: 286153 273469 215 467 0 285684 0
[pid=1515/tid=1522] ppid=1501 vsize=440036 CPUtime=0 cores=1,3,5,7
/proc/1515/task/1522/stat : 1522 (python) S 1501 1501 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338281495 450596864 26350 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1523] ppid=1501 vsize=515828 CPUtime=0.07 cores=1,3,5,7
/proc/1523/stat : 1523 (python) S 1501 1501 27000 0 -1 4202560 1098 0 0 0 4 3 0 0 20 0 2 0 338281498 528207872 26356 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1523/statm: 128957 26356 383 364 0 64174 0
[pid=1525] ppid=1523 vsize=60820 CPUtime=49.39 cores=1,3,5,7
/proc/1525/stat : 1525 (wbo1.4a) T 1523 1525 1525 0 -1 4202496 18093 0 0 0 4875 64 0 0 20 0 1 0 338281498 62279680 14389 33554432000 134512640 135751879 4294317664 4294315840 134616480 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1525/statm: 15205 14389 143 303 0 14900 0
[pid=1523/tid=1552] ppid=1501 vsize=515828 CPUtime=0 cores=1,3,5,7
/proc/1523/task/1552/stat : 1552 (python) S 1501 1501 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338286439 528207872 26356 33554432000 4194304 5683644 140734509508976 140268521398832 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1553] ppid=1501 vsize=591620 CPUtime=0.07 cores=1,3,5,7
/proc/1553/stat : 1553 (python) S 1501 1501 27000 0 -1 4202560 1084 0 0 0 3 4 0 0 20 0 2 0 338286441 605818880 26362 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1553/statm: 147905 26362 383 364 0 83119 0
[pid=1555] ppid=1553 vsize=844160 CPUtime=49.1 cores=1,3,5,7
/proc/1555/stat : 1555 (scip-2.0.1.linu) T 1553 1555 1555 0 -1 4202496 156296 0 0 0 4842 68 0 0 20 0 1 0 338286441 864419840 189404 33554432000 4194304 15243407 140737080108016 140737080103368 6457242 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1555/statm: 211040 189404 884 2698 0 208335 0
[pid=1553/tid=1634] ppid=1501 vsize=591620 CPUtime=0 cores=1,3,5,7
/proc/1553/task/1634/stat : 1634 (python) S 1501 1501 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338291353 605818880 26362 33554432000 4194304 5683644 140734509508976 140268510908976 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1635] ppid=1501 vsize=667412 CPUtime=0.08 cores=1,3,5,7
/proc/1635/stat : 1635 (python) S 1501 1501 27000 0 -1 4202560 1126 0 0 0 4 4 0 0 20 0 2 0 338291356 683429888 26368 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1635/statm: 166853 26368 383 364 0 102064 0
[pid=1638] ppid=1635 vsize=890868 CPUtime=49.16 cores=1,3,5,7
/proc/1638/stat : 1638 (scip-2.0.1.linu) T 1635 1638 1638 0 -1 4202496 155213 0 0 0 4838 78 0 0 20 0 1 0 338291356 912248832 213349 33554432000 4194304 13684479 140734771193728 140734771185864 7155211 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1638/statm: 222717 213349 767 2317 0 220395 0
[pid=1635/tid=1693] ppid=1501 vsize=667412 CPUtime=0 cores=1,3,5,7
/proc/1635/task/1693/stat : 1693 (python) S 1501 1501 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338296274 683429888 26368 33554432000 4194304 5683644 140734509508976 140268500419120 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1716] ppid=1501 vsize=818996 CPUtime=0.08 cores=1,3,5,7
/proc/1716/stat : 1716 (python) S 1501 1501 27000 0 -1 4202560 1143 0 0 0 4 4 0 0 20 0 2 0 338312459 838651904 26380 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/1716/statm: 204749 26380 383 364 0 139954 0
[pid=1718] ppid=1716 vsize=79532 CPUtime=48.25 cores=1,3,5,7
/proc/1718/stat : 1718 (bsolo_pb10) T 1716 1718 1718 0 -1 4202496 21018 0 0 0 4734 91 0 0 20 0 1 0 338312459 81440768 17432 33554432000 134512640 137138936 4289542944 4289542060 136570369 0 0 16781312 16384 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/1718/statm: 19883 17432 192 642 0 19239 0
[pid=1716/tid=1723] ppid=1501 vsize=818996 CPUtime=0 cores=1,3,5,7
/proc/1716/task/1723/stat : 1723 (python) S 1501 1501 27000 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338317286 838651904 26380 33554432000 4194304 5683644 140734509508976 140268479439408 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1724] ppid=1501 vsize=894788 CPUtime=0.16 cores=1,3,5,7
/proc/1724/stat : 1724 (python) S 1501 1501 27000 0 -1 4202560 1236 0 0 0 9 7 0 0 20 0 2 0 338317289 916262912 26392 33554432000 4194304 5683644 140734509508976 140734509500624 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/1724/statm: 223697 26392 389 364 0 158899 0
[pid=1726] ppid=1724 vsize=65608 CPUtime=95.25 cores=1,3,5,7
/proc/1726/stat : 1726 (clasp-1.3.7-x86) T 1724 1726 1726 0 -1 4202496 17768 0 0 0 9520 5 0 0 20 0 1 0 338317289 67182592 15807 33554432000 134512640 136139019 4291376528 4291375392 134608331 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/1726/statm: 16402 15807 183 398 0 16002 0
[pid=1724/tid=1729] ppid=1501 vsize=894788 CPUtime=0 cores=1,3,5,7
/proc/1724/task/1729/stat : 1729 (python) S 1501 1501 27000 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338322004 916262912 26392 33554432000 4194304 5683644 140734509508976 140268458451504 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1751] ppid=1501 vsize=970580 CPUtime=0 cores=1,3,5,7
/proc/1751/stat : 1751 (python) S 1501 1501 27000 0 -1 4202560 891 0 0 0 0 0 0 0 20 0 1 0 338326822 993873920 26350 33554432000 4194304 5683644 140734509508976 140734509500064 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/1751/statm: 242645 26350 342 364 0 177844 0
[pid=1753] ppid=1751 vsize=9558408 CPUtime=0.93 cores=1,3,5,7
/proc/1753/stat : 1753 (java) S 1751 1753 1753 0 -1 4202496 8840 0 1 0 91 2 0 0 20 0 18 0 338326822 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140736029948784 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 5 0 0 0 0 0
/proc/1753/statm: 2389602 22623 2230 9 0 2380296 0
[pid=1753/tid=1754] ppid=1751 vsize=9558408 CPUtime=0.44 cores=1,3,5,7
/proc/1753/task/1754/stat : 1754 (java) R 1751 1753 1753 0 -1 4202560 2453 0 1 0 44 0 0 0 20 0 18 0 338326823 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138980073408 219517347085 0 4 0 16800975 0 0 0 -1 5 0 0 0 0 0
[pid=1753/tid=1755] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1755/stat : 1755 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138952534952 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1756] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1756/stat : 1756 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138951482408 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1753/tid=1757] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1757/stat : 1757 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138950429352 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=1753/tid=1758] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1758/stat : 1758 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138949376808 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=1753/tid=1759] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1759/stat : 1759 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138948323752 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1760] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1760/stat : 1760 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138947271208 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=1753/tid=1761] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1761/stat : 1761 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138946218152 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1762] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1762/stat : 1762 (java) S 1751 1753 1753 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338326824 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138815990056 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1763] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1763/stat : 1763 (java) S 1751 1753 1753 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338326825 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138333043560 219517335465 0 0 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1764] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1764/stat : 1764 (java) S 1751 1753 1753 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338326825 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138331989880 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1765] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1765/stat : 1765 (java) S 1751 1753 1753 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338326825 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138330936712 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1766] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1766/stat : 1766 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326827 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138329885456 219517343520 0 4 0 16800975 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1767] ppid=1751 vsize=9558408 CPUtime=0.25 cores=1,3,5,7
/proc/1753/task/1767/stat : 1767 (java) R 1751 1753 1753 0 -1 4202560 2288 0 0 0 25 0 0 0 20 0 18 0 338326827 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138328831808 140138986042509 0 4 0 16800975 0 0 0 -1 1 0 0 0 0 0
[pid=1753/tid=1768] ppid=1751 vsize=9558408 CPUtime=0.21 cores=1,3,5,7
/proc/1753/task/1768/stat : 1768 (java) R 1751 1753 1753 0 -1 4202560 2677 0 0 0 21 0 0 0 20 0 18 0 338326827 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138327779336 140138986876779 0 4 0 16800975 0 0 0 -1 3 0 0 0 0 0
[pid=1753/tid=1769] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1769/stat : 1769 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326827 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138326726904 219517334588 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1753/tid=1770] ppid=1751 vsize=9558408 CPUtime=0 cores=1,3,5,7
/proc/1753/task/1770/stat : 1770 (java) S 1751 1753 1753 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338326827 9787809792 22623 33554432000 1073741824 1073778376 140736029957648 140138325674568 219517335465 0 4 0 16800975 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1508] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1508/stat : 1508 (python) S 1499 1501 27000 0 -1 4202560 27 1039909 0 0 0 0 545 377 20 0 10 0 338271770 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268531888688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1516] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1516/stat : 1516 (python) S 1499 1501 27000 0 -1 4202560 21 1039909 0 0 0 0 545 377 20 0 10 0 338276766 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268521398832 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1524] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1524/stat : 1524 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 10 0 338281498 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268510908976 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1554] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1554/stat : 1554 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 10 0 338286441 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268500419120 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1636] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1636/stat : 1636 (python) S 1499 1501 27000 0 -1 4202560 21 1039909 0 0 0 0 545 377 20 0 10 0 338291356 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268489929264 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1700] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1700/stat : 1700 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 10 0 338298013 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268479439408 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=1501/tid=1717] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1717/stat : 1717 (python) S 1499 1501 27000 0 -1 4202560 21 1039909 0 0 0 0 545 377 20 0 10 0 338312459 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268458451504 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=1501/tid=1725] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1725/stat : 1725 (python) S 1499 1501 27000 0 -1 4202560 22 1039909 0 0 0 0 545 377 20 0 10 0 338317289 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268447949360 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=1501/tid=1752] ppid=1499 vsize=1046360 CPUtime=9.22 cores=1,3,5,7
/proc/1501/task/1752/stat : 1752 (python) S 1499 1501 27000 0 -1 4202560 21 1039909 0 0 0 0 545 377 20 0 10 0 338326822 1071472640 27933 33554432000 4194304 5683644 140734509508976 140268437459504 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 422.44
Current children cumulated vsize (KiB) 19095732
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1501 and gives
#  childrusage.ru_utime.tv_sec=28
#  childrusage.ru_utime.tv_usec=571656
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=15389
# CPU time returned by wait4() is 32.587
# while last known CPU time is 422.44
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 557.669
CPU time (s): 422.44
CPU user time (s): 413.68
CPU system time (s): 8.76
CPU usage (%): 75.7511
Max. virtual memory (cumulated for all children) (KiB): 19095732

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 28.5717
system time used= 4.01539
maximum resident set size= 4158004
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1079861
page faults= 1
swaps= 0
block input operations= 0
block output operations= 36208
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1004960
involuntary context switches= 63

runsolver used 2.13368 second user time and 5.30219 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 13:03:55
IDJOB=3741163
IDBENCH=89796
IDSOLVER=2327
FILE ID=node114/3741163-1338721435
RUNJOBID= node114-1338713908-27017
PBS_JOBID= 14636748
Free space on /tmp= 65944 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-70.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3741163-1338721435/watcher-3741163-1338721435 -o /tmp/evaluation-result-3741163-1338721435/solver-3741163-1338721435 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741163-1338721435.opb -seed 399512522 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 4f155d4f8b05885565510d4fbd89c993
RANDOM SEED=399512522

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        22410540 kB
Buffers:         1020356 kB
Cached:          8007856 kB
SwapCached:         2180 kB
Active:          3370980 kB
Inactive:        5893668 kB
Active(anon):     220612 kB
Inactive(anon):    10668 kB
Active(file):    3150368 kB
Inactive(file):  5883000 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:           2271084 kB
Writeback:             0 kB
AnonPages:        234464 kB
Mapped:            17708 kB
Shmem:               136 kB
Slab:            1056320 kB
SReclaimable:     495228 kB
SUnreclaim:       561092 kB
KernelStack:        1472 kB
PageTables:         5924 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:     577232 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:     92160 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 65876 MiB
End job on node114 at 2012-06-03 13:13:17