Trace number 3742200

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 617.43 615.164

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/
normalized-TateBritain_K101.opb
MD5SUMa3de6c6ee42f8b511357d3857e721bb4
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark198.533
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables6868
Total number of constraints7072
Number of constraints which are clauses6971
Number of constraints which are cardinality constraints (but not clauses)101
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint101
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 102
Number of bits of the biggest sum of numbers7
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.41/1.30	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.41/1.30	c INFO: solving HOME/instance-3742200-1338723856.opb
0.41/1.38	c INFO: parsing took 0.07 s
0.62/1.52	c INFO: features took 0.15 CPU seconds
0.62/1.55	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.67)
51.65/52.51	c INFO: running clasp-1.3.7@0 for 45 with 1749 remaining (b = 0.36)
98.79/99.66	c INFO: running scip-2.0.1-spx@0 for 45 with 1702 remaining (b = 0.06)
145.24/146.07	c INFO: running clasp-1.3.7@47 for 45 with 1656 remaining (b = 0.01)
191.12/191.85	c INFO: wrote minion input file to HOME/tmprksm7a.minion
191.12/191.85	c INFO: running minion-0.12@0 for 45 with 1611 remaining (b = 0.09)
238.56/239.25	c INFO: running wbo1.4a@0 for 45 with 1563 remaining (b = 0.02)
289.02/289.61	c INFO: running scip-2.0.1-clp@50 for 45 with 1513 remaining (b = 0.01)
332.77/333.37	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1469 remaining (b = 0.03)
382.88/380.92	c INFO: running clasp-1.3.7@92 for 136 with 1420 remaining (b = 0.02)
520.50/518.33	c INFO: running scip-2.0.1-spx@45 for 45 with 1282 remaining (b = 0.01)
569.68/567.50	c INFO: running bsolo_pb10-l3@0 for 45 with 1233 remaining (b = 0.02)
617.32/615.01	c INFO: running bsolo_pb10-l1@0 for 91 with 1186 remaining (b = 0.02)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742200-1338723856/watcher-3742200-1338723856 -o /tmp/evaluation-result-3742200-1338723856/solver-3742200-1338723856 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742200-1338723856.opb -seed 830543092 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.00 2.03 2.12 3/217 2397
/proc/meminfo: memFree=25157820/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/2397/stat : 2397 (python) D 2395 2397 27000 0 -1 4202496 633 0 0 0 0 0 0 0 20 0 1 0 338513078 15642624 526 33554432000 4194304 5683644 140735723290384 140735723276296 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2397/statm: 3819 526 312 364 0 261 0

[startup+0.0947721 s]
/proc/loadavg: 2.00 2.03 2.12 3/217 2397
/proc/meminfo: memFree=25157820/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=19080 CPUtime=0.01 cores=1,3,5,7
/proc/2397/stat : 2397 (python) D 2395 2397 27000 0 -1 4202496 1369 0 0 0 1 0 0 0 20 0 1 0 338513078 19537920 1124 33554432000 4194304 5683644 140735723290384 140735723216648 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2397/statm: 4770 1124 430 364 0 694 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 19080

[startup+0.100282 s]
/proc/loadavg: 2.00 2.03 2.12 3/217 2397
/proc/meminfo: memFree=25157820/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=19080 CPUtime=0.01 cores=1,3,5,7
/proc/2397/stat : 2397 (python) R 2395 2397 27000 0 -1 4202496 1393 0 0 0 1 0 0 0 20 0 1 0 338513078 19537920 1147 33554432000 4194304 5683644 140735723290384 140735723220888 4864850 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2397/statm: 4770 1147 430 364 0 694 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 19080

[startup+0.30024 s]
/proc/loadavg: 2.00 2.03 2.12 3/217 2397
/proc/meminfo: memFree=25157820/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=96892 CPUtime=0.1 cores=1,3,5,7
/proc/2397/stat : 2397 (python) R 2395 2397 27000 0 -1 4202496 3913 1009 1 0 9 1 0 0 20 0 1 0 338513078 99217408 2647 33554432000 4194304 5683644 140735723290384 140735723194616 219513963408 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2397/statm: 24231 2652 822 364 0 1722 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 96892

[startup+0.700248 s]
/proc/loadavg: 2.00 2.03 2.12 3/217 2397
/proc/meminfo: memFree=25157820/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=125760 CPUtime=0.21 cores=1,3,5,7
/proc/2397/stat : 2397 (python) D 2395 2397 27000 0 -1 4202496 6058 1009 1 0 16 5 0 0 20 0 1 0 338513078 128778240 3918 33554432000 4194304 5683644 140735723290384 140735723195016 219504799015 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/2397/statm: 31440 3918 1017 364 0 2862 0
Current children cumulated CPU time (s) 0.21
Current children cumulated vsize (KiB) 125760

[startup+1.50026 s]
/proc/loadavg: 2.00 2.03 2.12 3/218 2402
/proc/meminfo: memFree=25140072/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=284104 CPUtime=0.62 cores=1,3,5,7
/proc/2397/stat : 2397 (python) R 2395 2397 27000 0 -1 4202496 11114 1009 1 0 51 11 0 0 20 0 1 0 338513078 290922496 7778 33554432000 4194304 5683644 140735723290384 140735723283336 139872645489368 0 0 16781312 33554434 0 0 0 17 1 0 0 2 0 0
/proc/2397/statm: 71026 7778 1846 364 0 6255 0
Current children cumulated CPU time (s) 0.62
Current children cumulated vsize (KiB) 284104

[startup+3.10042 s]
/proc/loadavg: 2.08 2.05 2.12 3/221 2405
/proc/meminfo: memFree=25054572/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=360072 CPUtime=0.67 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2395 2397 27000 0 -1 4202496 11436 1009 1 0 56 11 0 0 20 0 2 0 338513078 368713728 7879 33554432000 4194304 5683644 140735723290384 140735723285216 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 2 0 0
/proc/2397/statm: 90018 7879 1900 364 0 25241 0
[pid=2403] ppid=2397 vsize=284292 CPUtime=0 cores=1,3,5,7
/proc/2403/stat : 2403 (python) S 2397 2397 27000 0 -1 4202560 759 0 0 0 0 0 0 0 20 0 1 0 338513234 291115008 6314 33554432000 4194304 5683644 140735723290384 140735723281472 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2403/statm: 71073 6314 341 364 0 6296 0
[pid=2405] ppid=2403 vsize=112052 CPUtime=1.53 cores=1,3,5,7
/proc/2405/stat : 2405 (scip-2.0.1.linu) R 2403 2405 2405 0 -1 4202496 23664 0 0 0 148 5 0 0 20 0 1 0 338513234 114741248 23060 33554432000 4194304 15243407 140737399223792 140737399220104 4880915 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2405/statm: 28013 23060 682 2698 0 25308 0
[pid=2397/tid=2404] ppid=2395 vsize=360072 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338513234 368713728 7879 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 2.2
Current children cumulated vsize (KiB) 756416
heavy processes:

[startup+6.30041 s]
/proc/loadavg: 2.08 2.05 2.12 3/221 2405
/proc/meminfo: memFree=24387760/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=360072 CPUtime=0.67 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2395 2397 27000 0 -1 4202496 11436 1009 1 0 56 11 0 0 20 0 2 0 338513078 368713728 7879 33554432000 4194304 5683644 140735723290384 140735723285216 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 2 0 0
/proc/2397/statm: 90018 7879 1900 364 0 25241 0
[pid=2403] ppid=2397 vsize=284292 CPUtime=0 cores=1,3,5,7
/proc/2403/stat : 2403 (python) S 2397 2397 27000 0 -1 4202560 885 0 0 0 0 0 0 0 20 0 1 0 338513234 291115008 6337 33554432000 4194304 5683644 140735723290384 140735723281472 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2403/statm: 71073 6337 364 364 0 6296 0
[pid=2405] ppid=2403 vsize=173532 CPUtime=4.72 cores=1,3,5,7
/proc/2405/stat : 2405 (scip-2.0.1.linu) R 2403 2405 2405 0 -1 4202496 38670 0 0 0 460 12 0 0 20 0 1 0 338513234 177696768 38066 33554432000 4194304 15243407 140737399223792 140737399220104 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2405/statm: 43383 38066 686 2698 0 40678 0
[pid=2397/tid=2404] ppid=2395 vsize=360072 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338513234 368713728 7879 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 5.39
Current children cumulated vsize (KiB) 817896

[startup+12.7004 s]
/proc/loadavg: 2.14 2.06 2.13 3/221 2405
/proc/meminfo: memFree=24313952/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=360072 CPUtime=0.67 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2395 2397 27000 0 -1 4202496 11436 1009 1 0 56 11 0 0 20 0 2 0 338513078 368713728 7879 33554432000 4194304 5683644 140735723290384 140735723285216 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 2 0 0
/proc/2397/statm: 90018 7879 1900 364 0 25241 0
[pid=2403] ppid=2397 vsize=284292 CPUtime=0.01 cores=1,3,5,7
/proc/2403/stat : 2403 (python) S 2397 2397 27000 0 -1 4202560 889 0 0 0 1 0 0 0 20 0 1 0 338513234 291115008 6337 33554432000 4194304 5683644 140735723290384 140735723281472 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2403/statm: 71073 6337 364 364 0 6296 0
[pid=2405] ppid=2403 vsize=228152 CPUtime=11.12 cores=1,3,5,7
/proc/2405/stat : 2405 (scip-2.0.1.linu) R 2403 2405 2405 0 -1 4202496 52333 0 0 0 1090 22 0 0 20 0 1 0 338513234 233627648 51729 33554432000 4194304 15243407 140737399223792 140737399219176 13229099 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2405/statm: 57038 51729 686 2698 0 54333 0
[pid=2397/tid=2404] ppid=2395 vsize=360072 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338513234 368713728 7879 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.8
Current children cumulated vsize (KiB) 872516

[startup+25.5004 s]
/proc/loadavg: 2.12 2.06 2.12 3/221 2405
/proc/meminfo: memFree=24234060/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=360072 CPUtime=0.67 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2395 2397 27000 0 -1 4202496 11436 1009 1 0 56 11 0 0 20 0 2 0 338513078 368713728 7879 33554432000 4194304 5683644 140735723290384 140735723285216 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 2 0 0
/proc/2397/statm: 90018 7879 1900 364 0 25241 0
[pid=2403] ppid=2397 vsize=284292 CPUtime=0.02 cores=1,3,5,7
/proc/2403/stat : 2403 (python) S 2397 2397 27000 0 -1 4202560 995 0 0 0 1 1 0 0 20 0 1 0 338513234 291115008 6342 33554432000 4194304 5683644 140735723290384 140735723281472 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2403/statm: 71073 6342 369 364 0 6296 0
[pid=2405] ppid=2403 vsize=306672 CPUtime=23.93 cores=1,3,5,7
/proc/2405/stat : 2405 (scip-2.0.1.linu) R 2403 2405 2405 0 -1 4202496 71950 0 0 0 2349 44 0 0 20 0 1 0 338513234 314032128 71318 33554432000 4194304 15243407 140737399223792 140737399219880 13277498 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2405/statm: 76668 71318 874 2698 0 73963 0
[pid=2397/tid=2404] ppid=2395 vsize=360072 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338513234 368713728 7879 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.62
Current children cumulated vsize (KiB) 951036

[startup+51.1003 s]
/proc/loadavg: 2.08 2.05 2.12 3/222 2406
/proc/meminfo: memFree=24216476/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=360072 CPUtime=0.67 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2395 2397 27000 0 -1 4202496 11436 1009 1 0 56 11 0 0 20 0 2 0 338513078 368713728 7879 33554432000 4194304 5683644 140735723290384 140735723285216 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 2 0 0
/proc/2397/statm: 90018 7879 1900 364 0 25241 0
[pid=2403] ppid=2397 vsize=284292 CPUtime=0.07 cores=1,3,5,7
/proc/2403/stat : 2403 (python) S 2397 2397 27000 0 -1 4202560 1013 0 0 0 4 3 0 0 20 0 1 0 338513234 291115008 6342 33554432000 4194304 5683644 140735723290384 140735723281472 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2403/statm: 71073 6342 369 364 0 6296 0
[pid=2405] ppid=2403 vsize=335804 CPUtime=49.51 cores=1,3,5,7
/proc/2405/stat : 2405 (scip-2.0.1.linu) R 2403 2405 2405 0 -1 4202496 83359 0 0 0 4861 90 0 0 20 0 1 0 338513234 343863296 77556 33554432000 4194304 15243407 140737399223792 140737399218504 8126699 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2405/statm: 83951 77556 899 2698 0 81246 0
[pid=2397/tid=2404] ppid=2395 vsize=360072 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 2 0 338513234 368713728 7879 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.25
Current children cumulated vsize (KiB) 980168

[startup+102.301 s]
/proc/loadavg: 2.03 2.04 2.11 3/230 2414
/proc/meminfo: memFree=24089244/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=511656 CPUtime=0.71 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2395 2397 27000 0 -1 4202496 13189 1009 1 0 60 11 0 0 20 0 4 0 338513078 523935744 8404 33554432000 4194304 5683644 140735723290384 140735723285216 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 2 0 0
/proc/2397/statm: 127914 8404 1905 364 0 63131 0
[pid=2403] ppid=2397 vsize=360072 CPUtime=0.07 cores=1,3,5,7
/proc/2403/stat : 2403 (python) S 2397 2397 27000 0 -1 4202560 1148 0 0 0 4 3 0 0 20 0 2 0 338513234 368713728 6377 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2403/statm: 90018 6377 397 364 0 25241 0
[pid=2405] ppid=2403 vsize=335836 CPUtime=50.9 cores=1,3,5,7
/proc/2405/stat : 2405 (scip-2.0.1.linu) T 2403 2405 2405 0 -1 4202496 85129 0 0 0 5000 90 0 0 20 0 1 0 338513234 343896064 77615 33554432000 4194304 15243407 140737399223792 140737399218504 7128865 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2405/statm: 83959 77615 899 2698 0 81254 0
[pid=2403/tid=2407] ppid=2397 vsize=360072 CPUtime=0 cores=1,3,5,7
/proc/2403/task/2407/stat : 2407 (python) S 2397 2397 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338518327 368713728 6377 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2408] ppid=2397 vsize=360084 CPUtime=0.07 cores=1,3,5,7
/proc/2408/stat : 2408 (python) S 2397 2397 27000 0 -1 4202560 1080 0 0 0 4 3 0 0 20 0 2 0 338518329 368726016 6364 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2408/statm: 90021 6364 384 364 0 25241 0
[pid=2410] ppid=2408 vsize=17444 CPUtime=47.11 cores=1,3,5,7
/proc/2410/stat : 2410 (clasp-1.3.7-x86) T 2408 2410 2410 0 -1 4202496 4451 0 0 0 4711 0 0 0 20 0 1 0 338518330 17862656 4091 33554432000 134512640 136139019 4290787568 4290786500 134632473 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2410/statm: 4361 4091 182 398 0 3961 0
[pid=2408/tid=2411] ppid=2397 vsize=360084 CPUtime=0 cores=1,3,5,7
/proc/2408/task/2411/stat : 2411 (python) S 2397 2397 27000 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338523042 368726016 6364 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2412] ppid=2397 vsize=435876 CPUtime=0 cores=1,3,5,7
/proc/2412/stat : 2412 (python) S 2397 2397 27000 0 -1 4202560 800 0 0 0 0 0 0 0 20 0 1 0 338523045 446337024 6829 33554432000 4194304 5683644 140735723290384 140735723281472 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2412/statm: 108969 6829 336 364 0 44186 0
[pid=2414] ppid=2412 vsize=137452 CPUtime=2.62 cores=1,3,5,7
/proc/2414/stat : 2414 (scip-2.0.1.linu) R 2412 2414 2414 0 -1 4202496 30048 0 0 0 256 6 0 0 20 0 1 0 338523045 140750848 29432 33554432000 4194304 13684479 140733691190608 140733691186920 11876362 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/2414/statm: 34363 29432 635 2317 0 32041 0
[pid=2397/tid=2404] ppid=2395 vsize=511656 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 4 0 338513234 523935744 8404 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2409] ppid=2395 vsize=511656 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2409/stat : 2409 (python) S 2395 2397 27000 0 -1 4202560 20 1009 0 0 0 0 0 0 20 0 4 0 338518330 523935744 8404 33554432000 4194304 5683644 140735723290384 139872387981872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2413] ppid=2395 vsize=511656 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2413/stat : 2413 (python) S 2395 2397 27000 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 4 0 338523045 523935744 8404 33554432000 4194304 5683644 140735723290384 139872377492016 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.48
Current children cumulated vsize (KiB) 2158420

[startup+162.301 s]
/proc/loadavg: 2.01 2.03 2.10 3/231 2415
/proc/meminfo: memFree=23885908/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=511656 CPUtime=0.73 cores=1,3,5,7

################
# More data... #
################

[pid=2483/tid=2511] ppid=2397 vsize=663756 CPUtime=0 cores=1,3,5,7
/proc/2483/task/2511/stat : 2511 (python) S 2397 2397 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338551168 679686144 7022 33554432000 4194304 5683644 140735723290384 139872356512304 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2604] ppid=2397 vsize=739548 CPUtime=0.01 cores=1,3,5,7
/proc/2604/stat : 2604 (python) S 2397 2397 27000 0 -1 4202560 1000 0 0 0 1 0 0 0 20 0 1 0 338569828 757297152 7002 33554432000 4194304 5683644 140735723290384 140735723281472 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/2604/statm: 184887 7002 359 364 0 120092 0
[pid=2606] ppid=2604 vsize=53640 CPUtime=14.78 cores=1,3,5,7
/proc/2606/stat : 2606 (bsolo_pb10) R 2604 2606 2606 0 -1 4202496 11474 0 0 0 1469 9 0 0 20 0 1 0 338569829 54927360 10981 33554432000 134512640 137138936 4288836480 4288834904 134606663 0 0 16781312 16384 0 0 0 17 1 0 0 0 0 0
/proc/2606/statm: 13410 10981 192 642 0 12766 0
[pid=2397/tid=2404] ppid=2395 vsize=815328 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 31 1009 0 0 0 0 0 0 20 0 8 0 338513234 834895872 8562 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2397/tid=2409] ppid=2395 vsize=815328 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2409/stat : 2409 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 8 0 338518330 834895872 8562 33554432000 4194304 5683644 140735723290384 139872387981872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2397/tid=2413] ppid=2395 vsize=815328 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2413/stat : 2413 (python) S 2395 2397 27000 0 -1 4202560 25 1009 0 0 0 0 0 0 20 0 8 0 338523045 834895872 8562 33554432000 4194304 5683644 140735723290384 139872377492016 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2417] ppid=2395 vsize=815328 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2417/stat : 2417 (python) S 2395 2397 27000 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 8 0 338532264 834895872 8562 33554432000 4194304 5683644 140735723290384 139872367002160 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2425] ppid=2395 vsize=815328 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2425/stat : 2425 (python) S 2395 2397 27000 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 8 0 338537004 834895872 8562 33554432000 4194304 5683644 140735723290384 139872356512304 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2484] ppid=2395 vsize=815328 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2484/stat : 2484 (python) S 2395 2397 27000 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 8 0 338546416 834895872 8562 33554432000 4194304 5683644 140735723290384 139872346022448 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2605] ppid=2395 vsize=815328 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2605/stat : 2605 (python) S 2395 2397 27000 0 -1 4202560 23 1009 0 0 0 0 0 0 20 0 8 0 338569829 834895872 8562 33554432000 4194304 5683644 140735723290384 139872335532592 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 584.57
Current children cumulated vsize (KiB) 15069800



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+615.102 s]
/proc/loadavg: 2.01 2.02 2.06 3/233 2614
/proc/meminfo: memFree=23640932/32873648 swapFree=7120/7120
[pid=2397] ppid=2395 vsize=891120 CPUtime=1.56 cores=1,3,5,7
/proc/2397/stat : 2397 (python) S 2395 2397 27000 0 -1 4202496 18218 1009 1 0 142 14 0 0 20 0 9 0 338513078 912506880 9078 33554432000 4194304 5683644 140735723290384 140735723285216 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 5 0 0 2 0 0
/proc/2397/statm: 222780 9078 1915 364 0 157982 0
[pid=2403] ppid=2397 vsize=360072 CPUtime=0.15 cores=1,3,5,7
/proc/2403/stat : 2403 (python) S 2397 2397 27000 0 -1 4202560 1180 0 0 0 9 6 0 0 20 0 2 0 338513234 368713728 6378 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2403/statm: 90018 6378 398 364 0 25241 0
[pid=2405] ppid=2403 vsize=346116 CPUtime=94.63 cores=1,3,5,7
/proc/2405/stat : 2405 (scip-2.0.1.linu) T 2403 2405 2405 0 -1 4202496 108801 0 0 0 9362 101 0 0 20 0 1 0 338513234 354422784 83529 33554432000 4194304 15243407 140737399223792 140737399218184 8119102 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2405/statm: 86529 83529 947 2698 0 83824 0
[pid=2403/tid=2407] ppid=2397 vsize=360072 CPUtime=0 cores=1,3,5,7
/proc/2403/task/2407/stat : 2407 (python) S 2397 2397 27000 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338518327 368713728 6378 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2408] ppid=2397 vsize=360084 CPUtime=0.39 cores=1,3,5,7
/proc/2408/stat : 2408 (python) S 2397 2397 27000 0 -1 4202560 1181 0 0 0 21 18 0 0 20 0 2 0 338518329 368726016 6364 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2408/statm: 90021 6364 384 364 0 25241 0
[pid=2410] ppid=2408 vsize=31440 CPUtime=229.6 cores=1,3,5,7
/proc/2410/stat : 2410 (clasp-1.3.7-x86) T 2408 2410 2410 0 -1 4202496 7961 0 0 0 22959 1 0 0 20 0 1 0 338518330 32194560 7602 33554432000 134512640 136139019 4290787568 4290786192 134599797 0 0 16781312 16386 18446744071579387992 0 0 17 3 0 0 0 0 0
/proc/2410/statm: 7860 7602 182 398 0 7460 0
[pid=2408/tid=2411] ppid=2397 vsize=360084 CPUtime=0 cores=1,3,5,7
/proc/2408/task/2411/stat : 2411 (python) S 2397 2397 27000 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338523042 368726016 6364 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2412] ppid=2397 vsize=435876 CPUtime=0.17 cores=1,3,5,7
/proc/2412/stat : 2412 (python) S 2397 2397 27000 0 -1 4202560 1141 0 0 0 10 7 0 0 20 0 2 0 338523045 446337024 6877 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2412/statm: 108969 6877 384 364 0 44186 0
[pid=2414] ppid=2412 vsize=316716 CPUtime=95.51 cores=1,3,5,7
/proc/2414/stat : 2414 (scip-2.0.1.linu) T 2412 2414 2414 0 -1 4202496 77474 0 0 0 9508 43 0 0 20 0 1 0 338523045 324317184 75984 33554432000 4194304 13684479 140733691190608 140733691182520 7010167 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2414/statm: 79179 75984 759 2317 0 76857 0
[pid=2412/tid=2415] ppid=2397 vsize=435876 CPUtime=0 cores=1,3,5,7
/proc/2412/task/2415/stat : 2415 (python) S 2397 2397 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338527684 446337024 6877 33554432000 4194304 5683644 140735723290384 139872387981872 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2416] ppid=2397 vsize=511668 CPUtime=0.07 cores=1,3,5,7
/proc/2416/stat : 2416 (python) S 2397 2397 27000 0 -1 4202560 1136 0 0 0 4 3 0 0 20 0 2 0 338532264 523948032 6884 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2416/statm: 127917 6884 384 364 0 63131 0
[pid=2418] ppid=2416 vsize=91032 CPUtime=47.34 cores=1,3,5,7
/proc/2418/stat : 2418 (minion) T 2416 2418 2418 0 -1 4202496 9155 0 0 0 4733 1 0 0 20 0 1 0 338532265 93216768 15455 33554432000 134512640 165565024 4287465424 4287462672 142765104 0 0 16781312 8396802 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2418/statm: 22758 15455 336 7582 0 15173 0
[pid=2416/tid=2423] ppid=2397 vsize=511668 CPUtime=0 cores=1,3,5,7
/proc/2416/task/2423/stat : 2423 (python) S 2397 2397 27000 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338537001 523948032 6884 33554432000 4194304 5683644 140735723290384 139872377492016 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2424] ppid=2397 vsize=587964 CPUtime=0.07 cores=1,3,5,7
/proc/2424/stat : 2424 (python) S 2397 2397 27000 0 -1 4202560 1132 0 0 0 4 3 0 0 20 0 2 0 338537004 602075136 7016 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2424/statm: 146991 7016 384 364 0 82202 0
[pid=2426] ppid=2424 vsize=131588 CPUtime=50.31 cores=1,3,5,7
/proc/2426/stat : 2426 (wbo1.4a) T 2424 2426 2426 0 -1 4202496 32807 0 0 0 4962 69 0 0 20 0 1 0 338537004 134746112 32438 33554432000 134512640 135751879 4294754832 4294752444 134518928 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2426/statm: 32897 32438 143 303 0 32592 0
[pid=2424/tid=2478] ppid=2397 vsize=587964 CPUtime=0 cores=1,3,5,7
/proc/2424/task/2478/stat : 2478 (python) S 2397 2397 27000 0 -1 4202560 51 0 0 0 0 0 0 0 20 0 2 0 338542037 602075136 7016 33554432000 4194304 5683644 140735723290384 139872367002160 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2483] ppid=2397 vsize=663756 CPUtime=0.08 cores=1,3,5,7
/proc/2483/stat : 2483 (python) S 2397 2397 27000 0 -1 4202560 1155 0 0 0 5 3 0 0 20 0 2 0 338546416 679686144 7022 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/2483/statm: 165939 7022 384 364 0 101147 0
[pid=2485] ppid=2483 vsize=9624972 CPUtime=49.93 cores=1,3,5,7
/proc/2485/stat : 2485 (java) T 2483 2485 2485 0 -1 4202496 25893 0 1 0 4952 41 0 0 20 0 19 0 338546416 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140734470861216 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2485/statm: 2406243 427836 2267 9 0 2396937 0
[pid=2485/tid=2486] ppid=2483 vsize=9624972 CPUtime=47.22 cores=1,3,5,7
/proc/2485/task/2486/stat : 2486 (java) T 2483 2485 2485 0 -1 4202560 3404 0 1 0 4697 25 0 0 20 0 19 0 338546417 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583310204304 140583155429680 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2487] ppid=2483 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/2485/task/2487/stat : 2487 (java) T 2483 2485 2485 0 -1 4202560 139 0 0 0 7 2 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583282657704 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2485/tid=2488] ppid=2483 vsize=9624972 CPUtime=0.07 cores=1,3,5,7
/proc/2485/task/2488/stat : 2488 (java) T 2483 2485 2485 0 -1 4202560 56 0 0 0 6 1 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583281605160 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2489] ppid=2483 vsize=9624972 CPUtime=0.09 cores=1,3,5,7
/proc/2485/task/2489/stat : 2489 (java) T 2483 2485 2485 0 -1 4202560 38 0 0 0 8 1 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583280552104 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2490] ppid=2483 vsize=9624972 CPUtime=0.11 cores=1,3,5,7
/proc/2485/task/2490/stat : 2490 (java) T 2483 2485 2485 0 -1 4202560 136 0 0 0 9 2 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583279499560 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2491] ppid=2483 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2485/task/2491/stat : 2491 (java) T 2483 2485 2485 0 -1 4202560 47 0 0 0 7 1 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583278447528 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2492] ppid=2483 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2485/task/2492/stat : 2492 (java) T 2483 2485 2485 0 -1 4202560 122 0 0 0 6 2 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583277394984 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2485/tid=2493] ppid=2483 vsize=9624972 CPUtime=0.1 cores=1,3,5,7
/proc/2485/task/2493/stat : 2493 (java) T 2483 2485 2485 0 -1 4202560 157 0 0 0 8 2 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583276341928 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2494] ppid=2483 vsize=9624972 CPUtime=0.08 cores=1,3,5,7
/proc/2485/task/2494/stat : 2494 (java) T 2483 2485 2485 0 -1 4202560 61 0 0 0 8 0 0 0 20 0 19 0 338546419 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140583275289384 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2485/tid=2495] ppid=2483 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2485/task/2495/stat : 2495 (java) T 2483 2485 2485 0 -1 4202560 740 0 0 0 0 0 0 0 20 0 19 0 338546420 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582663989608 219517335465 0 0 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2496] ppid=2483 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2485/task/2496/stat : 2496 (java) T 2483 2485 2485 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338546420 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582662935928 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2497] ppid=2483 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2485/task/2497/stat : 2497 (java) T 2483 2485 2485 0 -1 4202560 9 0 0 0 0 0 0 0 20 0 19 0 338546420 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582661882760 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2498] ppid=2483 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2485/task/2498/stat : 2498 (java) T 2483 2485 2485 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338546422 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582660831504 219517343520 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2499] ppid=2483 vsize=9624972 CPUtime=0.93 cores=1,3,5,7
/proc/2485/task/2499/stat : 2499 (java) T 2483 2485 2485 0 -1 4202560 9727 0 0 0 91 2 0 0 20 0 19 0 338546422 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582659779384 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2485/tid=2500] ppid=2483 vsize=9624972 CPUtime=0.92 cores=1,3,5,7
/proc/2485/task/2500/stat : 2500 (java) T 2483 2485 2485 0 -1 4202560 9847 0 0 0 90 2 0 0 20 0 19 0 338546423 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582658726840 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2501] ppid=2483 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2485/task/2501/stat : 2501 (java) T 2483 2485 2485 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338546423 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582657673976 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 1 0 0 0 0 0
[pid=2485/tid=2502] ppid=2483 vsize=9624972 CPUtime=0.01 cores=1,3,5,7
/proc/2485/task/2502/stat : 2502 (java) T 2483 2485 2485 0 -1 4202560 3 0 0 0 1 0 0 0 20 0 19 0 338546423 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582656621640 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2485/tid=2503] ppid=2483 vsize=9624972 CPUtime=0 cores=1,3,5,7
/proc/2485/task/2503/stat : 2503 (java) T 2483 2485 2485 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338546456 9855971328 427836 33554432000 1073741824 1073778376 140734470870080 140582655566888 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 3 0 0 0 0 0
[pid=2483/tid=2511] ppid=2397 vsize=663756 CPUtime=0 cores=1,3,5,7
/proc/2483/task/2511/stat : 2511 (python) S 2397 2397 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338551168 679686144 7022 33554432000 4194304 5683644 140735723290384 139872356512304 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2604] ppid=2397 vsize=739548 CPUtime=0.07 cores=1,3,5,7
/proc/2604/stat : 2604 (python) S 2397 2397 27000 0 -1 4202560 1173 0 0 0 4 3 0 0 20 0 2 0 338569828 757297152 7028 33554432000 4194304 5683644 140735723290384 140735723282032 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/2604/statm: 184887 7028 384 364 0 120092 0
[pid=2606] ppid=2604 vsize=112648 CPUtime=47.47 cores=1,3,5,7
/proc/2606/stat : 2606 (bsolo_pb10) T 2604 2606 2606 0 -1 4202496 26229 0 0 0 4732 15 0 0 20 0 1 0 338569829 115351552 25736 33554432000 134512640 137138936 4288836480 4288835016 136504096 0 0 16781312 16384 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/2606/statm: 28162 25736 192 642 0 27518 0
[pid=2604/tid=2611] ppid=2397 vsize=739548 CPUtime=0 cores=1,3,5,7
/proc/2604/task/2611/stat : 2611 (python) S 2397 2397 27000 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338574578 757297152 7028 33554432000 4194304 5683644 140735723290384 139872346022448 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2612] ppid=2397 vsize=815340 CPUtime=0 cores=1,3,5,7
/proc/2612/stat : 2612 (python) S 2397 2397 27000 0 -1 4202560 859 0 0 0 0 0 0 0 20 0 1 0 338574580 834908160 6985 33554432000 4194304 5683644 140735723290384 140735723281472 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 7 0 0 0 0 0
/proc/2612/statm: 203835 6985 336 364 0 139037 0
[pid=2614] ppid=2612 vsize=13300 CPUtime=0.08 cores=1,3,5,7
/proc/2614/stat : 2614 (bsolo_pb10) R 2612 2614 2614 0 -1 4202496 1236 0 0 0 8 0 0 0 20 0 1 0 338574581 13619200 860 33554432000 134512640 137138936 4292864688 4292780972 136570369 0 0 16781312 16384 0 0 0 17 3 0 0 0 0 0
/proc/2614/statm: 3325 860 184 642 0 2681 0
[pid=2397/tid=2404] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2404/stat : 2404 (python) S 2395 2397 27000 0 -1 4202560 31 1009 0 0 0 0 0 0 20 0 9 0 338513234 912506880 9078 33554432000 4194304 5683644 140735723290384 139872470422064 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2397/tid=2409] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2409/stat : 2409 (python) S 2395 2397 27000 0 -1 4202560 26 1009 0 0 0 0 0 0 20 0 9 0 338518330 912506880 9078 33554432000 4194304 5683644 140735723290384 139872387981872 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=2397/tid=2413] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2413/stat : 2413 (python) S 2395 2397 27000 0 -1 4202560 25 1009 0 0 0 0 0 0 20 0 9 0 338523045 912506880 9078 33554432000 4194304 5683644 140735723290384 139872377492016 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2417] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2417/stat : 2417 (python) S 2395 2397 27000 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338532264 912506880 9078 33554432000 4194304 5683644 140735723290384 139872367002160 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2425] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2425/stat : 2425 (python) S 2395 2397 27000 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338537004 912506880 9078 33554432000 4194304 5683644 140735723290384 139872356512304 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2484] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2484/stat : 2484 (python) S 2395 2397 27000 0 -1 4202560 21 1009 0 0 0 0 0 0 20 0 9 0 338546416 912506880 9078 33554432000 4194304 5683644 140735723290384 139872346022448 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=2397/tid=2605] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2605/stat : 2605 (python) S 2395 2397 27000 0 -1 4202560 23 1009 0 0 0 0 0 0 20 0 9 0 338569829 912506880 9078 33554432000 4194304 5683644 140735723290384 139872335532592 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=2397/tid=2613] ppid=2395 vsize=891120 CPUtime=0 cores=1,3,5,7
/proc/2397/task/2613/stat : 2613 (python) S 2395 2397 27000 0 -1 4202560 19 1009 0 0 0 0 0 0 20 0 9 0 338574580 912506880 9078 33554432000 4194304 5683644 140735723290384 139871985328688 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 617.43
Current children cumulated vsize (KiB) 16033240

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 2397 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=428782
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=144977
# CPU time returned by wait4() is 1.57376
# while last known CPU time is 617.43
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 615.164
CPU time (s): 617.43
CPU user time (s): 614.15
CPU system time (s): 3.28
CPU usage (%): 100.368
Max. virtual memory (cumulated for all children) (KiB): 16033240

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.42878
system time used= 0.144977
maximum resident set size= 36312
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19733
page faults= 1
swaps= 0
block input operations= 0
block output operations= 1640
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2093
involuntary context switches= 19

runsolver used 2.6426 second user time and 6.60499 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 13:44:16
IDJOB=3742200
IDBENCH=90139
IDSOLVER=2327
FILE ID=node114/3742200-1338723856
RUNJOBID= node114-1338713908-27017
PBS_JOBID= 14636748
Free space on /tmp= 70040 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathTate/normalized-TateBritain_K101.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742200-1338723856/watcher-3742200-1338723856 -o /tmp/evaluation-result-3742200-1338723856/solver-3742200-1338723856 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742200-1338723856.opb -seed 830543092 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= a3de6c6ee42f8b511357d3857e721bb4
RANDOM SEED=830543092

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        25158348 kB
Buffers:         1037556 kB
Cached:          3819612 kB
SwapCached:         2180 kB
Active:          4914444 kB
Inactive:        1773928 kB
Active(anon):    1815736 kB
Inactive(anon):    10732 kB
Active(file):    3098708 kB
Inactive(file):  1763196 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:            102312 kB
Writeback:             0 kB
AnonPages:       1837528 kB
Mapped:            34736 kB
Shmem:               184 kB
Slab:             870596 kB
SReclaimable:     312428 kB
SUnreclaim:       558168 kB
KernelStack:        1760 kB
PageTables:        11880 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    3295496 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1093632 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 69940 MiB
End job on node114 at 2012-06-03 13:54:34