Trace number 3742226

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 285.14 282.862

General information on the benchmark

Name/DEC-SMALLINT-LIN/sroussel/
ShortestPathNG/normalized-NG_K128.opb
MD5SUM17c4ca9a6493665b57c1ccd2ab23e03d
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmark
Best value of the objective obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables15232
Total number of constraints15490
Number of constraints which are clauses15362
Number of constraints which are cardinality constraints (but not clauses)128
Number of constraints which are nor clauses,nor cardinality constraints0
Minimum length of a constraint1
Maximum length of a constraint128
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 1
Number of bits of the biggest number in a constraint 1
Biggest sum of numbers in a constraint 129
Number of bits of the biggest sum of numbers8
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.34/1.21	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.34/1.21	c INFO: solving HOME/instance-3742226-1338724165.opb
0.45/1.38	c INFO: parsing took 0.16 s
0.65/1.71	c INFO: features took 0.33 CPU seconds
0.65/1.74	c INFO: running scip-2.0.1-clp@0 for 45 with 1799 remaining (b = 0.61)
51.58/52.48	c INFO: running clasp-1.3.7@0 for 45 with 1748 remaining (b = 0.43)
98.85/99.64	c INFO: running scip-2.0.1-spx@0 for 45 with 1701 remaining (b = 0.05)
149.49/150.25	c INFO: running scip-2.0.1-clp@50 for 45 with 1651 remaining (b = 0.01)
192.93/193.60	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1608 remaining (b = 0.04)
239.17/237.29	c INFO: running clasp-1.3.7@47 for 45 with 1562 remaining (b = 0.01)
284.44/282.44	c INFO: running sat4j-pb-v20101225@0 for 45 with 1517 remaining (b = 0.14)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3742226-1338724165/watcher-3742226-1338724165 -o /tmp/evaluation-result-3742226-1338724165/solver-3742226-1338724165 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742226-1338724165.opb -seed 1927519361 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 1.99 2.00 2.08 2/218 9087
/proc/meminfo: memFree=29240484/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=15276 CPUtime=0 cores=0,2,4,6
/proc/9087/stat : 9087 (python) D 9085 9087 3958 0 -1 4202496 672 0 0 0 0 0 0 0 20 0 1 0 338542641 15642624 559 33554432000 4194304 5683644 140734291953520 140734291921464 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 3819 559 324 364 0 261 0

[startup+0.0614669 s]
/proc/loadavg: 1.99 2.00 2.08 2/218 9087
/proc/meminfo: memFree=29240484/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=16596 CPUtime=0.01 cores=0,2,4,6
/proc/9087/stat : 9087 (python) R 9085 9087 3958 0 -1 4202496 1198 0 0 0 1 0 0 0 20 0 1 0 338542641 16994304 997 33554432000 4194304 5683644 140734291953520 140734291920984 217410519952 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 4149 997 418 364 0 591 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16596

[startup+0.100164 s]
/proc/loadavg: 1.99 2.00 2.08 2/218 9087
/proc/meminfo: memFree=29240484/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=19088 CPUtime=0.02 cores=0,2,4,6
/proc/9087/stat : 9087 (python) R 9085 9087 3958 0 -1 4202496 1398 0 0 0 2 0 0 0 20 0 1 0 338542641 19546112 1152 33554432000 4194304 5683644 140734291953520 140734291884024 217410153232 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 4772 1152 430 364 0 696 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19088

[startup+0.300153 s]
/proc/loadavg: 1.99 2.00 2.08 2/218 9087
/proc/meminfo: memFree=29240484/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=99660 CPUtime=0.1 cores=0,2,4,6
/proc/9087/stat : 9087 (python) R 9085 9087 3958 0 -1 4202496 4001 1011 1 0 8 2 0 0 20 0 1 0 338542641 102051840 2689 33554432000 4194304 5683644 140734291953520 140734291827608 217410836543 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 24915 2689 847 364 0 1722 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 99660

[startup+0.700142 s]
/proc/loadavg: 1.99 2.00 2.08 2/218 9087
/proc/meminfo: memFree=29240484/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=138348 CPUtime=0.17 cores=0,2,4,6
/proc/9087/stat : 9087 (python) D 9085 9087 3958 0 -1 4202496 6248 1011 1 0 12 5 0 0 20 0 1 0 338542641 141668352 4084 33554432000 4194304 5683644 140734291953520 140734291853512 217410519952 0 0 16781312 2 18446744072100588836 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 34587 4084 1098 364 0 2953 0
Current children cumulated CPU time (s) 0.17
Current children cumulated vsize (KiB) 138348

[startup+1.50013 s]
/proc/loadavg: 1.99 2.00 2.08 3/219 9092
/proc/meminfo: memFree=29220504/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=287820 CPUtime=0.65 cores=0,2,4,6
/proc/9087/stat : 9087 (python) R 9085 9087 3958 0 -1 4202496 12706 1011 1 0 56 9 0 0 20 0 1 0 338542641 294727680 8706 33554432000 4194304 5683644 140734291953520 140734291946520 140645649230320 0 0 16781312 33554434 0 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 71955 8706 1846 364 0 7184 0
Current children cumulated CPU time (s) 0.65
Current children cumulated vsize (KiB) 287820

[startup+3.10033 s]
/proc/loadavg: 1.99 2.00 2.08 3/222 9095
/proc/meminfo: memFree=29127976/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=363868 CPUtime=0.88 cores=0,2,4,6
/proc/9087/stat : 9087 (python) S 9085 9087 3958 0 -1 4202496 13083 1011 1 0 79 9 0 0 20 0 2 0 338542641 372600832 8859 33554432000 4194304 5683644 140734291953520 140734291948352 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 90967 8859 1902 364 0 26190 0
[pid=9093] ppid=9087 vsize=288088 CPUtime=0 cores=0,2,4,6
/proc/9093/stat : 9093 (python) S 9087 9087 3958 0 -1 4202560 765 0 0 0 0 0 0 0 20 0 1 0 338542815 295002112 7291 33554432000 4194304 5683644 140734291953520 140734291944608 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 4 0 0 0 0 0
/proc/9093/statm: 72022 7291 340 364 0 7245 0
[pid=9095] ppid=9093 vsize=150300 CPUtime=1.34 cores=0,2,4,6
/proc/9095/stat : 9095 (scip-2.0.1.linu) R 9093 9095 9095 0 -1 4202496 34278 0 0 0 125 9 0 0 20 0 1 0 338542816 153907200 33567 33554432000 4194304 15243407 140733566515200 140733566510584 4880912 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9095/statm: 37575 33567 683 2698 0 34870 0
[pid=9087/tid=9094] ppid=9085 vsize=363868 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338542816 372600832 8859 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 2.22
Current children cumulated vsize (KiB) 802256
heavy processes:

[startup+6.30028 s]
/proc/loadavg: 1.99 2.00 2.08 3/222 9095
/proc/meminfo: memFree=29033492/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=363868 CPUtime=0.88 cores=0,2,4,6
/proc/9087/stat : 9087 (python) S 9085 9087 3958 0 -1 4202496 13083 1011 1 0 79 9 0 0 20 0 2 0 338542641 372600832 8859 33554432000 4194304 5683644 140734291953520 140734291948352 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 90967 8859 1902 364 0 26190 0
[pid=9093] ppid=9087 vsize=288088 CPUtime=0 cores=0,2,4,6
/proc/9093/stat : 9093 (python) S 9087 9087 3958 0 -1 4202560 890 0 0 0 0 0 0 0 20 0 1 0 338542815 295002112 7314 33554432000 4194304 5683644 140734291953520 140734291944608 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9093/statm: 72022 7314 363 364 0 7245 0
[pid=9095] ppid=9093 vsize=209340 CPUtime=4.55 cores=0,2,4,6
/proc/9095/stat : 9095 (scip-2.0.1.linu) R 9093 9095 9095 0 -1 4202496 47957 0 0 0 443 12 0 0 20 0 1 0 338542816 214364160 47246 33554432000 4194304 15243407 140733566515200 140733566510584 4880958 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9095/statm: 52335 47246 683 2698 0 49630 0
[pid=9087/tid=9094] ppid=9085 vsize=363868 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338542816 372600832 8859 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 5.43
Current children cumulated vsize (KiB) 861296

[startup+12.7004 s]
/proc/loadavg: 1.99 2.00 2.08 3/222 9095
/proc/meminfo: memFree=28962752/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=363868 CPUtime=0.88 cores=0,2,4,6
/proc/9087/stat : 9087 (python) S 9085 9087 3958 0 -1 4202496 13083 1011 1 0 79 9 0 0 20 0 2 0 338542641 372600832 8859 33554432000 4194304 5683644 140734291953520 140734291948352 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 90967 8859 1902 364 0 26190 0
[pid=9093] ppid=9087 vsize=288088 CPUtime=0.01 cores=0,2,4,6
/proc/9093/stat : 9093 (python) S 9087 9087 3958 0 -1 4202560 894 0 0 0 1 0 0 0 20 0 1 0 338542815 295002112 7314 33554432000 4194304 5683644 140734291953520 140734291944608 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9093/statm: 72022 7314 363 364 0 7245 0
[pid=9095] ppid=9093 vsize=280448 CPUtime=10.95 cores=0,2,4,6
/proc/9095/stat : 9095 (scip-2.0.1.linu) R 9093 9095 9095 0 -1 4202496 65294 0 0 0 1079 16 0 0 20 0 1 0 338542816 287178752 64583 33554432000 4194304 15243407 140733566515200 140733566511496 13277498 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9095/statm: 70112 64583 683 2698 0 67407 0
[pid=9087/tid=9094] ppid=9085 vsize=363868 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338542816 372600832 8859 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 11.84
Current children cumulated vsize (KiB) 932404

[startup+25.5069 s]
/proc/loadavg: 1.99 2.00 2.07 3/222 9095
/proc/meminfo: memFree=28845528/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=363868 CPUtime=0.88 cores=0,2,4,6
/proc/9087/stat : 9087 (python) S 9085 9087 3958 0 -1 4202496 13083 1011 1 0 79 9 0 0 20 0 2 0 338542641 372600832 8859 33554432000 4194304 5683644 140734291953520 140734291948352 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 90967 8859 1902 364 0 26190 0
[pid=9093] ppid=9087 vsize=288088 CPUtime=0.03 cores=0,2,4,6
/proc/9093/stat : 9093 (python) S 9087 9087 3958 0 -1 4202560 996 0 0 0 2 1 0 0 20 0 1 0 338542815 295002112 7319 33554432000 4194304 5683644 140734291953520 140734291944608 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9093/statm: 72022 7319 368 364 0 7245 0
[pid=9095] ppid=9093 vsize=389836 CPUtime=23.74 cores=0,2,4,6
/proc/9095/stat : 9095 (scip-2.0.1.linu) R 9093 9095 9095 0 -1 4202496 92669 0 0 0 2345 29 0 0 20 0 1 0 338542816 399192064 91958 33554432000 4194304 15243407 140733566515200 140733566511496 4508550 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9095/statm: 97459 91958 687 2698 0 94754 0
[pid=9087/tid=9094] ppid=9085 vsize=363868 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338542816 372600832 8859 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 24.65
Current children cumulated vsize (KiB) 1041792

[startup+51.1003 s]
/proc/loadavg: 1.99 2.00 2.07 3/226 9101
/proc/meminfo: memFree=28629416/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=363868 CPUtime=0.88 cores=0,2,4,6
/proc/9087/stat : 9087 (python) S 9085 9087 3958 0 -1 4202496 13083 1011 1 0 79 9 0 0 20 0 2 0 338542641 372600832 8859 33554432000 4194304 5683644 140734291953520 140734291948352 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9087/statm: 90967 8859 1902 364 0 26190 0
[pid=9093] ppid=9087 vsize=288088 CPUtime=0.07 cores=0,2,4,6
/proc/9093/stat : 9093 (python) S 9087 9087 3958 0 -1 4202560 1017 0 0 0 4 3 0 0 20 0 1 0 338542815 295002112 7319 33554432000 4194304 5683644 140734291953520 140734291944608 217410535848 0 0 16781312 33554946 18446744071580425449 0 0 17 2 0 0 0 0 0
/proc/9093/statm: 72022 7319 368 364 0 7245 0
[pid=9095] ppid=9093 vsize=547164 CPUtime=49.33 cores=0,2,4,6
/proc/9095/stat : 9095 (scip-2.0.1.linu) R 9093 9095 9095 0 -1 4202496 131975 0 0 0 4887 46 0 0 20 0 1 0 338542816 560295936 131264 33554432000 4194304 15243407 140733566515200 140733566511512 4868208 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/9095/statm: 136791 131264 687 2698 0 134086 0
[pid=9087/tid=9094] ppid=9085 vsize=363868 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 2 0 338542816 372600832 8859 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
Current children cumulated CPU time (s) 50.28
Current children cumulated vsize (KiB) 1199120

[startup+102.301 s]
/proc/loadavg: 1.99 2.00 2.07 3/191 9159
/proc/meminfo: memFree=30328668/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=515452 CPUtime=0.94 cores=0,2,4,6
/proc/9087/stat : 9087 (python) S 9085 9087 3958 0 -1 4202496 14931 1011 1 0 84 10 0 0 20 0 4 0 338542641 527822848 8876 33554432000 4194304 5683644 140734291953520 140734291948352 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/9087/statm: 128863 8876 1907 364 0 64080 0
[pid=9093] ppid=9087 vsize=363868 CPUtime=0.08 cores=0,2,4,6
/proc/9093/stat : 9093 (python) S 9087 9087 3958 0 -1 4202560 1155 0 0 0 5 3 0 0 20 0 2 0 338542815 372600832 7354 33554432000 4194304 5683644 140734291953520 140734291945168 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9093/statm: 90967 7354 396 364 0 26190 0
[pid=9095] ppid=9093 vsize=555268 CPUtime=50.69 cores=0,2,4,6
/proc/9095/stat : 9095 (scip-2.0.1.linu) T 9093 9095 9095 0 -1 4202496 133886 0 0 0 5022 47 0 0 20 0 1 0 338542816 568594432 133175 33554432000 4194304 15243407 140733566515200 140733566511512 4880912 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/9095/statm: 138817 133175 687 2698 0 136112 0
[pid=9093/tid=9102] ppid=9087 vsize=363868 CPUtime=0 cores=0,2,4,6
/proc/9093/task/9102/stat : 9102 (python) S 9087 9087 3958 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338547888 372600832 7354 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9103] ppid=9087 vsize=363880 CPUtime=0.07 cores=0,2,4,6
/proc/9103/stat : 9103 (python) S 9087 9087 3958 0 -1 4202560 1087 0 0 0 3 4 0 0 20 0 2 0 338547890 372613120 7341 33554432000 4194304 5683644 140734291953520 140734291945168 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9103/statm: 90970 7341 383 364 0 26190 0
[pid=9105] ppid=9103 vsize=21904 CPUtime=47.11 cores=0,2,4,6
/proc/9105/stat : 9105 (clasp-1.3.7-x86) T 9103 9105 9105 0 -1 4202496 5711 0 0 0 4710 1 0 0 20 0 1 0 338547890 22429696 5219 33554432000 134512640 136139019 4289832512 4289831236 135527656 0 0 16781312 16386 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/9105/statm: 5476 5219 182 398 0 5076 0
[pid=9103/tid=9156] ppid=9087 vsize=363880 CPUtime=0 cores=0,2,4,6
/proc/9103/task/9156/stat : 9156 (python) S 9087 9087 3958 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338552603 372613120 7341 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9157] ppid=9087 vsize=439672 CPUtime=0 cores=0,2,4,6
/proc/9157/stat : 9157 (python) S 9087 9087 3958 0 -1 4202560 807 0 0 0 0 0 0 0 20 0 1 0 338552606 450224128 7298 33554432000 4194304 5683644 140734291953520 140734291944608 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/9157/statm: 109918 7298 335 364 0 45135 0
[pid=9159] ppid=9157 vsize=187040 CPUtime=2.64 cores=0,2,4,6
/proc/9159/stat : 9159 (scip-2.0.1.linu) R 9157 9159 9159 0 -1 4202496 42432 0 0 0 254 10 0 0 20 0 1 0 338552606 191528960 41710 33554432000 4194304 13684479 140735329412064 140735329407448 4880923 0 0 16781312 2 0 0 0 17 2 0 0 0 0 0
/proc/9159/statm: 46760 41710 636 2317 0 44438 0
[pid=9087/tid=9094] ppid=9085 vsize=515452 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 4 0 338542816 527822848 8876 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=9087/tid=9104] ppid=9085 vsize=515452 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9104/stat : 9104 (python) S 9085 9087 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 4 0 338547890 527822848 8876 33554432000 4194304 5683644 140734291953520 140645458993712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9087/tid=9158] ppid=9085 vsize=515452 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9158/stat : 9158 (python) S 9085 9087 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 4 0 338552606 527822848 8876 33554432000 4194304 5683644 140734291953520 140645448503856 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
Current children cumulated CPU time (s) 101.53
Current children cumulated vsize (KiB) 2447084

[startup+162.301 s]
/proc/loadavg: 1.99 2.00 2.06 3/196 9164
/proc/meminfo: memFree=29612892/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=515452 CPUtime=0.96 cores=0,2,4,6

################
# More data... #
################

[pid=9172/tid=9187] ppid=9170 vsize=9624972 CPUtime=1.03 cores=0,2,4,6
/proc/9172/task/9187/stat : 9187 (java) T 9170 9172 9172 0 -1 4202560 8425 0 0 0 101 2 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581104077624 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=9172/tid=9188] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9188/stat : 9188 (java) T 9170 9172 9172 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581103025016 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=9172/tid=9189] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9189/stat : 9189 (java) T 9170 9172 9172 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581101972424 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=9172/tid=9190] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9190/stat : 9190 (java) T 9170 9172 9172 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338562059 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581100917928 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9170/tid=9192] ppid=9087 vsize=515464 CPUtime=0 cores=0,2,4,6
/proc/9170/task/9192/stat : 9192 (python) S 9087 9087 3958 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338566368 527835136 7353 33554432000 4194304 5683644 140734291953520 140645448503856 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9087/tid=9094] ppid=9085 vsize=591244 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 31 1011 0 0 0 0 0 0 20 0 5 0 338542816 605433856 8884 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=9087/tid=9104] ppid=9085 vsize=591244 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9104/stat : 9104 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 5 0 338547890 605433856 8884 33554432000 4194304 5683644 140734291953520 140645458993712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9087/tid=9158] ppid=9085 vsize=591244 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9158/stat : 9158 (python) S 9085 9087 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 5 0 338552606 605433856 8884 33554432000 4194304 5683644 140734291953520 140645448503856 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9087/tid=9171] ppid=9085 vsize=591244 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9171/stat : 9171 (python) S 9085 9087 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 5 0 338562002 605433856 8884 33554432000 4194304 5683644 140734291953520 140645438014000 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 284.34
Current children cumulated vsize (KiB) 13275184



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+282.801 s]
/proc/loadavg: 1.99 2.00 2.05 3/264 9233
/proc/meminfo: memFree=26451088/32873648 swapFree=7628/7628
[pid=9087] ppid=9085 vsize=667036 CPUtime=1.03 cores=0,2,4,6
/proc/9087/stat : 9087 (python) S 9085 9087 3958 0 -1 4202496 16873 1011 1 0 93 10 0 0 20 0 6 0 338542641 683044864 8892 33554432000 4194304 5683644 140734291953520 140734291948352 217413903629 0 0 16781312 33554434 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9087/statm: 166759 8892 1911 364 0 101970 0
[pid=9093] ppid=9087 vsize=363868 CPUtime=0.15 cores=0,2,4,6
/proc/9093/stat : 9093 (python) S 9087 9087 3958 0 -1 4202560 1191 0 0 0 8 7 0 0 20 0 2 0 338542815 372600832 7355 33554432000 4194304 5683644 140734291953520 140734291945168 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9093/statm: 90967 7355 397 364 0 26190 0
[pid=9095] ppid=9093 vsize=794076 CPUtime=93.98 cores=0,2,4,6
/proc/9095/stat : 9095 (scip-2.0.1.linu) T 9093 9095 9095 0 -1 4202496 193365 0 0 0 9326 72 0 0 20 0 1 0 338542816 813133824 194814 33554432000 4194304 15243407 140733566515200 140733566511512 6145123 0 0 16781312 2 18446744071579387992 0 0 17 0 0 0 0 0 0
/proc/9095/statm: 198519 194814 687 2698 0 195814 0
[pid=9093/tid=9102] ppid=9087 vsize=363868 CPUtime=0 cores=0,2,4,6
/proc/9093/task/9102/stat : 9102 (python) S 9087 9087 3958 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338547888 372600832 7355 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9103] ppid=9087 vsize=363880 CPUtime=0.16 cores=0,2,4,6
/proc/9103/stat : 9103 (python) S 9087 9087 3958 0 -1 4202560 1151 0 0 0 9 7 0 0 20 0 2 0 338547890 372613120 7341 33554432000 4194304 5683644 140734291953520 140734291945168 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9103/statm: 90970 7341 383 364 0 26190 0
[pid=9105] ppid=9103 vsize=27984 CPUtime=92.24 cores=0,2,4,6
/proc/9105/stat : 9105 (clasp-1.3.7-x86) T 9103 9105 9105 0 -1 4202496 7229 0 0 0 9222 2 0 0 20 0 1 0 338547890 28655616 6737 33554432000 134512640 136139019 4289832512 4289831488 134883704 0 0 16781312 16386 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/9105/statm: 6996 6737 182 398 0 6596 0
[pid=9103/tid=9156] ppid=9087 vsize=363880 CPUtime=0 cores=0,2,4,6
/proc/9103/task/9156/stat : 9156 (python) S 9087 9087 3958 0 -1 4202560 52 0 0 0 0 0 0 0 20 0 2 0 338552603 372613120 7341 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9157] ppid=9087 vsize=439672 CPUtime=0.08 cores=0,2,4,6
/proc/9157/stat : 9157 (python) S 9087 9087 3958 0 -1 4202560 1119 0 0 0 3 5 0 0 20 0 2 0 338552606 450224128 7347 33554432000 4194304 5683644 140734291953520 140734291945168 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9157/statm: 109918 7347 383 364 0 45135 0
[pid=9159] ppid=9157 vsize=554024 CPUtime=50.56 cores=0,2,4,6
/proc/9159/stat : 9159 (scip-2.0.1.linu) T 9157 9159 9159 0 -1 4202496 133857 0 0 0 5015 41 0 0 20 0 1 0 338552606 567320576 133135 33554432000 4194304 13684479 140735329412064 140735329408376 4880915 0 0 16781312 2 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/9159/statm: 138506 133135 640 2317 0 136184 0
[pid=9157/tid=9164] ppid=9087 vsize=439672 CPUtime=0 cores=0,2,4,6
/proc/9157/task/9164/stat : 9164 (python) S 9087 9087 3958 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338557664 450224128 7347 33554432000 4194304 5683644 140734291953520 140645458993712 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9170] ppid=9087 vsize=515464 CPUtime=0.07 cores=0,2,4,6
/proc/9170/stat : 9170 (python) S 9087 9087 3958 0 -1 4202560 1139 0 0 0 4 3 0 0 20 0 2 0 338562002 527835136 7353 33554432000 4194304 5683644 140734291953520 140734291945168 217413903629 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/9170/statm: 128866 7353 383 364 0 64080 0
[pid=9172] ppid=9170 vsize=9624972 CPUtime=46.22 cores=0,2,4,6
/proc/9172/stat : 9172 (java) T 9170 9172 9172 0 -1 4202496 22659 0 1 0 4581 41 0 0 20 0 19 0 338562002 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140735859959408 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/9172/statm: 2406243 436973 2269 9 0 2396937 0
[pid=9172/tid=9173] ppid=9170 vsize=9624972 CPUtime=43.37 cores=0,2,4,6
/proc/9172/task/9173/stat : 9173 (java) T 9170 9172 9172 0 -1 4202560 3812 0 1 0 4310 27 0 0 20 0 19 0 338562003 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581486967920 140581343511255 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9174] ppid=9170 vsize=9624972 CPUtime=0.13 cores=0,2,4,6
/proc/9172/task/9174/stat : 9174 (java) T 9170 9172 9172 0 -1 4202560 102 0 0 0 11 2 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581465070120 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9175] ppid=9170 vsize=9624972 CPUtime=0.09 cores=0,2,4,6
/proc/9172/task/9175/stat : 9175 (java) T 9170 9172 9172 0 -1 4202560 47 0 0 0 7 2 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581464017320 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9176] ppid=9170 vsize=9624972 CPUtime=0.07 cores=0,2,4,6
/proc/9172/task/9176/stat : 9176 (java) T 9170 9172 9172 0 -1 4202560 86 0 0 0 7 0 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581462964520 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=9172/tid=9177] ppid=9170 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/9172/task/9177/stat : 9177 (java) T 9170 9172 9172 0 -1 4202560 151 0 0 0 10 2 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581461911720 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9178] ppid=9170 vsize=9624972 CPUtime=0.07 cores=0,2,4,6
/proc/9172/task/9178/stat : 9178 (java) T 9170 9172 9172 0 -1 4202560 39 0 0 0 6 1 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581124697128 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=9172/tid=9179] ppid=9170 vsize=9624972 CPUtime=0.09 cores=0,2,4,6
/proc/9172/task/9179/stat : 9179 (java) T 9170 9172 9172 0 -1 4202560 152 0 0 0 7 2 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581123644328 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9180] ppid=9170 vsize=9624972 CPUtime=0.12 cores=0,2,4,6
/proc/9172/task/9180/stat : 9180 (java) T 9170 9172 9172 0 -1 4202560 155 0 0 0 10 2 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581122591528 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9181] ppid=9170 vsize=9624972 CPUtime=0.06 cores=0,2,4,6
/proc/9172/task/9181/stat : 9181 (java) T 9170 9172 9172 0 -1 4202560 28 0 0 0 6 0 0 0 20 0 19 0 338562005 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581121538728 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9182] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9182/stat : 9182 (java) T 9170 9172 9172 0 -1 4202560 1220 0 0 0 0 0 0 0 20 0 19 0 338562006 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581109340648 217413892009 0 0 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=9172/tid=9183] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9183/stat : 9183 (java) T 9170 9172 9172 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338562006 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581108286712 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9184] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9184/stat : 9184 (java) T 9170 9172 9172 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338562006 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581107233800 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9172/tid=9185] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9185/stat : 9185 (java) T 9170 9172 9172 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581106182288 217413900064 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=9172/tid=9186] ppid=9170 vsize=9624972 CPUtime=0.92 cores=0,2,4,6
/proc/9172/task/9186/stat : 9186 (java) T 9170 9172 9172 0 -1 4202560 7024 0 0 0 92 0 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581105130424 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=9172/tid=9187] ppid=9170 vsize=9624972 CPUtime=1.03 cores=0,2,4,6
/proc/9172/task/9187/stat : 9187 (java) T 9170 9172 9172 0 -1 4202560 8425 0 0 0 101 2 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581104077624 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=9172/tid=9188] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9188/stat : 9188 (java) T 9170 9172 9172 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581103025016 217413891132 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=9172/tid=9189] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9189/stat : 9189 (java) T 9170 9172 9172 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 19 0 338562008 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581101972424 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=9172/tid=9190] ppid=9170 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/9172/task/9190/stat : 9190 (java) T 9170 9172 9172 0 -1 4202560 12 0 0 0 0 0 0 0 20 0 19 0 338562059 9855971328 436973 33554432000 1073741824 1073778376 140735859968272 140581100917928 217413892009 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=9170/tid=9192] ppid=9087 vsize=515464 CPUtime=0 cores=0,2,4,6
/proc/9170/task/9192/stat : 9192 (python) S 9087 9087 3958 0 -1 4202560 50 0 0 0 0 0 0 0 20 0 2 0 338566368 527835136 7353 33554432000 4194304 5683644 140734291953520 140645448503856 217413900064 0 0 16781312 33554946 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9214] ppid=9087 vsize=591256 CPUtime=0 cores=0,2,4,6
/proc/9214/stat : 9214 (python) S 9087 9087 3958 0 -1 4202560 857 0 0 0 0 0 0 0 20 0 1 0 338570886 605446144 7310 33554432000 4194304 5683644 140734291953520 140734291944608 217410535907 0 0 16781312 33554946 18446744071580425449 0 0 17 6 0 0 0 0 0
/proc/9214/statm: 147814 7310 335 364 0 83025 0
[pid=9216] ppid=9214 vsize=9558408 CPUtime=0.65 cores=0,2,4,6
/proc/9216/stat : 9216 (java) S 9214 9216 9216 0 -1 4202496 7536 0 1 0 63 2 0 0 20 0 18 0 338570886 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 140735057590272 217413877821 0 0 0 16800975 18446744073709551615 0 0 17 4 0 0 0 0 0
/proc/9216/statm: 2389602 14921 2216 9 0 2380296 0
[pid=9216/tid=9217] ppid=9214 vsize=9558408 CPUtime=0.34 cores=0,2,4,6
/proc/9216/task/9217/stat : 9217 (java) R 9214 9216 9216 0 -1 4202560 2434 0 1 0 32 2 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139696118861368 139695975672917 0 4 0 16800975 0 0 0 -1 0 0 0 0 0 0
[pid=9216/tid=9218] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9218/stat : 9218 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139696096963624 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9219] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9219/stat : 9219 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139696095910824 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9216/tid=9220] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9220/stat : 9220 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139696094858024 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=9216/tid=9221] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9221/stat : 9221 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695757453992 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9216/tid=9222] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9222/stat : 9222 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695756401192 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9223] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9223/stat : 9223 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695755348392 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9216/tid=9224] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9224/stat : 9224 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695754295592 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9225] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9225/stat : 9225 (java) S 9214 9216 9216 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 18 0 338570887 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695753242792 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9226] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9226/stat : 9226 (java) S 9214 9216 9216 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 18 0 338570889 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695741045736 217413892009 0 0 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9227] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9227/stat : 9227 (java) S 9214 9216 9216 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 18 0 338570889 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695739991800 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9228] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9228/stat : 9228 (java) S 9214 9216 9216 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 18 0 338570889 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695738938888 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9229] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9229/stat : 9229 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570891 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695737887376 217413900064 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9230] ppid=9214 vsize=9558408 CPUtime=0.15 cores=0,2,4,6
/proc/9216/task/9230/stat : 9230 (java) S 9214 9216 9216 0 -1 4202560 1482 0 0 0 15 0 0 0 20 0 18 0 338570891 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695736834488 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9216/tid=9231] ppid=9214 vsize=9558408 CPUtime=0.13 cores=0,2,4,6
/proc/9216/task/9231/stat : 9231 (java) S 9214 9216 9216 0 -1 4202560 2204 0 0 0 13 0 0 0 20 0 18 0 338570891 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695735781688 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9216/tid=9232] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9232/stat : 9232 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570891 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695734729080 217413891132 0 4 0 16800975 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9216/tid=9233] ppid=9214 vsize=9558408 CPUtime=0 cores=0,2,4,6
/proc/9216/task/9233/stat : 9233 (java) S 9214 9216 9216 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 18 0 338570891 9787809792 14921 33554432000 1073741824 1073778376 140735057599136 139695733676488 217413892009 0 4 0 16800975 18446744071579510697 0 0 -1 2 0 0 0 0 0
[pid=9087/tid=9094] ppid=9085 vsize=667036 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9094/stat : 9094 (python) S 9085 9087 3958 0 -1 4202560 31 1011 0 0 0 0 0 0 20 0 6 0 338542816 683044864 8892 33554432000 4194304 5683644 140734291953520 140645469483568 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 6 0 0 0 0 0
[pid=9087/tid=9104] ppid=9085 vsize=667036 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9104/stat : 9104 (python) S 9085 9087 3958 0 -1 4202560 27 1011 0 0 0 0 0 0 20 0 6 0 338547890 683044864 8892 33554432000 4194304 5683644 140734291953520 140645458993712 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9087/tid=9158] ppid=9085 vsize=667036 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9158/stat : 9158 (python) S 9085 9087 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 6 0 338552606 683044864 8892 33554432000 4194304 5683644 140734291953520 140645448503856 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=9087/tid=9171] ppid=9085 vsize=667036 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9171/stat : 9171 (python) S 9085 9087 3958 0 -1 4202560 21 1011 0 0 0 0 0 0 20 0 6 0 338562002 683044864 8892 33554432000 4194304 5683644 140734291953520 140645438014000 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=9087/tid=9215] ppid=9085 vsize=667036 CPUtime=0 cores=0,2,4,6
/proc/9087/task/9215/stat : 9215 (python) S 9085 9087 3958 0 -1 4202560 22 1011 0 0 0 0 0 0 20 0 6 0 338570886 683044864 8892 33554432000 4194304 5683644 140734291953520 140645427524144 217413900064 0 0 16781312 33554434 18446744071579510697 0 0 -1 2 0 0 0 0 0
Current children cumulated CPU time (s) 285.14
Current children cumulated vsize (KiB) 23500640

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 9087 and gives
#  childrusage.ru_utime.tv_sec=0
#  childrusage.ru_utime.tv_usec=938857
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=110983
# CPU time returned by wait4() is 1.04984
# while last known CPU time is 285.14
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 282.862
CPU time (s): 285.14
CPU user time (s): 283.24
CPU system time (s): 1.9
CPU usage (%): 100.805
Max. virtual memory (cumulated for all children) (KiB): 23500640

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 0.938857
system time used= 0.110983
maximum resident set size= 36060
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 17884
page faults= 1
swaps= 0
block input operations= 0
block output operations= 448
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2079
involuntary context switches= 12

runsolver used 0.881865 second user time and 2.46462 second system time

The end

Launcher Data 

Begin job on node122 at 2012-06-03 13:49:25
IDJOB=3742226
IDBENCH=90142
IDSOLVER=2327
FILE ID=node122/3742226-1338724165
RUNJOBID= node122-1338713392-3974
PBS_JOBID= 14636732
Free space on /tmp= 71084 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB12/normalized-PB12/DEC-SMALLINT-LIN/sroussel/ShortestPathNG/normalized-NG_K128.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3742226-1338724165/watcher-3742226-1338724165 -o /tmp/evaluation-result-3742226-1338724165/solver-3742226-1338724165 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3742226-1338724165.opb -seed 1927519361 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 17c4ca9a6493665b57c1ccd2ab23e03d
RANDOM SEED=1927519361

node122.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5333.06
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5438.54
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.533
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        29240492 kB
Buffers:          206292 kB
Cached:           375860 kB
SwapCached:         1804 kB
Active:          2754308 kB
Inactive:          39432 kB
Active(anon):    2207444 kB
Inactive(anon):     7100 kB
Active(file):     546864 kB
Inactive(file):    32332 kB
Unevictable:        7628 kB
Mlocked:            7628 kB
SwapTotal:      67108856 kB
SwapFree:       67100164 kB
Dirty:            102764 kB
Writeback:             0 kB
AnonPages:       2219864 kB
Mapped:            32544 kB
Shmem:                84 kB
Slab:             689280 kB
SReclaimable:     131572 kB
SUnreclaim:       557708 kB
KernelStack:        1800 kB
PageTables:        12664 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    3009692 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1669120 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 70984 MiB
End job on node122 at 2012-06-03 13:54:11