Trace number 3741007

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 420.57 536.369

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-76.opb
MD5SUMf030651c159d27eae553d070b0927366
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark1408.44
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables12587
Total number of constraints283937
Number of constraints which are clauses255125
Number of constraints which are cardinality constraints (but not clauses)903
Number of constraints which are nor clauses,nor cardinality constraints27909
Minimum length of a constraint2
Maximum length of a constraint270
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 271
Number of bits of the biggest sum of numbers9
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.42/1.25	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.42/1.25	c INFO: solving HOME/instance-3741007-1338720776.opb
2.41/3.28	c INFO: parsing took 1.97 s
5.90/6.77	c INFO: features took 3.48 CPU seconds
6.00/6.80	c INFO: running wbo1.4b-fixed@0 for 45 with 1794 remaining (b = 0.96)
55.74/56.59	c INFO: running pbct-0.1.2-linear@0 for 45 with 1745 remaining (b = 0.96)
103.20/103.90	c INFO: running wbo1.4a@0 for 45 with 1698 remaining (b = 0.95)
153.17/153.81	c INFO: running scip-2.0.1-clp@0 for 45 with 1649 remaining (b = 0.83)
200.92/201.51	c INFO: running scip-2.0.1-spx@0 for 45 with 1602 remaining (b = 0.68)
266.80/267.33	c INFO: wrote minion input file to HOME/tmp4aq3oQ.minion
266.80/267.33	c INFO: running minion-0.12@0 for 45 with 1554 remaining (b = 0.55)
276.11/392.08	c INFO: running bsolo_pb10-l2@0 for 45 with 1531 remaining (b = 0.29)
324.97/440.88	c INFO: running clasp-1.3.7@0 for 45 with 1483 remaining (b = 0.18)
372.23/488.05	c INFO: running clasp-1.3.7@47 for 45 with 1436 remaining (b = 0.02)
420.52/536.21	c INFO: running sat4j-pb-v20101225-cutting@0 for 45 with 1388 remaining (b = 0.05)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3741007-1338720776/watcher-3741007-1338720776 -o /tmp/evaluation-result-3741007-1338720776/solver-3741007-1338720776 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741007-1338720776.opb -seed 2037121049 -budget 1800 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.33 2.88 2.56 2/204 5615
/proc/meminfo: memFree=25109368/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=15276 CPUtime=0 cores=1,3,5,7
/proc/5615/stat : 5615 (python) D 5613 5615 3621 0 -1 4202496 672 0 0 0 0 0 0 0 20 0 1 0 338202651 15642624 559 33554432000 4194304 5683644 140737190967600 140737190935544 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 3819 559 324 364 0 261 0

[startup+0.0691049 s]
/proc/loadavg: 2.33 2.88 2.56 2/204 5615
/proc/meminfo: memFree=25109368/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=16596 CPUtime=0.01 cores=1,3,5,7
/proc/5615/stat : 5615 (python) D 5613 5615 3621 0 -1 4202496 1205 0 0 0 1 0 0 0 20 0 1 0 338202651 16994304 1002 33554432000 4194304 5683644 140737190967600 140737190935880 213419639696 0 0 16781312 2 18446744072100609316 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 4149 1002 422 364 0 591 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16596

[startup+0.100279 s]
/proc/loadavg: 2.33 2.88 2.56 2/204 5615
/proc/meminfo: memFree=25109368/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=19088 CPUtime=0.02 cores=1,3,5,7
/proc/5615/stat : 5615 (python) R 5613 5615 3621 0 -1 4202496 1395 0 0 0 2 0 0 0 20 0 1 0 338202651 19546112 1149 33554432000 4194304 5683644 140737190967600 140737190898104 4509116 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 4772 1149 430 364 0 696 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19088

[startup+0.300215 s]
/proc/loadavg: 2.33 2.88 2.56 2/204 5615
/proc/meminfo: memFree=25109368/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=96916 CPUtime=0.11 cores=1,3,5,7
/proc/5615/stat : 5615 (python) R 5613 5615 3621 0 -1 4202496 3960 1008 1 0 9 2 0 0 20 0 1 0 338202651 99241984 2652 33554432000 4194304 5683644 140737190967600 140737190853928 213419638453 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 24229 2652 823 364 0 1720 0
Current children cumulated CPU time (s) 0.11
Current children cumulated vsize (KiB) 96916

[startup+0.700149 s]
/proc/loadavg: 2.33 2.88 2.56 2/204 5615
/proc/meminfo: memFree=25109368/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=136268 CPUtime=0.24 cores=1,3,5,7
/proc/5615/stat : 5615 (python) R 5613 5615 3621 0 -1 4202496 6235 1008 1 0 16 8 0 0 20 0 1 0 338202651 139538432 4073 33554432000 4194304 5683644 140737190967600 140737190873320 213410475818 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 34587 4077 1095 364 0 2953 0
Current children cumulated CPU time (s) 0.24
Current children cumulated vsize (KiB) 136268

[startup+1.50022 s]
/proc/loadavg: 2.33 2.88 2.56 3/205 5620
/proc/meminfo: memFree=25092968/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=300692 CPUtime=0.71 cores=1,3,5,7
/proc/5615/stat : 5615 (python) R 5613 5615 3621 0 -1 4202496 12263 1008 1 0 56 15 0 0 20 0 1 0 338202651 307908608 11850 33554432000 4194304 5683644 140737190967600 140737190960840 213419265940 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 75173 11850 1811 364 0 10402 0
Current children cumulated CPU time (s) 0.71
Current children cumulated vsize (KiB) 300692

[startup+3.10028 s]
/proc/loadavg: 2.33 2.88 2.56 3/205 5620
/proc/meminfo: memFree=25053092/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=362096 CPUtime=2.31 cores=1,3,5,7
/proc/5615/stat : 5615 (python) R 5613 5615 3621 0 -1 4202496 25208 1008 1 0 213 18 0 0 20 0 1 0 338202651 370786304 26328 33554432000 4194304 5683644 140737190967600 140737190960216 139827188083397 0 0 16781312 2 0 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 90524 26329 1811 364 0 25753 0
Current children cumulated CPU time (s) 2.31
Current children cumulated vsize (KiB) 362096
heavy processes:

[startup+6.30018 s]
/proc/loadavg: 2.30 2.86 2.55 3/205 5620
/proc/meminfo: memFree=25013900/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=359848 CPUtime=5.5 cores=1,3,5,7
/proc/5615/stat : 5615 (python) R 5613 5615 3621 0 -1 4202496 30644 1008 1 0 531 19 0 0 20 0 1 0 338202651 368484352 26714 33554432000 4194304 5683644 140737190967600 140737190960776 139827188203803 0 0 16781312 33554434 0 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 89962 26714 1847 364 0 25191 0
Current children cumulated CPU time (s) 5.5
Current children cumulated vsize (KiB) 359848

[startup+12.7004 s]
/proc/loadavg: 2.28 2.85 2.55 3/228 5643
/proc/meminfo: memFree=24831028/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=440088 CPUtime=6 cores=1,3,5,7
/proc/5615/stat : 5615 (python) S 5613 5615 3621 0 -1 4202496 32041 1008 1 0 580 20 0 0 20 0 2 0 338202651 450650112 27884 33554432000 4194304 5683644 140737190967600 140737190962432 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 110022 27884 1902 364 0 45245 0
[pid=5621] ppid=5615 vsize=364308 CPUtime=0 cores=1,3,5,7
/proc/5621/stat : 5621 (python) S 5615 5615 3621 0 -1 4202560 884 0 0 0 0 0 0 0 20 0 1 0 338203331 373051392 26339 33554432000 4194304 5683644 140737190967600 140737190958688 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/5621/statm: 91077 26339 363 364 0 26300 0
[pid=5623] ppid=5621 vsize=36928 CPUtime=5.88 cores=1,3,5,7
/proc/5623/stat : 5623 (wbo1.4b-fixed) R 5621 5623 5623 0 -1 4202496 12217 0 0 0 562 26 0 0 20 0 1 0 338203332 37814272 8508 33554432000 134512640 135752315 4290989120 4290987196 135270849 0 0 16781312 3 0 0 0 17 1 0 0 0 0 0
/proc/5623/statm: 9232 8508 146 303 0 8927 0
[pid=5615/tid=5622] ppid=5613 vsize=440088 CPUtime=0 cores=1,3,5,7
/proc/5615/task/5622/stat : 5622 (python) S 5613 5615 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338203332 450650112 27884 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 11.88
Current children cumulated vsize (KiB) 841324

[startup+25.5004 s]
/proc/loadavg: 2.22 2.81 2.54 3/180 5688
/proc/meminfo: memFree=26280760/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=440088 CPUtime=6 cores=1,3,5,7
/proc/5615/stat : 5615 (python) S 5613 5615 3621 0 -1 4202496 32041 1008 1 0 580 20 0 0 20 0 2 0 338202651 450650112 27884 33554432000 4194304 5683644 140737190967600 140737190962432 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 110022 27884 1902 364 0 45245 0
[pid=5621] ppid=5615 vsize=364308 CPUtime=0.02 cores=1,3,5,7
/proc/5621/stat : 5621 (python) S 5615 5615 3621 0 -1 4202560 895 0 0 0 1 1 0 0 20 0 1 0 338203331 373051392 26339 33554432000 4194304 5683644 140737190967600 140737190958688 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/5621/statm: 91077 26339 363 364 0 26300 0
[pid=5623] ppid=5621 vsize=45648 CPUtime=18.68 cores=1,3,5,7
/proc/5623/stat : 5623 (wbo1.4b-fixed) R 5621 5623 5623 0 -1 4202496 14370 0 0 0 1823 45 0 0 20 0 1 0 338203332 46743552 10661 33554432000 134512640 135752315 4290989120 4290987120 134611380 0 0 16781312 3 0 0 0 17 5 0 0 0 0 0
/proc/5623/statm: 11412 10661 146 303 0 11107 0
[pid=5615/tid=5622] ppid=5613 vsize=440088 CPUtime=0 cores=1,3,5,7
/proc/5615/task/5622/stat : 5622 (python) S 5613 5615 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338203332 450650112 27884 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 24.7
Current children cumulated vsize (KiB) 850044

[startup+51.1005 s]
/proc/loadavg: 2.14 2.74 2.52 3/180 5688
/proc/meminfo: memFree=25901856/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=440088 CPUtime=6 cores=1,3,5,7
/proc/5615/stat : 5615 (python) S 5613 5615 3621 0 -1 4202496 32041 1008 1 0 580 20 0 0 20 0 2 0 338202651 450650112 27884 33554432000 4194304 5683644 140737190967600 140737190962432 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 110022 27884 1902 364 0 45245 0
[pid=5621] ppid=5615 vsize=364308 CPUtime=0.06 cores=1,3,5,7
/proc/5621/stat : 5621 (python) S 5615 5615 3621 0 -1 4202560 1005 0 0 0 4 2 0 0 20 0 1 0 338203331 373051392 26344 33554432000 4194304 5683644 140737190967600 140737190958688 213419655592 0 0 16781312 33554946 18446744071580425449 0 0 17 1 0 0 0 0 0
/proc/5621/statm: 91077 26344 368 364 0 26300 0
[pid=5623] ppid=5621 vsize=58024 CPUtime=44.27 cores=1,3,5,7
/proc/5623/stat : 5623 (wbo1.4b-fixed) R 5621 5623 5623 0 -1 4202496 17435 0 0 0 4359 68 0 0 20 0 1 0 338203332 59416576 13726 33554432000 134512640 135752315 4290989120 4290987196 135270849 0 0 16781312 3 0 0 0 17 5 0 0 0 0 0
/proc/5623/statm: 14506 13726 146 303 0 14201 0
[pid=5615/tid=5622] ppid=5613 vsize=440088 CPUtime=0 cores=1,3,5,7
/proc/5615/task/5622/stat : 5622 (python) S 5613 5615 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 2 0 338203332 450650112 27884 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 50.33
Current children cumulated vsize (KiB) 862420

[startup+102.307 s]
/proc/loadavg: 2.06 2.62 2.50 3/187 5695
/proc/meminfo: memFree=24685260/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=515880 CPUtime=6.02 cores=1,3,5,7
/proc/5615/stat : 5615 (python) S 5613 5615 3621 0 -1 4202496 32971 1008 1 0 582 20 0 0 20 0 3 0 338202651 528261120 27893 33554432000 4194304 5683644 140737190967600 140737190962432 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 128970 27893 1905 364 0 64190 0
[pid=5621] ppid=5615 vsize=440088 CPUtime=0.08 cores=1,3,5,7
/proc/5621/stat : 5621 (python) S 5615 5615 3621 0 -1 4202560 1138 0 0 0 5 3 0 0 20 0 2 0 338203331 450650112 26379 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5621/statm: 110022 26379 396 364 0 45245 0
[pid=5623] ppid=5621 vsize=60032 CPUtime=49.74 cores=1,3,5,7
/proc/5623/stat : 5623 (wbo1.4b-fixed) T 5621 5623 5623 0 -1 4202496 17886 0 0 0 4905 69 0 0 20 0 1 0 338203332 61472768 14177 33554432000 134512640 135752315 4290989120 4290987120 134611201 0 0 16781312 3 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/5623/statm: 15008 14177 146 303 0 14703 0
[pid=5621/tid=5689] ppid=5615 vsize=440088 CPUtime=0 cores=1,3,5,7
/proc/5621/task/5689/stat : 5689 (python) S 5615 5615 3621 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338208308 450650112 26379 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5690] ppid=5615 vsize=440100 CPUtime=0.06 cores=1,3,5,7
/proc/5690/stat : 5690 (python) S 5615 5615 3621 0 -1 4202560 915 0 0 0 3 3 0 0 20 0 1 0 338208311 450662400 26340 33554432000 4194304 5683644 140737190967600 140737190958688 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 3 0 0 0 0 0
/proc/5690/statm: 110025 26340 358 364 0 45245 0
[pid=5692] ppid=5690 vsize=1137720 CPUtime=45.69 cores=1,3,5,7
/proc/5692/stat : 5692 (pbct-0.1.2-linu) R 5690 5692 5692 0 -1 4202496 267573 0 0 0 4500 69 0 0 20 0 1 0 338208311 1165025280 272648 33554432000 134512640 136422188 4290402768 4290400492 135874785 0 0 16781312 16387 0 0 0 17 1 0 0 0 0 0
/proc/5692/statm: 284430 272648 215 467 0 283961 0
[pid=5615/tid=5622] ppid=5613 vsize=515880 CPUtime=0 cores=1,3,5,7
/proc/5615/task/5622/stat : 5622 (python) S 5613 5615 3621 0 -1 4202560 27 1008 0 0 0 0 0 0 20 0 3 0 338203332 528261120 27893 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5691] ppid=5613 vsize=515880 CPUtime=0 cores=1,3,5,7
/proc/5615/task/5691/stat : 5691 (python) S 5613 5615 3621 0 -1 4202560 22 1008 0 0 0 0 0 0 20 0 3 0 338208311 528261120 27893 33554432000 4194304 5683644 140737190967600 139827004297776 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 101.59
Current children cumulated vsize (KiB) 2593820

[startup+162.301 s]
/proc/loadavg: 2.02 2.51 2.46 3/198 5707
/proc/meminfo: memFree=20158404/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=667464 CPUtime=6.08 cores=1,3,5,7
/proc/5615/stat : 5615 (python) S 5613 5615 3621 0 -1 4202496 34758 1008 1 0 587 21 0 0 20 0 5 0 338202651 683483136 27909 33554432000 4194304 5683644 140737190967600 140737190962432 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 166866 27909 1909 364 0 102080 0
[pid=5621] ppid=5615 vsize=440088 CPUtime=0.08 cores=1,3,5,7
/proc/5621/stat : 5621 (python) S 5615 5615 3621 0 -1 4202560 1138 0 0 0 5 3 0 0 20 0 2 0 338203331 450650112 26379 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5621/statm: 110022 26379 396 364 0 45245 0
[pid=5623] ppid=5621 vsize=60032 CPUtime=49.74 cores=1,3,5,7
/proc/5623/stat : 5623 (wbo1.4b-fixed) T 5621 5623 5623 0 -1 4202496 17886 0 0 0 4905 69 0 0 20 0 1 0 338203332 61472768 14180 33554432000 134512640 135752315 4290989120 4290987120 134611201 0 0 16781312 3 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/5623/statm: 15008 14180 146 303 0 14703 0
[pid=5621/tid=5689] ppid=5615 vsize=440088 CPUtime=0 cores=1,3,5,7
/proc/5621/task/5689/stat : 5689 (python) S 5615 5615 3621 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338208308 450650112 26379 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5690] ppid=5615 vsize=440100 CPUtime=0.08 cores=1,3,5,7
/proc/5690/stat : 5690 (python) S 5615 5615 3621 0 -1 4202560 1068 0 0 0 4 4 0 0 20 0 2 0 338208311 450662400 26366 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5690/statm: 110025 26366 383 364 0 45245 0
[pid=5692] ppid=5690 vsize=1137984 CPUtime=47.26 cores=1,3,5,7
/proc/5692/stat : 5692 (pbct-0.1.2-linu) T 5690 5692 5692 0 -1 4202496 267611 0 0 0 4656 70 0 0 20 0 1 0 338208311 1165295616 272686 33554432000 134512640 136422188 4290402768 4290400384 134868319 0 0 16781312 16387 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5692/statm: 284496 272686 215 467 0 284027 0
[pid=5690/tid=5696] ppid=5615 vsize=440100 CPUtime=0 cores=1,3,5,7
/proc/5690/task/5696/stat : 5696 (python) S 5615 5615 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338213039 450662400 26366 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5697] ppid=5615 vsize=515892 CPUtime=0.08 cores=1,3,5,7
/proc/5697/stat : 5697 (python) S 5615 5615 3621 0 -1 4202560 1092 0 0 0 5 3 0 0 20 0 2 0 338213042 528273408 26372 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5697/statm: 128973 26372 383 364 0 64190 0
[pid=5699] ppid=5697 vsize=55584 CPUtime=49.86 cores=1,3,5,7
/proc/5699/stat : 5699 (wbo1.4a) T 5697 5699 5699 0 -1 4202496 16755 0 0 0 4921 65 0 0 20 0 1 0 338213042 56918016 13045 33554432000 134512640 135751879 4293619456 4293617632 134616336 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5699/statm: 13896 13045 143 303 0 13591 0
[pid=5697/tid=5704] ppid=5615 vsize=515892 CPUtime=0 cores=1,3,5,7
/proc/5697/task/5704/stat : 5704 (python) S 5615 5615 3621 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338218030 528273408 26372 33554432000 4194304 5683644 140737190967600 139827004297776 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0

################
# More data... #
################

/proc/5723/statm: 204765 26396 383 364 0 139970 0
[pid=5725] ppid=5723 vsize=87268 CPUtime=48.75 cores=1,3,5,7
/proc/5725/stat : 5725 (bsolo_pb10) T 5723 5725 5725 0 -1 4202496 26987 0 0 0 4819 56 0 0 20 0 1 0 338241860 89362432 19362 33554432000 134512640 137138936 4289593552 4289592480 134542214 0 0 16781312 16384 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/5725/statm: 21817 19362 192 642 0 21173 0
[pid=5723/tid=5732] ppid=5615 vsize=819060 CPUtime=0 cores=1,3,5,7
/proc/5723/task/5732/stat : 5732 (python) S 5615 5615 3621 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338246737 838717440 26396 33554432000 4194304 5683644 140737190967600 139826944287280 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5733] ppid=5615 vsize=894852 CPUtime=0.14 cores=1,3,5,7
/proc/5733/stat : 5733 (python) S 5615 5615 3621 0 -1 4202560 1239 0 0 0 8 6 0 0 20 0 2 0 338246739 916328448 26408 33554432000 4194304 5683644 140737190967600 140737190958688 213419655651 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5733/statm: 223713 26408 389 364 0 158915 0
[pid=5735] ppid=5733 vsize=58380 CPUtime=81.36 cores=1,3,5,7
/proc/5735/stat : 5735 (clasp-1.3.7-x86) R 5733 5735 5735 0 -1 4202496 15980 0 0 0 8131 5 0 0 20 0 1 0 338246740 59781120 14012 33554432000 134512640 136139019 4290097072 4290096044 134622001 0 0 16781312 16386 0 0 0 17 1 0 0 0 0 0
/proc/5735/statm: 14595 14012 183 398 0 14195 0
[pid=5733/tid=5763] ppid=5615 vsize=894852 CPUtime=0 cores=1,3,5,7
/proc/5733/task/5763/stat : 5763 (python) S 5615 5615 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338251454 916328448 26408 33554432000 4194304 5683644 140737190967600 139826933797424 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5615/tid=5622] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5622/stat : 5622 (python) S 5613 5615 3621 0 -1 4202560 27 1039946 0 0 0 0 548 387 20 0 9 0 338203332 993927168 27941 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5691] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5691/stat : 5691 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 9 0 338208311 993927168 27941 33554432000 4194304 5683644 140737190967600 139827004297776 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5698] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5698/stat : 5698 (python) S 5613 5615 3621 0 -1 4202560 21 1039946 0 0 0 0 548 387 20 0 9 0 338213042 993927168 27941 33554432000 4194304 5683644 140737190967600 139826993807920 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5706] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5706/stat : 5706 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 9 0 338218033 993927168 27941 33554432000 4194304 5683644 140737190967600 139826983318064 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5713] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5713/stat : 5713 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 9 0 338222803 993927168 27941 33554432000 4194304 5683644 140737190967600 139826972828208 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5718] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5718/stat : 5718 (python) S 5613 5615 3621 0 -1 4202560 21 1039946 0 0 0 0 548 387 20 0 9 0 338229385 993927168 27941 33554432000 4194304 5683644 140737190967600 139826944287280 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5724] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5724/stat : 5724 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 9 0 338241860 993927168 27941 33554432000 4194304 5683644 140737190967600 139826933797424 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5734] ppid=5613 vsize=970632 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5734/stat : 5734 (python) S 5613 5615 3621 0 -1 4202560 21 1039946 0 0 0 0 548 387 20 0 9 0 338246740 993927168 27941 33554432000 4194304 5683644 140737190967600 139826923307568 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
Current children cumulated CPU time (s) 406.59
Current children cumulated vsize (KiB) 8417864
heavy processes:



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+536.301 s]
/proc/loadavg: 2.23 3.08 2.84 3/238 5782
/proc/meminfo: memFree=12352676/32873844 swapFree=6608/6608
[pid=5615] ppid=5613 vsize=1046424 CPUtime=32.92 cores=1,3,5,7
/proc/5615/stat : 5615 (python) S 5613 5615 3621 0 -1 4202496 39464 1039946 1 0 2330 27 548 387 20 0 10 0 338202651 1071538176 27949 33554432000 4194304 5683644 140737190967600 140737190962432 213423023373 0 0 16781312 33554434 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5615/statm: 261606 27949 1919 364 0 196805 0
[pid=5621] ppid=5615 vsize=440088 CPUtime=0.08 cores=1,3,5,7
/proc/5621/stat : 5621 (python) S 5615 5615 3621 0 -1 4202560 1138 0 0 0 5 3 0 0 20 0 2 0 338203331 450650112 26379 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5621/statm: 110022 26379 396 364 0 45245 0
[pid=5623] ppid=5621 vsize=60032 CPUtime=49.74 cores=1,3,5,7
/proc/5623/stat : 5623 (wbo1.4b-fixed) T 5621 5623 5623 0 -1 4202496 17886 0 0 0 4905 69 0 0 20 0 1 0 338203332 61472768 14254 33554432000 134512640 135752315 4290989120 4290987120 134611201 0 0 16781312 3 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/5623/statm: 15008 14254 146 303 0 14703 0
[pid=5621/tid=5689] ppid=5615 vsize=440088 CPUtime=0 cores=1,3,5,7
/proc/5621/task/5689/stat : 5689 (python) S 5615 5615 3621 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338208308 450650112 26379 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5690] ppid=5615 vsize=440100 CPUtime=0.08 cores=1,3,5,7
/proc/5690/stat : 5690 (python) S 5615 5615 3621 0 -1 4202560 1068 0 0 0 4 4 0 0 20 0 2 0 338208311 450662400 26366 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5690/statm: 110025 26366 383 364 0 45245 0
[pid=5692] ppid=5690 vsize=1137984 CPUtime=47.26 cores=1,3,5,7
/proc/5692/stat : 5692 (pbct-0.1.2-linu) T 5690 5692 5692 0 -1 4202496 267611 0 0 0 4656 70 0 0 20 0 1 0 338208311 1165295616 272686 33554432000 134512640 136422188 4290402768 4290400384 134868319 0 0 16781312 16387 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5692/statm: 284496 272686 215 467 0 284027 0
[pid=5690/tid=5696] ppid=5615 vsize=440100 CPUtime=0 cores=1,3,5,7
/proc/5690/task/5696/stat : 5696 (python) S 5615 5615 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338213039 450662400 26366 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5697] ppid=5615 vsize=515892 CPUtime=0.08 cores=1,3,5,7
/proc/5697/stat : 5697 (python) S 5615 5615 3621 0 -1 4202560 1092 0 0 0 5 3 0 0 20 0 2 0 338213042 528273408 26372 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5697/statm: 128973 26372 383 364 0 64190 0
[pid=5699] ppid=5697 vsize=55584 CPUtime=49.86 cores=1,3,5,7
/proc/5699/stat : 5699 (wbo1.4a) T 5697 5699 5699 0 -1 4202496 16755 0 0 0 4921 65 0 0 20 0 1 0 338213042 56918016 13045 33554432000 134512640 135751879 4293619456 4293617632 134616336 0 0 16781312 3 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5699/statm: 13896 13045 143 303 0 13591 0
[pid=5697/tid=5704] ppid=5615 vsize=515892 CPUtime=0 cores=1,3,5,7
/proc/5697/task/5704/stat : 5704 (python) S 5615 5615 3621 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338218030 528273408 26372 33554432000 4194304 5683644 140737190967600 139827004297776 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5705] ppid=5615 vsize=591684 CPUtime=0.08 cores=1,3,5,7
/proc/5705/stat : 5705 (python) S 5615 5615 3621 0 -1 4202560 1086 0 0 0 4 4 0 0 20 0 2 0 338218032 605884416 26378 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5705/statm: 147921 26378 383 364 0 83135 0
[pid=5707] ppid=5705 vsize=827028 CPUtime=47.65 cores=1,3,5,7
/proc/5707/stat : 5707 (scip-2.0.1.linu) T 5705 5707 5707 0 -1 4202496 153010 0 0 0 4711 54 0 0 20 0 1 0 338218033 846876672 193514 33554432000 4194304 15243407 140736966312304 140736966307016 8088801 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5707/statm: 206757 193514 896 2698 0 204052 0
[pid=5705/tid=5711] ppid=5615 vsize=591684 CPUtime=0 cores=1,3,5,7
/proc/5705/task/5711/stat : 5711 (python) S 5615 5615 3621 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338222800 605884416 26378 33554432000 4194304 5683644 140737190967600 139826993807920 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5712] ppid=5615 vsize=667476 CPUtime=0.08 cores=1,3,5,7
/proc/5712/stat : 5712 (python) S 5615 5615 3621 0 -1 4202560 1126 0 0 0 4 4 0 0 20 0 2 0 338222802 683495424 26384 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5712/statm: 166869 26384 383 364 0 102080 0
[pid=5714] ppid=5712 vsize=851804 CPUtime=48.46 cores=1,3,5,7
/proc/5714/stat : 5714 (scip-2.0.1.linu) T 5712 5714 5714 0 -1 4202496 148442 0 0 0 4791 55 0 0 20 0 1 0 338222803 872247296 204762 33554432000 4194304 13684479 140733989871520 140733989863384 7358124 0 0 16781312 2 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5714/statm: 212951 204762 770 2317 0 210629 0
[pid=5712/tid=5715] ppid=5615 vsize=667476 CPUtime=0 cores=1,3,5,7
/proc/5712/task/5715/stat : 5715 (python) S 5615 5615 3621 0 -1 4202560 47 0 0 0 0 0 0 0 20 0 2 0 338227651 683495424 26384 33554432000 4194304 5683644 140737190967600 139826983318064 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5723] ppid=5615 vsize=819060 CPUtime=0.08 cores=1,3,5,7
/proc/5723/stat : 5723 (python) S 5615 5615 3621 0 -1 4202560 1141 0 0 0 4 4 0 0 20 0 2 0 338241859 838717440 26396 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 1 0 0 0 0 0
/proc/5723/statm: 204765 26396 383 364 0 139970 0
[pid=5725] ppid=5723 vsize=87268 CPUtime=48.75 cores=1,3,5,7
/proc/5725/stat : 5725 (bsolo_pb10) T 5723 5725 5725 0 -1 4202496 26987 0 0 0 4819 56 0 0 20 0 1 0 338241860 89362432 19362 33554432000 134512640 137138936 4289593552 4289592480 134542214 0 0 16781312 16384 18446744071579387992 0 0 17 5 0 0 0 0 0
/proc/5725/statm: 21817 19362 192 642 0 21173 0
[pid=5723/tid=5732] ppid=5615 vsize=819060 CPUtime=0 cores=1,3,5,7
/proc/5723/task/5732/stat : 5732 (python) S 5615 5615 3621 0 -1 4202560 44 0 0 0 0 0 0 0 20 0 2 0 338246737 838717440 26396 33554432000 4194304 5683644 140737190967600 139826944287280 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5733] ppid=5615 vsize=894852 CPUtime=0.17 cores=1,3,5,7
/proc/5733/stat : 5733 (python) S 5615 5615 3621 0 -1 4202560 1244 0 0 0 10 7 0 0 20 0 2 0 338246739 916328448 26408 33554432000 4194304 5683644 140737190967600 140737190959248 213423023373 0 0 16781312 33554946 18446744073709551615 0 0 17 3 0 0 0 0 0
/proc/5733/statm: 223713 26408 389 364 0 158915 0
[pid=5735] ppid=5733 vsize=59796 CPUtime=95.25 cores=1,3,5,7
/proc/5735/stat : 5735 (clasp-1.3.7-x86) T 5733 5735 5735 0 -1 4202496 16274 0 0 0 9520 5 0 0 20 0 1 0 338246740 61231104 14306 33554432000 134512640 136139019 4290097072 4290096048 134883787 0 0 16781312 16386 18446744071579387992 0 0 17 1 0 0 0 0 0
/proc/5735/statm: 14949 14306 183 398 0 14549 0
[pid=5733/tid=5763] ppid=5615 vsize=894852 CPUtime=0 cores=1,3,5,7
/proc/5733/task/5763/stat : 5763 (python) S 5615 5615 3621 0 -1 4202560 45 0 0 0 0 0 0 0 20 0 2 0 338251454 916328448 26408 33554432000 4194304 5683644 140737190967600 139826933797424 213423019808 0 0 16781312 33554946 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5768] ppid=5615 vsize=970644 CPUtime=0 cores=1,3,5,7
/proc/5768/stat : 5768 (python) S 5615 5615 3621 0 -1 4202560 838 0 0 0 0 0 0 0 20 0 1 0 338256273 993939456 26363 33554432000 4194304 5683644 140737190967600 140737190958688 213419655651 0 0 16781312 33554946 18446744071580425449 0 0 17 5 0 0 0 0 0
/proc/5768/statm: 242661 26363 339 364 0 177860 0
[pid=5770] ppid=5768 vsize=9225560 CPUtime=0.03 cores=1,3,5,7
/proc/5770/stat : 5770 (java) S 5768 5770 5770 0 -1 4202496 3290 0 10 0 2 1 0 0 20 0 13 0 338256273 9446973440 3758 33554432000 1073741824 1073778376 140737157631072 140737157622208 213422997565 0 0 0 16800971 18446744073709551615 0 0 17 3 0 0 1 0 0
/proc/5770/statm: 2306390 3758 1707 9 0 2297091 0
[pid=5770/tid=5771] ppid=5768 vsize=9225560 CPUtime=0.02 cores=1,3,5,7
/proc/5770/task/5771/stat : 5771 (java) D 5768 5770 5770 0 -1 4202560 1876 0 5 0 2 0 0 0 20 0 13 0 338256278 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677686568536 213419638453 0 4 0 16800971 18446744072100609316 0 0 -1 5 0 0 0 0 0
[pid=5770/tid=5772] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5772/stat : 5772 (java) S 5768 5770 5770 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677648063272 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5770/tid=5773] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5773/stat : 5773 (java) S 5768 5770 5770 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677647010472 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5770/tid=5774] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5774/stat : 5774 (java) S 5768 5770 5770 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677645958184 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5770/tid=5775] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5775/stat : 5775 (java) S 5768 5770 5770 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677644905384 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5770/tid=5776] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5776/stat : 5776 (java) S 5768 5770 5770 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677643852072 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5770/tid=5777] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5777/stat : 5777 (java) S 5768 5770 5770 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677642799272 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 5 0 0 0 0 0
[pid=5770/tid=5778] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5778/stat : 5778 (java) S 5768 5770 5770 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677641746984 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5770/tid=5779] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5779/stat : 5779 (java) S 5768 5770 5770 0 -1 4202560 3 0 0 0 0 0 0 0 20 0 13 0 338256280 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677640694184 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5770/tid=5780] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5780/stat : 5780 (java) S 5768 5770 5770 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 13 0 338256281 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677510527720 213423011753 0 0 0 16800971 18446744071579510697 0 0 -1 7 0 0 0 0 0
[pid=5770/tid=5781] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5781/stat : 5781 (java) S 5768 5770 5770 0 -1 4202560 16 0 0 0 0 0 0 0 20 0 13 0 338256281 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677509473784 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5770/tid=5782] ppid=5768 vsize=9225560 CPUtime=0 cores=1,3,5,7
/proc/5770/task/5782/stat : 5782 (java) S 5768 5770 5770 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 13 0 338256281 9446973440 3759 33554432000 1073741824 1073778376 140737157631072 139677508421384 213423010876 0 4 0 16800971 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5615/tid=5622] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5622/stat : 5622 (python) S 5613 5615 3621 0 -1 4202560 27 1039946 0 0 0 0 548 387 20 0 10 0 338203332 1071538176 27949 33554432000 4194304 5683644 140737190967600 139827014787632 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5691] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5691/stat : 5691 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 10 0 338208311 1071538176 27949 33554432000 4194304 5683644 140737190967600 139827004297776 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5698] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5698/stat : 5698 (python) S 5613 5615 3621 0 -1 4202560 21 1039946 0 0 0 0 548 387 20 0 10 0 338213042 1071538176 27949 33554432000 4194304 5683644 140737190967600 139826993807920 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5706] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5706/stat : 5706 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 10 0 338218033 1071538176 27949 33554432000 4194304 5683644 140737190967600 139826983318064 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5713] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5713/stat : 5713 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 10 0 338222803 1071538176 27949 33554432000 4194304 5683644 140737190967600 139826972828208 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5718] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5718/stat : 5718 (python) S 5613 5615 3621 0 -1 4202560 21 1039946 0 0 0 0 548 387 20 0 10 0 338229385 1071538176 27949 33554432000 4194304 5683644 140737190967600 139826944287280 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5724] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5724/stat : 5724 (python) S 5613 5615 3621 0 -1 4202560 22 1039946 0 0 0 0 548 387 20 0 10 0 338241860 1071538176 27949 33554432000 4194304 5683644 140737190967600 139826933797424 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
[pid=5615/tid=5734] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5734/stat : 5734 (python) S 5613 5615 3621 0 -1 4202560 21 1039946 0 0 0 0 548 387 20 0 10 0 338246740 1071538176 27949 33554432000 4194304 5683644 140737190967600 139826923307568 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 1 0 0 0 0 0
[pid=5615/tid=5769] ppid=5613 vsize=1046424 CPUtime=9.35 cores=1,3,5,7
/proc/5615/task/5769/stat : 5769 (python) S 5613 5615 3621 0 -1 4202560 21 1039946 0 0 0 0 548 387 20 0 10 0 338256273 1071538176 27949 33554432000 4194304 5683644 140737190967600 139826912817712 213423019808 0 0 16781312 33554434 18446744071579510697 0 0 -1 3 0 0 0 0 0
Current children cumulated CPU time (s) 420.57
Current children cumulated vsize (KiB) 18691276
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 5615 and gives
#  childrusage.ru_utime.tv_sec=28
#  childrusage.ru_utime.tv_usec=786623
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=150369
# CPU time returned by wait4() is 32.937
# while last known CPU time is 420.57
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 536.369
CPU time (s): 420.57
CPU user time (s): 412.39
CPU system time (s): 8.18
CPU usage (%): 78.4106
Max. virtual memory (cumulated for all children) (KiB): 18691276

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 28.7866
system time used= 4.15037
maximum resident set size= 4158040
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1079410
page faults= 1
swaps= 0
block input operations= 0
block output operations= 36240
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 800154
involuntary context switches= 78

runsolver used 2.29065 second user time and 5.21621 second system time

The end

Launcher Data 

Begin job on node140 at 2012-06-03 12:52:56
IDJOB=3741007
IDBENCH=89728
IDSOLVER=2327
FILE ID=node140/3741007-1338720776
RUNJOBID= node140-1338713222-3638
PBS_JOBID= 14636725
Free space on /tmp= 70504 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-76.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3741007-1338720776/watcher-3741007-1338720776 -o /tmp/evaluation-result-3741007-1338720776/solver-3741007-1338720776 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741007-1338720776.opb -seed 2037121049 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f030651c159d27eae553d070b0927366
RANDOM SEED=2037121049

node140.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5334.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.92
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.88
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2667.094
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873844 kB
MemFree:        25109748 kB
Buffers:          476964 kB
Cached:          4580324 kB
SwapCached:         1664 kB
Active:          3420016 kB
Inactive:        3406908 kB
Active(anon):    1761928 kB
Inactive(anon):     9500 kB
Active(file):    1658088 kB
Inactive(file):  3397408 kB
Unevictable:        6608 kB
Mlocked:            6608 kB
SwapTotal:      67108856 kB
SwapFree:       67105156 kB
Dirty:            113536 kB
Writeback:             0 kB
AnonPages:       1774300 kB
Mapped:            26740 kB
Shmem:               104 kB
Slab:             786600 kB
SReclaimable:     223988 kB
SUnreclaim:       562612 kB
KernelStack:        1656 kB
PageTables:        12528 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545776 kB
Committed_AS:    2524920 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346368 kB
VmallocChunk:   34341987984 kB
HardwareCorrupted:     0 kB
AnonHugePages:    890880 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7872 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 66312 MiB
End job on node140 at 2012-06-03 13:01:56