Trace number 3741111

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
PB11: borg pb-dec-11.04.03? (MO) 255.6 412.455

General information on the benchmark

Name/DEC-SMALLINT-LIN/lopes/
normalized-170.opb
MD5SUMec3e1a4dea6a8a4871a8c3a868360cb3
Bench CategoryDEC-SMALLINT-LIN (no optimisation, small integers, linear constraints)
Best result obtained on this benchmarkSAT
Best value of the objective obtained on this benchmark0
Best CPU time to get the best result obtained on this benchmark58.4911
Has Objective FunctionNO
Satisfiable
(Un)Satisfiability was proved
Best value of the objective function
Optimality of the best value was proved
Number of variables26339
Total number of constraints612374
Number of constraints which are clauses549666
Number of constraints which are cardinality constraints (but not clauses)2270
Number of constraints which are nor clauses,nor cardinality constraints60438
Minimum length of a constraint1
Maximum length of a constraint1485
Number of terms in the objective function 0
Biggest coefficient in the objective function 0
Number of bits for the biggest coefficient in the objective function 0
Sum of the numbers in the objective function 0
Number of bits of the sum of numbers in the objective function 0
Biggest number in a constraint 3
Number of bits of the biggest number in a constraint 2
Biggest sum of numbers in a constraint 1486
Number of bits of the biggest sum of numbers11
Number of products (including duplicates)0
Sum of products size (including duplicates)0
Number of different products0
Sum of products size0

Solver Data 

0.34/1.28	c INFO: loaded portfolio from HOME/etc/borg-mix+class.dec.pickle
0.34/1.28	c INFO: solving HOME/instance-3741111-1338721210.opb
4.91/5.84	c INFO: parsing took 4.47 s
12.41/13.38	c INFO: features took 7.52 CPU seconds
50.39/51.30	c INFO: wrote minion input file to HOME/tmpHUOK3H.minion
50.39/51.30	c INFO: running minion-0.12@0 for 45 with 1787 remaining (b = 0.74)
64.20/223.62	c INFO: running pbct-0.1.2-linear@0 for 45 with 1738 remaining (b = 0.91)
112.05/271.44	c INFO: running clasp-1.3.7@0 for 45 with 1692 remaining (b = 0.91)
159.19/318.59	c INFO: running wbo1.4b-fixed@0 for 45 with 1644 remaining (b = 0.91)
208.55/367.80	c INFO: running sat4j-pb-v20101225@0 for 45 with 1597 remaining (b = 0.55)
255.20/412.00	c INFO: running wbo1.4a@0 for 45 with 1551 remaining (b = 0.41)

Verifier Data 

ERROR: no interpretation found !

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3741111-1338721210/watcher-3741111-1338721210 -o /tmp/evaluation-result-3741111-1338721210/solver-3741111-1338721210 -C 1800 -W 1900 -M 15500 python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741111-1338721210.opb -seed 1358277646 -budget 1800 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 1800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 1830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1900 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.11 2.31 2.44 3/196 1352
/proc/meminfo: memFree=20417192/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=15016 CPUtime=0 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 531 0 0 0 0 0 0 0 20 0 1 0 338248639 15376384 435 33554432000 4194304 5683644 140736324309968 140736324296744 5054470 0 0 4096 0 0 0 0 17 4 0 0 0 0 0
/proc/1352/statm: 3754 439 273 364 0 196 0

[startup+0.100338 s]
/proc/loadavg: 2.11 2.31 2.44 3/196 1352
/proc/meminfo: memFree=20417192/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=19084 CPUtime=0.02 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 1395 0 0 0 2 0 0 0 20 0 1 0 338248639 19542016 1149 33554432000 4194304 5683644 140736324309968 140736324240472 4446008 0 0 16781312 2 0 0 0 17 4 0 0 0 0 0
/proc/1352/statm: 4771 1149 430 364 0 695 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 19084

[startup+0.20019 s]
/proc/loadavg: 2.11 2.31 2.44 3/196 1352
/proc/meminfo: memFree=20417192/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=56944 CPUtime=0.04 cores=0,2,4,6
/proc/1352/stat : 1352 (python) D 1350 1352 27000 0 -1 4202496 2300 0 0 0 4 0 0 0 20 0 1 0 338248639 58310656 1869 33554432000 4194304 5683644 140736324309968 140736324236232 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 4 0 0 0 0 0
/proc/1352/statm: 14236 1869 508 364 0 1313 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 56944

[startup+0.300218 s]
/proc/loadavg: 2.11 2.31 2.44 3/196 1352
/proc/meminfo: memFree=20417192/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=96892 CPUtime=0.1 cores=0,2,4,6
/proc/1352/stat : 1352 (python) D 1350 1352 27000 0 -1 4202496 3909 1008 1 0 9 1 0 0 20 0 1 0 338248639 99217408 2647 33554432000 4194304 5683644 140736324309968 140736324209960 219513963408 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/1352/statm: 24223 2647 822 364 0 1714 0
Current children cumulated CPU time (s) 0.1
Current children cumulated vsize (KiB) 96892

[startup+0.700152 s]
/proc/loadavg: 2.11 2.31 2.44 3/196 1352
/proc/meminfo: memFree=20417192/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=122344 CPUtime=0.2 cores=0,2,4,6
/proc/1352/stat : 1352 (python) D 1350 1352 27000 0 -1 4202496 5982 1008 1 0 15 5 0 0 20 0 1 0 338248639 125280256 3855 33554432000 4194304 5683644 140736324309968 140736324255320 219513963408 0 0 16781312 2 18446744072100609316 0 0 17 0 0 0 0 0 0
/proc/1352/statm: 30586 3855 1015 364 0 2829 0
Current children cumulated CPU time (s) 0.2
Current children cumulated vsize (KiB) 122344

[startup+1.50019 s]
/proc/loadavg: 2.11 2.31 2.44 2/197 1357
/proc/meminfo: memFree=20296452/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=314428 CPUtime=0.61 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 12099 1008 1 0 51 10 0 0 20 0 1 0 338248639 321974272 15267 33554432000 4194304 5683644 140736324309968 140736324303208 4425072 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/1352/statm: 78607 15269 1811 364 0 13836 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 314428

[startup+3.10018 s]
/proc/loadavg: 2.11 2.31 2.44 3/197 1357
/proc/meminfo: memFree=20229208/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=365424 CPUtime=2.21 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 24367 1008 1 0 209 12 0 0 20 0 1 0 338248639 374194176 27535 33554432000 4194304 5683644 140736324309968 140736324302584 219514280015 0 0 16781312 2 0 0 0 17 0 0 0 0 0 0
/proc/1352/statm: 91356 27535 1811 364 0 26585 0
Current children cumulated CPU time (s) 2.21
Current children cumulated vsize (KiB) 365424
heavy processes:

[startup+6.30023 s]
/proc/loadavg: 2.10 2.31 2.44 3/197 1357
/proc/meminfo: memFree=20158696/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=445324 CPUtime=5.4 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 46597 1008 1 0 522 18 0 0 20 0 1 0 338248639 456011776 48083 33554432000 4194304 5683644 140736324309968 140736324300392 4643481 0 0 16781312 33554434 0 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 111331 48083 1846 364 0 46560 0
Current children cumulated CPU time (s) 5.4
Current children cumulated vsize (KiB) 445324

[startup+12.7002 s]
/proc/loadavg: 2.09 2.30 2.44 3/197 1357
/proc/meminfo: memFree=20157184/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=450108 CPUtime=11.8 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 47283 1008 1 0 1162 18 0 0 20 0 1 0 338248639 460910592 49280 33554432000 4194304 5683644 140736324309968 140736324303144 219513633945 0 0 16781312 33554434 0 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 112527 49280 1847 364 0 47756 0
Current children cumulated CPU time (s) 11.8
Current children cumulated vsize (KiB) 450108

[startup+25.5002 s]
/proc/loadavg: 2.07 2.29 2.43 3/197 1357
/proc/meminfo: memFree=20136776/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=459692 CPUtime=24.6 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 47708 1008 1 0 2438 22 0 0 20 0 1 0 338248639 470724608 51719 33554432000 4194304 5683644 140736324309968 140736324304296 140650534700852 0 0 16781312 33554434 0 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 114923 51719 1893 364 0 50149 0
Current children cumulated CPU time (s) 24.6
Current children cumulated vsize (KiB) 459692

[startup+51.1003 s]
/proc/loadavg: 2.04 2.26 2.42 3/198 1384
/proc/meminfo: memFree=19335256/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=459692 CPUtime=50.19 cores=0,2,4,6
/proc/1352/stat : 1352 (python) R 1350 1352 27000 0 -1 4202496 47708 1008 1 0 4990 29 0 0 20 0 1 0 338248639 470724608 51719 33554432000 4194304 5683644 140736324309968 140736324304296 4865436 0 0 16781312 33554434 0 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 114923 51719 1893 364 0 50149 0
Current children cumulated CPU time (s) 50.19
Current children cumulated vsize (KiB) 459692

[startup+102.3 s]
/proc/loadavg: 3.49 2.58 2.52 2/205 1393
/proc/meminfo: memFree=6696604/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=535480 CPUtime=50.41 cores=0,2,4,6
/proc/1352/stat : 1352 (python) S 1350 1352 27000 0 -1 4202496 47922 1008 1 0 5011 30 0 0 20 0 2 0 338248639 548331520 51733 33554432000 4194304 5683644 140736324309968 140736324304800 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 133870 51733 1902 364 0 69093 0
[pid=1385] ppid=1352 vsize=459700 CPUtime=0.08 cores=0,2,4,6
/proc/1385/stat : 1385 (python) S 1352 1352 27000 0 -1 4202560 999 0 0 0 5 3 0 0 20 0 1 0 338253770 470732800 50187 33554432000 4194304 5683644 140736324309968 140736324301056 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/1385/statm: 114925 50187 362 364 0 50148 0
[pid=1387] ppid=1385 vsize=4174764 CPUtime=13.26 cores=0,2,4,6
/proc/1387/stat : 1387 (minion) S 1385 1387 1387 0 -1 4204032 590786 0 0 0 1104 222 0 0 20 0 1 0 338253770 4274958336 1035466 33554432000 134512640 165565024 4291735600 4291732832 159737071 0 0 16781312 8396802 18446744071580380219 0 0 17 4 0 0 0 0 0
/proc/1387/statm: 1043691 1035466 359 7582 0 1036106 0
[pid=1352/tid=1386] ppid=1350 vsize=535480 CPUtime=0 cores=0,2,4,6
/proc/1352/task/1386/stat : 1386 (python) S 1350 1352 27000 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 2 0 338253770 548331520 51733 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 63.75
Current children cumulated vsize (KiB) 5169944
heavy processes:
  %CPU=35 pid=1389 uid=0 cmd=

[startup+162.3 s]
/proc/loadavg: 4.52 3.05 2.68 2/184 1441
/proc/meminfo: memFree=13647304/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=535480 CPUtime=50.41 cores=0,2,4,6
/proc/1352/stat : 1352 (python) S 1350 1352 27000 0 -1 4202496 47922 1008 1 0 5011 30 0 0 20 0 2 0 338248639 548331520 51733 33554432000 4194304 5683644 140736324309968 140736324304800 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 133870 51733 1902 364 0 69093 0
[pid=1385] ppid=1352 vsize=459700 CPUtime=0.17 cores=0,2,4,6
/proc/1385/stat : 1385 (python) S 1352 1352 27000 0 -1 4202560 1032 0 0 0 10 7 0 0 20 0 1 0 338253770 470732800 50187 33554432000 4194304 5683644 140736324309968 140736324301056 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/1385/statm: 114925 50187 362 364 0 50148 0
[pid=1387] ppid=1385 vsize=4174764 CPUtime=13.37 cores=0,2,4,6
/proc/1387/stat : 1387 (minion) S 1385 1387 1387 0 -1 4204032 590786 0 0 0 1104 233 0 0 20 0 1 0 338253770 4274958336 1035466 33554432000 134512640 165565024 4291735600 4291732832 159737071 0 0 16781312 8396802 18446744071580380219 0 0 17 2 0 0 0 0 0
/proc/1387/statm: 1043691 1035466 359 7582 0 1036106 0
[pid=1352/tid=1386] ppid=1350 vsize=535480 CPUtime=0 cores=0,2,4,6
/proc/1352/task/1386/stat : 1386 (python) S 1350 1352 27000 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 2 0 338253770 548331520 51733 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 63.95
Current children cumulated vsize (KiB) 5169944
heavy processes:
  %CPU=14 pid=1389 uid=0 cmd=/usr/libexec/abrt-hook-ccpp /var/spool/abrt 1387 11 506 18446744073709551615 

[startup+222.3 s]
/proc/loadavg: 4.82 3.40 2.82 2/176 1487
/proc/meminfo: memFree=14102428/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=535480 CPUtime=50.41 cores=0,2,4,6
/proc/1352/stat : 1352 (python) S 1350 1352 27000 0 -1 4202496 47922 1008 1 0 5011 30 0 0 20 0 2 0 338248639 548331520 51733 33554432000 4194304 5683644 140736324309968 140736324304800 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 133870 51733 1902 364 0 69093 0
[pid=1385] ppid=1352 vsize=459700 CPUtime=0.27 cores=0,2,4,6
/proc/1385/stat : 1385 (python) S 1352 1352 27000 0 -1 4202560 1045 0 0 0 16 11 0 0 20 0 1 0 338253770 470732800 50187 33554432000 4194304 5683644 140736324309968 140736324301056 219513979304 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/1385/statm: 114925 50187 362 364 0 50148 0
[pid=1387] ppid=1385 vsize=4174764 CPUtime=13.37 cores=0,2,4,6
/proc/1387/stat : 1387 (minion) S 1385 1387 1387 0 -1 4204032 590786 0 0 0 1104 233 0 0 20 0 1 0 338253770 4274958336 1035466 33554432000 134512640 165565024 4291735600 4291732832 159737071 0 0 16781312 8396802 18446744071580380219 0 0 17 2 0 0 0 0 0
/proc/1387/statm: 1043691 1035466 359 7582 0 1036106 0
[pid=1352/tid=1386] ppid=1350 vsize=535480 CPUtime=0 cores=0,2,4,6
/proc/1352/task/1386/stat : 1386 (python) S 1350 1352 27000 0 -1 4202560 26 1008 0 0 0 0 0 0 20 0 2 0 338253770 548331520 51733 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 64.05
Current children cumulated vsize (KiB) 5169944
heavy processes:

[startup+282.3 s]
/proc/loadavg: 3.21 3.20 2.79 3/193 1517
/proc/meminfo: memFree=19870096/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=687064 CPUtime=64.25 cores=0,2,4,6
/proc/1352/stat : 1352 (python) S 1350 1352 27000 0 -1 4202496 49551 593049 1 0 5015 30 1120 260 20 0 4 0 338248639 703553536 51750 33554432000 4194304 5683644 140736324309968 140736324304800 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 171766 51750 1907 364 0 106983 0
[pid=1491] ppid=1352 vsize=535492 CPUtime=0.07 cores=0,2,4,6
/proc/1491/stat : 1491 (python) S 1352 1352 27000 0 -1 4202560 1065 0 0 0 4 3 0 0 20 0 2 0 338271001 548343808 50215 33554432000 4194304 5683644 140736324309968 140736324301616 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/1491/statm: 133873 50215 383 364 0 69093 0
[pid=1493] ppid=1491 vsize=2447792 CPUtime=47.77 cores=0,2,4,6
/proc/1493/stat : 1493 (pbct-0.1.2-linu) T 1491 1493 1493 0 -1 4202496 559471 0 0 0 4677 100 0 0 20 0 1 0 338271001 2506539008 583577 33554432000 134512640 136422188 4287634960 4287632480 135006609 0 0 16781312 16387 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/1493/statm: 611948 583577 215 467 0 611479 0
[pid=1491/tid=1510] ppid=1352 vsize=535492 CPUtime=0 cores=0,2,4,6
/proc/1491/task/1510/stat : 1510 (python) S 1352 1352 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338275781 548343808 50215 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1511] ppid=1352 vsize=611284 CPUtime=0 cores=0,2,4,6
/proc/1511/stat : 1511 (python) S 1352 1352 27000 0 -1 4202560 906 0 0 0 0 0 0 0 20 0 1 0 338275783 625954816 50195 33554432000 4194304 5683644 140736324309968 140736324301056 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 0 0 0 0 0 0
/proc/1511/statm: 152821 50195 358 364 0 88038 0
[pid=1513] ppid=1511 vsize=42676 CPUtime=10.84 cores=0,2,4,6
/proc/1513/stat : 1513 (clasp-1.3.7-x86) R 1511 1513 1513 0 -1 4202496 13750 0 0 0 1081 3 0 0 20 0 1 0 338275784 43700224 10246 33554432000 134512640 136139019 4287548496 4287547376 134867637 0 0 16781312 16386 0 0 0 17 2 0 0 0 0 0
/proc/1513/statm: 10669 10246 183 398 0 10269 0
[pid=1352/tid=1386] ppid=1350 vsize=687064 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1386/stat : 1386 (python) S 1350 1352 27000 0 -1 4202560 26 593049 0 0 0 0 1120 260 20 0 4 0 338253770 703553536 51750 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1492] ppid=1350 vsize=687064 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1492/stat : 1492 (python) S 1350 1352 27000 0 -1 4202560 22 593049 0 0 0 0 1120 260 20 0 4 0 338271001 703553536 51750 33554432000 4194304 5683644 140736324309968 140650351153712 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1512] ppid=1350 vsize=687064 CPUtime=13.8 cores=0,2,4,6

################
# More data... #
################

/proc/1529/task/1531/stat : 1531 (java) T 1527 1529 1529 0 -1 4202560 98 0 0 0 19 5 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110765251496 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1532] ppid=1527 vsize=9624972 CPUtime=0.25 cores=0,2,4,6
/proc/1529/task/1532/stat : 1532 (java) T 1527 1529 1529 0 -1 4202560 111 0 0 0 21 4 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110764198952 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=1529/tid=1533] ppid=1527 vsize=9624972 CPUtime=0.19 cores=0,2,4,6
/proc/1529/task/1533/stat : 1533 (java) T 1527 1529 1529 0 -1 4202560 86 0 0 0 16 3 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110763145896 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1535] ppid=1527 vsize=9624972 CPUtime=0.23 cores=0,2,4,6
/proc/1529/task/1535/stat : 1535 (java) T 1527 1529 1529 0 -1 4202560 337 0 0 0 20 3 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110762093352 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=1529/tid=1536] ppid=1527 vsize=9624972 CPUtime=0.27 cores=0,2,4,6
/proc/1529/task/1536/stat : 1536 (java) T 1527 1529 1529 0 -1 4202560 122 0 0 0 23 4 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110761040296 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=1529/tid=1537] ppid=1527 vsize=9624972 CPUtime=0.21 cores=0,2,4,6
/proc/1529/task/1537/stat : 1537 (java) T 1527 1529 1529 0 -1 4202560 282 0 0 0 17 4 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110759987752 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1538] ppid=1527 vsize=9624972 CPUtime=0.24 cores=0,2,4,6
/proc/1529/task/1538/stat : 1538 (java) T 1527 1529 1529 0 -1 4202560 104 0 0 0 19 5 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110641227944 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1539] ppid=1527 vsize=9624972 CPUtime=0.27 cores=0,2,4,6
/proc/1529/task/1539/stat : 1539 (java) T 1527 1529 1529 0 -1 4202560 130 0 0 0 22 5 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110640175400 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=1529/tid=1540] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1540/stat : 1540 (java) T 1527 1529 1529 0 -1 4202560 1267 0 0 0 0 0 0 0 20 0 19 0 338285424 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110627978088 219517335465 0 0 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1541] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1541/stat : 1541 (java) T 1527 1529 1529 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338285424 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110626924408 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1542] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1542/stat : 1542 (java) T 1527 1529 1529 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338285424 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110625871240 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1543] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1543/stat : 1543 (java) T 1527 1529 1529 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109886253840 219517343520 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1547] ppid=1527 vsize=9624972 CPUtime=0.44 cores=0,2,4,6
/proc/1529/task/1547/stat : 1547 (java) T 1527 1529 1529 0 -1 4202560 1968 0 0 0 44 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109885200696 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1548] ppid=1527 vsize=9624972 CPUtime=0.49 cores=0,2,4,6
/proc/1529/task/1548/stat : 1548 (java) T 1527 1529 1529 0 -1 4202560 3823 0 0 0 47 2 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109884148152 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=1529/tid=1549] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1549/stat : 1549 (java) T 1527 1529 1529 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109883095288 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=1529/tid=1550] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1550/stat : 1550 (java) T 1527 1529 1529 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109882042952 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1551] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1551/stat : 1551 (java) T 1527 1529 1529 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338285829 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109880989224 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=1527/tid=1557] ppid=1352 vsize=762868 CPUtime=0 cores=0,2,4,6
/proc/1527/task/1557/stat : 1557 (python) S 1352 1352 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338289837 781176832 50233 33554432000 4194304 5683644 140736324309968 140650330174000 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1558] ppid=1352 vsize=838660 CPUtime=0 cores=0,2,4,6
/proc/1558/stat : 1558 (python) S 1352 1352 27000 0 -1 4202560 801 0 0 0 0 0 0 0 20 0 1 0 338289840 858787840 50698 33554432000 4194304 5683644 140736324309968 140736324301056 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 6 0 0 0 0 0
/proc/1558/statm: 209665 50698 335 364 0 144873 0
[pid=1560] ppid=1558 vsize=39648 CPUtime=0.38 cores=0,2,4,6
/proc/1560/stat : 1560 (wbo1.4a) R 1558 1560 1560 0 -1 4202496 4531 0 0 0 38 0 0 0 20 0 1 0 338289840 40599552 4273 33554432000 134512640 135751879 4290402896 4289352408 135206651 0 0 16781312 3 0 0 0 17 2 0 0 0 0 0
/proc/1560/statm: 9912 4273 1578 303 0 3057 0
[pid=1352/tid=1386] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1386/stat : 1386 (python) S 1350 1352 27000 0 -1 4202560 26 593049 0 0 0 0 1120 260 20 0 7 0 338253770 936386560 52282 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1492] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1492/stat : 1492 (python) S 1350 1352 27000 0 -1 4202560 22 593049 0 0 0 0 1120 260 20 0 7 0 338271001 936386560 52282 33554432000 4194304 5683644 140736324309968 140650351153712 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1512] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1512/stat : 1512 (python) S 1350 1352 27000 0 -1 4202560 21 593049 0 0 0 0 1120 260 20 0 7 0 338275784 936386560 52282 33554432000 4194304 5683644 140736324309968 140650340663856 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1520] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1520/stat : 1520 (python) S 1350 1352 27000 0 -1 4202560 22 593049 0 0 0 0 1120 260 20 0 7 0 338280499 936386560 52282 33554432000 4194304 5683644 140736324309968 140650330174000 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1528] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1528/stat : 1528 (python) S 1350 1352 27000 0 -1 4202560 19 593049 0 0 0 0 1120 260 20 0 7 0 338285420 936386560 52282 33554432000 4194304 5683644 140736324309968 140650246824496 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1559] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1559/stat : 1559 (python) S 1350 1352 27000 0 -1 4202560 21 593049 0 0 0 0 1120 260 20 0 7 0 338289840 936386560 52282 33554432000 4194304 5683644 140736324309968 140650236334640 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 255.6
Current children cumulated vsize (KiB) 16570396
heavy processes:

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+412.401 s]
/proc/loadavg: 2.14 2.77 2.68 3/230 1560
/proc/meminfo: memFree=15123888/32873648 swapFree=7120/7120
[pid=1352] ppid=1350 vsize=914440 CPUtime=64.34 cores=0,2,4,6
/proc/1352/stat : 1352 (python) S 1350 1352 27000 0 -1 4202496 51971 593049 1 0 5022 32 1120 260 20 0 7 0 338248639 936386560 52282 33554432000 4194304 5683644 140736324309968 140736324304800 219517347085 0 0 16781312 33554434 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/1352/statm: 228610 52282 1913 364 0 163818 0
[pid=1491] ppid=1352 vsize=535492 CPUtime=0.07 cores=0,2,4,6
/proc/1491/stat : 1491 (python) S 1352 1352 27000 0 -1 4202560 1065 0 0 0 4 3 0 0 20 0 2 0 338271001 548343808 50215 33554432000 4194304 5683644 140736324309968 140736324301616 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/1491/statm: 133873 50215 383 364 0 69093 0
[pid=1493] ppid=1491 vsize=2447792 CPUtime=47.77 cores=0,2,4,6
/proc/1493/stat : 1493 (pbct-0.1.2-linu) T 1491 1493 1493 0 -1 4202496 559471 0 0 0 4677 100 0 0 20 0 1 0 338271001 2506539008 583578 33554432000 134512640 136422188 4287634960 4287632480 135006609 0 0 16781312 16387 18446744071579387992 0 0 17 4 0 0 0 0 0
/proc/1493/statm: 611948 583578 215 467 0 611479 0
[pid=1491/tid=1510] ppid=1352 vsize=535492 CPUtime=0 cores=0,2,4,6
/proc/1491/task/1510/stat : 1510 (python) S 1352 1352 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338275781 548343808 50215 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1511] ppid=1352 vsize=611284 CPUtime=0.07 cores=0,2,4,6
/proc/1511/stat : 1511 (python) S 1352 1352 27000 0 -1 4202560 1091 0 0 0 4 3 0 0 20 0 2 0 338275783 625954816 50221 33554432000 4194304 5683644 140736324309968 140736324301616 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/1511/statm: 152821 50221 383 364 0 88038 0
[pid=1513] ppid=1511 vsize=45780 CPUtime=47.11 cores=0,2,4,6
/proc/1513/stat : 1513 (clasp-1.3.7-x86) T 1511 1513 1513 0 -1 4202496 14489 0 0 0 4707 4 0 0 20 0 1 0 338275784 46878720 10985 33554432000 134512640 136139019 4287548496 4287547476 134883821 0 0 16781312 16386 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/1513/statm: 11445 10985 183 398 0 11045 0
[pid=1511/tid=1518] ppid=1352 vsize=611284 CPUtime=0 cores=0,2,4,6
/proc/1511/task/1518/stat : 1518 (python) S 1352 1352 27000 0 -1 4202560 43 0 0 0 0 0 0 0 20 0 2 0 338280496 625954816 50221 33554432000 4194304 5683644 140736324309968 140650351153712 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=1519] ppid=1352 vsize=687076 CPUtime=0.07 cores=0,2,4,6
/proc/1519/stat : 1519 (python) S 1352 1352 27000 0 -1 4202560 1106 0 0 0 4 3 0 0 20 0 2 0 338280499 703565824 50227 33554432000 4194304 5683644 140736324309968 140736324301616 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/1519/statm: 171769 50227 383 364 0 106983 0
[pid=1521] ppid=1519 vsize=62384 CPUtime=49.16 cores=0,2,4,6
/proc/1521/stat : 1521 (wbo1.4b-fixed) T 1519 1521 1521 0 -1 4202496 21796 0 0 0 4754 162 0 0 20 0 1 0 338280499 63881216 14511 33554432000 134512640 135752315 4292648416 4292646432 134605105 0 0 16781312 3 18446744071579387992 0 0 17 2 0 0 0 0 0
/proc/1521/statm: 15596 14511 146 303 0 15291 0
[pid=1519/tid=1526] ppid=1352 vsize=687076 CPUtime=0 cores=0,2,4,6
/proc/1519/task/1526/stat : 1526 (python) S 1352 1352 27000 0 -1 4202560 49 0 0 0 0 0 0 0 20 0 2 0 338285417 703565824 50227 33554432000 4194304 5683644 140736324309968 140650340663856 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 0 0 0 0 0 0
[pid=1527] ppid=1352 vsize=762868 CPUtime=0.08 cores=0,2,4,6
/proc/1527/stat : 1527 (python) S 1352 1352 27000 0 -1 4202560 1123 0 0 0 4 4 0 0 20 0 2 0 338285420 781176832 50233 33554432000 4194304 5683644 140736324309968 140736324301616 219517347085 0 0 16781312 33554946 18446744073709551615 0 0 17 2 0 0 0 0 0
/proc/1527/statm: 190717 50233 383 364 0 125928 0
[pid=1529] ppid=1527 vsize=9624972 CPUtime=46.55 cores=0,2,4,6
/proc/1529/stat : 1529 (java) T 1527 1529 1529 0 -1 4202496 13824 0 1 0 4580 75 0 0 20 0 19 0 338285420 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140737040157136 219517321277 0 0 0 16800975 18446744073709551615 0 0 17 0 0 0 0 0 0
/proc/1529/statm: 2406243 683657 2259 9 0 2396937 0
[pid=1529/tid=1530] ppid=1527 vsize=9624972 CPUtime=43.57 cores=0,2,4,6
/proc/1529/task/1530/stat : 1530 (java) T 1527 1529 1529 0 -1 4202560 4086 0 1 0 4315 42 0 0 20 0 19 0 338285422 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110803758448 140110641943208 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1531] ppid=1527 vsize=9624972 CPUtime=0.24 cores=0,2,4,6
/proc/1529/task/1531/stat : 1531 (java) T 1527 1529 1529 0 -1 4202560 98 0 0 0 19 5 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110765251496 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1532] ppid=1527 vsize=9624972 CPUtime=0.25 cores=0,2,4,6
/proc/1529/task/1532/stat : 1532 (java) T 1527 1529 1529 0 -1 4202560 111 0 0 0 21 4 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110764198952 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=1529/tid=1533] ppid=1527 vsize=9624972 CPUtime=0.19 cores=0,2,4,6
/proc/1529/task/1533/stat : 1533 (java) T 1527 1529 1529 0 -1 4202560 86 0 0 0 16 3 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110763145896 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1535] ppid=1527 vsize=9624972 CPUtime=0.23 cores=0,2,4,6
/proc/1529/task/1535/stat : 1535 (java) T 1527 1529 1529 0 -1 4202560 337 0 0 0 20 3 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110762093352 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=1529/tid=1536] ppid=1527 vsize=9624972 CPUtime=0.27 cores=0,2,4,6
/proc/1529/task/1536/stat : 1536 (java) T 1527 1529 1529 0 -1 4202560 122 0 0 0 23 4 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110761040296 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 6 0 0 0 0 0
[pid=1529/tid=1537] ppid=1527 vsize=9624972 CPUtime=0.21 cores=0,2,4,6
/proc/1529/task/1537/stat : 1537 (java) T 1527 1529 1529 0 -1 4202560 282 0 0 0 17 4 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110759987752 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1538] ppid=1527 vsize=9624972 CPUtime=0.24 cores=0,2,4,6
/proc/1529/task/1538/stat : 1538 (java) T 1527 1529 1529 0 -1 4202560 104 0 0 0 19 5 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110641227944 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1539] ppid=1527 vsize=9624972 CPUtime=0.27 cores=0,2,4,6
/proc/1529/task/1539/stat : 1539 (java) T 1527 1529 1529 0 -1 4202560 130 0 0 0 22 5 0 0 20 0 19 0 338285423 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110640175400 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=1529/tid=1540] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1540/stat : 1540 (java) T 1527 1529 1529 0 -1 4202560 1267 0 0 0 0 0 0 0 20 0 19 0 338285424 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110627978088 219517335465 0 0 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1541] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1541/stat : 1541 (java) T 1527 1529 1529 0 -1 4202560 17 0 0 0 0 0 0 0 20 0 19 0 338285424 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110626924408 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1542] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1542/stat : 1542 (java) T 1527 1529 1529 0 -1 4202560 8 0 0 0 0 0 0 0 20 0 19 0 338285424 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140110625871240 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1543] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1543/stat : 1543 (java) T 1527 1529 1529 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109886253840 219517343520 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1547] ppid=1527 vsize=9624972 CPUtime=0.44 cores=0,2,4,6
/proc/1529/task/1547/stat : 1547 (java) T 1527 1529 1529 0 -1 4202560 1968 0 0 0 44 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109885200696 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1548] ppid=1527 vsize=9624972 CPUtime=0.49 cores=0,2,4,6
/proc/1529/task/1548/stat : 1548 (java) T 1527 1529 1529 0 -1 4202560 3823 0 0 0 47 2 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109884148152 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=1529/tid=1549] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1549/stat : 1549 (java) T 1527 1529 1529 0 -1 4202560 1 0 0 0 0 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109883095288 219517334588 0 4 0 16800975 18446744071579387992 0 0 -1 4 0 0 0 0 0
[pid=1529/tid=1550] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1550/stat : 1550 (java) T 1527 1529 1529 0 -1 4202560 2 0 0 0 0 0 0 0 20 0 19 0 338285427 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109882042952 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 2 0 0 0 0 0
[pid=1529/tid=1551] ppid=1527 vsize=9624972 CPUtime=0 cores=0,2,4,6
/proc/1529/task/1551/stat : 1551 (java) T 1527 1529 1529 0 -1 4202560 7 0 0 0 0 0 0 0 20 0 19 0 338285829 9855971328 683657 33554432000 1073741824 1073778376 140737040166000 140109880989224 219517335465 0 4 0 16800975 18446744071579387992 0 0 -1 0 0 0 0 0 0
[pid=1527/tid=1557] ppid=1352 vsize=762868 CPUtime=0 cores=0,2,4,6
/proc/1527/task/1557/stat : 1557 (python) S 1352 1352 27000 0 -1 4202560 46 0 0 0 0 0 0 0 20 0 2 0 338289837 781176832 50233 33554432000 4194304 5683644 140736324309968 140650330174000 219517343520 0 0 16781312 33554946 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1558] ppid=1352 vsize=838660 CPUtime=0 cores=0,2,4,6
/proc/1558/stat : 1558 (python) S 1352 1352 27000 0 -1 4202560 801 0 0 0 0 0 0 0 20 0 1 0 338289840 858787840 50698 33554432000 4194304 5683644 140736324309968 140736324301056 219513979363 0 0 16781312 33554946 18446744071580425449 0 0 17 6 0 0 0 0 0
/proc/1558/statm: 209665 50698 335 364 0 144873 0
[pid=1560] ppid=1558 vsize=39648 CPUtime=0.38 cores=0,2,4,6
/proc/1560/stat : 1560 (wbo1.4a) R 1558 1560 1560 0 -1 4202496 4531 0 0 0 38 0 0 0 20 0 1 0 338289840 40599552 4273 33554432000 134512640 135751879 4290402896 4289352408 135206651 0 0 16781312 3 0 0 0 17 2 0 0 0 0 0
/proc/1560/statm: 9912 4273 1578 303 0 3057 0
[pid=1352/tid=1386] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1386/stat : 1386 (python) S 1350 1352 27000 0 -1 4202560 26 593049 0 0 0 0 1120 260 20 0 7 0 338253770 936386560 52282 33554432000 4194304 5683644 140736324309968 140650361643568 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1492] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1492/stat : 1492 (python) S 1350 1352 27000 0 -1 4202560 22 593049 0 0 0 0 1120 260 20 0 7 0 338271001 936386560 52282 33554432000 4194304 5683644 140736324309968 140650351153712 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1512] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1512/stat : 1512 (python) S 1350 1352 27000 0 -1 4202560 21 593049 0 0 0 0 1120 260 20 0 7 0 338275784 936386560 52282 33554432000 4194304 5683644 140736324309968 140650340663856 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1520] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1520/stat : 1520 (python) S 1350 1352 27000 0 -1 4202560 22 593049 0 0 0 0 1120 260 20 0 7 0 338280499 936386560 52282 33554432000 4194304 5683644 140736324309968 140650330174000 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1528] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1528/stat : 1528 (python) S 1350 1352 27000 0 -1 4202560 19 593049 0 0 0 0 1120 260 20 0 7 0 338285420 936386560 52282 33554432000 4194304 5683644 140736324309968 140650246824496 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
[pid=1352/tid=1559] ppid=1350 vsize=914440 CPUtime=13.8 cores=0,2,4,6
/proc/1352/task/1559/stat : 1559 (python) S 1350 1352 27000 0 -1 4202560 21 593049 0 0 0 0 1120 260 20 0 7 0 338289840 936386560 52282 33554432000 4194304 5683644 140736324309968 140650236334640 219517343520 0 0 16781312 33554434 18446744071579510697 0 0 -1 4 0 0 0 0 0
Current children cumulated CPU time (s) 255.6
Current children cumulated vsize (KiB) 16570396

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 1352 and gives
#  childrusage.ru_utime.tv_sec=61
#  childrusage.ru_utime.tv_usec=439659
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=925555
# CPU time returned by wait4() is 64.3652
# while last known CPU time is 255.6
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 412.455
CPU time (s): 255.6
CPU user time (s): 249.14
CPU system time (s): 6.46
CPU usage (%): 61.9704
Max. virtual memory (cumulated for all children) (KiB): 16570396

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 61.4397
system time used= 2.92556
maximum resident set size= 4141864
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 645432
page faults= 1
swaps= 0
block input operations= 0
block output operations= 78728
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 995450
involuntary context switches= 117

runsolver used 1.05084 second user time and 2.74858 second system time

The end

Launcher Data 

Begin job on node114 at 2012-06-03 13:00:11
IDJOB=3741111
IDBENCH=89775
IDSOLVER=2327
FILE ID=node114/3741111-1338721210
RUNJOBID= node114-1338713908-27016
PBS_JOBID= 14636748
Free space on /tmp= 69980 MiB

SOLVER NAME= PB11: borg pb-dec-11.04.03
BENCH NAME= PB11/normalized-PB11/DEC-SMALLINT-LIN/lopes/normalized-170.opb
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.dec.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3741111-1338721210/watcher-3741111-1338721210 -o /tmp/evaluation-result-3741111-1338721210/solver-3741111-1338721210 -C 1800 -W 1900 -M 15500  python HOME/solve HOME/etc/borg-mix+class.dec.pickle HOME/instance-3741111-1338721210.opb -seed 1358277646 -budget 1800

TIME LIMIT= 1800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ec3e1a4dea6a8a4871a8c3a868360cb3
RANDOM SEED=1358277646

node114.alineos.net Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5332.48
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5451.90
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.242
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5331.91
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


/proc/meminfo:
MemTotal:       32873648 kB
MemFree:        20421208 kB
Buffers:         1010276 kB
Cached:          3879824 kB
SwapCached:         2180 kB
Active:          9653404 kB
Inactive:        1779056 kB
Active(anon):    6526888 kB
Inactive(anon):    10720 kB
Active(file):    3126516 kB
Inactive(file):  1768336 kB
Unevictable:        7120 kB
Mlocked:            7120 kB
SwapTotal:      67108856 kB
SwapFree:       67103576 kB
Dirty:            128044 kB
Writeback:             0 kB
AnonPages:       6540564 kB
Mapped:            24980 kB
Shmem:               172 kB
Slab:             864384 kB
SReclaimable:     303824 kB
SUnreclaim:       560560 kB
KernelStack:        1592 kB
PageTables:        22644 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    83545680 kB
Committed_AS:    8499636 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      346400 kB
VmallocChunk:   34341988052 kB
HardwareCorrupted:     0 kB
AnonHugePages:    739328 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7680 kB
DirectMap2M:    33538048 kB

Free space on /tmp at the end= 65844 MiB
End job on node114 at 2012-06-03 13:07:07