Trace number 3498254

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-industrial (fixed)? (TO) 5000.07 5024.26

General information on the benchmark

Nameindustrial/fuhs/hard/
AProVE07-01.cnf
MD5SUMd2f77e32530988f77dca5112174eee00
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark2476.89
Satisfiable
(Un)Satisfiability was proved
Number of variables7502
Number of clauses28770
Sum of the clauses size76290
Maximum clause length847
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 215814
Number of clauses of size 311119
Number of clauses of size 4733
Number of clauses of size 5172
Number of clauses of size over 5931

Solver Data 

0.51/31.19	c INFO: loaded portfolio from HOME/etc/borg-mix+class.industrial.pickle
0.51/31.19	c INFO: solving HOME/instance-3498254-1307953113.cnf
0.51/31.19	c INFO: computing task features
0.54/31.27	c INFO: computed variable-clause graph statistics
0.54/31.28	c INFO: computed clause balance statistics
1.03/31.74	c INFO: computed variable balance statistics
1.34/32.08	c INFO: computed small-clause counts
1.34/32.09	c INFO: computed Horn-clause counts
1.53/32.20	c INFO: computed variable graph statistics
1.53/32.20	c INFO: collected features for HOME/instance-3498254-1307953113.cnf in 1.01 s
1.53/32.27	c INFO: running lingeling-276@0 for 97 with 4998 remaining (b = 0.29)
101.77/132.40	c INFO: running lingeling-276@100 for 97 with 4898 remaining (b = 0.15)
197.01/227.52	c INFO: running lingeling-276@0 for 97 with 4803 remaining (b = 0.18)
297.19/327.65	c INFO: running lingeling-276@195 for 388 with 4703 remaining (b = 0.30)
689.05/719.07	c INFO: running cryptominisat-2.9.0@0 for 485 with 4312 remaining (b = 0.33)
1176.86/1206.23	c INFO: running lingeling-276@100 for 97 with 3825 remaining (b = 0.10)
1274.09/1303.36	c INFO: running precosat-570@0 for 194 with 3728 remaining (b = 0.21)
1470.62/1499.61	c INFO: running SApperloT@0 for 97 with 3532 remaining (b = 0.06)
1570.83/1599.76	c INFO: running lingeling-276@197 for 97 with 3431 remaining (b = 0.08)
1668.07/1696.89	c INFO: running cryptominisat-2.9.0@486 for 97 with 3334 remaining (b = 0.08)
1763.83/1792.45	c INFO: running lingeling-276@294 for 194 with 3239 remaining (b = 0.12)
1957.27/1985.70	c INFO: running precosat-570@196 for 97 with 3046 remaining (b = 0.07)
2054.62/2082.83	c INFO: running precosat-570@293 for 97 with 2948 remaining (b = 0.07)
2149.85/2177.96	c INFO: running lingeling-276@0 for 97 with 2853 remaining (b = 0.09)
2250.18/2278.10	c INFO: running cryptominisat-2.9.0@582 for 194 with 2753 remaining (b = 0.10)
2447.87/2475.56	c INFO: running lingeling-276@100 for 97 with 2556 remaining (b = 0.05)
2545.11/2572.69	c INFO: running lingeling-276@487 for 97 with 2459 remaining (b = 0.05)
2642.46/2669.83	c INFO: running cryptominisat-2.9.0@0 for 485 with 2362 remaining (b = 0.23)
3131.20/3157.96	c INFO: running cryptominisat-2.9.0@487 for 97 with 1874 remaining (b = 0.05)
3226.24/3252.90	c INFO: running precosat-570@388 for 388 with 1779 remaining (b = 0.17)
3618.27/3644.37	c INFO: running precosat-570@0 for 194 with 1387 remaining (b = 0.10)
3814.83/3840.62	c INFO: running cryptominisat-2.9.0@779 for 1191 with 1191 remaining (b = 0.35)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3498254-1307953113/watcher-3498254-1307953113 -o /tmp/evaluation-result-3498254-1307953113/solver-3498254-1307953113 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3498254-1307953113.cnf -seed 1825549681 -budget 5000 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 1/173 8987
/proc/meminfo: memFree=4127396/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=11804 CPUtime=0 cores=1,3,5,7
/proc/8987/stat : 8987 (runsolver) S 8985 8987 8950 0 -1 4202560 47 0 0 0 0 0 0 0 17 0 1 0 887727786 12087296 55 33554432000 4194304 5436572 140736215842288 18446744073709551615 5057063 0 2147483391 4096 24578 18446744071699199166 0 0 17 3 0 0 0
/proc/8987/statm: 2951 55 36 304 0 2643 0

[startup+0.063319 s]
/proc/loadavg: 0.00 0.00 0.00 1/173 8987
/proc/meminfo: memFree=4127396/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=15184 CPUtime=0 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 723 0 6 0 0 0 0 0 17 0 1 0 887727786 15548416 617 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039653376 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 1
/proc/8987/statm: 3796 617 322 364 0 283 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15184

[startup+0.100324 s]
/proc/loadavg: 0.00 0.00 0.00 1/173 8987
/proc/meminfo: memFree=4127396/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=15776 CPUtime=0 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 899 0 6 0 0 0 0 0 17 0 1 0 887727786 16154624 784 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039653376 0 2147483391 16781312 2 18446744071699199166 0 0 17 5 0 0 2
/proc/8987/statm: 3944 784 346 364 0 431 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 15776

[startup+0.300296 s]
/proc/loadavg: 0.00 0.00 0.00 1/173 8987
/proc/meminfo: memFree=4127396/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=16400 CPUtime=0.01 cores=1,3,5,7
/proc/8987/stat : 8987 (python) D 8985 8987 8950 0 -1 4202496 1112 0 6 0 1 0 0 0 16 0 1 0 887727786 16793600 985 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039655168 0 0 16781312 2 18446744071562234166 0 0 17 7 0 0 7
/proc/8987/statm: 4100 985 408 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+0.700218 s]
/proc/loadavg: 0.00 0.00 0.00 1/173 8987
/proc/meminfo: memFree=4127396/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=16400 CPUtime=0.01 cores=1,3,5,7
/proc/8987/stat : 8987 (python) D 8985 8987 8950 0 -1 4202496 1117 0 6 0 1 0 0 0 15 0 1 0 887727786 16793600 989 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039655168 0 0 16781312 2 18446744071562234166 0 0 17 7 0 0 8
/proc/8987/statm: 4100 989 413 364 0 587 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16400

[startup+1.50102 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 8997
/proc/meminfo: memFree=4088264/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=16812 CPUtime=0.01 cores=1,3,5,7
/proc/8987/stat : 8987 (python) D 8985 8987 8950 0 -1 4202496 1209 0 6 0 1 0 0 0 16 0 1 0 887727786 17215488 1072 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039651173 0 0 16781312 2 0 0 0 17 7 0 0 35
/proc/8987/statm: 4203 1072 417 364 0 690 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16812

[startup+3.10074 s]
/proc/loadavg: 0.00 0.00 0.00 1/178 8997
/proc/meminfo: memFree=4087068/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=21104 CPUtime=0.02 cores=1,3,5,7
/proc/8987/stat : 8987 (python) D 8985 8987 8950 0 -1 4202496 1296 0 8 0 2 0 0 0 16 0 1 0 887727786 21610496 1154 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039653376 0 0 16781312 2 0 0 0 17 7 0 0 47
/proc/8987/statm: 5276 1154 428 364 0 723 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 21104
heavy processes:

[startup+6.30024 s]
/proc/loadavg: 0.16 0.03 0.01 1/178 8997
/proc/meminfo: memFree=4085632/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=34424 CPUtime=0.05 cores=1,3,5,7
/proc/8987/stat : 8987 (python) R 8985 8987 8950 0 -1 4202496 1542 0 12 0 4 1 0 0 15 0 1 0 887727786 35250176 1388 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039653376 0 2147483391 16781312 2 0 0 0 17 7 0 0 64
/proc/8987/statm: 8606 1388 448 364 0 944 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 34424
heavy processes:

[startup+12.7011 s]
/proc/loadavg: 0.23 0.05 0.02 1/178 8998
/proc/meminfo: memFree=4076584/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=85620 CPUtime=0.09 cores=1,3,5,7
/proc/8987/stat : 8987 (python) R 8985 8987 8950 0 -1 4202496 2611 0 21 0 6 3 0 0 15 0 1 0 887727786 87674880 2348 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039651173 0 2147483391 16781312 2 0 0 0 17 7 0 0 99
/proc/8987/statm: 21405 2348 713 364 0 1622 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 85620
heavy processes:

[startup+25.5068 s]
/proc/loadavg: 0.48 0.11 0.04 1/178 9006
/proc/meminfo: memFree=4047100/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=160668 CPUtime=0.33 cores=1,3,5,7
/proc/8987/stat : 8987 (python) D 8985 8987 8950 0 -1 4202496 6561 927 52 8 22 11 0 0 16 0 1 0 887727786 164524032 4700 33554432000 4194304 5683644 140733304753920 18446744073709551615 212034733072 0 0 16781312 2 18446744071562234166 0 0 17 7 0 0 340
/proc/8987/statm: 40167 4700 1258 364 0 3454 0
Current children cumulated CPU time (s) 0.33
Current children cumulated vsize (KiB) 160668
heavy processes:

[startup+51.1012 s]
/proc/loadavg: 0.88 0.23 0.08 3/181 9010
/proc/meminfo: memFree=3726512/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=291704 CPUtime=1.55 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 11819 927 111 8 138 17 0 0 18 0 2 0 887727786 298704896 7797 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 487
/proc/8987/statm: 72926 7797 1875 364 0 8844 0
[pid=9007] ppid=8987 vsize=281460 CPUtime=0.01 cores=1,3,5,7
/proc/9007/stat : 9007 (python) S 8987 8987 8950 0 -1 4202560 969 0 0 0 0 1 0 0 15 0 1 0 887731014 288215040 6300 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039675407 0 0 16781312 514 3669342980772167944 0 0 17 7 0 0 0
/proc/9007/statm: 70365 6300 382 364 0 6283 0
[pid=9009] ppid=9007 vsize=11968 CPUtime=18.81 cores=1,3,5,7
/proc/9009/stat : 9009 (lingeling) R 9007 9009 9009 0 -1 4202496 4075 0 0 0 1881 0 0 0 25 0 1 0 887731014 12255232 2162 33554432000 4194304 4352236 140734276966864 18446744073709551615 4278664 0 0 16781312 17506 0 0 0 17 1 0 0 0
/proc/9009/statm: 2992 2162 129 39 0 2072 0
[pid=8987/tid=9008] ppid=8985 vsize=291704 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9008/stat : 9008 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 2 0 887731014 298704896 7797 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 20.37
Current children cumulated vsize (KiB) 585132
heavy processes:

[startup+102.301 s]
/proc/loadavg: 1.56 0.52 0.18 3/181 9011
/proc/meminfo: memFree=3724740/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=291704 CPUtime=1.55 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 11819 927 111 8 138 17 0 0 18 0 2 0 887727786 298704896 7797 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 487
/proc/8987/statm: 72926 7797 1875 364 0 8844 0
[pid=9007] ppid=8987 vsize=281460 CPUtime=0.07 cores=1,3,5,7
/proc/9007/stat : 9007 (python) S 8987 8987 8950 0 -1 4202560 986 0 0 0 4 3 0 0 15 0 1 0 887731014 288215040 6300 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039675407 0 0 16781312 514 3669342980772167944 0 0 17 7 0 0 0
/proc/9007/statm: 70365 6300 382 364 0 6283 0
[pid=9009] ppid=9007 vsize=14932 CPUtime=70.01 cores=1,3,5,7
/proc/9009/stat : 9009 (lingeling) R 9007 9009 9009 0 -1 4202496 12240 0 0 0 7000 1 0 0 25 0 1 0 887731014 15290368 2935 33554432000 4194304 4352236 140734276966864 18446744073709551615 4328194 0 0 16781312 17506 0 0 0 17 1 0 0 0
/proc/9009/statm: 3733 2935 129 39 0 2813 0
[pid=8987/tid=9008] ppid=8985 vsize=291704 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9008/stat : 9008 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 2 0 887731014 298704896 7797 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 71.63
Current children cumulated vsize (KiB) 588096
heavy processes:

[startup+162.3 s]
/proc/loadavg: 1.88 0.80 0.30 3/182 9014
/proc/meminfo: memFree=3720640/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=291704 CPUtime=1.58 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 12314 927 111 8 141 17 0 0 18 0 2 0 887727786 298704896 7798 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 487
/proc/8987/statm: 72926 7798 1876 364 0 8844 0
[pid=9007] ppid=8987 vsize=291704 CPUtime=0.15 cores=1,3,5,7
/proc/9007/stat : 9007 (python) S 8987 8987 8950 0 -1 4202560 1168 0 0 0 8 7 0 0 15 0 2 0 887731014 298704896 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039675462 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9007/statm: 72926 6339 416 364 0 8844 0
[pid=9009] ppid=9007 vsize=16980 CPUtime=129.99 cores=1,3,5,7
/proc/9009/stat : 9009 (lingeling) R 9007 9009 9009 0 -1 4202496 19728 0 0 0 12997 2 0 0 25 0 1 0 887731014 17387520 3447 33554432000 4194304 4352236 140734276966864 18446744073709551615 4327899 0 0 16781312 17506 0 0 0 17 1 0 0 0
/proc/9009/statm: 4245 3447 129 39 0 3325 0
[pid=9007/tid=9013] ppid=8987 vsize=291704 CPUtime=0 cores=1,3,5,7
/proc/9007/task/9013/stat : 9013 (python) S 8987 8987 8950 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 887741024 298704896 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454224044224 0 0 -1 7 0 0 0
[pid=8987/tid=9008] ppid=8985 vsize=291704 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9008/stat : 9008 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 2 0 887731014 298704896 7798 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 131.72
Current children cumulated vsize (KiB) 600388

[startup+222.301 s]
/proc/loadavg: 1.95 1.01 0.40 3/185 9018
/proc/meminfo: memFree=2998316/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=291704 CPUtime=1.58 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 12314 927 111 8 141 17 0 0 18 0 2 0 887727786 298704896 7798 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 5 0 0 487
/proc/8987/statm: 72926 7798 1876 364 0 8844 0
[pid=9007] ppid=8987 vsize=291704 CPUtime=0.22 cores=1,3,5,7
/proc/9007/stat : 9007 (python) S 8987 8987 8950 0 -1 4202560 1185 0 0 0 12 10 0 0 15 0 2 0 887731014 298704896 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039675462 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9007/statm: 72926 6339 416 364 0 8844 0
[pid=9009] ppid=9007 vsize=24920 CPUtime=190 cores=1,3,5,7
/proc/9009/stat : 9009 (lingeling) R 9007 9009 9009 0 -1 4202496 27889 0 0 0 18997 3 0 0 25 0 1 0 887731014 25518080 5432 33554432000 4194304 4352236 140734276966864 18446744073709551615 4329149 0 0 16781312 17506 0 0 0 17 1 0 0 0
/proc/9009/statm: 6230 5432 129 39 0 5310 0
[pid=9007/tid=9013] ppid=8987 vsize=291704 CPUtime=0 cores=1,3,5,7
/proc/9007/task/9013/stat : 9013 (python) S 8987 8987 8950 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 887741024 298704896 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454224044224 0 0 -1 7 0 0 0
[pid=8987/tid=9008] ppid=8985 vsize=291704 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9008/stat : 9008 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 2 0 887731014 298704896 7798 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 191.8
Current children cumulated vsize (KiB) 608328

[startup+282.301 s]
/proc/loadavg: 1.98 1.19 0.50 3/188 9023
/proc/meminfo: memFree=2938476/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=301960 CPUtime=1.6 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 12617 927 111 8 143 17 0 0 20 0 3 0 887727786 309207040 7804 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 1 0 0 487
/proc/8987/statm: 75490 7804 1878 364 0 11405 0
[pid=9007] ppid=8987 vsize=291704 CPUtime=0.23 cores=1,3,5,7
/proc/9007/stat : 9007 (python) S 8987 8987 8950 0 -1 4202560 1188 0 0 0 12 11 0 0 15 0 2 0 887731014 298704896 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 7 0 0 0
/proc/9007/statm: 72926 6339 416 364 0 8844 0
[pid=9009] ppid=9007 vsize=22360 CPUtime=195.2 cores=1,3,5,7
/proc/9009/stat : 9009 (lingeling) T 9007 9009 9009 0 -1 4202496 28401 0 0 0 19517 3 0 0 25 0 1 0 887731014 22896640 4792 33554432000 4194304 4352236 140734276966864 18446744073709551615 4276514 0 0 16781312 17506 18446744071562691360 0 0 17 1 0 0 0
/proc/9009/statm: 5590 4792 129 39 0 4670 0
[pid=9007/tid=9013] ppid=8987 vsize=291704 CPUtime=0 cores=1,3,5,7

################
# More data... #
################

[pid=9068] ppid=8987 vsize=312228 CPUtime=1.05 cores=1,3,5,7
/proc/9068/stat : 9068 (python) S 8987 8987 8950 0 -1 4202560 1151 0 0 0 57 48 0 0 18 0 2 0 887858122 319721472 6331 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9068/statm: 78057 6331 400 364 0 13966 0
[pid=9070] ppid=9068 vsize=50344 CPUtime=779.93 cores=1,3,5,7
/proc/9070/stat : 9070 (precosat) T 9068 9070 9070 0 -1 4202496 13143 0 0 0 77991 2 0 0 25 0 1 0 887858122 51552256 12396 33554432000 4194304 4890832 140734283503504 18446744073709551615 4238686 0 0 16781312 17442 18446744071562691360 0 0 17 1 0 0 0
/proc/9070/statm: 12586 12396 86 171 0 12413 0
[pid=9068/tid=9076] ppid=8987 vsize=312228 CPUtime=0 cores=1,3,5,7
/proc/9068/task/9076/stat : 9076 (python) S 8987 8987 8950 0 -1 4202560 51 0 0 0 0 0 0 0 18 0 2 0 887877745 319721472 6331 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604471127114176 0 0 -1 3 0 0 0
[pid=9077] ppid=8987 vsize=322484 CPUtime=0.13 cores=1,3,5,7
/proc/9077/stat : 9077 (python) S 8987 8987 8950 0 -1 4202560 1108 0 0 0 7 6 0 0 16 0 2 0 887877747 330223616 6335 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9077/statm: 80621 6335 400 364 0 16527 0
[pid=9079] ppid=9077 vsize=12740 CPUtime=100.11 cores=1,3,5,7
/proc/9079/stat : 9079 (SApperloT-base) T 9077 9079 9079 0 -1 4202496 3138 0 0 0 10011 0 0 0 25 0 1 0 887877748 13045760 2872 33554432000 134512640 135674940 4292584640 18446744073709551615 134551862 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9079/statm: 3185 2872 100 284 0 2899 0
[pid=9077/tid=9083] ppid=8987 vsize=322484 CPUtime=0 cores=1,3,5,7
/proc/9077/task/9083/stat : 9083 (python) S 8987 8987 8950 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 887887760 330223616 6335 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454223471552 0 0 -1 5 0 0 0
[pid=9099] ppid=8987 vsize=332740 CPUtime=0.26 cores=1,3,5,7
/proc/9099/stat : 9099 (python) S 8987 8987 8950 0 -1 4202560 1126 0 0 0 14 12 0 0 18 0 2 0 887945582 340725760 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/9099/statm: 83185 6339 400 364 0 19088 0
[pid=9101] ppid=9099 vsize=21004 CPUtime=197.23 cores=1,3,5,7
/proc/9101/stat : 9101 (lingeling) T 9099 9101 9101 0 -1 4202496 27970 0 0 0 19720 3 0 0 25 0 1 0 887945583 21508096 4421 33554432000 4194304 4352236 140733941391616 18446744073709551615 4328009 0 0 16781312 17506 18446744071562691360 0 0 17 3 0 0 0
/proc/9101/statm: 5251 4421 129 39 0 4331 0
[pid=9099/tid=9105] ppid=8987 vsize=332740 CPUtime=0 cores=1,3,5,7
/proc/9099/task/9105/stat : 9105 (python) S 8987 8987 8950 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 887955595 340725760 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454223473344 0 0 -1 3 0 0 0
[pid=9125] ppid=8987 vsize=342996 CPUtime=0.8 cores=1,3,5,7
/proc/9125/stat : 9125 (python) S 8987 8987 8950 0 -1 4202560 1167 0 0 0 40 40 0 0 15 0 2 0 887994769 351227904 6366 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9125/statm: 85749 6366 400 364 0 21649 0
[pid=9127] ppid=9125 vsize=185444 CPUtime=583.04 cores=1,3,5,7
/proc/9127/stat : 9127 (cryptominisat-2) T 9125 9127 9127 0 -1 4202496 170391 0 0 0 58274 30 0 0 25 0 1 0 887994769 189894656 27748 33554432000 4194304 4677096 140734540015088 18446744073709551615 4319156 0 0 16781312 2 18446744071562691360 0 0 17 1 0 0 0
/proc/9127/statm: 46361 27748 393 118 0 41320 0
[pid=9125/tid=9141] ppid=8987 vsize=342996 CPUtime=0 cores=1,3,5,7
/proc/9125/task/9141/stat : 9141 (python) S 8987 8987 8950 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 888043581 351227904 6366 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604471125182272 0 0 -1 3 0 0 0
[pid=9158] ppid=8987 vsize=353252 CPUtime=0.26 cores=1,3,5,7
/proc/9158/stat : 9158 (python) S 8987 8987 8950 0 -1 4202560 1177 0 0 0 14 12 0 0 19 0 2 0 888092223 361730048 6376 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/9158/statm: 88313 6376 406 364 0 24210 0
[pid=9160] ppid=9158 vsize=29680 CPUtime=196.23 cores=1,3,5,7
/proc/9160/stat : 9160 (precosat) T 9158 9160 9160 0 -1 4202496 7581 0 0 0 19622 1 0 0 25 0 1 0 888092224 30392320 7222 33554432000 4194304 4890832 140736228792480 18446744073709551615 4238599 0 0 16781312 17442 18446744071562691360 0 0 17 3 0 0 0
/proc/9160/statm: 7420 7222 86 171 0 7247 0
[pid=9158/tid=9167] ppid=8987 vsize=353252 CPUtime=0 cores=1,3,5,7
/proc/9158/task/9167/stat : 9167 (python) S 8987 8987 8950 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 888111847 361730048 6376 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604471125180480 0 0 -1 1 0 0 0
[pid=8987/tid=9008] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9008/stat : 9008 (python) S 8985 8987 8950 0 -1 4202560 35 927 0 8 0 0 0 0 18 0 9 0 887731014 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
[pid=8987/tid=9020] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9020/stat : 9020 (python) S 8985 8987 8950 0 -1 4202560 36 927 0 8 0 0 0 0 15 0 9 0 887750539 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9044] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9044/stat : 9044 (python) S 8985 8987 8950 0 -1 4202560 44 927 0 8 0 0 0 0 18 0 9 0 887799693 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9069] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9069/stat : 9069 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 9 0 887858122 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
[pid=8987/tid=9078] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9078/stat : 9078 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 20 0 9 0 887877748 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9100] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9100/stat : 9100 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 9 0 887945583 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9126] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9126/stat : 9126 (python) S 8985 8987 8950 0 -1 4202560 29 927 0 8 0 0 0 0 18 0 9 0 887994769 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9159] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9159/stat : 9159 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 20 0 9 0 888092223 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4998.16
Current children cumulated vsize (KiB) 3523500



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5024.2 s]
/proc/loadavg: 2.08 2.02 1.97 2/208 9202
/proc/meminfo: memFree=3550208/32951124 swapFree=67050916/67111528
[pid=8987] ppid=8985 vsize=363496 CPUtime=1.93 cores=1,3,5,7
/proc/8987/stat : 8987 (python) S 8985 8987 8950 0 -1 4202496 18351 927 111 8 174 19 0 0 18 0 9 0 887727786 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 2 18446744073709551615 0 0 17 7 0 0 487
/proc/8987/statm: 90874 7864 1890 364 0 26771 0
[pid=9007] ppid=8987 vsize=291704 CPUtime=0.69 cores=1,3,5,7
/proc/9007/stat : 9007 (python) S 8987 8987 8950 0 -1 4202560 1198 0 0 0 38 31 0 0 15 0 2 0 887731014 298704896 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9007/statm: 72926 6339 416 364 0 8844 0
[pid=9009] ppid=9007 vsize=24920 CPUtime=586.6 cores=1,3,5,7
/proc/9009/stat : 9009 (lingeling) T 9007 9009 9009 0 -1 4202496 59761 0 0 0 58653 7 0 0 25 0 1 0 887731014 25518080 5432 33554432000 4194304 4352236 140734276966864 18446744073709551615 4328126 0 0 16781312 17506 18446744071562691360 0 0 17 1 0 0 0
/proc/9009/statm: 6230 5432 129 39 0 5310 0
[pid=9007/tid=9013] ppid=8987 vsize=291704 CPUtime=0 cores=1,3,5,7
/proc/9007/task/9013/stat : 9013 (python) S 8987 8987 8950 0 -1 4202560 48 0 0 0 0 0 0 0 15 0 2 0 887741024 298704896 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454224044224 0 0 -1 7 0 0 0
[pid=9019] ppid=8987 vsize=291716 CPUtime=0.76 cores=1,3,5,7
/proc/9019/stat : 9019 (python) S 8987 8987 8950 0 -1 4202560 1138 0 0 0 39 37 0 0 18 0 2 0 887750538 298717184 6323 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9019/statm: 72929 6323 400 364 0 8844 0
[pid=9021] ppid=9019 vsize=30096 CPUtime=584.68 cores=1,3,5,7
/proc/9021/stat : 9021 (lingeling) T 9019 9021 9021 0 -1 4202496 61697 0 0 0 58460 8 0 0 25 0 1 0 887750539 30818304 6726 33554432000 4194304 4352236 140735092545136 18446744073709551615 4329089 0 0 16781312 17506 18446744071562691360 0 0 17 1 0 0 0
/proc/9021/statm: 7524 6726 129 39 0 6604 0
[pid=9019/tid=9025] ppid=8987 vsize=291716 CPUtime=0 cores=1,3,5,7
/proc/9019/task/9025/stat : 9025 (python) S 8987 8987 8950 0 -1 4202560 50 0 0 0 0 0 0 0 18 0 2 0 887760549 298717184 6323 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604471145675712 0 0 -1 5 0 0 0
[pid=9043] ppid=8987 vsize=301972 CPUtime=2.61 cores=1,3,5,7
/proc/9043/stat : 9043 (python) S 8987 8987 8950 0 -1 4202560 1166 0 0 0 134 127 0 0 15 0 2 0 887799693 309219328 6328 33554432000 4194304 5683644 140733304753920 18446744073709551615 212039675462 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/9043/statm: 75493 6328 401 364 0 11405 0
[pid=9045] ppid=9043 vsize=256684 CPUtime=1963.76 cores=1,3,5,7
/proc/9045/stat : 9045 (cryptominisat-2) R 9043 9045 9045 0 -1 4202496 405853 0 0 0 196299 77 0 0 25 0 1 0 887799693 262844416 43194 33554432000 4194304 4677096 140734437088000 18446744073709551615 4337908 0 0 16781312 2 0 0 0 17 3 0 0 0
/proc/9045/statm: 64171 43194 393 118 0 59130 0
[pid=9043/tid=9063] ppid=8987 vsize=301972 CPUtime=0 cores=1,3,5,7
/proc/9043/task/9063/stat : 9063 (python) S 8987 8987 8950 0 -1 4202560 44 0 0 0 0 0 0 0 15 0 2 0 887848408 309219328 6328 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454224046912 0 0 -1 1 0 0 0
[pid=9068] ppid=8987 vsize=312228 CPUtime=1.05 cores=1,3,5,7
/proc/9068/stat : 9068 (python) S 8987 8987 8950 0 -1 4202560 1151 0 0 0 57 48 0 0 18 0 2 0 887858122 319721472 6331 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9068/statm: 78057 6331 400 364 0 13966 0
[pid=9070] ppid=9068 vsize=50344 CPUtime=779.93 cores=1,3,5,7
/proc/9070/stat : 9070 (precosat) T 9068 9070 9070 0 -1 4202496 13143 0 0 0 77991 2 0 0 25 0 1 0 887858122 51552256 12396 33554432000 4194304 4890832 140734283503504 18446744073709551615 4238686 0 0 16781312 17442 18446744071562691360 0 0 17 1 0 0 0
/proc/9070/statm: 12586 12396 86 171 0 12413 0
[pid=9068/tid=9076] ppid=8987 vsize=312228 CPUtime=0 cores=1,3,5,7
/proc/9068/task/9076/stat : 9076 (python) S 8987 8987 8950 0 -1 4202560 51 0 0 0 0 0 0 0 18 0 2 0 887877745 319721472 6331 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604471127114176 0 0 -1 3 0 0 0
[pid=9077] ppid=8987 vsize=322484 CPUtime=0.13 cores=1,3,5,7
/proc/9077/stat : 9077 (python) S 8987 8987 8950 0 -1 4202560 1108 0 0 0 7 6 0 0 16 0 2 0 887877747 330223616 6335 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 5 0 0 0
/proc/9077/statm: 80621 6335 400 364 0 16527 0
[pid=9079] ppid=9077 vsize=12740 CPUtime=100.11 cores=1,3,5,7
/proc/9079/stat : 9079 (SApperloT-base) T 9077 9079 9079 0 -1 4202496 3138 0 0 0 10011 0 0 0 25 0 1 0 887877748 13045760 2872 33554432000 134512640 135674940 4292584640 18446744073709551615 134551862 0 0 16781312 0 18446744071562691360 0 0 17 3 0 0 0
/proc/9079/statm: 3185 2872 100 284 0 2899 0
[pid=9077/tid=9083] ppid=8987 vsize=322484 CPUtime=0 cores=1,3,5,7
/proc/9077/task/9083/stat : 9083 (python) S 8987 8987 8950 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 887887760 330223616 6335 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454223471552 0 0 -1 5 0 0 0
[pid=9099] ppid=8987 vsize=332740 CPUtime=0.26 cores=1,3,5,7
/proc/9099/stat : 9099 (python) S 8987 8987 8950 0 -1 4202560 1126 0 0 0 14 12 0 0 18 0 2 0 887945582 340725760 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/9099/statm: 83185 6339 400 364 0 19088 0
[pid=9101] ppid=9099 vsize=21004 CPUtime=197.23 cores=1,3,5,7
/proc/9101/stat : 9101 (lingeling) T 9099 9101 9101 0 -1 4202496 27970 0 0 0 19720 3 0 0 25 0 1 0 887945583 21508096 4421 33554432000 4194304 4352236 140733941391616 18446744073709551615 4328009 0 0 16781312 17506 18446744071562691360 0 0 17 3 0 0 0
/proc/9101/statm: 5251 4421 129 39 0 4331 0
[pid=9099/tid=9105] ppid=8987 vsize=332740 CPUtime=0 cores=1,3,5,7
/proc/9099/task/9105/stat : 9105 (python) S 8987 8987 8950 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 887955595 340725760 6339 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604454223473344 0 0 -1 3 0 0 0
[pid=9125] ppid=8987 vsize=342996 CPUtime=0.8 cores=1,3,5,7
/proc/9125/stat : 9125 (python) S 8987 8987 8950 0 -1 4202560 1167 0 0 0 40 40 0 0 15 0 2 0 887994769 351227904 6366 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 3 0 0 0
/proc/9125/statm: 85749 6366 400 364 0 21649 0
[pid=9127] ppid=9125 vsize=185444 CPUtime=583.04 cores=1,3,5,7
/proc/9127/stat : 9127 (cryptominisat-2) T 9125 9127 9127 0 -1 4202496 170391 0 0 0 58274 30 0 0 25 0 1 0 887994769 189894656 27748 33554432000 4194304 4677096 140734540015088 18446744073709551615 4319156 0 0 16781312 2 18446744071562691360 0 0 17 1 0 0 0
/proc/9127/statm: 46361 27748 393 118 0 41320 0
[pid=9125/tid=9141] ppid=8987 vsize=342996 CPUtime=0 cores=1,3,5,7
/proc/9125/task/9141/stat : 9141 (python) S 8987 8987 8950 0 -1 4202560 49 0 0 0 0 0 0 0 15 0 2 0 888043581 351227904 6366 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604471125182272 0 0 -1 3 0 0 0
[pid=9158] ppid=8987 vsize=353252 CPUtime=0.26 cores=1,3,5,7
/proc/9158/stat : 9158 (python) S 8987 8987 8950 0 -1 4202560 1177 0 0 0 14 12 0 0 19 0 2 0 888092223 361730048 6376 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051482059 0 0 16781312 514 18446744073709551615 0 0 17 1 0 0 0
/proc/9158/statm: 88313 6376 406 364 0 24210 0
[pid=9160] ppid=9158 vsize=29680 CPUtime=196.23 cores=1,3,5,7
/proc/9160/stat : 9160 (precosat) T 9158 9160 9160 0 -1 4202496 7581 0 0 0 19622 1 0 0 25 0 1 0 888092224 30392320 7222 33554432000 4194304 4890832 140736228792480 18446744073709551615 4238599 0 0 16781312 17442 18446744071562691360 0 0 17 3 0 0 0
/proc/9160/statm: 7420 7222 86 171 0 7247 0
[pid=9158/tid=9167] ppid=8987 vsize=353252 CPUtime=0 cores=1,3,5,7
/proc/9158/task/9167/stat : 9167 (python) S 8987 8987 8950 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 888111847 361730048 6376 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 514 18446604471125180480 0 0 -1 1 0 0 0
[pid=8987/tid=9008] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9008/stat : 9008 (python) S 8985 8987 8950 0 -1 4202560 35 927 0 8 0 0 0 0 18 0 9 0 887731014 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
[pid=8987/tid=9020] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9020/stat : 9020 (python) S 8985 8987 8950 0 -1 4202560 36 927 0 8 0 0 0 0 15 0 9 0 887750539 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9044] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9044/stat : 9044 (python) S 8985 8987 8950 0 -1 4202560 44 927 0 8 0 0 0 0 18 0 9 0 887799693 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9069] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9069/stat : 9069 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 9 0 887858122 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 5 0 0 0
[pid=8987/tid=9078] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9078/stat : 9078 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 20 0 9 0 887877748 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9100] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9100/stat : 9100 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 18 0 9 0 887945583 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9126] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9126/stat : 9126 (python) S 8985 8987 8950 0 -1 4202560 29 927 0 8 0 0 0 0 18 0 9 0 887994769 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 1 0 0 0
[pid=8987/tid=9159] ppid=8985 vsize=363496 CPUtime=0 cores=1,3,5,7
/proc/8987/task/9159/stat : 9159 (python) S 8985 8987 8950 0 -1 4202560 31 927 0 8 0 0 0 0 20 0 9 0 888092223 372219904 7864 33554432000 4194304 5683644 140733304753920 18446744073709551615 212051478961 0 0 16781312 2 18446604471145673920 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 5000.07
Current children cumulated vsize (KiB) 3523500

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 8987 and gives
#  childrusage.ru_utime.tv_sec=1
#  childrusage.ru_utime.tv_usec=741735
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=197969
# CPU time returned by wait4() is 1.9397
# while last known CPU time is 5000.07
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5024.26
CPU time (s): 5000.07
CPU user time (s): 4995.47
CPU system time (s): 4.6
CPU usage (%): 99.5186
Max. virtual memory (cumulated for all children) (KiB): 3537808

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 1.74173
system time used= 0.197969
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 19283
page faults= 119
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 16330
involuntary context switches= 79

runsolver used 14.1458 second user time and 31.5432 second system time

The end

Launcher Data 

Begin job on node146 at 2011-06-13 10:18:33
IDJOB=3498254
IDBENCH=20399
IDSOLVER=1991
FILE ID=node146/3498254-1307953113
RUNJOBID= node146-1307953113-8969
PBS_JOBID= 13559682
Free space on /tmp= 6544 MiB

SOLVER NAME= borg-sat 11.03.02-seq-industrial (fixed)
BENCH NAME= SAT07/industrial/fuhs/hard/AProVE07-01.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.industrial.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3498254-1307953113/watcher-3498254-1307953113 -o /tmp/evaluation-result-3498254-1307953113/solver-3498254-1307953113 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3498254-1307953113.cnf -seed 1825549681 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d2f77e32530988f77dca5112174eee00
RANDOM SEED=1825549681

node146.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.86
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:       4128080 kB
Buffers:        124848 kB
Cached:       27773572 kB
SwapCached:      13448 kB
Active:       12760632 kB
Inactive:     15155780 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:       4128080 kB
SwapTotal:    67111528 kB
SwapFree:     67050916 kB
Dirty:           58300 kB
Writeback:           0 kB
AnonPages:       16560 kB
Mapped:           8656 kB
Slab:           842364 kB
PageTables:       4656 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   187140 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 6512 MiB
End job on node146 at 2011-06-13 11:42:22