Trace number 3498460

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
borg-sat 11.03.02-seq-industrial (fixed)? (TO) 5000.03 4993.56

General information on the benchmark

Nameapplication/kullmann/AES/Benchmarks/
128/aes_128_10_keyfind_1.cnf
MD5SUMf35f3aee72237c1bf3d673feb4883756
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables8080
Number of clauses96704
Sum of the clauses size539496
Maximum clause length8
Minimum clause length1
Number of clauses of size 1336
Number of clauses of size 21152
Number of clauses of size 315040
Number of clauses of size 42944
Number of clauses of size 518432
Number of clauses of size over 558800

Solver Data 

0.36/0.68	c INFO: loaded portfolio from HOME/etc/borg-mix+class.industrial.pickle
0.36/0.68	c INFO: solving HOME/instance-3498460-1307954511.cnf
0.36/0.68	c INFO: computing task features
0.94/1.23	c INFO: computed variable-clause graph statistics
0.94/1.24	c INFO: computed clause balance statistics
3.74/4.07	c INFO: computed variable balance statistics
5.04/5.39	c INFO: computed small-clause counts
5.04/5.39	c INFO: computed Horn-clause counts
5.25/5.56	c INFO: computed variable graph statistics
5.25/5.56	c INFO: collected features for HOME/instance-3498460-1307954511.cnf in 4.87 s
5.25/5.59	c INFO: running lingeling-276@0 for 97 with 4995 remaining (b = 0.26)
105.55/105.72	c INFO: running lingeling-276@100 for 97 with 4895 remaining (b = 0.13)
200.75/200.83	c INFO: running SApperloT@0 for 97 with 4799 remaining (b = 0.10)
300.95/300.98	c INFO: running lingeling-276@195 for 1067 with 4699 remaining (b = 0.45)
1372.34/1371.07	c INFO: running cryptominisat-2.9.0@0 for 1649 with 3629 remaining (b = 0.59)
3029.67/3025.88	c INFO: running lingeling-276@0 for 485 with 1975 remaining (b = 0.38)
3520.86/3516.42	c INFO: running lingeling-276@0 for 97 with 1485 remaining (b = 0.11)
3620.06/3615.55	c INFO: running lingeling-276@490 for 97 with 1386 remaining (b = 0.05)
3717.27/3712.68	c INFO: running lingeling-276@99 for 194 with 1289 remaining (b = 0.12)
3909.86/3904.92	c INFO: running cryptominisat-2.9.0@0 for 485 with 1097 remaining (b = 0.27)
4396.51/4390.88	c INFO: running precosat-570@0 for 291 with 611 remaining (b = 0.17)
4689.35/4683.29	c INFO: running precosat-570@292 for 97 with 319 remaining (b = 0.05)
4790.71/4784.43	c INFO: running lingeling-276@0 for 97 with 218 remaining (b = 0.05)
4890.84/4884.56	c INFO: running cryptominisat-2.9.0@485 for 118 with 118 remaining (b = 0.04)

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.1 (svn: 989)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /var/tmp/evaluation-result-3498460-1307954511/watcher-3498460-1307954511 -o /var/tmp/evaluation-result-3498460-1307954511/solver-3498460-1307954511 -C 5000 -W 5100 -M 15500 python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3498460-1307954511.cnf -seed 145861740 -budget 5000 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 5030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 2.00 1.98 1.53 3/192 15889
/proc/meminfo: memFree=15998412/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=15184 CPUtime=0 cores=0,2,4,6
/proc/15889/stat : 15889 (python) R 15887 15889 15646 0 -1 4202496 708 0 0 0 0 0 0 0 25 0 1 0 985681799 15548416 599 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895184350 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15889/statm: 3796 603 319 364 0 283 0

[startup+0.0265561 s]
/proc/loadavg: 2.00 1.98 1.53 3/192 15889
/proc/meminfo: memFree=15998412/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=16396 CPUtime=0.01 cores=0,2,4,6
/proc/15889/stat : 15889 (python) R 15887 15889 15646 0 -1 4202496 1080 0 0 0 1 0 0 0 25 0 1 0 985681799 16789504 950 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895488512 0 2147483391 16781312 2 0 0 0 17 4 0 0 0
/proc/15889/statm: 4099 950 385 364 0 586 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 16396

[startup+0.100547 s]
/proc/loadavg: 2.00 1.98 1.53 3/192 15889
/proc/meminfo: memFree=15998412/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=85360 CPUtime=0.06 cores=0,2,4,6
/proc/15889/stat : 15889 (python) R 15887 15889 15646 0 -1 4202496 2569 0 0 0 5 1 0 0 18 0 1 0 985681799 87408640 2295 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895187211 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15889/statm: 21340 2300 707 364 0 1557 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 85360

[startup+0.300468 s]
/proc/loadavg: 2.00 1.98 1.53 3/192 15889
/proc/meminfo: memFree=15998412/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=135748 CPUtime=0.19 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 5971 934 1 0 14 5 0 0 17 0 1 0 985681799 139005952 4149 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895488512 0 2147483391 16781312 2 18446744071699215550 0 0 17 4 0 0 0
/proc/15889/statm: 33937 4149 1032 364 0 3123 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 135748

[startup+0.700453 s]
/proc/loadavg: 2.00 1.98 1.53 3/192 15889
/proc/meminfo: memFree=15998412/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=282252 CPUtime=0.44 cores=0,2,4,6
/proc/15889/stat : 15889 (python) R 15887 15889 15646 0 -1 4202496 10193 934 1 0 32 12 0 0 16 0 1 0 985681799 289026048 7866 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895187575 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15889/statm: 70612 7888 1761 364 0 6536 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 282252

[startup+1.5003 s]
/proc/loadavg: 2.00 1.98 1.53 3/193 15894
/proc/meminfo: memFree=15954284/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=313964 CPUtime=1.24 cores=0,2,4,6
/proc/15889/stat : 15889 (python) R 15887 15889 15646 0 -1 4202496 22642 934 1 0 111 13 0 0 17 0 1 0 985681799 321499136 15876 33554432000 4194304 5683644 140734998173568 18446744073709551615 47839562566387 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15889/statm: 78491 15876 1826 364 0 14415 0
Current children cumulated CPU time (s) 1.24
Current children cumulated vsize (KiB) 313964

[startup+3.101 s]
/proc/loadavg: 2.00 1.98 1.53 3/193 15894
/proc/meminfo: memFree=15941016/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=313964 CPUtime=2.84 cores=0,2,4,6
/proc/15889/stat : 15889 (python) R 15887 15889 15646 0 -1 4202496 22642 934 1 0 271 13 0 0 19 0 1 0 985681799 321499136 15876 33554432000 4194304 5683644 140734998173568 18446744073709551615 47839562568008 0 0 16781312 2 0 0 0 17 4 0 0 0
/proc/15889/statm: 78491 15876 1826 364 0 14415 0
Current children cumulated CPU time (s) 2.84
Current children cumulated vsize (KiB) 313964

[startup+6.30034 s]
/proc/loadavg: 2.00 1.98 1.53 3/193 15894
/proc/meminfo: memFree=15941388/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=321596 CPUtime=5.34 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 23012 934 1 0 520 14 0 0 25 0 2 0 985681799 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15889/statm: 80399 15262 1867 364 0 16317 0
Current children cumulated CPU time (s) 5.34
Current children cumulated vsize (KiB) 321596

[startup+12.7012 s]
/proc/loadavg: 2.00 1.98 1.54 3/196 15898
/proc/meminfo: memFree=15928128/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=321596 CPUtime=5.34 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 23012 934 1 0 520 14 0 0 25 0 2 0 985681799 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15889/statm: 80399 15262 1867 364 0 16317 0
[pid=15895] ppid=15889 vsize=311352 CPUtime=0 cores=0,2,4,6
/proc/15895/stat : 15895 (python) S 15889 15889 15646 0 -1 4202560 970 0 0 0 0 0 0 0 18 0 1 0 985682358 318824448 13772 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510543 0 0 16781312 514 4177801417093775619 0 0 17 4 0 0 0
/proc/15895/statm: 77838 13772 381 364 0 13756 0
[pid=15897] ppid=15895 vsize=18120 CPUtime=7.1 cores=0,2,4,6
/proc/15897/stat : 15897 (lingeling) R 15895 15897 15897 0 -1 4202496 4827 0 0 0 709 1 0 0 25 0 1 0 985682358 18554880 3698 33554432000 4194304 4352236 140736129422160 18446744073709551615 4328009 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/15897/statm: 4530 3698 129 39 0 3610 0
[pid=15889/tid=15896] ppid=15887 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 29 934 0 0 0 0 0 0 25 0 2 0 985682358 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 12.44
Current children cumulated vsize (KiB) 651068

[startup+25.5008 s]
/proc/loadavg: 2.00 1.98 1.54 3/196 15898
/proc/meminfo: memFree=15926972/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=321596 CPUtime=5.34 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 23012 934 1 0 520 14 0 0 25 0 2 0 985681799 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15889/statm: 80399 15262 1867 364 0 16317 0
[pid=15895] ppid=15889 vsize=311352 CPUtime=0.02 cores=0,2,4,6
/proc/15895/stat : 15895 (python) S 15889 15889 15646 0 -1 4202560 978 0 0 0 1 1 0 0 18 0 1 0 985682358 318824448 13772 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510543 0 0 16781312 514 9290433450056057091 0 0 17 4 0 0 0
/proc/15895/statm: 77838 13772 381 364 0 13756 0
[pid=15897] ppid=15895 vsize=20040 CPUtime=19.89 cores=0,2,4,6
/proc/15897/stat : 15897 (lingeling) R 15895 15897 15897 0 -1 4202496 5307 0 0 0 1988 1 0 0 25 0 1 0 985682358 20520960 4178 33554432000 4194304 4352236 140736129422160 18446744073709551615 4278835 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/15897/statm: 5010 4178 129 39 0 4090 0
[pid=15889/tid=15896] ppid=15887 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 29 934 0 0 0 0 0 0 25 0 2 0 985682358 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 25.25
Current children cumulated vsize (KiB) 652988

[startup+51.101 s]
/proc/loadavg: 2.00 1.98 1.55 3/196 15899
/proc/meminfo: memFree=15923776/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=321596 CPUtime=5.34 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 23012 934 1 0 520 14 0 0 25 0 2 0 985681799 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15889/statm: 80399 15262 1867 364 0 16317 0
[pid=15895] ppid=15889 vsize=311352 CPUtime=0.05 cores=0,2,4,6
/proc/15895/stat : 15895 (python) S 15889 15889 15646 0 -1 4202560 988 0 0 0 3 2 0 0 18 0 1 0 985682358 318824448 13772 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510543 0 0 16781312 514 9290433450056057091 0 0 17 4 0 0 0
/proc/15895/statm: 77838 13772 381 364 0 13756 0
[pid=15897] ppid=15895 vsize=31064 CPUtime=45.49 cores=0,2,4,6
/proc/15897/stat : 15897 (lingeling) R 15895 15897 15897 0 -1 4202496 12159 0 0 0 4547 2 0 0 25 0 1 0 985682358 31809536 6934 33554432000 4194304 4352236 140736129422160 18446744073709551615 4234523 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/15897/statm: 7766 6934 129 39 0 6846 0
[pid=15889/tid=15896] ppid=15887 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 29 934 0 0 0 0 0 0 25 0 2 0 985682358 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 50.88
Current children cumulated vsize (KiB) 664012

[startup+102.307 s]
/proc/loadavg: 2.00 1.98 1.57 3/196 15900
/proc/meminfo: memFree=15889132/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=321596 CPUtime=5.34 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 23012 934 1 0 520 14 0 0 25 0 2 0 985681799 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15889/statm: 80399 15262 1867 364 0 16317 0
[pid=15895] ppid=15889 vsize=311352 CPUtime=0.1 cores=0,2,4,6
/proc/15895/stat : 15895 (python) S 15889 15889 15646 0 -1 4202560 1012 0 0 0 6 4 0 0 18 0 1 0 985682358 318824448 13772 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510543 0 0 16781312 514 9290433450056057091 0 0 17 4 0 0 0
/proc/15895/statm: 77838 13772 381 364 0 13756 0
[pid=15897] ppid=15895 vsize=32604 CPUtime=96.7 cores=0,2,4,6
/proc/15897/stat : 15897 (lingeling) R 15895 15897 15897 0 -1 4202496 19097 0 0 0 9667 3 0 0 25 0 1 0 985682358 33386496 7344 33554432000 4194304 4352236 140736129422160 18446744073709551615 4328026 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/15897/statm: 8151 7344 129 39 0 7231 0
[pid=15889/tid=15896] ppid=15887 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 29 934 0 0 0 0 0 0 25 0 2 0 985682358 329314304 15262 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 102.14
Current children cumulated vsize (KiB) 665552

[startup+162.301 s]
/proc/loadavg: 2.00 1.98 1.60 3/198 15904
/proc/meminfo: memFree=15874188/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=321596 CPUtime=5.36 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 23512 934 1 0 522 14 0 0 18 0 2 0 985681799 329314304 15271 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15889/statm: 80399 15271 1876 364 0 16317 0
[pid=15895] ppid=15889 vsize=321596 CPUtime=0.17 cores=0,2,4,6
/proc/15895/stat : 15895 (python) S 15889 15889 15646 0 -1 4202560 1196 0 0 0 9 8 0 0 18 0 2 0 985682358 329314304 13811 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510598 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15895/statm: 80399 13811 415 364 0 16317 0
[pid=15897] ppid=15895 vsize=33892 CPUtime=156.68 cores=0,2,4,6
/proc/15897/stat : 15897 (lingeling) R 15895 15897 15897 0 -1 4202496 24987 0 0 0 15664 4 0 0 25 0 1 0 985682358 34705408 7666 33554432000 4194304 4352236 140736129422160 18446744073709551615 4231179 0 0 16781312 17506 0 0 0 17 0 0 0 0
/proc/15897/statm: 8473 7666 129 39 0 7553 0
[pid=15895/tid=15901] ppid=15889 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15895/task/15901/stat : 15901 (python) S 15889 15889 15646 0 -1 4202560 47 0 0 0 0 0 0 0 19 0 2 0 985692369 329314304 13811 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453885413440 0 0 -1 0 0 0 0
[pid=15889/tid=15896] ppid=15887 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 2 0 985682358 329314304 15271 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 162.21
Current children cumulated vsize (KiB) 677084

[startup+222.301 s]
/proc/loadavg: 2.06 2.00 1.63 3/201 15908
/proc/meminfo: memFree=15817440/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=331852 CPUtime=5.38 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 23898 934 1 0 524 14 0 0 20 0 3 0 985681799 339816448 15277 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 6 0 0 0
/proc/15889/statm: 82963 15277 1878 364 0 18878 0
[pid=15895] ppid=15889 vsize=321596 CPUtime=0.23 cores=0,2,4,6
/proc/15895/stat : 15895 (python) S 15889 15889 15646 0 -1 4202560 1210 0 0 0 13 10 0 0 18 0 2 0 985682358 329314304 13811 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15895/statm: 80399 13811 415 364 0 16317 0
[pid=15897] ppid=15895 vsize=35176 CPUtime=195.19 cores=0,2,4,6
/proc/15897/stat : 15897 (lingeling) T 15895 15897 15897 0 -1 4202496 26588 0 0 0 19515 4 0 0 25 0 1 0 985682358 36020224 7987 33554432000 4194304 4352236 140736129422160 18446744073709551615 4329043 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/15897/statm: 8794 7987 129 39 0 7874 0
[pid=15895/tid=15901] ppid=15889 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15895/task/15901/stat : 15901 (python) S 15889 15889 15646 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 985692369 329314304 13811 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453885413440 0 0 -1 0 0 0 0
[pid=15906] ppid=15889 vsize=321608 CPUtime=0.02 cores=0,2,4,6
/proc/15906/stat : 15906 (python) S 15889 15889 15646 0 -1 4202560 900 0 0 0 1 1 0 0 18 0 1 0 985701883 329326592 13765 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510598 0 0 16781312 514 9290433450056057092 0 0 17 4 0 0 0
/proc/15906/statm: 80402 13765 370 364 0 16317 0
[pid=15908] ppid=15906 vsize=31712 CPUtime=21.44 cores=0,2,4,6
/proc/15908/stat : 15908 (SApperloT-base) R 15906 15908 15908 0 -1 4202496 7727 0 0 0 2143 1 0 0 25 0 1 0 985701883 32473088 7363 33554432000 134512640 135674940 4289044832 18446744073709551615 134564608 0 0 16781312 0 0 0 0 17 0 0 0 0
/proc/15908/statm: 7928 7363 100 284 0 7642 0
[pid=15889/tid=15896] ppid=15887 vsize=331852 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 29 934 0 0 0 0 0 0 18 0 3 0 985682358 339816448 15277 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/15952/stat : 15952 (cryptominisat-2) T 15950 15952 15952 0 -1 4202496 490657 0 0 0 165384 100 0 0 25 0 1 0 985818907 291262464 62273 33554432000 4194304 4677096 140735501646864 18446744073709551615 4320199 0 0 16781312 2 18446744071562691360 0 0 17 4 0 0 0
/proc/15952/statm: 71109 62273 392 118 0 66068 0
[pid=15950/tid=16097] ppid=15889 vsize=331864 CPUtime=0 cores=0,2,4,6
/proc/15950/task/16097/stat : 16097 (python) S 15889 15889 15646 0 -1 4202560 43 0 0 0 0 0 0 0 17 0 2 0 985984386 339828736 13800 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453849887744 0 0 -1 0 0 0 0
[pid=16098] ppid=15889 vsize=342120 CPUtime=0.72 cores=0,2,4,6
/proc/16098/stat : 16098 (python) S 15889 15889 15646 0 -1 4202560 1152 0 0 0 41 31 0 0 18 0 2 0 985984387 350330880 13803 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16098/statm: 85530 13803 399 364 0 21439 0
[pid=16100] ppid=16098 vsize=39832 CPUtime=587.64 cores=0,2,4,6
/proc/16100/stat : 16100 (lingeling) T 16098 16100 16100 0 -1 4202496 71607 0 0 0 58755 9 0 0 25 0 1 0 985984388 40787968 9158 33554432000 4194304 4352236 140734415895552 18446744073709551615 4329043 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/16100/statm: 9958 9158 129 39 0 9038 0
[pid=16098/tid=16117] ppid=15889 vsize=342120 CPUtime=0 cores=0,2,4,6
/proc/16098/task/16117/stat : 16117 (python) S 15889 15889 15646 0 -1 4202560 43 0 0 0 0 0 0 0 18 0 2 0 986033440 350330880 13803 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453871348480 0 0 -1 2 0 0 0
[pid=16118] ppid=15889 vsize=352376 CPUtime=0.39 cores=0,2,4,6
/proc/16118/stat : 16118 (python) S 15889 15889 15646 0 -1 4202560 1152 0 0 0 23 16 0 0 18 0 2 0 986033441 360833024 13807 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/16118/statm: 88094 13807 399 364 0 24000 0
[pid=16120] ppid=16118 vsize=28332 CPUtime=291.35 cores=0,2,4,6
/proc/16120/stat : 16120 (lingeling) T 16118 16120 16120 0 -1 4202496 39379 0 0 0 29129 6 0 0 25 0 1 0 986033441 29011968 6251 33554432000 4194304 4352236 140737481663344 18446744073709551615 4328114 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/16120/statm: 7083 6251 129 39 0 6163 0
[pid=16118/tid=16127] ppid=15889 vsize=352376 CPUtime=0 cores=0,2,4,6
/proc/16118/task/16127/stat : 16127 (python) S 15889 15889 15646 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 986043353 360833024 13807 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453867217152 0 0 -1 6 0 0 0
[pid=16139] ppid=15889 vsize=362632 CPUtime=0.93 cores=0,2,4,6
/proc/16139/stat : 16139 (python) R 15889 15889 15646 0 -1 4202560 1187 0 0 0 55 38 0 0 15 0 2 0 986072291 371335168 13812 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510598 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/16139/statm: 90658 13812 400 364 0 26561 0
[pid=16141] ppid=16139 vsize=165480 CPUtime=594.89 cores=0,2,4,6
/proc/16141/stat : 16141 (cryptominisat-2) R 16139 16141 16141 0 -1 4202496 225961 0 0 0 59441 48 0 0 25 0 1 0 986072292 169451520 34625 33554432000 4194304 4677096 140736468338576 18446744073709551615 4318809 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/16141/statm: 41370 34625 392 118 0 36329 0
[pid=16139/tid=16160] ppid=15889 vsize=362632 CPUtime=0 cores=0,2,4,6
/proc/16139/task/16160/stat : 16160 (python) S 15889 15889 15646 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 986120887 371335168 13812 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453979627648 0 0 -1 6 0 0 0
[pid=16161] ppid=15889 vsize=372888 CPUtime=0.58 cores=0,2,4,6
/proc/16161/stat : 16161 (python) S 15889 15889 15646 0 -1 4202560 1144 0 0 0 32 26 0 0 18 0 2 0 986120887 381837312 13815 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/16161/statm: 93222 13815 399 364 0 29122 0
[pid=16163] ppid=16161 vsize=59408 CPUtime=393.52 cores=0,2,4,6
/proc/16163/stat : 16163 (precosat) T 16161 16163 16163 0 -1 4202496 19320 0 0 0 39348 4 0 0 25 0 1 0 986120888 60833792 14686 33554432000 4194304 4890832 140736775953728 18446744073709551615 4238599 0 0 16781312 17442 18446744071562691360 0 0 17 4 0 0 0
/proc/16163/statm: 14852 14686 86 171 0 14679 0
[pid=16161/tid=16172] ppid=15889 vsize=372888 CPUtime=0 cores=0,2,4,6
/proc/16161/task/16172/stat : 16172 (python) S 15889 15889 15646 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 986150127 381837312 13815 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453973771200 0 0 -1 4 0 0 0
[pid=16185] ppid=15889 vsize=383144 CPUtime=0.13 cores=0,2,4,6
/proc/16185/stat : 16185 (python) S 15889 15889 15646 0 -1 4202560 1156 0 0 0 7 6 0 0 19 0 2 0 986160242 392339456 13825 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/16185/statm: 95786 13825 405 364 0 31683 0
[pid=16187] ppid=16185 vsize=25628 CPUtime=100.04 cores=0,2,4,6
/proc/16187/stat : 16187 (lingeling) T 16185 16187 16187 0 -1 4202496 24267 0 0 0 10002 2 0 0 25 0 1 0 986160242 26243072 5599 33554432000 4194304 4352236 140734562358000 18446744073709551615 4328194 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/16187/statm: 6407 5599 129 39 0 5487 0
[pid=16185/tid=16191] ppid=15889 vsize=383144 CPUtime=0 cores=0,2,4,6
/proc/16185/task/16191/stat : 16191 (python) S 15889 15889 15646 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 986170255 392339456 13825 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453871346688 0 0 -1 0 0 0 0
[pid=15889/tid=15896] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 9 0 985682358 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
[pid=15889/tid=15907] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15907/stat : 15907 (python) S 15887 15889 15646 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 9 0 985701883 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 6 0 0 0
[pid=15889/tid=15951] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15951/stat : 15951 (python) S 15887 15889 15646 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 9 0 985818906 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 6 0 0 0
[pid=15889/tid=16099] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16099/stat : 16099 (python) S 15887 15889 15646 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 9 0 985984388 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
[pid=15889/tid=16119] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16119/stat : 16119 (python) S 15887 15889 15646 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 9 0 986033441 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 0 0 0 0
[pid=15889/tid=16140] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16140/stat : 16140 (python) S 15887 15889 15646 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 9 0 986072291 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 0 0 0 0
[pid=15889/tid=16162] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16162/stat : 16162 (python) S 15887 15889 15646 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 9 0 986120888 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 0 0 0 0
[pid=15889/tid=16186] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16186/stat : 16186 (python) S 15887 15889 15646 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 9 0 986160242 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5000.03
Current children cumulated vsize (KiB) 3903608

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+4993.5 s]
/proc/loadavg: 2.06 2.01 1.97 3/230 16193
/proc/meminfo: memFree=22802896/32950928 swapFree=67054844/67111528
[pid=15889] ppid=15887 vsize=393388 CPUtime=5.51 cores=0,2,4,6
/proc/15889/stat : 15889 (python) S 15887 15889 15646 0 -1 4202496 29498 934 1 0 535 16 0 0 18 0 9 0 985681799 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 2 18446744073709551615 0 0 17 4 0 0 0
/proc/15889/statm: 98347 15313 1890 364 0 34244 0
[pid=15895] ppid=15889 vsize=321596 CPUtime=1.48 cores=0,2,4,6
/proc/15895/stat : 15895 (python) S 15889 15889 15646 0 -1 4202560 1215 0 0 0 82 66 0 0 18 0 2 0 985682358 329314304 13811 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15895/statm: 80399 13811 415 364 0 16317 0
[pid=15897] ppid=15895 vsize=56176 CPUtime=1265.29 cores=0,2,4,6
/proc/15897/stat : 15897 (lingeling) T 15895 15897 15897 0 -1 4202496 103557 0 0 0 126513 16 0 0 25 0 1 0 985682358 57524224 13212 33554432000 4194304 4352236 140736129422160 18446744073709551615 4329060 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/15897/statm: 14044 13212 129 39 0 13124 0
[pid=15895/tid=15901] ppid=15889 vsize=321596 CPUtime=0 cores=0,2,4,6
/proc/15895/task/15901/stat : 15901 (python) S 15889 15889 15646 0 -1 4202560 47 0 0 0 0 0 0 0 18 0 2 0 985692369 329314304 13811 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453885413440 0 0 -1 0 0 0 0
[pid=15906] ppid=15889 vsize=321608 CPUtime=0.12 cores=0,2,4,6
/proc/15906/stat : 15906 (python) S 15889 15889 15646 0 -1 4202560 1090 0 0 0 7 5 0 0 19 0 2 0 985701883 329326592 13795 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/15906/statm: 80402 13795 399 364 0 16317 0
[pid=15908] ppid=15906 vsize=62700 CPUtime=100.11 cores=0,2,4,6
/proc/15908/stat : 15908 (SApperloT-base) T 15906 15908 15908 0 -1 4202496 15158 0 0 0 10008 3 0 0 25 0 1 0 985701883 64204800 14794 33554432000 134512640 135674940 4289044832 18446744073709551615 134559192 0 0 16781312 0 18446744071562691360 0 0 17 0 0 0 0
/proc/15908/statm: 15675 14794 100 284 0 15389 0
[pid=15906/tid=15911] ppid=15889 vsize=321608 CPUtime=0 cores=0,2,4,6
/proc/15906/task/15911/stat : 15911 (python) S 15889 15889 15646 0 -1 4202560 46 0 0 0 0 0 0 0 19 0 2 0 985711895 329326592 13795 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453840870080 0 0 -1 4 0 0 0
[pid=15950] ppid=15889 vsize=331864 CPUtime=2.49 cores=0,2,4,6
/proc/15950/stat : 15950 (python) S 15889 15889 15646 0 -1 4202560 1235 0 0 0 132 117 0 0 17 0 2 0 985818906 339828736 13800 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/15950/statm: 82966 13800 400 364 0 18878 0
[pid=15952] ppid=15950 vsize=284436 CPUtime=1654.84 cores=0,2,4,6
/proc/15952/stat : 15952 (cryptominisat-2) T 15950 15952 15952 0 -1 4202496 490657 0 0 0 165384 100 0 0 25 0 1 0 985818907 291262464 62273 33554432000 4194304 4677096 140735501646864 18446744073709551615 4320199 0 0 16781312 2 18446744071562691360 0 0 17 4 0 0 0
/proc/15952/statm: 71109 62273 392 118 0 66068 0
[pid=15950/tid=16097] ppid=15889 vsize=331864 CPUtime=0 cores=0,2,4,6
/proc/15950/task/16097/stat : 16097 (python) S 15889 15889 15646 0 -1 4202560 43 0 0 0 0 0 0 0 17 0 2 0 985984386 339828736 13800 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453849887744 0 0 -1 0 0 0 0
[pid=16098] ppid=15889 vsize=342120 CPUtime=0.72 cores=0,2,4,6
/proc/16098/stat : 16098 (python) S 15889 15889 15646 0 -1 4202560 1152 0 0 0 41 31 0 0 18 0 2 0 985984387 350330880 13803 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 2 0 0 0
/proc/16098/statm: 85530 13803 399 364 0 21439 0
[pid=16100] ppid=16098 vsize=39832 CPUtime=587.64 cores=0,2,4,6
/proc/16100/stat : 16100 (lingeling) T 16098 16100 16100 0 -1 4202496 71607 0 0 0 58755 9 0 0 25 0 1 0 985984388 40787968 9158 33554432000 4194304 4352236 140734415895552 18446744073709551615 4329043 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/16100/statm: 9958 9158 129 39 0 9038 0
[pid=16098/tid=16117] ppid=15889 vsize=342120 CPUtime=0 cores=0,2,4,6
/proc/16098/task/16117/stat : 16117 (python) S 15889 15889 15646 0 -1 4202560 43 0 0 0 0 0 0 0 18 0 2 0 986033440 350330880 13803 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453871348480 0 0 -1 2 0 0 0
[pid=16118] ppid=15889 vsize=352376 CPUtime=0.39 cores=0,2,4,6
/proc/16118/stat : 16118 (python) S 15889 15889 15646 0 -1 4202560 1152 0 0 0 23 16 0 0 18 0 2 0 986033441 360833024 13807 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/16118/statm: 88094 13807 399 364 0 24000 0
[pid=16120] ppid=16118 vsize=28332 CPUtime=291.35 cores=0,2,4,6
/proc/16120/stat : 16120 (lingeling) T 16118 16120 16120 0 -1 4202496 39379 0 0 0 29129 6 0 0 25 0 1 0 986033441 29011968 6251 33554432000 4194304 4352236 140737481663344 18446744073709551615 4328114 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/16120/statm: 7083 6251 129 39 0 6163 0
[pid=16118/tid=16127] ppid=15889 vsize=352376 CPUtime=0 cores=0,2,4,6
/proc/16118/task/16127/stat : 16127 (python) S 15889 15889 15646 0 -1 4202560 48 0 0 0 0 0 0 0 18 0 2 0 986043353 360833024 13807 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453867217152 0 0 -1 6 0 0 0
[pid=16139] ppid=15889 vsize=362632 CPUtime=0.93 cores=0,2,4,6
/proc/16139/stat : 16139 (python) R 15889 15889 15646 0 -1 4202560 1187 0 0 0 55 38 0 0 15 0 2 0 986072291 371335168 13812 33554432000 4194304 5683644 140734998173568 18446744073709551615 231895510598 0 0 16781312 514 18446744073709551615 0 0 17 6 0 0 0
/proc/16139/statm: 90658 13812 400 364 0 26561 0
[pid=16141] ppid=16139 vsize=165480 CPUtime=594.89 cores=0,2,4,6
/proc/16141/stat : 16141 (cryptominisat-2) R 16139 16141 16141 0 -1 4202496 225961 0 0 0 59441 48 0 0 25 0 1 0 986072292 169451520 34625 33554432000 4194304 4677096 140736468338576 18446744073709551615 4318809 0 0 16781312 2 0 0 0 17 0 0 0 0
/proc/16141/statm: 41370 34625 392 118 0 36329 0
[pid=16139/tid=16160] ppid=15889 vsize=362632 CPUtime=0 cores=0,2,4,6
/proc/16139/task/16160/stat : 16160 (python) S 15889 15889 15646 0 -1 4202560 47 0 0 0 0 0 0 0 16 0 2 0 986120887 371335168 13812 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453979627648 0 0 -1 6 0 0 0
[pid=16161] ppid=15889 vsize=372888 CPUtime=0.58 cores=0,2,4,6
/proc/16161/stat : 16161 (python) S 15889 15889 15646 0 -1 4202560 1144 0 0 0 32 26 0 0 18 0 2 0 986120887 381837312 13815 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 0 0 0 0
/proc/16161/statm: 93222 13815 399 364 0 29122 0
[pid=16163] ppid=16161 vsize=59408 CPUtime=393.52 cores=0,2,4,6
/proc/16163/stat : 16163 (precosat) T 16161 16163 16163 0 -1 4202496 19320 0 0 0 39348 4 0 0 25 0 1 0 986120888 60833792 14686 33554432000 4194304 4890832 140736775953728 18446744073709551615 4238599 0 0 16781312 17442 18446744071562691360 0 0 17 4 0 0 0
/proc/16163/statm: 14852 14686 86 171 0 14679 0
[pid=16161/tid=16172] ppid=15889 vsize=372888 CPUtime=0 cores=0,2,4,6
/proc/16161/task/16172/stat : 16172 (python) S 15889 15889 15646 0 -1 4202560 46 0 0 0 0 0 0 0 18 0 2 0 986150127 381837312 13815 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453973771200 0 0 -1 4 0 0 0
[pid=16185] ppid=15889 vsize=383144 CPUtime=0.13 cores=0,2,4,6
/proc/16185/stat : 16185 (python) S 15889 15889 15646 0 -1 4202560 1156 0 0 0 7 6 0 0 19 0 2 0 986160242 392339456 13825 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907317195 0 0 16781312 514 18446744073709551615 0 0 17 4 0 0 0
/proc/16185/statm: 95786 13825 405 364 0 31683 0
[pid=16187] ppid=16185 vsize=25628 CPUtime=100.04 cores=0,2,4,6
/proc/16187/stat : 16187 (lingeling) T 16185 16187 16187 0 -1 4202496 24267 0 0 0 10002 2 0 0 25 0 1 0 986160242 26243072 5599 33554432000 4194304 4352236 140734562358000 18446744073709551615 4328194 0 0 16781312 17506 18446744071562691360 0 0 17 0 0 0 0
/proc/16187/statm: 6407 5599 129 39 0 5487 0
[pid=16185/tid=16191] ppid=15889 vsize=383144 CPUtime=0 cores=0,2,4,6
/proc/16185/task/16191/stat : 16191 (python) S 15889 15889 15646 0 -1 4202560 49 0 0 0 0 0 0 0 19 0 2 0 986170255 392339456 13825 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 514 18446604453871346688 0 0 -1 0 0 0 0
[pid=15889/tid=15896] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15896/stat : 15896 (python) S 15887 15889 15646 0 -1 4202560 33 934 0 0 0 0 0 0 18 0 9 0 985682358 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
[pid=15889/tid=15907] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15907/stat : 15907 (python) S 15887 15889 15646 0 -1 4202560 28 934 0 0 0 0 0 0 20 0 9 0 985701883 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 6 0 0 0
[pid=15889/tid=15951] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/15951/stat : 15951 (python) S 15887 15889 15646 0 -1 4202560 32 934 0 0 0 0 0 0 20 0 9 0 985818906 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 6 0 0 0
[pid=15889/tid=16099] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16099/stat : 16099 (python) S 15887 15889 15646 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 9 0 985984388 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
[pid=15889/tid=16119] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16119/stat : 16119 (python) S 15887 15889 15646 0 -1 4202560 30 934 0 0 0 0 0 0 18 0 9 0 986033441 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 0 0 0 0
[pid=15889/tid=16140] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16140/stat : 16140 (python) S 15887 15889 15646 0 -1 4202560 35 934 0 0 0 0 0 0 18 0 9 0 986072291 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 0 0 0 0
[pid=15889/tid=16162] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16162/stat : 16162 (python) S 15887 15889 15646 0 -1 4202560 28 934 0 0 0 0 0 0 18 0 9 0 986120888 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 0 0 0 0
[pid=15889/tid=16186] ppid=15887 vsize=393388 CPUtime=0 cores=0,2,4,6
/proc/15889/task/16186/stat : 16186 (python) S 15887 15889 15646 0 -1 4202560 27 934 0 0 0 0 0 0 20 0 9 0 986160242 402829312 15313 33554432000 4194304 5683644 140734998173568 18446744073709551615 231907314097 0 0 16781312 2 18446604453853488064 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5000.03
Current children cumulated vsize (KiB) 3903608

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 15889 and gives
#  childrusage.ru_utime.tv_sec=5
#  childrusage.ru_utime.tv_usec=356185
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=173973
# CPU time returned by wait4() is 5.53016
# while last known CPU time is 5000.03
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 4993.56
CPU time (s): 5000.03
CPU user time (s): 4994.94
CPU system time (s): 5.09
CPU usage (%): 100.13
Max. virtual memory (cumulated for all children) (KiB): 3925140

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5.35618
system time used= 0.173973
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 30436
page faults= 1
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2929
involuntary context switches= 30

runsolver used 11.0523 second user time and 21.8957 second system time

The end

Launcher Data 

Begin job on node114 at 2011-06-13 10:41:51
IDJOB=3498460
IDBENCH=83413
IDSOLVER=1991
FILE ID=node114/3498460-1307954511
RUNJOBID= node114-1307953126-15664
PBS_JOBID= 13559714
Free space on /var/tmp= 2836 MiB

SOLVER NAME= borg-sat 11.03.02-seq-industrial (fixed)
BENCH NAME= SAT11/application/kullmann/AES/Benchmarks/128/aes_128_10_keyfind_1.cnf
COMMAND LINE= python DIR/solve DIR/etc/borg-mix+class.industrial.pickle BENCHNAME -seed RANDOMSEED -budget TIMEOUT
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /var/tmp/evaluation-result-3498460-1307954511/watcher-3498460-1307954511 -o /var/tmp/evaluation-result-3498460-1307954511/solver-3498460-1307954511 -C 5000 -W 5100 -M 15500  python HOME/solve HOME/etc/borg-mix+class.industrial.pickle HOME/instance-3498460-1307954511.cnf -seed 145861740 -budget 5000

TIME LIMIT= 5000 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f35f3aee72237c1bf3d673feb4883756
RANDOM SEED=145861740

node114.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      15998840 kB
Buffers:         28940 kB
Cached:        8006456 kB
SwapCached:      10332 kB
Active:        8627960 kB
Inactive:      7934144 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      15998840 kB
SwapTotal:    67111528 kB
SwapFree:     67054844 kB
Dirty:           34612 kB
Writeback:           4 kB
AnonPages:     8524604 kB
Mapped:          19164 kB
Slab:           299320 kB
PageTables:      26896 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 10334672 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /var/tmp at the end= 2796 MiB
End job on node114 at 2011-06-13 12:05:07