SAT 2009 Competition: preprocessors track: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "applications instances" (APPLICATION): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances
Detail % of all instances% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)16367 SAT, 96 UNSAT56%100%33886.97207.90
1kw_pre2009-03-2114958 SAT, 91 UNSAT51%91%34591.65232.16
2ReVivAl 0.23 + SatElite2009-03-1812151 SAT, 70 UNSAT41%74%39093.24323.08
3SatElite + ReVivAl 0.232009-03-1811948 SAT, 71 UNSAT41%73%38374.13322.47
4ReVivAl 0.232009-03-1811753 SAT, 64 UNSAT40%72%44067.36376.64
5minisat2-simp07072111646 SAT, 70 UNSAT40%71%25111.90216.48
6IUT_BMB_SIM1.011146 SAT, 65 UNSAT38%68%30273.14272.73
7minisat2-core07072110647 SAT, 59 UNSAT36%65%23477.71221.49

Category "applications instances" (APPLICATION): SAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)6723%100%9895.12147.69
1kw_pre2009-03-215820%87%10876.22187.52
2ReVivAl 0.232009-03-185318%79%17449.50329.24
3ReVivAl 0.23 + SatElite2009-03-185117%76%14541.78285.13
4SatElite + ReVivAl 0.232009-03-184816%72%12696.73264.52
5minisat2-core0707214716%70%7279.67154.89
6minisat2-simp0707214616%69%6411.34139.38
7IUT_BMB_SIM1.04616%69%10218.94222.15

Category "applications instances" (APPLICATION): UNSAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)9633%100%23991.85249.92
1kw_pre2009-03-219131%95%23715.44260.61
2SatElite + ReVivAl 0.232009-03-187124%74%25677.40361.65
3minisat2-simp0707217024%73%18700.55267.15
4ReVivAl 0.23 + SatElite2009-03-187024%73%24551.46350.74
5IUT_BMB_SIM1.06522%68%20054.20308.53
6ReVivAl 0.232009-03-186422%67%26617.86415.90
7minisat2-core0707215920%61%16198.05274.54

Category "crafted instances" (CRAFTED): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances
Detail % of all instances% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)13792 SAT, 45 UNSAT49%100%20732.67151.33
1minisat2-simp07072111976 SAT, 43 UNSAT42%87%23212.54195.06
2SatElite + ReVivAl 0.232009-03-1811975 SAT, 44 UNSAT42%87%24059.71202.18
3ReVivAl 0.23 + SatElite2009-03-1811975 SAT, 44 UNSAT42%87%24622.54206.91
4ReVivAl 0.232009-03-1811472 SAT, 42 UNSAT41%83%20435.40179.26
5IUT_BMB_SIM1.010774 SAT, 33 UNSAT38%78%23163.33216.48
6kw_pre2009-03-2110672 SAT, 34 UNSAT38%77%16298.74153.76
7minisat2-core07072110069 SAT, 31 UNSAT36%73%17639.05176.39

Category "crafted instances" (CRAFTED): SAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)9233%100%11272.05122.52
1minisat2-simp0707217627%83%13552.50178.32
2SatElite + ReVivAl 0.232009-03-187527%82%12098.88161.32
3ReVivAl 0.23 + SatElite2009-03-187527%82%13413.59178.85
4IUT_BMB_SIM1.07426%80%13332.28180.17
5kw_pre2009-03-217226%78%9945.50138.13
6ReVivAl 0.232009-03-187226%78%10031.58139.33
7minisat2-core0707216925%75%9718.24140.84

Category "crafted instances" (CRAFTED): UNSAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)4516%100%9460.62210.24
1ReVivAl 0.23 + SatElite2009-03-184416%98%11208.96254.75
2SatElite + ReVivAl 0.232009-03-184416%98%11960.83271.84
3minisat2-simp0707214315%96%9660.04224.65
4ReVivAl 0.232009-03-184215%93%10403.82247.71
5kw_pre2009-03-213412%76%6353.24186.86
6IUT_BMB_SIM1.03312%73%9831.05297.91
7minisat2-core0707213111%69%7920.82255.51