SAT 2009 Competition: main competition (phase2): solvers page (tabular format, summary)

Some explanations

#Benchs: Number of benchs for this solver
Solved: Total number of solved instances (this is the sum of the SAT and UNSAT columns)
UNSAT: Total number of UNSAT answers
SAT: Total number of SAT answers
UNKNOWN: Total number of UNKNOWN answers
Bad Answers: Total number of bad answers

Category "applications instances" (APPLICATION)

Solver Name	#benchs	Solved	UNSAT	SAT	UNKNOWN
CircUs 2009-03-23 (complete)	292	196 (67.12%)	119 (40.75%)	77 (26.37%)	96 (32.88%)
clasp 1.2.0-SAT09-32 (complete)	292	175 (59.93%)	115 (39.38%)	60 (20.55%)	117 (40.07%)
glucose 1.0 (complete)	292	204 (69.86%)	127 (43.49%)	77 (26.37%)	88 (30.14%)
kw 2009-03-20 (complete)	292	175 (59.93%)	108 (36.99%)	67 (22.95%)	117 (40.07%)
LySAT i/2009-03-20 (complete)	292	197 (67.47%)	124 (42.47%)	73 (25%)	95 (32.53%)
ManySAT 1.1 aimd 1/2009-03-20 (complete)	292	193 (66.1%)	122 (41.78%)	71 (24.32%)	99 (33.9%)
MiniSat 2.1 (Sat-race'08 Edition) (complete)	292	194 (66.44%)	116 (39.73%)	78 (26.71%)	98 (33.56%)
MiniSAT 09z 2009-03-22 (complete)	292	193 (66.1%)	115 (39.38%)	78 (26.71%)	99 (33.9%)
minisat_cumr p-2009-03-18 (complete)	292	190 (65.07%)	115 (39.38%)	75 (25.68%)	102 (34.93%)
MXC 2009-03-10 (complete)	292	190 (65.07%)	111 (38.01%)	79 (27.05%)	102 (34.93%)
precosat 236 (complete)	292	204 (69.86%)	125 (42.81%)	79 (27.05%)	88 (30.14%)
Rsat 2009-03-22 (complete)	292	188 (64.38%)	114 (39.04%)	74 (25.34%)	104 (35.62%)
SApperloT base (complete)	292	186 (63.7%)	108 (36.99%)	78 (26.71%)	106 (36.3%)
SAT07 reference solver: picosat 535 (complete)	292	171 (58.56%)	95 (32.53%)	76 (26.03%)	121 (41.44%)
SAT07 reference solver: Rsat 2007-02-08 (complete)	292	180 (61.64%)	111 (38.01%)	69 (23.63%)	112 (38.36%)
SATzilla2009_I 2009-03-22 (complete)	292	195 (66.78%)	114 (39.04%)	81 (27.74%)	97 (33.22%)

Solver Name

Progress

CircUs
2009-03-23
(complete)

done
292

UNSAT
119

SAT
77

?
96

clasp
1.2.0-SAT09-32
(complete)

done
292

UNSAT
115

SAT
60

?
117

glucose
1.0
(complete)

done
292

UNSAT
127

SAT
77

?
88

kw
2009-03-20
(complete)

done
292

UNSAT
108

SAT
67

?
117

LySAT
i/2009-03-20
(complete)

done
292

UNSAT
124

SAT
73

?
95

ManySAT 1.1
aimd 1/2009-03-20
(complete)

done
292

UNSAT
122

SAT
71

?
99

MiniSat
2.1 (Sat-race'08 Edition)
(complete)

done
292

UNSAT
116

SAT
78

?
98

MiniSAT 09z
2009-03-22
(complete)

done
292

UNSAT
115

SAT
78

?
99

minisat_cumr
p-2009-03-18
(complete)

done
292

UNSAT
115

SAT
75

?
102

MXC
2009-03-10
(complete)

done
292

UNSAT
111

SAT
79

?
102

precosat
236
(complete)

done
292

UNSAT
125

SAT
79

?
88

Rsat
2009-03-22
(complete)

done
292

UNSAT
114

SAT
74

?
104

SApperloT
base
(complete)

done
292

UNSAT
108

SAT
78

?
106

SAT07 reference solver: picosat
535
(complete)

done
292

UNSAT
95

SAT
76

?
121

SAT07 reference solver: Rsat
2007-02-08
(complete)

done
292

UNSAT
111

SAT
69

?
112

SATzilla2009_I
2009-03-22
(complete)

done
292

UNSAT
114

SAT
81

?
97

Category "crafted instances" (CRAFTED)

Solver Name	#benchs	Solved	UNSAT	SAT	UNKNOWN
clasp 1.2.0-SAT09-32 (complete)	281	156 (55.52%)	64 (22.78%)	92 (32.74%)	125 (44.48%)
glucose 1.0 (complete)	281	135 (48.04%)	49 (17.44%)	86 (30.6%)	146 (51.96%)
IUT_BMB_SAT 1.0 (complete)	281	149 (53.02%)	60 (21.35%)	89 (31.67%)	132 (46.98%)
LySAT c/2009-03-20 (complete)	281	141 (50.18%)	58 (20.64%)	83 (29.54%)	140 (49.82%)
MiniSat 2.1 (Sat-race'08 Edition) (complete)	281	137 (48.75%)	50 (17.79%)	87 (30.96%)	144 (51.25%)
MXC 2009-03-10 (complete)	281	146 (51.96%)	55 (19.57%)	91 (32.38%)	135 (48.04%)
precosat 236 (complete)	281	141 (50.18%)	51 (18.15%)	90 (32.03%)	140 (49.82%)
SApperloT base (complete)	281	149 (53.02%)	57 (20.28%)	92 (32.74%)	132 (46.98%)
SAT07 reference solver: minisat SAT 2007 (complete)	281	150 (53.38%)	60 (21.35%)	90 (32.03%)	131 (46.62%)
SAT07 reference solver: SATzilla CRAFTED (complete)	281	137 (48.75%)	53 (18.86%)	84 (29.89%)	144 (51.25%)
SATzilla2009_C 2009-03-22 (complete)	281	155 (55.16%)	72 (25.62%)	83 (29.54%)	126 (44.84%)
VARSAT-industrial 2009-03-22 (complete)	281	145 (51.6%)	60 (21.35%)	85 (30.25%)	136 (48.4%)

Solver Name

Progress

clasp
1.2.0-SAT09-32
(complete)

done
281

UNSAT
64

SAT
92

?
125

glucose
1.0
(complete)

done
281

UNSAT
49

SAT
86

?
146

IUT_BMB_SAT
1.0
(complete)

done
281

UNSAT
60

SAT
89

?
132

LySAT
c/2009-03-20
(complete)

done
281

UNSAT
58

SAT
83

?
140

MiniSat
2.1 (Sat-race'08 Edition)
(complete)

done
281

UNSAT
50

SAT
87

?
144

MXC
2009-03-10
(complete)

done
281

UNSAT
55

SAT
91

?
135

precosat
236
(complete)

done
281

UNSAT
51

SAT
90

?
140

SApperloT
base
(complete)

done
281

UNSAT
57

SAT
92

?
132

SAT07 reference solver: minisat
SAT 2007
(complete)

done
281

UNSAT
60

SAT
90

?
131

SAT07 reference solver: SATzilla
CRAFTED
(complete)

done
281

UNSAT
53

SAT
84

?
144

SATzilla2009_C
2009-03-22
(complete)

done
281

UNSAT
72

SAT
83

?
126

VARSAT-industrial
2009-03-22
(complete)

done
281

UNSAT
60

SAT
85

?
136

Category "random instances" (RANDOM)

Solver Name	#benchs	Solved	UNSAT	SAT	UNKNOWN
adaptg2wsat2009++ 2009-03-23 (incomplete)	610	338 (55.41%)	0 (0%)	338 (55.41%)	272 (44.59%)
gnovelty+2 2009-03-22 (incomplete)	610	355 (58.2%)	0 (0%)	355 (58.2%)	255 (41.8%)
gNovelty+-T 2009-03-22 (incomplete)	610	314 (51.48%)	0 (0%)	314 (51.48%)	296 (48.52%)
hybridGM3 3 (incomplete)	610	340 (55.74%)	0 (0%)	340 (55.74%)	270 (44.26%)
iPAWS 2009-03-22 (incomplete)	610	288 (47.21%)	0 (0%)	288 (47.21%)	322 (52.79%)
march_hi hi (complete)	610	313 (51.31%)	140 (22.95%)	173 (28.36%)	297 (48.69%)
SAT07 reference solver: adaptg2wsat+ 2007-02-08 (incomplete)	610	298 (48.85%)	0 (0%)	298 (48.85%)	312 (51.15%)
SAT07 reference solver: gnovelty+ 2007-02-08 (incomplete)	610	318 (52.13%)	0 (0%)	318 (52.13%)	292 (47.87%)
SAT07 reference solver: March KS 2007-02-08 (complete)	610	308 (50.49%)	131 (21.48%)	177 (29.02%)	302 (49.51%)
SAT07 reference solver: SATzilla RANDOM (complete)	610	308 (50.49%)	127 (20.82%)	181 (29.67%)	302 (49.51%)
SATzilla2009_R 2009-03-22 (complete)	610	435 (71.31%)	96 (15.74%)	339 (55.57%)	175 (28.69%)
TNM 2009-03-22 (incomplete)	610	379 (62.13%)	0 (0%)	379 (62.13%)	231 (37.87%)

Solver Name

Progress

adaptg2wsat2009++
2009-03-23
(incomplete)

done
610

SAT
338

?
272

gnovelty+2
2009-03-22
(incomplete)

done
610

SAT
355

?
255

gNovelty+-T
2009-03-22
(incomplete)

done
610

SAT
314

?
296

hybridGM3
3
(incomplete)

done
610

SAT
340

?
270

iPAWS
2009-03-22
(incomplete)

done
610

SAT
288

?
322

march_hi
hi
(complete)

done
610

UNSAT
140

SAT
173

?
297

SAT07 reference solver: adaptg2wsat+
2007-02-08
(incomplete)

done
610

SAT
298

?
312

SAT07 reference solver: gnovelty+
2007-02-08
(incomplete)

done
610

SAT
318

?
292

SAT07 reference solver: March KS
2007-02-08
(complete)

done
610

UNSAT
131

SAT
177

?
302

SAT07 reference solver: SATzilla
RANDOM
(complete)

done
610

UNSAT
127

SAT
181

?
302

SATzilla2009_R
2009-03-22
(complete)

done
610

UNSAT
96

SAT
339

?
175

TNM
2009-03-22
(incomplete)

done
610

SAT
379

?
231

All categories for solvers registered in all categories

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name	#benchs	Solved	UNSAT	SAT	UNKNOWN
adaptg2wsat2009++ 2009-03-23 (incomplete)	610	338 (55.41%)	0 (0%)	338 (55.41%)	272 (44.59%)
CircUs 2009-03-23 (complete)	292	196 (67.12%)	119 (40.75%)	77 (26.37%)	96 (32.88%)
clasp 1.2.0-SAT09-32 (complete)	573	331 (57.77%)	179 (31.24%)	152 (26.53%)	242 (42.23%)
glucose 1.0 (complete)	573	339 (59.16%)	176 (30.72%)	163 (28.45%)	234 (40.84%)
gnovelty+2 2009-03-22 (incomplete)	610	355 (58.2%)	0 (0%)	355 (58.2%)	255 (41.8%)
gNovelty+-T 2009-03-22 (incomplete)	610	314 (51.48%)	0 (0%)	314 (51.48%)	296 (48.52%)
hybridGM3 3 (incomplete)	610	340 (55.74%)	0 (0%)	340 (55.74%)	270 (44.26%)
iPAWS 2009-03-22 (incomplete)	610	288 (47.21%)	0 (0%)	288 (47.21%)	322 (52.79%)
IUT_BMB_SAT 1.0 (complete)	281	149 (53.02%)	60 (21.35%)	89 (31.67%)	132 (46.98%)
kw 2009-03-20 (complete)	292	175 (59.93%)	108 (36.99%)	67 (22.95%)	117 (40.07%)
LySAT c/2009-03-20 (complete)	281	141 (50.18%)	58 (20.64%)	83 (29.54%)	140 (49.82%)
LySAT i/2009-03-20 (complete)	292	197 (67.47%)	124 (42.47%)	73 (25%)	95 (32.53%)
ManySAT 1.1 aimd 1/2009-03-20 (complete)	292	193 (66.1%)	122 (41.78%)	71 (24.32%)	99 (33.9%)
march_hi hi (complete)	610	313 (51.31%)	140 (22.95%)	173 (28.36%)	297 (48.69%)
MiniSat 2.1 (Sat-race'08 Edition) (complete)	573	331 (57.77%)	166 (28.97%)	165 (28.8%)	242 (42.23%)
MiniSAT 09z 2009-03-22 (complete)	292	193 (66.1%)	115 (39.38%)	78 (26.71%)	99 (33.9%)
minisat_cumr p-2009-03-18 (complete)	292	190 (65.07%)	115 (39.38%)	75 (25.68%)	102 (34.93%)
MXC 2009-03-10 (complete)	573	336 (58.64%)	166 (28.97%)	170 (29.67%)	237 (41.36%)
precosat 236 (complete)	573	345 (60.21%)	176 (30.72%)	169 (29.49%)	228 (39.79%)
Rsat 2009-03-22 (complete)	292	188 (64.38%)	114 (39.04%)	74 (25.34%)	104 (35.62%)
SApperloT base (complete)	573	335 (58.46%)	165 (28.8%)	170 (29.67%)	238 (41.54%)
SAT07 reference solver: adaptg2wsat+ 2007-02-08 (incomplete)	610	298 (48.85%)	0 (0%)	298 (48.85%)	312 (51.15%)
SAT07 reference solver: gnovelty+ 2007-02-08 (incomplete)	610	318 (52.13%)	0 (0%)	318 (52.13%)	292 (47.87%)
SAT07 reference solver: March KS 2007-02-08 (complete)	610	308 (50.49%)	131 (21.48%)	177 (29.02%)	302 (49.51%)
SAT07 reference solver: minisat SAT 2007 (complete)	281	150 (53.38%)	60 (21.35%)	90 (32.03%)	131 (46.62%)
SAT07 reference solver: picosat 535 (complete)	292	171 (58.56%)	95 (32.53%)	76 (26.03%)	121 (41.44%)
SAT07 reference solver: Rsat 2007-02-08 (complete)	292	180 (61.64%)	111 (38.01%)	69 (23.63%)	112 (38.36%)
SAT07 reference solver: SATzilla CRAFTED (complete)	281	137 (48.75%)	53 (18.86%)	84 (29.89%)	144 (51.25%)
SAT07 reference solver: SATzilla RANDOM (complete)	610	308 (50.49%)	127 (20.82%)	181 (29.67%)	302 (49.51%)
SATzilla2009_C 2009-03-22 (complete)	281	155 (55.16%)	72 (25.62%)	83 (29.54%)	126 (44.84%)
SATzilla2009_I 2009-03-22 (complete)	292	195 (66.78%)	114 (39.04%)	81 (27.74%)	97 (33.22%)
SATzilla2009_R 2009-03-22 (complete)	610	435 (71.31%)	96 (15.74%)	339 (55.57%)	175 (28.69%)
TNM 2009-03-22 (incomplete)	610	379 (62.13%)	0 (0%)	379 (62.13%)	231 (37.87%)
VARSAT-industrial 2009-03-22 (complete)	281	145 (51.6%)	60 (21.35%)	85 (30.25%)	136 (48.4%)