SAT 2009 Competition: main competition (phase2): ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "applications instances" (APPLICATION): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)			229	91 SAT, 138 UNSAT	78%	100%	153127.06	668.68
1	precosat	236	204	79 SAT, 125 UNSAT	70%	89%	180345.80	884.05
2	glucose	1.0	204	77 SAT, 127 UNSAT	70%	89%	218826.10	1072.68
3	LySAT	i/2009-03-20	197	73 SAT, 124 UNSAT	67%	86%	198491.53	1007.57
4	CircUs	2009-03-23	196	77 SAT, 119 UNSAT	67%	86%	229285.44	1169.82
5	SATzilla2009_I	2009-03-22	195	81 SAT, 114 UNSAT	67%	85%	234743.41	1203.81
6	MiniSat	2.1 (Sat-race'08 Edition)	194	78 SAT, 116 UNSAT	66%	85%	144548.45	745.10
7	ManySAT 1.1	aimd 1/2009-03-20	193	71 SAT, 122 UNSAT	66%	84%	173344.71	898.16
8	MiniSAT 09z	2009-03-22	193	78 SAT, 115 UNSAT	66%	84%	184696.75	956.98
9	MXC	2009-03-10	190	79 SAT, 111 UNSAT	65%	83%	180409.82	949.53
10	minisat_cumr	p-2009-03-18	190	75 SAT, 115 UNSAT	65%	83%	206371.06	1086.16
11	Rsat	2009-03-22	188	74 SAT, 114 UNSAT	64%	82%	187726.95	998.55
12	SApperloT	base	186	78 SAT, 108 UNSAT	64%	81%	282488.39	1518.75
13	SAT07 reference solver: Rsat	2007-02-08	180	69 SAT, 111 UNSAT	62%	79%	195748.38	1087.49
14	kw	2009-03-20	175	67 SAT, 108 UNSAT	60%	76%	90213.34	515.50
15	clasp	1.2.0-SAT09-32	175	60 SAT, 115 UNSAT	60%	76%	163460.74	934.06
16	SAT07 reference solver: picosat	535	171	76 SAT, 95 UNSAT	59%	75%	209004.97	1222.25

Category "applications instances" (APPLICATION): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)			91	31%	100%	52336.24	575.12
1	SATzilla2009_I	2009-03-22	81	28%	89%	96609.87	1192.71
2	precosat	236	79	27%	87%	52903.18	669.66
3	MXC	2009-03-10	79	27%	87%	75203.55	951.94
4	MiniSat	2.1 (Sat-race'08 Edition)	78	27%	86%	42218.37	541.26
5	MiniSAT 09z	2009-03-22	78	27%	86%	75075.48	962.51
6	SApperloT	base	78	27%	86%	111286.45	1426.75
7	CircUs	2009-03-23	77	26%	85%	74720.59	970.40
8	glucose	1.0	77	26%	85%	90532.72	1175.75
9	SAT07 reference solver: picosat	535	76	26%	84%	84382.33	1110.29
10	minisat_cumr	p-2009-03-18	75	26%	82%	67373.20	898.31
11	Rsat	2009-03-22	74	25%	81%	85363.26	1153.56
12	LySAT	i/2009-03-20	73	25%	80%	81793.98	1120.47
13	ManySAT 1.1	aimd 1/2009-03-20	71	24%	78%	62994.30	887.24
14	SAT07 reference solver: Rsat	2007-02-08	69	24%	76%	47294.67	685.43
15	kw	2009-03-20	67	23%	74%	31254.87	466.49
16	clasp	1.2.0-SAT09-32	60	21%	66%	25529.94	425.50

Category "applications instances" (APPLICATION): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)			138	47%	100%	100790.82	730.37
1	glucose	1.0	127	43%	92%	128293.39	1010.18
2	precosat	236	125	43%	91%	127442.62	1019.54
3	LySAT	i/2009-03-20	124	42%	90%	116697.55	941.11
4	ManySAT 1.1	aimd 1/2009-03-20	122	42%	88%	110350.41	904.51
5	CircUs	2009-03-23	119	41%	86%	154564.85	1298.86
6	MiniSat	2.1 (Sat-race'08 Edition)	116	40%	84%	102330.08	882.16
7	MiniSAT 09z	2009-03-22	115	39%	83%	109621.27	953.23
8	clasp	1.2.0-SAT09-32	115	39%	83%	137930.80	1199.40
9	minisat_cumr	p-2009-03-18	115	39%	83%	138997.86	1208.68
10	Rsat	2009-03-22	114	39%	83%	102363.69	897.93
11	SATzilla2009_I	2009-03-22	114	39%	83%	138133.54	1211.70
12	MXC	2009-03-10	111	38%	80%	105206.27	947.80
13	SAT07 reference solver: Rsat	2007-02-08	111	38%	80%	148453.71	1337.42
14	kw	2009-03-20	108	37%	78%	58958.47	545.91
15	SApperloT	base	108	37%	78%	171201.93	1585.20
16	SAT07 reference solver: picosat	535	95	33%	69%	124622.64	1311.82

Category "crafted instances" (CRAFTED): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)			187	108 SAT, 79 UNSAT	67%	100%	62264.60	332.97
1	clasp	1.2.0-SAT09-32	156	92 SAT, 64 UNSAT	56%	83%	89194.49	571.76
2	SATzilla2009_C	2009-03-22	155	83 SAT, 72 UNSAT	55%	83%	94762.27	611.37
3	SAT07 reference solver: minisat	SAT 2007	150	90 SAT, 60 UNSAT	53%	80%	99960.89	666.41
4	IUT_BMB_SAT	1.0	149	89 SAT, 60 UNSAT	53%	80%	93502.16	627.53
5	SApperloT	base	149	92 SAT, 57 UNSAT	53%	80%	108298.52	726.84
6	MXC	2009-03-10	146	91 SAT, 55 UNSAT	52%	78%	76965.59	527.16
7	VARSAT-industrial	2009-03-22	145	85 SAT, 60 UNSAT	52%	78%	119365.13	823.21
8	precosat	236	141	90 SAT, 51 UNSAT	50%	75%	66318.44	470.34
9	LySAT	c/2009-03-20	141	83 SAT, 58 UNSAT	50%	75%	89925.84	637.77
10	SAT07 reference solver: SATzilla	CRAFTED	137	84 SAT, 53 UNSAT	49%	73%	76856.90	561.00
11	MiniSat	2.1 (Sat-race'08 Edition)	137	87 SAT, 50 UNSAT	49%	73%	78381.80	572.13
12	glucose	1.0	135	86 SAT, 49 UNSAT	48%	72%	70385.63	521.38

Category "crafted instances" (CRAFTED): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)			108	38%	100%	21224.84	196.53
1	clasp	1.2.0-SAT09-32	92	33%	85%	49775.04	541.03
2	SApperloT	base	92	33%	85%	54682.14	594.37
3	MXC	2009-03-10	91	32%	84%	39227.16	431.07
4	precosat	236	90	32%	83%	34447.16	382.75
5	SAT07 reference solver: minisat	SAT 2007	90	32%	83%	48346.20	537.18
6	IUT_BMB_SAT	1.0	89	32%	82%	45287.01	508.84
7	MiniSat	2.1 (Sat-race'08 Edition)	87	31%	81%	41994.77	482.70
8	glucose	1.0	86	31%	80%	37779.61	439.30
9	VARSAT-industrial	2009-03-22	85	30%	79%	54521.77	641.43
10	SAT07 reference solver: SATzilla	CRAFTED	84	30%	78%	21726.48	258.65
11	SATzilla2009_C	2009-03-22	83	30%	77%	39383.44	474.50
12	LySAT	c/2009-03-20	83	30%	77%	42073.80	506.91

Category "crafted instances" (CRAFTED): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)			79	28%	100%	41039.76	519.49
1	SATzilla2009_C	2009-03-22	72	26%	91%	55378.83	769.15
2	clasp	1.2.0-SAT09-32	64	23%	81%	39419.45	615.93
3	IUT_BMB_SAT	1.0	60	21%	76%	48215.14	803.59
4	SAT07 reference solver: minisat	SAT 2007	60	21%	76%	51614.69	860.24
5	VARSAT-industrial	2009-03-22	60	21%	76%	64843.36	1080.72
6	LySAT	c/2009-03-20	58	21%	73%	47852.03	825.04
7	SApperloT	base	57	20%	72%	53616.38	940.64
8	MXC	2009-03-10	55	20%	70%	37738.43	686.15
9	SAT07 reference solver: SATzilla	CRAFTED	53	19%	67%	55130.42	1040.20
10	precosat	236	51	18%	65%	31871.28	624.93
11	MiniSat	2.1 (Sat-race'08 Edition)	50	18%	63%	36387.03	727.74
12	glucose	1.0	49	17%	62%	32606.02	665.43

Category "random instances" (RANDOM): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 610
Virtual Best Solver (VBS)			544	404 SAT, 140 UNSAT	89%	100%	260652.75	479.14
1	SATzilla2009_R	2009-03-22	435	339 SAT, 96 UNSAT	71%	80%	231051.45	531.15
2	TNM	2009-03-22	379	379 SAT	62%	70%	194780.22	513.93
3	gnovelty+2	2009-03-22	355	355 SAT	58%	65%	154503.93	435.22
4	hybridGM3	3	340	340 SAT	56%	63%	101986.32	299.96
5	adaptg2wsat2009++	2009-03-23	338	338 SAT	55%	62%	133641.90	395.39
6	SAT07 reference solver: gnovelty+	2007-02-08	318	318 SAT	52%	58%	130357.30	409.93
7	gNovelty+-T	2009-03-22	314	314 SAT	51%	58%	143439.69	456.81
8	march_hi	hi	313	173 SAT, 140 UNSAT	51%	58%	261826.59	836.51
9	SAT07 reference solver: SATzilla	RANDOM	308	181 SAT, 127 UNSAT	50%	57%	186335.14	604.98
10	SAT07 reference solver: March KS	2007-02-08	308	177 SAT, 131 UNSAT	50%	57%	258763.45	840.14
11	SAT07 reference solver: adaptg2wsat+	2007-02-08	298	298 SAT	49%	55%	117302.89	393.63
12	iPAWS	2009-03-22	288	288 SAT	47%	53%	93855.93	325.89

Category "random instances" (RANDOM): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 610
Virtual Best Solver (VBS)			404	66%	100%	97656.83	241.72
1	TNM	2009-03-22	379	62%	94%	194780.22	513.93
2	gnovelty+2	2009-03-22	355	58%	88%	154503.93	435.22
3	hybridGM3	3	340	56%	84%	101986.32	299.96
4	SATzilla2009_R	2009-03-22	339	56%	84%	122158.36	360.35
5	adaptg2wsat2009++	2009-03-23	338	55%	84%	133641.90	395.39
6	SAT07 reference solver: gnovelty+	2007-02-08	318	52%	79%	130357.30	409.93
7	gNovelty+-T	2009-03-22	314	51%	78%	143439.69	456.81
8	SAT07 reference solver: adaptg2wsat+	2007-02-08	298	49%	74%	117302.89	393.63
9	iPAWS	2009-03-22	288	47%	71%	93855.93	325.89
10	SAT07 reference solver: SATzilla	RANDOM	181	30%	45%	23793.38	131.46
11	SAT07 reference solver: March KS	2007-02-08	177	29%	44%	98629.25	557.23
12	march_hi	hi	173	28%	43%	90433.09	522.73

Category "random instances" (RANDOM): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 610
Virtual Best Solver (VBS)			140	23%	100%	162995.92	1164.26
1	march_hi	hi	140	23%	100%	171393.50	1224.24
2	SAT07 reference solver: March KS	2007-02-08	131	21%	94%	160134.20	1222.40
3	SAT07 reference solver: SATzilla	RANDOM	127	21%	91%	162541.76	1279.86
4	SATzilla2009_R	2009-03-22	96	16%	69%	108893.09	1134.30
5	hybridGM3	3	0	0%	0%	0.00	-
6	adaptg2wsat2009++	2009-03-23	0	0%	0%	0.00	-
7	gnovelty+2	2009-03-22	0	0%	0%	0.00	-
8	iPAWS	2009-03-22	0	0%	0%	0.00	-
9	TNM	2009-03-22	0	0%	0%	0.00	-
10	SAT07 reference solver: gnovelty+	2007-02-08	0	0%	0%	0.00	-
11	SAT07 reference solver: adaptg2wsat+	2007-02-08	0	0%	0%	0.00	-
12	gNovelty+-T	2009-03-22	0	0%	0%	0.00	-