SAT 2009 Competition: main competition (phase 1): ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "applications instances" (APPLICATION): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)			196	79 SAT, 117 UNSAT	67%	100%	33863.84	172.77
1	precosat	236	164	65 SAT, 99 UNSAT	56%	84%	37379.67	227.92
2	MiniSat	2.1 (Sat-race'08 Edition)	155	65 SAT, 90 UNSAT	53%	79%	27011.56	174.27
3	LySAT	i/2009-03-20	153	57 SAT, 96 UNSAT	52%	78%	35271.11	230.53
4	glucose	1.0	152	54 SAT, 98 UNSAT	52%	78%	34784.84	228.85
5	MiniSAT 09z	2009-03-22	152	59 SAT, 93 UNSAT	52%	78%	37872.87	249.16
6	kw	2009-03-20	150	58 SAT, 92 UNSAT	51%	77%	35080.23	233.87
7	ManySAT 1.1	aimd 1/2009-03-20	149	54 SAT, 95 UNSAT	51%	76%	34834.19	233.79
8	ManySAT 1.1	aimd 0/2009-03-20	149	54 SAT, 95 UNSAT	51%	76%	38639.59	259.33
9	MXC	2009-03-10	147	62 SAT, 85 UNSAT	50%	75%	27968.90	190.26
10	ManySAT 1.1	aimd 2/2009-03-20	145	51 SAT, 94 UNSAT	50%	74%	34242.50	236.16
11	CircUs	2009-03-23	144	59 SAT, 85 UNSAT	49%	73%	36680.28	254.72
12	Rsat	2009-03-22	143	53 SAT, 90 UNSAT	49%	73%	31000.89	216.79
13	SATzilla2009_I	2009-03-22	142	60 SAT, 82 UNSAT	49%	72%	33608.36	236.68
14	minisat_cumr	p-2009-03-18	141	58 SAT, 83 UNSAT	48%	72%	29304.08	207.83
15	picosat	913	139	63 SAT, 76 UNSAT	48%	71%	34013.47	244.70
16	clasp	1.2.0-SAT09-32	138	53 SAT, 85 UNSAT	47%	70%	33317.37	241.43
17	SAT07 reference solver: Rsat	2007-02-08	133	56 SAT, 77 UNSAT	46%	68%	28975.23	217.86
18	SApperloT	base	129	55 SAT, 74 UNSAT	44%	66%	31762.78	246.22
19	SAT07 reference solver: picosat	535	126	59 SAT, 67 UNSAT	43%	64%	33871.13	268.82
20	LySAT	c/2009-03-20	123	51 SAT, 72 UNSAT	42%	63%	26865.49	218.42
21	IUT_BMB_SAT	1.0	116	46 SAT, 70 UNSAT	40%	59%	20974.40	180.81
22	HydraSAT	2009-03-22-Base	116	53 SAT, 63 UNSAT	40%	59%	26856.33	231.52
23	HydraSAT-Flat	2009-03-22-Flat	115	51 SAT, 64 UNSAT	39%	59%	26016.34	226.23
24	VARSAT-industrial	2009-03-22	110	49 SAT, 61 UNSAT	38%	56%	22753.77	206.85
25	SApperloT	hrp	107	42 SAT, 65 UNSAT	37%	55%	20954.19	195.83
26	HydraSAT-Multi	2009-03-22-Multi	106	49 SAT, 57 UNSAT	36%	54%	16308.48	153.85
27	SATzilla2009_C	2009-03-22	106	45 SAT, 61 UNSAT	36%	54%	25974.72	245.04
28	VARSAT-crafted	2009-03-22	99	44 SAT, 55 UNSAT	34%	51%	23553.01	237.91
29	SAT4J CORE	2.1 RC1	95	46 SAT, 49 UNSAT	33%	48%	25380.84	267.17
30	satake	2009-03-22	92	40 SAT, 52 UNSAT	32%	47%	18309.62	199.02
31	CSat	2009-03-22	91	40 SAT, 51 UNSAT	31%	46%	20461.14	224.85
32	SATzilla2009_R	2009-03-22	59	36 SAT, 23 UNSAT	20%	30%	6260.03	106.10
33	VARSAT-random	2009-03-22	59	25 SAT, 34 UNSAT	20%	30%	16836.65	285.37
34	march_hi	hi	21	9 SAT, 12 UNSAT	7%	11%	5170.80	246.23
35	march_nn	nn	21	10 SAT, 11 UNSAT	7%	11%	6189.51	294.74
36	Hybrid2	2009-03-22	12	11 SAT, 1 UNSAT	4%	6%	3851.84	320.99
37	adaptg2wsat2009	2009-03-23	11	8 SAT, 3 UNSAT	4%	6%	1746.45	158.77
38	adaptg2wsat2009++	2009-03-23	11	8 SAT, 3 UNSAT	4%	6%	1806.37	164.22
39	slstc	1.0	10	9 SAT, 1 UNSAT	3%	5%	2093.29	209.33
40	tts	tts-5-0	10	6 SAT, 4 UNSAT	3%	5%	2539.03	253.90
41	NCVWr	2009-03-22	10	9 SAT, 1 UNSAT	3%	5%	2973.72	297.37
42	iPAWS	2009-03-22	8	8 SAT	3%	4%	1400.34	175.04
43	ttsth-5-0	2009-03-19	8	4 SAT, 4 UNSAT	3%	4%	2937.42	367.18
44	hybridGM7	7	7	7 SAT	2%	4%	468.76	66.97
45	gnovelty+	2009-03-22	7	7 SAT	2%	4%	1586.83	226.69
46	gNovelty+-T	2009-03-22	7	7 SAT	2%	4%	1826.46	260.92
47	TNM	2009-03-22	6	5 SAT, 1 UNSAT	2%	3%	1157.83	192.97
48	hybridGM	1	5	5 SAT	2%	3%	731.62	146.32
49	hybridGM3	3	5	5 SAT	2%	3%	1103.11	220.62
50	gnovelty+2	2009-03-22	4	4 SAT	1%	2%	91.85	22.96

Category "applications instances" (APPLICATION): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)			79	27%	100%	8981.63	113.69
1	MiniSat	2.1 (Sat-race'08 Edition)	65	22%	82%	8104.11	124.68
2	precosat	236	65	22%	82%	11641.55	179.10
3	picosat	913	63	22%	80%	12779.67	202.85
4	MXC	2009-03-10	62	21%	78%	8494.77	137.01
5	SATzilla2009_I	2009-03-22	60	21%	76%	10593.32	176.56
6	MiniSAT 09z	2009-03-22	59	20%	75%	9724.50	164.82
7	CircUs	2009-03-23	59	20%	75%	13064.44	221.43
8	SAT07 reference solver: picosat	535	59	20%	75%	14213.77	240.91
9	minisat_cumr	p-2009-03-18	58	20%	73%	9145.58	157.68
10	kw	2009-03-20	58	20%	73%	10004.54	172.49
11	LySAT	i/2009-03-20	57	20%	72%	11536.87	202.40
12	SAT07 reference solver: Rsat	2007-02-08	56	19%	71%	6923.51	123.63
13	SApperloT	base	55	19%	70%	9889.91	179.82
14	glucose	1.0	54	18%	68%	7478.34	138.49
15	ManySAT 1.1	aimd 0/2009-03-20	54	18%	68%	7983.87	147.85
16	ManySAT 1.1	aimd 1/2009-03-20	54	18%	68%	8270.36	153.15
17	Rsat	2009-03-22	53	18%	67%	6813.66	128.56
18	clasp	1.2.0-SAT09-32	53	18%	67%	8597.36	162.21
19	HydraSAT	2009-03-22-Base	53	18%	67%	8759.45	165.27
20	ManySAT 1.1	aimd 2/2009-03-20	51	17%	65%	6647.69	130.35
21	HydraSAT-Flat	2009-03-22-Flat	51	17%	65%	8488.27	166.44
22	LySAT	c/2009-03-20	51	17%	65%	9499.56	186.27
23	HydraSAT-Multi	2009-03-22-Multi	49	17%	62%	5579.12	113.86
24	VARSAT-industrial	2009-03-22	49	17%	62%	7133.73	145.59
25	IUT_BMB_SAT	1.0	46	16%	58%	4370.12	95.00
26	SAT4J CORE	2.1 RC1	46	16%	58%	8889.09	193.24
27	SATzilla2009_C	2009-03-22	45	15%	57%	7869.43	174.88
28	VARSAT-crafted	2009-03-22	44	15%	56%	8875.81	201.72
29	SApperloT	hrp	42	14%	53%	5571.58	132.66
30	satake	2009-03-22	40	14%	51%	6311.56	157.79
31	CSat	2009-03-22	40	14%	51%	7022.57	175.56
32	SATzilla2009_R	2009-03-22	36	12%	46%	3643.77	101.22
33	VARSAT-random	2009-03-22	25	9%	32%	6512.66	260.51
34	Hybrid2	2009-03-22	11	4%	14%	3851.04	350.09
35	march_nn	nn	10	3%	13%	3627.52	362.75
36	slstc	1.0	9	3%	11%	2092.23	232.47
37	NCVWr	2009-03-22	9	3%	11%	2972.93	330.33
38	march_hi	hi	9	3%	11%	3063.43	340.38
39	iPAWS	2009-03-22	8	3%	10%	1400.34	175.04
40	adaptg2wsat2009	2009-03-23	8	3%	10%	1651.93	206.49
41	adaptg2wsat2009++	2009-03-23	8	3%	10%	1707.51	213.44
42	hybridGM7	7	7	2%	9%	468.76	66.97
43	gnovelty+	2009-03-22	7	2%	9%	1586.83	226.69
44	gNovelty+-T	2009-03-22	7	2%	9%	1826.46	260.92
45	tts	tts-5-0	6	2%	8%	2460.64	410.11
46	hybridGM	1	5	2%	6%	731.62	146.32
47	hybridGM3	3	5	2%	6%	1103.11	220.62
48	TNM	2009-03-22	5	2%	6%	1157.00	231.40
49	gnovelty+2	2009-03-22	4	1%	5%	91.85	22.96
50	ttsth-5-0	2009-03-19	4	1%	5%	2841.45	710.36

Category "applications instances" (APPLICATION): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 292
Virtual Best Solver (VBS)			117	40%	100%	24882.22	212.67
1	precosat	236	99	34%	85%	25738.12	259.98
2	glucose	1.0	98	34%	84%	27306.50	278.64
3	LySAT	i/2009-03-20	96	33%	82%	23734.24	247.23
4	ManySAT 1.1	aimd 1/2009-03-20	95	33%	81%	26563.83	279.62
5	ManySAT 1.1	aimd 0/2009-03-20	95	33%	81%	30655.72	322.69
6	ManySAT 1.1	aimd 2/2009-03-20	94	32%	80%	27594.81	293.56
7	MiniSAT 09z	2009-03-22	93	32%	79%	28148.36	302.67
8	kw	2009-03-20	92	32%	79%	25075.69	272.56
9	MiniSat	2.1 (Sat-race'08 Edition)	90	31%	77%	18907.45	210.08
10	Rsat	2009-03-22	90	31%	77%	24187.23	268.75
11	MXC	2009-03-10	85	29%	73%	19474.13	229.11
12	CircUs	2009-03-23	85	29%	73%	23615.84	277.83
13	clasp	1.2.0-SAT09-32	85	29%	73%	24720.01	290.82
14	minisat_cumr	p-2009-03-18	83	28%	71%	20158.50	242.87
15	SATzilla2009_I	2009-03-22	82	28%	70%	23015.04	280.67
16	SAT07 reference solver: Rsat	2007-02-08	77	26%	66%	22051.72	286.39
17	picosat	913	76	26%	65%	21233.80	279.39
18	SApperloT	base	74	25%	63%	21872.87	295.58
19	LySAT	c/2009-03-20	72	25%	62%	17365.93	241.19
20	IUT_BMB_SAT	1.0	70	24%	60%	16604.28	237.20
21	SAT07 reference solver: picosat	535	67	23%	57%	19657.36	293.39
22	SApperloT	hrp	65	22%	56%	15382.61	236.66
23	HydraSAT-Flat	2009-03-22-Flat	64	22%	55%	17528.07	273.88
24	HydraSAT	2009-03-22-Base	63	22%	54%	18096.88	287.25
25	VARSAT-industrial	2009-03-22	61	21%	52%	15620.04	256.07
26	SATzilla2009_C	2009-03-22	61	21%	52%	18105.29	296.81
27	HydraSAT-Multi	2009-03-22-Multi	57	20%	49%	10729.36	188.23
28	VARSAT-crafted	2009-03-22	55	19%	47%	14677.20	266.86
29	satake	2009-03-22	52	18%	44%	11998.06	230.73
30	CSat	2009-03-22	51	17%	44%	13438.57	263.50
31	SAT4J CORE	2.1 RC1	49	17%	42%	16491.75	336.57
32	VARSAT-random	2009-03-22	34	12%	29%	10323.99	303.65
33	SATzilla2009_R	2009-03-22	23	8%	20%	2616.26	113.75
34	march_hi	hi	12	4%	10%	2107.36	175.61
35	march_nn	nn	11	4%	9%	2561.99	232.91
36	tts	tts-5-0	4	1%	3%	78.39	19.60
37	ttsth-5-0	2009-03-19	4	1%	3%	95.97	23.99
38	adaptg2wsat2009	2009-03-23	3	1%	3%	94.52	31.51
39	adaptg2wsat2009++	2009-03-23	3	1%	3%	98.86	32.95
40	NCVWr	2009-03-22	1	0%	1%	0.80	0.80
41	Hybrid2	2009-03-22	1	0%	1%	0.80	0.80
42	TNM	2009-03-22	1	0%	1%	0.83	0.83
43	slstc	1.0	1	0%	1%	1.06	1.06
44	iPAWS	2009-03-22	0	0%	0%	0.00	-
45	hybridGM	1	0	0%	0%	0.00	-
46	hybridGM3	3	0	0%	0%	0.00	-
47	hybridGM7	7	0	0%	0%	0.00	-
48	gnovelty+	2009-03-22	0	0%	0%	0.00	-
49	gnovelty+2	2009-03-22	0	0%	0%	0.00	-
50	gNovelty+-T	2009-03-22	0	0%	0%	0.00	-

Category "crafted instances" (CRAFTED): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)			194	124 SAT, 70 UNSAT	69%	100%	19204.67	98.99
1	clasp	1.2.0-SAT09-32	131	78 SAT, 53 UNSAT	47%	68%	22257.76	169.91
2	SATzilla2009_I	2009-03-22	128	86 SAT, 42 UNSAT	46%	66%	21700.11	169.53
3	SATzilla2009_C	2009-03-22	125	73 SAT, 52 UNSAT	44%	64%	16701.85	133.61
4	MXC	2009-03-10	124	80 SAT, 44 UNSAT	44%	64%	22256.57	179.49
5	precosat	236	122	81 SAT, 41 UNSAT	43%	63%	22844.50	187.25
6	IUT_BMB_SAT	1.0	120	76 SAT, 44 UNSAT	43%	62%	22395.97	186.63
7	SAT07 reference solver: minisat	SAT 2007	119	76 SAT, 43 UNSAT	42%	61%	22930.58	192.69
8	SAT07 reference solver: SATzilla	CRAFTED	114	82 SAT, 32 UNSAT	41%	59%	18066.80	158.48
9	MiniSat	2.1 (Sat-race'08 Edition)	114	74 SAT, 40 UNSAT	41%	59%	18107.02	158.83
10	glucose	1.0	114	75 SAT, 39 UNSAT	41%	59%	20823.96	182.67
11	VARSAT-industrial	2009-03-22	113	73 SAT, 40 UNSAT	40%	58%	22306.77	197.41
12	SApperloT	base	113	73 SAT, 40 UNSAT	40%	58%	22826.65	202.01
13	picosat	913	112	80 SAT, 32 UNSAT	40%	58%	17111.73	152.78
14	LySAT	c/2009-03-20	112	70 SAT, 42 UNSAT	40%	58%	21080.61	188.22
15	CircUs	2009-03-23	107	70 SAT, 37 UNSAT	38%	55%	16148.01	150.92
16	kw	2009-03-20	106	72 SAT, 34 UNSAT	38%	55%	16460.37	155.29
17	Rsat	2009-03-22	105	71 SAT, 34 UNSAT	37%	54%	14010.73	133.44
18	SATzilla2009_R	2009-03-22	104	78 SAT, 26 UNSAT	37%	54%	14460.38	139.04
19	ManySAT 1.1	aimd 1/2009-03-20	103	72 SAT, 31 UNSAT	37%	53%	14991.64	145.55
20	HydraSAT-Multi	2009-03-22-Multi	103	70 SAT, 33 UNSAT	37%	53%	20825.53	202.19
21	HydraSAT-Flat	2009-03-22-Flat	102	70 SAT, 32 UNSAT	36%	53%	17796.15	174.47
22	SApperloT	hrp	102	69 SAT, 33 UNSAT	36%	53%	20647.84	202.43
23	minisat_cumr	p-2009-03-18	102	75 SAT, 27 UNSAT	36%	53%	23176.38	227.22
24	VARSAT-crafted	2009-03-22	102	61 SAT, 41 UNSAT	36%	53%	23304.40	228.47
25	LySAT	i/2009-03-20	100	69 SAT, 31 UNSAT	36%	52%	14874.18	148.74
26	ManySAT 1.1	aimd 2/2009-03-20	99	70 SAT, 29 UNSAT	35%	51%	14211.48	143.55
27	ManySAT 1.1	aimd 0/2009-03-20	99	71 SAT, 28 UNSAT	35%	51%	15251.61	154.06
28	HydraSAT	2009-03-22-Base	99	66 SAT, 33 UNSAT	35%	51%	16718.94	168.88
29	MiniSAT 09z	2009-03-22	99	72 SAT, 27 UNSAT	35%	51%	17027.31	171.99
30	VARSAT-random	2009-03-22	84	47 SAT, 37 UNSAT	30%	43%	14023.19	166.94
31	satake	2009-03-22	75	55 SAT, 20 UNSAT	27%	39%	16261.12	216.81
32	iPAWS	2009-03-22	71	71 SAT	25%	37%	7352.89	103.56
33	SAT4J CORE	2.1 RC1	71	50 SAT, 21 UNSAT	25%	37%	15136.95	213.20
34	adaptg2wsat2009	2009-03-23	70	68 SAT, 2 UNSAT	25%	36%	9425.51	134.65
35	adaptg2wsat2009++	2009-03-23	66	64 SAT, 2 UNSAT	23%	34%	5796.69	87.83
36	Hybrid2	2009-03-22	66	66 SAT	23%	34%	10425.56	157.96
37	CSat	2009-03-22	65	50 SAT, 15 UNSAT	23%	34%	10319.33	158.76
38	march_hi	hi	63	45 SAT, 18 UNSAT	22%	32%	10622.02	168.60
39	TNM	2009-03-22	62	62 SAT	22%	32%	8181.19	131.95
40	SAT07 reference solver: March KS	2007-02-08	61	42 SAT, 19 UNSAT	22%	31%	9021.93	147.90
41	march_nn	nn	58	43 SAT, 15 UNSAT	21%	30%	6232.17	107.45
42	gnovelty+	2009-03-22	54	54 SAT	19%	28%	5853.95	108.41
43	gNovelty+-T	2009-03-22	53	53 SAT	19%	27%	5073.82	95.73
44	hybridGM	1	51	51 SAT	18%	26%	5298.30	103.89
45	hybridGM3	3	51	51 SAT	18%	26%	6737.29	132.10
46	NCVWr	2009-03-22	48	48 SAT	17%	25%	12116.63	252.43
47	tts	tts-5-0	46	25 SAT, 21 UNSAT	16%	24%	2507.80	54.52
48	ttsth-5-0	2009-03-19	46	24 SAT, 22 UNSAT	16%	24%	4020.68	87.41
49	gnovelty+2	2009-03-22	46	44 SAT, 2 UNSAT	16%	24%	4840.28	105.22
50	hybridGM7	7	38	38 SAT	14%	20%	4385.10	115.40
51	slstc	1.0	33	33 SAT	12%	17%	4228.67	128.14

Category "crafted instances" (CRAFTED): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)			124	44%	100%	6981.50	56.30
1	SATzilla2009_I	2009-03-22	86	31%	69%	13179.69	153.25
2	SAT07 reference solver: SATzilla	CRAFTED	82	29%	66%	10771.00	131.35
3	precosat	236	81	29%	65%	14640.31	180.74
4	picosat	913	80	28%	65%	10666.41	133.33
5	MXC	2009-03-10	80	28%	65%	11550.05	144.38
6	SATzilla2009_R	2009-03-22	78	28%	63%	8500.41	108.98
7	clasp	1.2.0-SAT09-32	78	28%	63%	9083.59	116.46
8	IUT_BMB_SAT	1.0	76	27%	61%	11919.48	156.84
9	SAT07 reference solver: minisat	SAT 2007	76	27%	61%	13440.20	176.84
10	glucose	1.0	75	27%	60%	12274.80	163.66
11	minisat_cumr	p-2009-03-18	75	27%	60%	14494.53	193.26
12	MiniSat	2.1 (Sat-race'08 Edition)	74	26%	60%	6563.28	88.69
13	SATzilla2009_C	2009-03-22	73	26%	59%	7808.26	106.96
14	SApperloT	base	73	26%	59%	10374.21	142.11
15	VARSAT-industrial	2009-03-22	73	26%	59%	13192.38	180.72
16	ManySAT 1.1	aimd 1/2009-03-20	72	26%	58%	7924.60	110.06
17	MiniSAT 09z	2009-03-22	72	26%	58%	9768.54	135.67
18	kw	2009-03-20	72	26%	58%	10059.34	139.71
19	Rsat	2009-03-22	71	25%	57%	6069.79	85.49
20	iPAWS	2009-03-22	71	25%	57%	7352.89	103.56
21	ManySAT 1.1	aimd 0/2009-03-20	71	25%	57%	8967.06	126.30
22	ManySAT 1.1	aimd 2/2009-03-20	70	25%	56%	7071.88	101.03
23	CircUs	2009-03-23	70	25%	56%	9182.71	131.18
24	LySAT	c/2009-03-20	70	25%	56%	9188.34	131.26
25	HydraSAT-Flat	2009-03-22-Flat	70	25%	56%	11042.24	157.75
26	HydraSAT-Multi	2009-03-22-Multi	70	25%	56%	12819.38	183.13
27	LySAT	i/2009-03-20	69	25%	56%	8309.51	120.43
28	SApperloT	hrp	69	25%	56%	10369.28	150.28
29	adaptg2wsat2009	2009-03-23	68	24%	55%	9425.50	138.61
30	HydraSAT	2009-03-22-Base	66	23%	53%	8839.42	133.93
31	Hybrid2	2009-03-22	66	23%	53%	10425.56	157.96
32	adaptg2wsat2009++	2009-03-23	64	23%	52%	5796.69	90.57
33	TNM	2009-03-22	62	22%	50%	8181.19	131.95
34	VARSAT-crafted	2009-03-22	61	22%	49%	14553.59	238.58
35	satake	2009-03-22	55	20%	44%	10012.44	182.04
36	gnovelty+	2009-03-22	54	19%	44%	5853.95	108.41
37	gNovelty+-T	2009-03-22	53	19%	43%	5073.82	95.73
38	hybridGM	1	51	18%	41%	5298.30	103.89
39	hybridGM3	3	51	18%	41%	6737.29	132.10
40	CSat	2009-03-22	50	18%	40%	7905.65	158.11
41	SAT4J CORE	2.1 RC1	50	18%	40%	9373.75	187.47
42	NCVWr	2009-03-22	48	17%	39%	12116.63	252.43
43	VARSAT-random	2009-03-22	47	17%	38%	8110.69	172.57
44	march_hi	hi	45	16%	36%	5335.21	118.56
45	gnovelty+2	2009-03-22	44	16%	35%	4840.28	110.01
46	march_nn	nn	43	15%	35%	3399.05	79.05
47	SAT07 reference solver: March KS	2007-02-08	42	15%	34%	3699.64	88.09
48	hybridGM7	7	38	14%	31%	4385.10	115.40
49	slstc	1.0	33	12%	27%	4228.67	128.14
50	tts	tts-5-0	25	9%	20%	1183.31	47.33
51	ttsth-5-0	2009-03-19	24	9%	19%	1268.49	52.85

Category "crafted instances" (CRAFTED): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 281
Virtual Best Solver (VBS)			70	25%	100%	12223.17	174.62
1	clasp	1.2.0-SAT09-32	53	19%	76%	13174.17	248.57
2	SATzilla2009_C	2009-03-22	52	19%	74%	8893.59	171.03
3	IUT_BMB_SAT	1.0	44	16%	63%	10476.49	238.10
4	MXC	2009-03-10	44	16%	63%	10706.52	243.33
5	SAT07 reference solver: minisat	SAT 2007	43	15%	61%	9490.38	220.71
6	SATzilla2009_I	2009-03-22	42	15%	60%	8520.42	202.87
7	LySAT	c/2009-03-20	42	15%	60%	11892.27	283.15
8	precosat	236	41	15%	59%	8204.19	200.10
9	VARSAT-crafted	2009-03-22	41	15%	59%	8750.80	213.43
10	VARSAT-industrial	2009-03-22	40	14%	57%	9114.38	227.86
11	MiniSat	2.1 (Sat-race'08 Edition)	40	14%	57%	11543.74	288.59
12	SApperloT	base	40	14%	57%	12452.44	311.31
13	glucose	1.0	39	14%	56%	8549.16	219.21
14	VARSAT-random	2009-03-22	37	13%	53%	5912.50	159.80
15	CircUs	2009-03-23	37	13%	53%	6965.29	188.25
16	kw	2009-03-20	34	12%	49%	6401.03	188.27
17	Rsat	2009-03-22	34	12%	49%	7940.94	233.56
18	HydraSAT	2009-03-22-Base	33	12%	47%	7879.52	238.77
19	HydraSAT-Multi	2009-03-22-Multi	33	12%	47%	8006.15	242.61
20	SApperloT	hrp	33	12%	47%	10278.56	311.47
21	picosat	913	32	11%	46%	6445.33	201.42
22	HydraSAT-Flat	2009-03-22-Flat	32	11%	46%	6753.91	211.06
23	SAT07 reference solver: SATzilla	CRAFTED	32	11%	46%	7295.80	227.99
24	LySAT	i/2009-03-20	31	11%	44%	6564.66	211.76
25	ManySAT 1.1	aimd 1/2009-03-20	31	11%	44%	7067.04	227.97
26	ManySAT 1.1	aimd 2/2009-03-20	29	10%	41%	7139.60	246.19
27	ManySAT 1.1	aimd 0/2009-03-20	28	10%	40%	6284.55	224.45
28	MiniSAT 09z	2009-03-22	27	10%	39%	7258.77	268.84
29	minisat_cumr	p-2009-03-18	27	10%	39%	8681.85	321.55
30	SATzilla2009_R	2009-03-22	26	9%	37%	5959.97	229.23
31	ttsth-5-0	2009-03-19	22	8%	31%	2752.19	125.10
32	tts	tts-5-0	21	7%	30%	1324.49	63.07
33	SAT4J CORE	2.1 RC1	21	7%	30%	5763.20	274.44
34	satake	2009-03-22	20	7%	29%	6248.68	312.43
35	SAT07 reference solver: March KS	2007-02-08	19	7%	27%	5322.29	280.12
36	march_hi	hi	18	6%	26%	5286.81	293.71
37	CSat	2009-03-22	15	5%	21%	2413.68	160.91
38	march_nn	nn	15	5%	21%	2833.13	188.88
39	adaptg2wsat2009	2009-03-23	2	1%	3%	0.00	0.00
40	adaptg2wsat2009++	2009-03-23	2	1%	3%	0.00	0.00
41	gnovelty+2	2009-03-22	2	1%	3%	0.00	0.00
42	iPAWS	2009-03-22	0	0%	0%	0.00	-
43	Hybrid2	2009-03-22	0	0%	0%	0.00	-
44	NCVWr	2009-03-22	0	0%	0%	0.00	-
45	TNM	2009-03-22	0	0%	0%	0.00	-
46	hybridGM	1	0	0%	0%	0.00	-
47	hybridGM3	3	0	0%	0%	0.00	-
48	hybridGM7	7	0	0%	0%	0.00	-
49	gnovelty+	2009-03-22	0	0%	0%	0.00	-
50	slstc	1.0	0	0%	0%	0.00	-
51	gNovelty+-T	2009-03-22	0	0%	0%	0.00	-

Category "random instances" (RANDOM): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 570
Virtual Best Solver (VBS)			459	359 SAT, 100 UNSAT	81%	100%	62339.75	135.82
1	SATzilla2009_R	2009-03-22	365	299 SAT, 66 UNSAT	64%	80%	51997.72	142.46
2	TNM	2009-03-22	317	317 SAT	56%	69%	35346.17	111.50
3	gnovelty+2	2009-03-22	305	305 SAT	54%	66%	26616.48	87.27
4	hybridGM3	3	299	299 SAT	52%	65%	23272.79	77.84
5	hybridGM7	7	298	298 SAT	52%	65%	25567.23	85.80
6	adaptg2wsat2009++	2009-03-23	297	297 SAT	52%	65%	26432.65	89.00
7	hybridGM	1	294	294 SAT	52%	64%	23732.78	80.72
8	adaptg2wsat2009	2009-03-23	294	294 SAT	52%	64%	26658.47	90.68
9	Hybrid2	2009-03-22	290	290 SAT	51%	63%	30134.40	103.91
10	gnovelty+	2009-03-22	281	281 SAT	49%	61%	25523.72	90.83
11	NCVWr	2009-03-22	278	278 SAT	49%	61%	31132.10	111.99
12	SAT07 reference solver: gnovelty+	2007-02-08	272	272 SAT	48%	59%	21956.28	80.72
13	SAT07 reference solver: SATzilla	RANDOM	268	177 SAT, 91 UNSAT	47%	58%	42919.16	160.15
14	gNovelty+-T	2009-03-22	266	266 SAT	47%	58%	22823.37	85.80
15	SAT07 reference solver: adaptg2wsat+	2007-02-08	265	265 SAT	46%	58%	22333.18	84.28
16	iPAWS	2009-03-22	258	258 SAT	45%	56%	19296.93	74.79
17	march_hi	hi	247	147 SAT, 100 UNSAT	43%	54%	65568.89	265.46
18	march_nn	nn	243	145 SAT, 98 UNSAT	43%	53%	66494.85	273.64
19	SAT07 reference solver: March KS	2007-02-08	239	149 SAT, 90 UNSAT	42%	52%	57869.03	242.13
20	SATzilla2009_I	2009-03-22	145	90 SAT, 55 UNSAT	25%	32%	37645.86	259.63
21	slstc	1.0	118	118 SAT	21%	26%	13250.77	112.29
22	clasp	1.2.0-SAT09-32	84	66 SAT, 18 UNSAT	15%	18%	32979.32	392.61
23	VARSAT-random	2009-03-22	83	72 SAT, 11 UNSAT	15%	18%	30273.41	364.74
24	picosat	913	79	57 SAT, 22 UNSAT	14%	17%	29440.52	372.66
25	SATzilla2009_C	2009-03-22	73	61 SAT, 12 UNSAT	13%	16%	22395.73	306.79
26	VARSAT-industrial	2009-03-22	71	61 SAT, 10 UNSAT	12%	15%	27295.84	384.45
27	VARSAT-crafted	2009-03-22	70	60 SAT, 10 UNSAT	12%	15%	27367.38	390.96
28	SApperloT	base	70	53 SAT, 17 UNSAT	12%	15%	28249.79	403.57
29	IUT_BMB_SAT	1.0	63	50 SAT, 13 UNSAT	11%	14%	25630.38	406.83
30	MXC	2009-03-10	61	50 SAT, 11 UNSAT	11%	13%	28069.37	460.15
31	LySAT	c/2009-03-20	60	48 SAT, 12 UNSAT	11%	13%	24329.68	405.49
32	MiniSat	2.1 (Sat-race'08 Edition)	41	37 SAT, 4 UNSAT	7%	9%	16957.09	413.59
33	minisat_cumr	p-2009-03-18	29	29 SAT	5%	6%	14078.15	485.45
34	precosat	236	27	25 SAT, 2 UNSAT	5%	6%	9522.84	352.70
35	satake	2009-03-22	24	24 SAT	4%	5%	11034.05	459.75
36	SApperloT	hrp	17	13 SAT, 4 UNSAT	3%	4%	7724.34	454.37
37	glucose	1.0	17	17 SAT	3%	4%	7772.56	457.21
38	HydraSAT-Flat	2009-03-22-Flat	16	15 SAT, 1 UNSAT	3%	3%	5738.82	358.68
39	HydraSAT-Multi	2009-03-22-Multi	16	16 SAT	3%	3%	7836.19	489.76
40	HydraSAT	2009-03-22-Base	13	13 SAT	2%	3%	4930.65	379.28
41	CircUs	2009-03-23	8	8 SAT	1%	2%	2553.24	319.16
42	ManySAT 1.1	aimd 0/2009-03-20	7	7 SAT	1%	2%	1783.47	254.78
43	ManySAT 1.1	aimd 2/2009-03-20	6	6 SAT	1%	1%	957.09	159.51
44	LySAT	i/2009-03-20	6	6 SAT	1%	1%	2124.49	354.08
45	CSat	2009-03-22	6	6 SAT	1%	1%	2263.92	377.32
46	Rsat	2009-03-22	5	5 SAT	1%	1%	1801.20	360.24
47	ManySAT 1.1	aimd 1/2009-03-20	5	5 SAT	1%	1%	4144.36	828.87
48	kw	2009-03-20	4	4 SAT	1%	1%	635.52	158.88
49	SAT4J CORE	2.1 RC1	4	4 SAT	1%	1%	1440.19	360.05
50	MiniSAT 09z	2009-03-22	3	3 SAT	1%	1%	1096.04	365.35
51	tts	tts-5-0	0		0%	0%	0.00	-
52	ttsth-5-0	2009-03-19	0		0%	0%	0.00	-

Category "random instances" (RANDOM): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 570
Virtual Best Solver (VBS)			359	63%	100%	23984.32	66.81
1	TNM	2009-03-22	317	56%	88%	35346.17	111.50
2	gnovelty+2	2009-03-22	305	54%	85%	26616.48	87.27
3	hybridGM3	3	299	52%	83%	23272.79	77.84
4	SATzilla2009_R	2009-03-22	299	52%	83%	26986.91	90.26
5	hybridGM7	7	298	52%	83%	25567.23	85.80
6	adaptg2wsat2009++	2009-03-23	297	52%	83%	26432.65	89.00
7	hybridGM	1	294	52%	82%	23732.78	80.72
8	adaptg2wsat2009	2009-03-23	294	52%	82%	26658.47	90.68
9	Hybrid2	2009-03-22	290	51%	81%	30134.40	103.91
10	gnovelty+	2009-03-22	281	49%	78%	25523.72	90.83
11	NCVWr	2009-03-22	278	49%	77%	31132.10	111.99
12	SAT07 reference solver: gnovelty+	2007-02-08	272	48%	76%	21956.28	80.72
13	gNovelty+-T	2009-03-22	266	47%	74%	22823.37	85.80
14	SAT07 reference solver: adaptg2wsat+	2007-02-08	265	46%	74%	22333.18	84.28
15	iPAWS	2009-03-22	258	45%	72%	19296.93	74.79
16	SAT07 reference solver: SATzilla	RANDOM	177	31%	49%	4555.67	25.74
17	SAT07 reference solver: March KS	2007-02-08	149	26%	42%	22567.89	151.46
18	march_hi	hi	147	26%	41%	25375.87	172.62
19	march_nn	nn	145	25%	40%	24089.39	166.13
20	slstc	1.0	118	21%	33%	13250.77	112.29
21	SATzilla2009_I	2009-03-22	90	16%	25%	20081.92	223.13
22	VARSAT-random	2009-03-22	72	13%	20%	22619.27	314.16
23	clasp	1.2.0-SAT09-32	66	12%	18%	23046.03	349.18
24	SATzilla2009_C	2009-03-22	61	11%	17%	16574.52	271.71
25	VARSAT-industrial	2009-03-22	61	11%	17%	21950.46	359.84
26	VARSAT-crafted	2009-03-22	60	11%	17%	22433.48	373.89
27	picosat	913	57	10%	16%	18768.71	329.28
28	SApperloT	base	53	9%	15%	18847.23	355.61
29	IUT_BMB_SAT	1.0	50	9%	14%	19072.95	381.46
30	MXC	2009-03-10	50	9%	14%	21951.88	439.04
31	LySAT	c/2009-03-20	48	8%	13%	18352.61	382.35
32	MiniSat	2.1 (Sat-race'08 Edition)	37	6%	10%	12866.00	347.73
33	minisat_cumr	p-2009-03-18	29	5%	8%	14078.15	485.45
34	precosat	236	25	4%	7%	7912.41	316.50
35	satake	2009-03-22	24	4%	7%	11034.05	459.75
36	glucose	1.0	17	3%	5%	7772.56	457.21
37	HydraSAT-Multi	2009-03-22-Multi	16	3%	4%	7836.19	489.76
38	HydraSAT-Flat	2009-03-22-Flat	15	3%	4%	4847.49	323.17
39	HydraSAT	2009-03-22-Base	13	2%	4%	4930.65	379.28
40	SApperloT	hrp	13	2%	4%	5395.32	415.02
41	CircUs	2009-03-23	8	1%	2%	2553.24	319.16
42	ManySAT 1.1	aimd 0/2009-03-20	7	1%	2%	1783.47	254.78
43	ManySAT 1.1	aimd 2/2009-03-20	6	1%	2%	957.09	159.51
44	LySAT	i/2009-03-20	6	1%	2%	2124.49	354.08
45	CSat	2009-03-22	6	1%	2%	2263.92	377.32
46	Rsat	2009-03-22	5	1%	1%	1801.20	360.24
47	ManySAT 1.1	aimd 1/2009-03-20	5	1%	1%	4144.36	828.87
48	kw	2009-03-20	4	1%	1%	635.52	158.88
49	SAT4J CORE	2.1 RC1	4	1%	1%	1440.19	360.05
50	MiniSAT 09z	2009-03-22	3	1%	1%	1096.04	365.35
51	tts	tts-5-0	0	0%	0%	0.00	-
52	ttsth-5-0	2009-03-19	0	0%	0%	0.00	-

Category "random instances" (RANDOM): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Total number of instances in the category: 570
Virtual Best Solver (VBS)			100	18%	100%	38355.43	383.55
1	march_hi	hi	100	18%	100%	40193.02	401.93
2	march_nn	nn	98	17%	98%	42405.46	432.71
3	SAT07 reference solver: SATzilla	RANDOM	91	16%	91%	38363.49	421.58
4	SAT07 reference solver: March KS	2007-02-08	90	16%	90%	35301.14	392.23
5	SATzilla2009_R	2009-03-22	66	12%	66%	25010.81	378.95
6	SATzilla2009_I	2009-03-22	55	10%	55%	17563.94	319.34
7	picosat	913	22	4%	22%	10671.81	485.08
8	clasp	1.2.0-SAT09-32	18	3%	18%	9933.29	551.85
9	SApperloT	base	17	3%	17%	9402.56	553.09
10	IUT_BMB_SAT	1.0	13	2%	13%	6557.43	504.42
11	SATzilla2009_C	2009-03-22	12	2%	12%	5821.21	485.10
12	LySAT	c/2009-03-20	12	2%	12%	5977.07	498.09
13	MXC	2009-03-10	11	2%	11%	6117.49	556.14
14	VARSAT-random	2009-03-22	11	2%	11%	7654.14	695.83
15	VARSAT-crafted	2009-03-22	10	2%	10%	4933.90	493.39
16	VARSAT-industrial	2009-03-22	10	2%	10%	5345.39	534.54
17	SApperloT	hrp	4	1%	4%	2329.01	582.25
18	MiniSat	2.1 (Sat-race'08 Edition)	4	1%	4%	4091.09	1022.77
19	precosat	236	2	0%	2%	1610.43	805.21
20	HydraSAT-Flat	2009-03-22-Flat	1	0%	1%	891.33	891.33
21	SAT4J CORE	2.1 RC1	0	0%	0%	0.00	-
22	kw	2009-03-20	0	0%	0%	0.00	-
23	glucose	1.0	0	0%	0%	0.00	-
24	tts	tts-5-0	0	0%	0%	0.00	-
25	LySAT	i/2009-03-20	0	0%	0%	0.00	-
26	ManySAT 1.1	aimd 0/2009-03-20	0	0%	0%	0.00	-
27	ManySAT 1.1	aimd 1/2009-03-20	0	0%	0%	0.00	-
28	ManySAT 1.1	aimd 2/2009-03-20	0	0%	0%	0.00	-
29	iPAWS	2009-03-22	0	0%	0%	0.00	-
30	CSat	2009-03-22	0	0%	0%	0.00	-
31	HydraSAT-Multi	2009-03-22-Multi	0	0%	0%	0.00	-
32	HydraSAT	2009-03-22-Base	0	0%	0%	0.00	-
33	Hybrid2	2009-03-22	0	0%	0%	0.00	-
34	NCVWr	2009-03-22	0	0%	0%	0.00	-
35	TNM	2009-03-22	0	0%	0%	0.00	-
36	hybridGM	1	0	0%	0%	0.00	-
37	hybridGM3	3	0	0%	0%	0.00	-
38	hybridGM7	7	0	0%	0%	0.00	-
39	gnovelty+	2009-03-22	0	0%	0%	0.00	-
40	slstc	1.0	0	0%	0%	0.00	-
41	adaptg2wsat2009	2009-03-23	0	0%	0%	0.00	-
42	adaptg2wsat2009++	2009-03-23	0	0%	0%	0.00	-
43	CircUs	2009-03-23	0	0%	0%	0.00	-
44	Rsat	2009-03-22	0	0%	0%	0.00	-
45	gnovelty+2	2009-03-22	0	0%	0%	0.00	-
46	ttsth-5-0	2009-03-19	0	0%	0%	0.00	-
47	gNovelty+-T	2009-03-22	0	0%	0%	0.00	-
48	satake	2009-03-22	0	0%	0%	0.00	-
49	SAT07 reference solver: gnovelty+	2007-02-08	0	0%	0%	0.00	-
50	SAT07 reference solver: adaptg2wsat+	2007-02-08	0	0%	0%	0.00	-
51	minisat_cumr	p-2009-03-18	0	0%	0%	0.00	-
52	MiniSAT 09z	2009-03-22	0	0%	0%	0.00	-