SAT 2011 Competition: main competition (phase 2): ranking of solvers
max CPU time for any solver=5000 s
max WC time for any solver=5000 s

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "applications instances" (APPLICATION): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			258	116 SAT, 142 UNSAT	86%	100%	181310.92	41.24	702.76	-	-	-	-
1	glucose	2	215	94 SAT, 121 UNSAT	72%	83%	131771.28	80.17	612.89	131811.24	80.18	613.08	1.00
2	glueminisat	2.2.5	211	85 SAT, 126 UNSAT	70%	82%	141937.40	56.28	672.69	141934.06	56.29	672.67	1.00
3	Lingeling	587f (fixed)	208	91 SAT, 117 UNSAT	69%	81%	117889.23	71.82	566.78	117894.89	72.06	566.80	1.00
4	CryptoMiniSat	Strange-Night2-st (fixed)	207	88 SAT, 119 UNSAT	69%	80%	106101.58	96.69	512.57	106107.81	96.69	512.60	1.00
5	QuteRSat	2011-05-12 (fixed)	206	86 SAT, 120 UNSAT	69%	80%	114120.09	93.79	553.98	114118.44	93.80	553.97	1.00
6	SAT09 reference solver:precosat	236	206	96 SAT, 110 UNSAT	69%	80%	122104.77	113.43	592.74	122109.68	113.49	592.77	1.00
7	minisat hack: cir_minisat	2011-05-13 (simp)	202	99 SAT, 103 UNSAT	67%	78%	128125.68	69.99	634.29	128126.19	69.99	634.29	1.00
8	RestartSAT	B95	201	97 SAT, 104 UNSAT	67%	78%	103443.89	83.48	514.65	103442.81	83.50	514.64	1.00
9	MPhaseSAT64	2011-05-14 (fixed)	201	99 SAT, 102 UNSAT	67%	78%	126764.36	104.29	630.67	126763.18	104.41	630.66	1.00
10	Satisfiability Solver Selector	2011-03-18	200	91 SAT, 109 UNSAT	67%	78%	174515.04	159.89	872.58	174434.65	160.05	872.17	1.00
11	minisat hack: contrasat	2011-03-02	199	99 SAT, 100 UNSAT	66%	77%	132726.28	79.05	666.97	132725.67	79.05	666.96	1.00
12	EBGlucose	1.0	199	91 SAT, 108 UNSAT	66%	77%	142439.18	59.44	715.77	142454.28	59.45	715.85	1.00
13	minisat hack: minisat_psm	2011-04-01	197	95 SAT, 102 UNSAT	66%	76%	115782.74	59.63	587.73	115922.43	59.63	588.44	1.00
14	SAT09 reference solver:glucose	1.0	197	87 SAT, 110 UNSAT	66%	76%	128390.81	97.47	651.73	128440.56	97.50	651.98	1.00
15	minisat hack: EBMiniSAT	2011-03-02	196	98 SAT, 98 UNSAT	65%	76%	107673.20	67.01	549.35	107676.29	67.01	549.37	1.00
16	minisat hack reference: minisat	2.2.0	194	95 SAT, 99 UNSAT	65%	75%	101499.07	62.12	523.19	101492.22	62.12	523.16	1.00
17	minisat hack: LR_GL_SHR	2011-03-02	191	81 SAT, 110 UNSAT	64%	74%	111723.47	60.94	584.94	111722.82	60.95	584.94	1.00
18	borg-sat	11.03.02-seq-industrial (fixed)	190	90 SAT, 100 UNSAT	63%	74%	110627.44	204.09	582.25	111259.55	209.38	585.58	0.99
19	minisat hack: MiniSAT	2.2.0-agile-26	189	95 SAT, 94 UNSAT	63%	73%	91483.85	49.63	484.04	91483.73	49.64	484.04	1.00
20	CryptoMiniSat	Strange-Night-2-mt (fixed)	186	85 SAT, 101 UNSAT	62%	72%	187039.02	365.99	1005.59	25322.02	48.05	136.14	7.39
21	rcl	2011-04-01	184	93 SAT, 91 UNSAT	61%	71%	98231.37	67.98	533.87	98236.99	67.98	533.90	1.00
22	Plingeling	587f (fixed)	180	92 SAT, 88 UNSAT	60%	70%	127482.02	192.48	708.23	16660.53	26.90	92.56	7.65
23	ppfolio	par	178	85 SAT, 93 UNSAT	59%	69%	129238.28	205.57	726.06	16575.09	26.54	93.12	7.80
24	SAT09 reference solver: ManySAT 1.1	aimd 1/2009-03-20	166	75 SAT, 91 UNSAT	55%	64%	92886.59	91.87	559.56	25017.87	24.20	150.71	3.71
25	borg-sat	11.03.02-par-industrial (fixed)	165	78 SAT, 87 UNSAT	55%	64%	150027.94	380.65	909.26	35849.00	106.52	217.27	4.18
26	claspmt	2.0-R4095-patched (fixed)	161	75 SAT, 86 UNSAT	54%	62%	101878.03	178.50	632.78	33377.85	36.51	207.32	3.05

Category "applications instances" (APPLICATION): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			116	39%	100%	38135.96	22.37	328.76	-	-	-	-
1	minisat hack: contrasat	2011-03-02	99	33%	85%	41344.03	52.26	417.62	41344.13	52.25	417.62	1.00
2	minisat hack: cir_minisat	2011-05-13 (simp)	99	33%	85%	51118.56	47.44	516.35	51119.87	47.44	516.36	1.00
3	MPhaseSAT64	2011-05-14 (fixed)	99	33%	85%	57294.00	104.29	578.73	57292.89	104.41	578.72	1.00
4	minisat hack: EBMiniSAT	2011-03-02	98	33%	84%	47966.14	62.04	489.45	47968.94	62.04	489.48	1.00
5	RestartSAT	B95	97	32%	84%	46782.80	76.11	482.30	46782.58	76.11	482.29	1.00
6	SAT09 reference solver:precosat	236	96	32%	83%	51252.31	101.45	533.88	51258.50	101.54	533.94	1.00
7	minisat hack reference: minisat	2.2.0	95	32%	82%	35475.64	48.10	373.43	35477.25	48.11	373.44	1.00
8	minisat hack: MiniSAT	2.2.0-agile-26	95	32%	82%	37128.06	46.54	390.82	37128.64	46.54	390.83	1.00
9	minisat hack: minisat_psm	2011-04-01	95	32%	82%	39742.31	38.50	418.34	39788.17	38.49	418.82	1.00
10	glucose	2	94	31%	81%	43473.72	36.51	462.49	43514.41	41.81	462.92	1.00
11	rcl	2011-04-01	93	31%	80%	36633.36	47.89	393.91	36639.86	47.89	393.98	1.00
12	Plingeling	587f (fixed)	92	31%	79%	67199.32	154.91	730.43	8942.84	21.72	97.20	7.51
13	Lingeling	587f (fixed)	91	30%	78%	37172.20	59.71	408.49	37179.20	59.76	408.56	1.00
14	EBGlucose	1.0	91	30%	78%	54600.13	38.11	600.00	54603.02	38.11	600.03	1.00
15	Satisfiability Solver Selector	2011-03-18	91	30%	78%	58060.23	155.12	638.02	58083.68	155.30	638.28	1.00
16	borg-sat	11.03.02-seq-industrial (fixed)	90	30%	78%	45700.56	213.31	507.78	46107.60	215.74	512.31	0.99
17	CryptoMiniSat	Strange-Night2-st (fixed)	88	29%	76%	30990.02	75.07	352.16	30992.83	75.08	352.19	1.00
18	SAT09 reference solver:glucose	1.0	87	29%	75%	37731.46	47.23	433.69	37742.80	47.23	433.83	1.00
19	QuteRSat	2011-05-12 (fixed)	86	29%	74%	38780.09	83.53	450.93	38779.54	83.53	450.92	1.00
20	glueminisat	2.2.5	85	28%	73%	42157.38	38.80	495.97	42156.46	38.80	495.96	1.00
21	ppfolio	par	85	28%	73%	49012.28	135.98	576.62	6474.49	20.65	76.17	7.57
22	CryptoMiniSat	Strange-Night-2-mt (fixed)	85	28%	73%	74902.01	289.07	881.20	9989.17	40.43	117.52	7.50
23	minisat hack: LR_GL_SHR	2011-03-02	81	27%	70%	43907.08	40.43	542.06	43907.56	40.43	542.07	1.00
24	borg-sat	11.03.02-par-industrial (fixed)	78	26%	67%	62345.58	384.50	799.30	15834.76	116.05	203.01	3.94
25	SAT09 reference solver: ManySAT 1.1	aimd 1/2009-03-20	75	25%	65%	31589.38	54.34	421.19	8135.99	21.14	108.48	3.88
26	claspmt	2.0-R4095-patched (fixed)	75	25%	65%	47380.47	169.81	631.74	19284.25	41.18	257.12	2.46

Category "applications instances" (APPLICATION): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			142	47%	100%	143174.96	103.72	1008.27	-	-	-	-
1	glueminisat	2.2.5	126	42%	89%	99780.02	143.38	791.90	99777.60	143.38	791.89	1.00
2	glucose	2	121	40%	85%	88297.55	132.28	729.73	88296.83	132.29	729.73	1.00
3	QuteRSat	2011-05-12 (fixed)	120	40%	85%	75340.00	103.31	627.83	75338.89	103.31	627.82	1.00
4	CryptoMiniSat	Strange-Night2-st (fixed)	119	40%	84%	75111.56	108.40	631.19	75114.98	108.47	631.22	1.00
5	Lingeling	587f (fixed)	117	39%	82%	80717.03	128.51	689.89	80715.69	128.51	689.88	1.00
6	minisat hack: LR_GL_SHR	2011-03-02	110	37%	77%	67816.39	96.99	616.51	67815.26	96.99	616.50	1.00
7	SAT09 reference solver:precosat	236	110	37%	77%	70852.46	125.38	644.11	70851.17	125.38	644.10	1.00
8	SAT09 reference solver:glucose	1.0	110	37%	77%	90659.34	195.14	824.18	90697.76	195.15	824.53	1.00
9	Satisfiability Solver Selector	2011-03-18	109	36%	77%	116454.80	162.73	1068.39	116350.97	162.86	1067.44	1.00
10	EBGlucose	1.0	108	36%	76%	87839.05	150.45	813.32	87851.26	150.47	813.44	1.00
11	RestartSAT	B95	104	35%	73%	56661.09	99.46	544.82	56660.24	99.49	544.81	1.00
12	minisat hack: cir_minisat	2011-05-13 (simp)	103	34%	73%	77007.13	98.14	747.64	77006.32	98.14	747.63	1.00
13	MPhaseSAT64	2011-05-14 (fixed)	102	34%	72%	69470.37	106.95	681.08	69470.29	107.24	681.08	1.00
14	minisat hack: minisat_psm	2011-04-01	102	34%	72%	76040.44	97.87	745.49	76134.26	97.88	746.41	1.00
15	CryptoMiniSat	Strange-Night-2-mt (fixed)	101	34%	71%	112137.01	429.95	1110.27	15332.84	54.00	151.81	7.31
16	borg-sat	11.03.02-seq-industrial (fixed)	100	33%	70%	64926.88	199.87	649.27	65151.95	204.30	651.52	1.00
17	minisat hack: contrasat	2011-03-02	100	33%	70%	91382.25	167.44	913.82	91381.54	167.47	913.82	1.00
18	minisat hack reference: minisat	2.2.0	99	33%	70%	66023.43	92.07	666.90	66014.97	92.07	666.82	1.00
19	minisat hack: EBMiniSAT	2011-03-02	98	33%	69%	59707.06	83.02	609.26	59707.35	83.02	609.26	1.00
20	minisat hack: MiniSAT	2.2.0-agile-26	94	31%	66%	54355.80	71.57	578.25	54355.09	71.57	578.25	1.00
21	ppfolio	par	93	31%	65%	80226.00	274.68	862.65	10100.61	34.69	108.61	7.94
22	SAT09 reference solver: ManySAT 1.1	aimd 1/2009-03-20	91	30%	64%	61297.21	125.91	673.60	16881.88	32.71	185.52	3.63
23	rcl	2011-04-01	91	30%	64%	61598.01	105.69	676.90	61597.13	105.69	676.89	1.00
24	Plingeling	587f (fixed)	88	29%	62%	60282.70	237.76	685.03	7717.69	32.03	87.70	7.81
25	borg-sat	11.03.02-par-industrial (fixed)	87	29%	61%	87682.36	366.55	1007.84	20014.24	95.96	230.05	4.38
26	claspmt	2.0-R4095-patched (fixed)	86	29%	61%	54497.56	208.48	633.69	14093.60	34.20	163.88	3.87

Category "crafted instances" (CRAFTED): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			232	146 SAT, 86 UNSAT	77%	100%	168366.50	4.31	725.72	-	-	-	-
1	Satisfiability Solver Selector	2011-03-18	163	112 SAT, 51 UNSAT	54%	70%	95298.33	39.84	584.65	95323.70	40.02	584.81	1.00
2	ppfolio	par	155	120 SAT, 35 UNSAT	52%	67%	87278.52	13.39	563.09	10935.45	1.79	70.55	7.98
3	ppfolio	seq	152	114 SAT, 38 UNSAT	51%	66%	70616.85	9.20	464.58	70617.62	9.25	464.59	1.00
4	clasp	2.0-R4092-crafted	149	87 SAT, 62 UNSAT	50%	64%	143801.90	173.21	965.11	143799.50	173.21	965.10	1.00
5	SAT09 reference solver: clasp	1.2.0-SAT09-32	149	85 SAT, 64 UNSAT	50%	64%	174493.98	328.76	1171.10	174489.72	328.75	1171.07	1.00
6	MPhaseSAT	2011-02-15	132	93 SAT, 39 UNSAT	44%	57%	85067.06	45.18	644.45	85067.17	45.18	644.45	1.00
7	glucose	2	125	83 SAT, 42 UNSAT	42%	54%	104343.97	163.86	834.75	104342.45	163.85	834.74	1.00
8	SAT07 reference solver: minisat	SAT 2007	122	78 SAT, 44 UNSAT	41%	53%	110026.47	103.10	901.86	110023.91	103.10	901.84	1.00
9	PicoSAT	941	121	81 SAT, 40 UNSAT	40%	52%	103099.28	94.45	852.06	103099.04	94.45	852.06	1.00
10	claspmt	2.0-R4095-patched (fixed)	119	78 SAT, 41 UNSAT	40%	51%	76559.87	146.20	643.36	9622.03	18.34	80.86	7.96
11	Sol	2011-04-04	116	79 SAT, 37 UNSAT	39%	50%	41526.46	6.31	357.99	40991.82	5.82	353.38	1.01
12	SAT09 reference solver: SATzilla2009_C	2009-03-22	116	73 SAT, 43 UNSAT	39%	50%	100720.22	62.57	868.28	100714.27	62.57	868.23	1.00
13	SArTagnan	2011-05-15 (fixed)	112	83 SAT, 29 UNSAT	37%	48%	43895.71	0.77	391.93	5506.75	0.18	49.17	7.97
14	RestartSAT	B95	112	77 SAT, 35 UNSAT	37%	48%	67761.33	72.09	605.01	67760.34	72.08	605.00	1.00
15	CryptoMiniSat	Strange-Night2-st (fixed)	110	70 SAT, 40 UNSAT	37%	47%	56189.51	66.49	510.81	56188.98	66.49	510.81	1.00
16	QuteRSat	2011-05-12 (fixed)	110	70 SAT, 40 UNSAT	37%	47%	62230.60	45.83	565.73	62229.70	45.83	565.72	1.00
17	sattime	2011-03-02	109	109 SAT	36%	47%	22998.70	1.77	211.00	22998.62	1.77	211.00	1.00
18	SApperloT2010	2011-05-15 (fixed)	109	76 SAT, 33 UNSAT	36%	47%	58317.14	31.21	535.02	58316.92	31.21	535.02	1.00
19	sattime+	2011-03-02	106	106 SAT	35%	46%	17523.59	2.26	165.32	17523.33	2.26	165.31	1.00
20	CryptoMiniSat	Strange-Night-2-mt (fixed)	104	70 SAT, 34 UNSAT	35%	45%	60794.52	38.89	584.56	7683.21	4.91	73.88	7.91
21	Plingeling	587f (fixed)	102	75 SAT, 27 UNSAT	34%	44%	49572.24	26.42	486.00	6238.48	3.65	61.16	7.95
22	jMiniSat	2011	98	70 SAT, 28 UNSAT	33%	42%	53798.17	37.63	548.96	53722.49	37.15	548.19	1.00
23	sathys	2011-04-01	96	73 SAT, 23 UNSAT	32%	41%	27948.96	2.79	291.13	27949.17	2.80	291.14	1.00
24	priss	1.0	82	61 SAT, 21 UNSAT	27%	35%	45707.75	10.91	557.41	11433.66	2.78	139.43	4.00

Category "crafted instances" (CRAFTED): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			146	49%	100%	16219.25	0.45	111.09	-	-	-	-
1	ppfolio	par	120	40%	82%	60244.57	12.89	502.04	7550.22	1.70	62.92	7.98
2	ppfolio	seq	114	38%	78%	42748.08	5.47	374.98	42749.20	5.55	374.99	1.00
3	Satisfiability Solver Selector	2011-03-18	112	37%	77%	30625.58	32.77	273.44	30643.81	33.34	273.61	1.00
4	sattime	2011-03-02	109	36%	75%	22998.70	1.77	211.00	22998.62	1.77	211.00	1.00
5	sattime+	2011-03-02	106	35%	73%	17523.59	2.26	165.32	17523.33	2.26	165.31	1.00
6	MPhaseSAT	2011-02-15	93	31%	64%	44250.09	16.71	475.81	44250.75	16.97	475.81	1.00
7	clasp	2.0-R4092-crafted	87	29%	60%	63277.05	118.80	727.32	63275.98	118.80	727.31	1.00
8	SAT09 reference solver: clasp	1.2.0-SAT09-32	85	28%	58%	79260.72	105.00	932.48	79258.80	105.00	932.46	1.00
9	SArTagnan	2011-05-15 (fixed)	83	28%	57%	20667.07	0.26	249.00	2593.90	0.09	31.25	7.97
10	glucose	2	83	28%	57%	69764.82	150.41	840.54	69763.88	150.43	840.53	1.00
11	PicoSAT	941	81	27%	55%	52974.15	38.52	654.00	52974.65	38.52	654.01	1.00
12	Sol	2011-04-04	79	26%	54%	21815.85	4.37	276.15	21487.77	4.34	272.00	1.02
13	claspmt	2.0-R4095-patched (fixed)	78	26%	53%	41410.10	72.19	530.90	5215.34	11.41	66.86	7.94
14	SAT07 reference solver: minisat	SAT 2007	78	26%	53%	61282.26	29.56	785.67	61280.88	29.56	785.65	1.00
15	RestartSAT	B95	77	26%	53%	33016.95	18.18	428.79	33016.81	18.18	428.79	1.00
16	SApperloT2010	2011-05-15 (fixed)	76	25%	52%	32644.09	13.56	429.53	32644.07	13.57	429.53	1.00
17	Plingeling	587f (fixed)	75	25%	51%	22198.95	16.97	295.99	2798.27	2.35	37.31	7.93
18	sathys	2011-04-01	73	24%	50%	18578.52	0.63	254.50	18578.86	0.63	254.50	1.00
19	SAT09 reference solver: SATzilla2009_C	2009-03-22	73	24%	50%	64400.03	24.07	882.19	64395.01	24.07	882.12	1.00
20	CryptoMiniSat	Strange-Night2-st (fixed)	70	23%	48%	29216.21	29.26	417.37	29216.02	29.26	417.37	1.00
21	CryptoMiniSat	Strange-Night-2-mt (fixed)	70	23%	48%	29472.40	19.16	421.03	3756.30	2.51	53.66	7.85
22	jMiniSat	2011	70	23%	48%	37706.12	20.42	538.66	37652.02	19.82	537.89	1.00
23	QuteRSat	2011-05-12 (fixed)	70	23%	48%	38283.34	22.60	546.90	38282.89	22.61	546.90	1.00
24	priss	1.0	61	20%	42%	32363.25	10.82	530.55	8095.98	2.80	132.72	4.00

Category "crafted instances" (CRAFTED): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			86	29%	100%	152147.24	683.34	1769.15	-	-	-	-
1	SAT09 reference solver: clasp	1.2.0-SAT09-32	64	21%	74%	95233.26	1007.65	1488.02	95230.93	1007.62	1487.98	1.00
2	clasp	2.0-R4092-crafted	62	21%	72%	80524.85	817.87	1298.79	80523.53	817.84	1298.77	1.00
3	Satisfiability Solver Selector	2011-03-18	51	17%	59%	64672.75	821.55	1268.09	64679.90	821.67	1268.23	1.00
4	SAT07 reference solver: minisat	SAT 2007	44	15%	51%	48744.21	206.54	1107.82	48743.03	206.53	1107.80	1.00
5	SAT09 reference solver: SATzilla2009_C	2009-03-22	43	14%	50%	36320.19	198.17	844.66	36319.27	198.17	844.63	1.00
6	glucose	2	42	14%	49%	34579.15	189.63	823.31	34578.57	189.62	823.30	1.00
7	claspmt	2.0-R4095-patched (fixed)	41	14%	48%	35149.77	235.72	857.31	4406.69	29.67	107.48	7.98
8	QuteRSat	2011-05-12 (fixed)	40	13%	47%	23947.25	202.60	598.68	23946.81	202.61	598.67	1.00
9	CryptoMiniSat	Strange-Night2-st (fixed)	40	13%	47%	26973.30	233.96	674.33	26972.96	233.95	674.32	1.00
10	PicoSAT	941	40	13%	47%	50125.13	417.44	1253.13	50124.39	417.49	1253.11	1.00
11	MPhaseSAT	2011-02-15	39	13%	45%	40816.97	347.60	1046.59	40816.41	347.61	1046.57	1.00
12	ppfolio	seq	38	13%	44%	27868.77	241.92	733.39	27868.43	241.91	733.38	1.00
13	Sol	2011-04-04	37	12%	43%	19710.61	37.91	532.72	19504.05	37.32	527.14	1.01
14	ppfolio	par	35	12%	41%	27033.95	91.06	772.40	3385.23	11.40	96.72	7.99
15	RestartSAT	B95	35	12%	41%	34744.38	303.69	992.70	34743.53	303.68	992.67	1.00
16	CryptoMiniSat	Strange-Night-2-mt (fixed)	34	11%	40%	31322.12	180.85	921.24	3926.91	23.00	115.50	7.98
17	SApperloT2010	2011-05-15 (fixed)	33	11%	38%	25673.05	217.19	777.97	25672.85	217.20	777.97	1.00
18	SArTagnan	2011-05-15 (fixed)	29	10%	34%	23228.65	174.03	800.99	2912.85	21.82	100.44	7.97
19	jMiniSat	2011	28	9%	33%	16092.05	291.69	574.72	16070.47	289.48	573.95	1.00
20	Plingeling	587f (fixed)	27	9%	31%	27373.29	50.18	1013.83	3440.21	6.69	127.42	7.96
21	sathys	2011-04-01	23	8%	27%	9370.44	28.55	407.41	9370.32	28.55	407.41	1.00
22	priss	1.0	21	7%	24%	13344.50	11.01	635.45	3337.67	2.76	158.94	4.00
23	sattime	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
24	sattime+	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "random instances" (RANDOM): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 600
Virtual Best Solver (VBS)			492	369 SAT, 123 UNSAT	82%	100%	109607.71	3.57	222.78	-	-	-	-
1	Satisfiability Solver Selector	2011-03-18	408	312 SAT, 96 UNSAT	68%	83%	141266.71	93.36	346.24	141334.92	93.90	346.41	1.00
2	SAT09 reference solver: SATzilla2009_R	2009-03-22	384	287 SAT, 97 UNSAT	64%	78%	133181.59	12.99	346.83	133179.19	12.99	346.82	1.00
3	ppfolio	par	377	276 SAT, 101 UNSAT	63%	77%	184063.52	15.88	488.23	23089.60	2.01	61.25	7.97
4	ppfolio	seq	375	308 SAT, 67 UNSAT	63%	76%	171810.28	8.85	458.16	171811.56	8.85	458.16	1.00
5	sparrow2011	sparrow2011_ubcsat1.2_2011-03-02	362	362 SAT	60%	74%	49813.56	4.48	137.61	49813.15	4.48	137.61	1.00
6	MPhaseSAT_M	2011-02-16	359	255 SAT, 104 UNSAT	60%	73%	77986.56	9.36	217.23	77985.99	9.36	217.23	1.00
7	sattime2011	2011-03-02	334	334 SAT	56%	68%	119975.54	2.84	359.21	119973.55	2.84	359.20	1.00
8	EagleUP	1.565.350	328	328 SAT	55%	67%	46537.56	1.32	141.88	46536.51	1.32	141.88	1.00
9	adaptg2wsat2011	2011-03-02	328	328 SAT	55%	67%	88737.65	2.60	270.54	88736.72	2.60	270.54	1.00
10	SAT09 reference solver: TNM	2009-03-22	324	324 SAT	54%	66%	113421.46	1.96	350.07	113419.07	1.97	350.06	1.00
11	csls	pnorm-8cores	309	309 SAT	52%	63%	99581.50	1.71	322.27	12476.83	0.23	40.38	7.98
12	SAT09 reference solver: gnovelty+2	2009-03-22	298	298 SAT	50%	61%	86017.56	1.42	288.65	86015.33	1.42	288.64	1.00
13	march_rw	2011-03-02	251	128 SAT, 123 UNSAT	42%	51%	130200.54	36.10	518.73	130199.50	36.10	518.72	1.00
14	SAT09 reference solver: march_hi	hi	251	128 SAT, 123 UNSAT	42%	51%	135632.45	35.46	540.37	135629.18	35.47	540.36	1.00

Category "random instances" (RANDOM): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 600
Virtual Best Solver (VBS)			369	62%	100%	44479.65	0.82	120.54	-	-	-	-
1	sparrow2011	sparrow2011_ubcsat1.2_2011-03-02	362	60%	98%	49813.56	4.48	137.61	49813.15	4.48	137.61	1.00
2	sattime2011	2011-03-02	334	56%	91%	119975.54	2.84	359.21	119973.55	2.84	359.20	1.00
3	EagleUP	1.565.350	328	55%	89%	46537.56	1.32	141.88	46536.51	1.32	141.88	1.00
4	adaptg2wsat2011	2011-03-02	328	55%	89%	88737.65	2.60	270.54	88736.72	2.60	270.54	1.00
5	SAT09 reference solver: TNM	2009-03-22	324	54%	88%	113421.46	1.96	350.07	113419.07	1.97	350.06	1.00
6	Satisfiability Solver Selector	2011-03-18	312	52%	85%	113172.74	36.05	362.73	113215.83	36.23	362.87	1.00
7	csls	pnorm-8cores	309	52%	84%	99581.50	1.71	322.27	12476.83	0.23	40.38	7.98
8	ppfolio	seq	308	51%	83%	129702.81	5.00	421.11	129703.06	5.03	421.11	1.00
9	SAT09 reference solver: gnovelty+2	2009-03-22	298	50%	81%	86017.56	1.42	288.65	86015.33	1.42	288.64	1.00
10	SAT09 reference solver: SATzilla2009_R	2009-03-22	287	48%	78%	78298.89	1.02	272.82	78297.53	1.02	272.81	1.00
11	ppfolio	par	276	46%	75%	102283.23	5.83	370.59	12814.90	0.87	46.43	7.98
12	MPhaseSAT_M	2011-02-16	255	43%	69%	45756.38	1.35	179.44	45756.33	1.35	179.44	1.00
13	march_rw	2011-03-02	128	21%	35%	64017.27	46.66	500.13	64016.51	46.66	500.13	1.00
14	SAT09 reference solver: march_hi	hi	128	21%	35%	67316.96	46.41	525.91	67315.52	46.41	525.90	1.00

Category "random instances" (RANDOM): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 600
Virtual Best Solver (VBS)			123	21%	100%	65128.06	32.96	529.50	-	-	-	-
1	march_rw	2011-03-02	123	21%	100%	66183.27	33.53	538.08	66182.99	33.53	538.07	1.00
2	SAT09 reference solver: march_hi	hi	123	21%	100%	68315.49	34.10	555.41	68313.66	34.10	555.40	1.00
3	MPhaseSAT_M	2011-02-16	104	17%	85%	32230.18	116.81	309.91	32229.65	116.81	309.90	1.00
4	ppfolio	par	101	17%	82%	81780.29	180.54	809.71	10274.70	22.59	101.73	7.96
5	SAT09 reference solver: SATzilla2009_R	2009-03-22	97	16%	79%	54882.70	120.70	565.80	54881.66	120.71	565.79	1.00
6	Satisfiability Solver Selector	2011-03-18	96	16%	78%	28093.96	169.38	292.65	28119.10	170.52	292.91	1.00
7	ppfolio	seq	67	11%	54%	42107.47	226.08	628.47	42108.51	226.08	628.49	1.00
8	SAT09 reference solver: gnovelty+2	2009-03-22	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
9	SAT09 reference solver: TNM	2009-03-22	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
10	sparrow2011	sparrow2011_ubcsat1.2_2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
11	EagleUP	1.565.350	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
12	csls	pnorm-8cores	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
13	sattime2011	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
14	adaptg2wsat2011	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

SAT 2011 Competition: main competition (phase 2): ranking of solvers max CPU time for any solver=5000 s max WC time for any solver=5000 s