SAT 2011 Competition: main competition (phase 2): ranking of solvers
max CPU time for sequential solvers=5000 s
max CPU time for parallel solvers=40000 s
max WC time for any solver=5000 s

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "applications instances" (APPLICATION): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			258	116 SAT, 142 UNSAT	86%	100%	181310.92	41.24	702.76	-	-	-	-
1	Plingeling	587f (fixed)	230	106 SAT, 124 UNSAT	77%	89%	905080.32	343.82	3935.13	114263.55	52.28	496.80	7.92
2	CryptoMiniSat	Strange-Night-2-mt (fixed)	230	97 SAT, 133 UNSAT	77%	89%	906888.53	598.91	3942.99	119854.10	79.40	521.10	7.57
3	ppfolio	par	229	107 SAT, 122 UNSAT	76%	89%	984028.43	363.16	4297.07	126415.99	46.32	552.03	7.78
4	glucose	2	215	94 SAT, 121 UNSAT	72%	83%	131771.28	80.17	612.89	131811.24	80.18	613.08	1.00
5	claspmt	2.0-R4095-patched (fixed)	211	95 SAT, 116 UNSAT	70%	82%	831560.43	481.46	3941.04	137339.21	105.12	650.90	6.05
6	glueminisat	2.2.5	211	85 SAT, 126 UNSAT	70%	82%	141937.40	56.28	672.69	141934.06	56.29	672.67	1.00
7	Lingeling	587f (fixed)	208	91 SAT, 117 UNSAT	69%	81%	117889.23	71.82	566.78	117894.89	72.06	566.80	1.00
8	CryptoMiniSat	Strange-Night2-st (fixed)	207	88 SAT, 119 UNSAT	69%	80%	106101.58	96.69	512.57	106107.81	96.69	512.60	1.00
9	QuteRSat	2011-05-12 (fixed)	206	86 SAT, 120 UNSAT	69%	80%	114120.09	93.79	553.98	114118.44	93.80	553.97	1.00
10	SAT09 reference solver:precosat	236	206	96 SAT, 110 UNSAT	69%	80%	122104.77	113.43	592.74	122109.68	113.49	592.77	1.00
11	minisat hack: cir_minisat	2011-05-13 (simp)	202	99 SAT, 103 UNSAT	67%	78%	128125.68	69.99	634.29	128126.19	69.99	634.29	1.00
12	RestartSAT	B95	201	97 SAT, 104 UNSAT	67%	78%	103443.89	83.48	514.65	103442.81	83.50	514.64	1.00
13	MPhaseSAT64	2011-05-14 (fixed)	201	99 SAT, 102 UNSAT	67%	78%	126764.36	104.29	630.67	126763.18	104.41	630.66	1.00
14	Satisfiability Solver Selector	2011-03-18	200	91 SAT, 109 UNSAT	67%	78%	174515.04	159.89	872.58	174434.65	160.05	872.17	1.00
15	minisat hack: contrasat	2011-03-02	199	99 SAT, 100 UNSAT	66%	77%	132726.28	79.05	666.97	132725.67	79.05	666.96	1.00
16	EBGlucose	1.0	199	91 SAT, 108 UNSAT	66%	77%	142439.18	59.44	715.77	142454.28	59.45	715.85	1.00
17	minisat hack: minisat_psm	2011-04-01	197	95 SAT, 102 UNSAT	66%	76%	115782.74	59.63	587.73	115922.43	59.63	588.44	1.00
18	SAT09 reference solver:glucose	1.0	197	87 SAT, 110 UNSAT	66%	76%	128390.81	97.47	651.73	128440.56	97.50	651.98	1.00
19	minisat hack: EBMiniSAT	2011-03-02	196	98 SAT, 98 UNSAT	65%	76%	107673.20	67.01	549.35	107676.29	67.01	549.37	1.00
20	borg-sat	11.03.02-par-industrial (fixed)	195	90 SAT, 105 UNSAT	65%	76%	574448.93	507.23	2945.89	92546.56	157.06	474.60	6.21
21	SAT09 reference solver: ManySAT 1.1	aimd 1/2009-03-20	194	85 SAT, 109 UNSAT	65%	75%	373511.19	176.92	1925.32	95503.73	47.00	492.29	3.91
22	minisat hack reference: minisat	2.2.0	194	95 SAT, 99 UNSAT	65%	75%	101499.07	62.12	523.19	101492.22	62.12	523.16	1.00
23	minisat hack: LR_GL_SHR	2011-03-02	191	81 SAT, 110 UNSAT	64%	74%	111723.47	60.94	584.94	111722.82	60.95	584.94	1.00
24	borg-sat	11.03.02-seq-industrial (fixed)	190	90 SAT, 100 UNSAT	63%	74%	110627.44	204.09	582.25	111259.55	209.38	585.58	0.99
25	minisat hack: MiniSAT	2.2.0-agile-26	189	95 SAT, 94 UNSAT	63%	73%	91483.85	49.63	484.04	91483.73	49.64	484.04	1.00
26	rcl	2011-04-01	184	93 SAT, 91 UNSAT	61%	71%	98231.37	67.98	533.87	98236.99	67.98	533.90	1.00

Category "applications instances" (APPLICATION): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			116	39%	100%	38135.96	22.37	328.76	-	-	-	-
1	ppfolio	par	107	36%	92%	403943.48	216.95	3775.17	52281.98	39.97	488.62	7.73
2	Plingeling	587f (fixed)	106	35%	91%	254003.02	205.42	2396.25	32388.79	34.61	305.55	7.84
3	minisat hack: contrasat	2011-03-02	99	33%	85%	41344.03	52.26	417.62	41344.13	52.25	417.62	1.00
4	minisat hack: cir_minisat	2011-05-13 (simp)	99	33%	85%	51118.56	47.44	516.35	51119.87	47.44	516.36	1.00
5	MPhaseSAT64	2011-05-14 (fixed)	99	33%	85%	57294.00	104.29	578.73	57292.89	104.41	578.72	1.00
6	minisat hack: EBMiniSAT	2011-03-02	98	33%	84%	47966.14	62.04	489.45	47968.94	62.04	489.48	1.00
7	CryptoMiniSat	Strange-Night-2-mt (fixed)	97	32%	84%	231744.00	364.66	2389.11	29889.92	46.31	308.14	7.75
8	RestartSAT	B95	97	32%	84%	46782.80	76.11	482.30	46782.58	76.11	482.29	1.00
9	SAT09 reference solver:precosat	236	96	32%	83%	51252.31	101.45	533.88	51258.50	101.54	533.94	1.00
10	minisat hack reference: minisat	2.2.0	95	32%	82%	35475.64	48.10	373.43	35477.25	48.11	373.44	1.00
11	minisat hack: MiniSAT	2.2.0-agile-26	95	32%	82%	37128.06	46.54	390.82	37128.64	46.54	390.83	1.00
12	minisat hack: minisat_psm	2011-04-01	95	32%	82%	39742.31	38.50	418.34	39788.17	38.49	418.82	1.00
13	claspmt	2.0-R4095-patched (fixed)	95	32%	82%	321233.38	378.00	3381.40	61769.50	100.59	650.21	5.20
14	glucose	2	94	31%	81%	43473.72	36.51	462.49	43514.41	41.81	462.92	1.00
15	rcl	2011-04-01	93	31%	80%	36633.36	47.89	393.91	36639.86	47.89	393.98	1.00
16	Lingeling	587f (fixed)	91	30%	78%	37172.20	59.71	408.49	37179.20	59.76	408.56	1.00
17	EBGlucose	1.0	91	30%	78%	54600.13	38.11	600.00	54603.02	38.11	600.03	1.00
18	Satisfiability Solver Selector	2011-03-18	91	30%	78%	58060.23	155.12	638.02	58083.68	155.30	638.28	1.00
19	borg-sat	11.03.02-par-industrial (fixed)	90	30%	78%	188393.10	508.04	2093.26	34344.99	153.32	381.61	5.49
20	borg-sat	11.03.02-seq-industrial (fixed)	90	30%	78%	45700.56	213.31	507.78	46107.60	215.74	512.31	0.99
21	CryptoMiniSat	Strange-Night2-st (fixed)	88	29%	76%	30990.02	75.07	352.16	30992.83	75.08	352.19	1.00
22	SAT09 reference solver:glucose	1.0	87	29%	75%	37731.46	47.23	433.69	37742.80	47.23	433.83	1.00
23	QuteRSat	2011-05-12 (fixed)	86	29%	74%	38780.09	83.53	450.93	38779.54	83.53	450.92	1.00
24	SAT09 reference solver: ManySAT 1.1	aimd 1/2009-03-20	85	28%	73%	119386.54	115.11	1404.55	30129.12	33.09	354.46	3.96
25	glueminisat	2.2.5	85	28%	73%	42157.38	38.80	495.97	42156.46	38.80	495.96	1.00
26	minisat hack: LR_GL_SHR	2011-03-02	81	27%	70%	43907.08	40.43	542.06	43907.56	40.43	542.07	1.00

Category "applications instances" (APPLICATION): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			142	47%	100%	143174.96	103.72	1008.27	-	-	-	-
1	CryptoMiniSat	Strange-Night-2-mt (fixed)	133	44%	94%	675144.53	959.97	5076.27	89964.17	126.32	676.42	7.50
2	glueminisat	2.2.5	126	42%	89%	99780.02	143.38	791.90	99777.60	143.38	791.89	1.00
3	Plingeling	587f (fixed)	124	41%	87%	651077.30	698.52	5250.62	81874.76	96.33	660.28	7.95
4	ppfolio	par	122	41%	86%	580084.95	465.54	4754.79	74134.01	64.94	607.66	7.82
5	glucose	2	121	40%	85%	88297.55	132.28	729.73	88296.83	132.29	729.73	1.00
6	QuteRSat	2011-05-12 (fixed)	120	40%	85%	75340.00	103.31	627.83	75338.89	103.31	627.82	1.00
7	CryptoMiniSat	Strange-Night2-st (fixed)	119	40%	84%	75111.56	108.40	631.19	75114.98	108.47	631.22	1.00
8	Lingeling	587f (fixed)	117	39%	82%	80717.03	128.51	689.89	80715.69	128.51	689.88	1.00
9	claspmt	2.0-R4095-patched (fixed)	116	39%	82%	510327.05	591.01	4399.37	75569.70	111.95	651.46	6.75
10	minisat hack: LR_GL_SHR	2011-03-02	110	37%	77%	67816.39	96.99	616.51	67815.26	96.99	616.50	1.00
11	SAT09 reference solver:precosat	236	110	37%	77%	70852.46	125.38	644.11	70851.17	125.38	644.10	1.00
12	SAT09 reference solver:glucose	1.0	110	37%	77%	90659.34	195.14	824.18	90697.76	195.15	824.53	1.00
13	SAT09 reference solver: ManySAT 1.1	aimd 1/2009-03-20	109	36%	77%	254124.65	410.21	2331.42	65374.61	109.62	599.77	3.89
14	Satisfiability Solver Selector	2011-03-18	109	36%	77%	116454.80	162.73	1068.39	116350.97	162.86	1067.44	1.00
15	EBGlucose	1.0	108	36%	76%	87839.05	150.45	813.32	87851.26	150.47	813.44	1.00
16	borg-sat	11.03.02-par-industrial (fixed)	105	35%	74%	386055.83	506.42	3676.72	58201.57	160.61	554.30	6.63
17	RestartSAT	B95	104	35%	73%	56661.09	99.46	544.82	56660.24	99.49	544.81	1.00
18	minisat hack: cir_minisat	2011-05-13 (simp)	103	34%	73%	77007.13	98.14	747.64	77006.32	98.14	747.63	1.00
19	MPhaseSAT64	2011-05-14 (fixed)	102	34%	72%	69470.37	106.95	681.08	69470.29	107.24	681.08	1.00
20	minisat hack: minisat_psm	2011-04-01	102	34%	72%	76040.44	97.87	745.49	76134.26	97.88	746.41	1.00
21	borg-sat	11.03.02-seq-industrial (fixed)	100	33%	70%	64926.88	199.87	649.27	65151.95	204.30	651.52	1.00
22	minisat hack: contrasat	2011-03-02	100	33%	70%	91382.25	167.44	913.82	91381.54	167.47	913.82	1.00
23	minisat hack reference: minisat	2.2.0	99	33%	70%	66023.43	92.07	666.90	66014.97	92.07	666.82	1.00
24	minisat hack: EBMiniSAT	2011-03-02	98	33%	69%	59707.06	83.02	609.26	59707.35	83.02	609.26	1.00
25	minisat hack: MiniSAT	2.2.0-agile-26	94	31%	66%	54355.80	71.57	578.25	54355.09	71.57	578.25	1.00
26	rcl	2011-04-01	91	30%	64%	61598.01	105.69	676.90	61597.13	105.69	676.89	1.00

Category "crafted instances" (CRAFTED): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			232	146 SAT, 86 UNSAT	77%	100%	168366.50	4.31	725.72	-	-	-	-
1	ppfolio	par	187	127 SAT, 60 UNSAT	62%	81%	651564.93	44.12	3484.30	82245.80	5.53	439.82	7.92
2	claspmt	2.0-R4095-patched (fixed)	170	100 SAT, 70 UNSAT	57%	73%	878007.69	677.24	5164.75	110025.30	84.95	647.21	7.98
3	Satisfiability Solver Selector	2011-03-18	163	112 SAT, 51 UNSAT	54%	70%	95298.33	39.84	584.65	95323.70	40.02	584.81	1.00
4	ppfolio	seq	152	114 SAT, 38 UNSAT	51%	66%	70616.85	9.20	464.58	70617.62	9.25	464.59	1.00
5	clasp	2.0-R4092-crafted	149	87 SAT, 62 UNSAT	50%	64%	143801.90	173.21	965.11	143799.50	173.21	965.10	1.00
6	SAT09 reference solver: clasp	1.2.0-SAT09-32	149	85 SAT, 64 UNSAT	50%	64%	174493.98	328.76	1171.10	174489.72	328.75	1171.07	1.00
7	SArTagnan	2011-05-15 (fixed)	148	101 SAT, 47 UNSAT	49%	64%	623409.97	22.02	4212.23	78122.84	2.81	527.86	7.98
8	MPhaseSAT	2011-02-15	132	93 SAT, 39 UNSAT	44%	57%	85067.06	45.18	644.45	85067.17	45.18	644.45	1.00
9	CryptoMiniSat	Strange-Night-2-mt (fixed)	131	86 SAT, 45 UNSAT	44%	56%	532951.67	228.86	4068.33	66949.50	28.82	511.06	7.96
10	Plingeling	587f (fixed)	131	91 SAT, 40 UNSAT	44%	56%	543456.56	112.92	4148.52	68173.69	14.34	520.41	7.97
11	glucose	2	125	83 SAT, 42 UNSAT	42%	54%	104343.97	163.86	834.75	104342.45	163.85	834.74	1.00
12	SAT07 reference solver: minisat	SAT 2007	122	78 SAT, 44 UNSAT	41%	53%	110026.47	103.10	901.86	110023.91	103.10	901.84	1.00
13	PicoSAT	941	121	81 SAT, 40 UNSAT	40%	52%	103099.28	94.45	852.06	103099.04	94.45	852.06	1.00
14	Sol	2011-04-04	116	79 SAT, 37 UNSAT	39%	50%	41526.46	6.31	357.99	40991.82	5.82	353.38	1.01
15	SAT09 reference solver: SATzilla2009_C	2009-03-22	116	73 SAT, 43 UNSAT	39%	50%	100720.22	62.57	868.28	100714.27	62.57	868.23	1.00
16	RestartSAT	B95	112	77 SAT, 35 UNSAT	37%	48%	67761.33	72.09	605.01	67760.34	72.08	605.00	1.00
17	CryptoMiniSat	Strange-Night2-st (fixed)	110	70 SAT, 40 UNSAT	37%	47%	56189.51	66.49	510.81	56188.98	66.49	510.81	1.00
18	QuteRSat	2011-05-12 (fixed)	110	70 SAT, 40 UNSAT	37%	47%	62230.60	45.83	565.73	62229.70	45.83	565.72	1.00
19	sattime	2011-03-02	109	109 SAT	36%	47%	22998.70	1.77	211.00	22998.62	1.77	211.00	1.00
20	SApperloT2010	2011-05-15 (fixed)	109	76 SAT, 33 UNSAT	36%	47%	58317.14	31.21	535.02	58316.92	31.21	535.02	1.00
21	sattime+	2011-03-02	106	106 SAT	35%	46%	17523.59	2.26	165.32	17523.33	2.26	165.31	1.00
22	priss	1.0	104	73 SAT, 31 UNSAT	35%	45%	285971.88	124.22	2749.73	71502.34	31.11	687.52	4.00
23	jMiniSat	2011	98	70 SAT, 28 UNSAT	33%	42%	53798.17	37.63	548.96	53722.49	37.15	548.19	1.00
24	sathys	2011-04-01	96	73 SAT, 23 UNSAT	32%	41%	27948.96	2.79	291.13	27949.17	2.80	291.14	1.00

Category "crafted instances" (CRAFTED): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			146	49%	100%	16219.25	0.45	111.09	-	-	-	-
1	ppfolio	par	127	42%	87%	158178.53	14.94	1245.50	19800.87	1.89	155.91	7.99
2	ppfolio	seq	114	38%	78%	42748.08	5.47	374.98	42749.20	5.55	374.99	1.00
3	Satisfiability Solver Selector	2011-03-18	112	37%	77%	30625.58	32.77	273.44	30643.81	33.34	273.61	1.00
4	sattime	2011-03-02	109	36%	75%	22998.70	1.77	211.00	22998.62	1.77	211.00	1.00
5	sattime+	2011-03-02	106	35%	73%	17523.59	2.26	165.32	17523.33	2.26	165.31	1.00
6	SArTagnan	2011-05-15 (fixed)	101	34%	69%	303855.88	0.47	3008.47	38088.82	0.16	377.12	7.98
7	claspmt	2.0-R4095-patched (fixed)	100	33%	68%	407651.86	290.57	4076.52	51090.38	36.71	510.90	7.98
8	MPhaseSAT	2011-02-15	93	31%	64%	44250.09	16.71	475.81	44250.75	16.97	475.81	1.00
9	Plingeling	587f (fixed)	91	30%	62%	294787.89	34.29	3239.43	36970.49	4.48	406.27	7.97
10	clasp	2.0-R4092-crafted	87	29%	60%	63277.05	118.80	727.32	63275.98	118.80	727.31	1.00
11	CryptoMiniSat	Strange-Night-2-mt (fixed)	86	29%	59%	307342.16	116.38	3573.75	38638.95	15.43	449.29	7.95
12	SAT09 reference solver: clasp	1.2.0-SAT09-32	85	28%	58%	79260.72	105.00	932.48	79258.80	105.00	932.46	1.00
13	glucose	2	83	28%	57%	69764.82	150.41	840.54	69763.88	150.43	840.53	1.00
14	PicoSAT	941	81	27%	55%	52974.15	38.52	654.00	52974.65	38.52	654.01	1.00
15	Sol	2011-04-04	79	26%	54%	21815.85	4.37	276.15	21487.77	4.34	272.00	1.02
16	SAT07 reference solver: minisat	SAT 2007	78	26%	53%	61282.26	29.56	785.67	61280.88	29.56	785.65	1.00
17	RestartSAT	B95	77	26%	53%	33016.95	18.18	428.79	33016.81	18.18	428.79	1.00
18	SApperloT2010	2011-05-15 (fixed)	76	25%	52%	32644.09	13.56	429.53	32644.07	13.57	429.53	1.00
19	sathys	2011-04-01	73	24%	50%	18578.52	0.63	254.50	18578.86	0.63	254.50	1.00
20	priss	1.0	73	24%	50%	140251.03	111.82	1921.25	35069.42	27.99	480.40	4.00
21	SAT09 reference solver: SATzilla2009_C	2009-03-22	73	24%	50%	64400.03	24.07	882.19	64395.01	24.07	882.12	1.00
22	CryptoMiniSat	Strange-Night2-st (fixed)	70	23%	48%	29216.21	29.26	417.37	29216.02	29.26	417.37	1.00
23	jMiniSat	2011	70	23%	48%	37706.12	20.42	538.66	37652.02	19.82	537.89	1.00
24	QuteRSat	2011-05-12 (fixed)	70	23%	48%	38283.34	22.60	546.90	38282.89	22.61	546.90	1.00

Category "crafted instances" (CRAFTED): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)			86	29%	100%	152147.24	683.34	1769.15	-	-	-	-
1	claspmt	2.0-R4095-patched (fixed)	70	23%	81%	470355.83	2304.39	6719.37	58934.92	288.82	841.93	7.98
2	SAT09 reference solver: clasp	1.2.0-SAT09-32	64	21%	74%	95233.26	1007.65	1488.02	95230.93	1007.62	1487.98	1.00
3	clasp	2.0-R4092-crafted	62	21%	72%	80524.85	817.87	1298.79	80523.53	817.84	1298.77	1.00
4	ppfolio	par	60	20%	70%	493386.40	2025.82	8223.11	62444.92	253.43	1040.75	7.90
5	Satisfiability Solver Selector	2011-03-18	51	17%	59%	64672.75	821.55	1268.09	64679.90	821.67	1268.23	1.00
6	SArTagnan	2011-05-15 (fixed)	47	16%	55%	319554.10	1938.54	6799.02	40034.01	242.90	851.79	7.98
7	CryptoMiniSat	Strange-Night-2-mt (fixed)	45	15%	52%	225609.51	1104.22	5013.54	28310.56	138.33	629.12	7.97
8	SAT07 reference solver: minisat	SAT 2007	44	15%	51%	48744.21	206.54	1107.82	48743.03	206.53	1107.80	1.00
9	SAT09 reference solver: SATzilla2009_C	2009-03-22	43	14%	50%	36320.19	198.17	844.66	36319.27	198.17	844.63	1.00
10	glucose	2	42	14%	49%	34579.15	189.63	823.31	34578.57	189.62	823.30	1.00
11	QuteRSat	2011-05-12 (fixed)	40	13%	47%	23947.25	202.60	598.68	23946.81	202.61	598.67	1.00
12	CryptoMiniSat	Strange-Night2-st (fixed)	40	13%	47%	26973.30	233.96	674.33	26972.96	233.95	674.32	1.00
13	Plingeling	587f (fixed)	40	13%	47%	248668.67	1243.90	6216.72	31203.20	156.21	780.08	7.97
14	PicoSAT	941	40	13%	47%	50125.13	417.44	1253.13	50124.39	417.49	1253.11	1.00
15	MPhaseSAT	2011-02-15	39	13%	45%	40816.97	347.60	1046.59	40816.41	347.61	1046.57	1.00
16	ppfolio	seq	38	13%	44%	27868.77	241.92	733.39	27868.43	241.91	733.38	1.00
17	Sol	2011-04-04	37	12%	43%	19710.61	37.91	532.72	19504.05	37.32	527.14	1.01
18	RestartSAT	B95	35	12%	41%	34744.38	303.69	992.70	34743.53	303.68	992.67	1.00
19	SApperloT2010	2011-05-15 (fixed)	33	11%	38%	25673.05	217.19	777.97	25672.85	217.20	777.97	1.00
20	priss	1.0	31	10%	36%	145720.85	348.00	4700.67	36432.91	87.07	1175.26	4.00
21	jMiniSat	2011	28	9%	33%	16092.05	291.69	574.72	16070.47	289.48	573.95	1.00
22	sathys	2011-04-01	23	8%	27%	9370.44	28.55	407.41	9370.32	28.55	407.41	1.00
23	sattime	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
24	sattime+	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "random instances" (RANDOM): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 600
Virtual Best Solver (VBS)			492	369 SAT, 123 UNSAT	82%	100%	109607.71	3.57	222.78	-	-	-	-
1	ppfolio	par	444	322 SAT, 122 UNSAT	74%	90%	1353071.40	45.07	3047.46	170090.00	5.80	383.09	7.96
2	Satisfiability Solver Selector	2011-03-18	408	312 SAT, 96 UNSAT	68%	83%	141266.71	93.36	346.24	141334.92	93.90	346.41	1.00
3	SAT09 reference solver: SATzilla2009_R	2009-03-22	384	287 SAT, 97 UNSAT	64%	78%	133181.59	12.99	346.83	133179.19	12.99	346.82	1.00
4	ppfolio	seq	375	308 SAT, 67 UNSAT	63%	76%	171810.28	8.85	458.16	171811.56	8.85	458.16	1.00
5	sparrow2011	sparrow2011_ubcsat1.2_2011-03-02	362	362 SAT	60%	74%	49813.56	4.48	137.61	49813.15	4.48	137.61	1.00
6	MPhaseSAT_M	2011-02-16	359	255 SAT, 104 UNSAT	60%	73%	77986.56	9.36	217.23	77985.99	9.36	217.23	1.00
7	csls	pnorm-8cores	334	334 SAT	56%	68%	478685.42	2.30	1433.19	60011.40	0.32	179.67	7.98
8	sattime2011	2011-03-02	334	334 SAT	56%	68%	119975.54	2.84	359.21	119973.55	2.84	359.20	1.00
9	EagleUP	1.565.350	328	328 SAT	55%	67%	46537.56	1.32	141.88	46536.51	1.32	141.88	1.00
10	adaptg2wsat2011	2011-03-02	328	328 SAT	55%	67%	88737.65	2.60	270.54	88736.72	2.60	270.54	1.00
11	SAT09 reference solver: TNM	2009-03-22	324	324 SAT	54%	66%	113421.46	1.96	350.07	113419.07	1.97	350.06	1.00
12	SAT09 reference solver: gnovelty+2	2009-03-22	298	298 SAT	50%	61%	86017.56	1.42	288.65	86015.33	1.42	288.64	1.00
13	march_rw	2011-03-02	251	128 SAT, 123 UNSAT	42%	51%	130200.54	36.10	518.73	130199.50	36.10	518.72	1.00
14	SAT09 reference solver: march_hi	hi	251	128 SAT, 123 UNSAT	42%	51%	135632.45	35.46	540.37	135629.18	35.47	540.36	1.00

Category "random instances" (RANDOM): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 600
Virtual Best Solver (VBS)			369	62%	100%	44479.65	0.82	120.54	-	-	-	-
1	sparrow2011	sparrow2011_ubcsat1.2_2011-03-02	362	60%	98%	49813.56	4.48	137.61	49813.15	4.48	137.61	1.00
2	csls	pnorm-8cores	334	56%	91%	478685.42	2.30	1433.19	60011.40	0.32	179.67	7.98
3	sattime2011	2011-03-02	334	56%	91%	119975.54	2.84	359.21	119973.55	2.84	359.20	1.00
4	EagleUP	1.565.350	328	55%	89%	46537.56	1.32	141.88	46536.51	1.32	141.88	1.00
5	adaptg2wsat2011	2011-03-02	328	55%	89%	88737.65	2.60	270.54	88736.72	2.60	270.54	1.00
6	SAT09 reference solver: TNM	2009-03-22	324	54%	88%	113421.46	1.96	350.07	113419.07	1.97	350.06	1.00
7	ppfolio	par	322	54%	87%	832894.21	16.00	2586.63	104976.62	2.02	326.01	7.93
8	Satisfiability Solver Selector	2011-03-18	312	52%	85%	113172.74	36.05	362.73	113215.83	36.23	362.87	1.00
9	ppfolio	seq	308	51%	83%	129702.81	5.00	421.11	129703.06	5.03	421.11	1.00
10	SAT09 reference solver: gnovelty+2	2009-03-22	298	50%	81%	86017.56	1.42	288.65	86015.33	1.42	288.64	1.00
11	SAT09 reference solver: SATzilla2009_R	2009-03-22	287	48%	78%	78298.89	1.02	272.82	78297.53	1.02	272.81	1.00
12	MPhaseSAT_M	2011-02-16	255	43%	69%	45756.38	1.35	179.44	45756.33	1.35	179.44	1.00
13	march_rw	2011-03-02	128	21%	35%	64017.27	46.66	500.13	64016.51	46.66	500.13	1.00
14	SAT09 reference solver: march_hi	hi	128	21%	35%	67316.96	46.41	525.91	67315.52	46.41	525.90	1.00

Category "random instances" (RANDOM): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 600
Virtual Best Solver (VBS)			123	21%	100%	65128.06	32.96	529.50	-	-	-	-
1	march_rw	2011-03-02	123	21%	100%	66183.27	33.53	538.08	66182.99	33.53	538.07	1.00
2	SAT09 reference solver: march_hi	hi	123	21%	100%	68315.49	34.10	555.41	68313.66	34.10	555.40	1.00
3	ppfolio	par	122	20%	99%	520177.19	267.35	4263.75	65113.38	33.45	533.72	7.99
4	MPhaseSAT_M	2011-02-16	104	17%	85%	32230.18	116.81	309.91	32229.65	116.81	309.90	1.00
5	SAT09 reference solver: SATzilla2009_R	2009-03-22	97	16%	79%	54882.70	120.70	565.80	54881.66	120.71	565.79	1.00
6	Satisfiability Solver Selector	2011-03-18	96	16%	78%	28093.96	169.38	292.65	28119.10	170.52	292.91	1.00
7	ppfolio	seq	67	11%	54%	42107.47	226.08	628.47	42108.51	226.08	628.49	1.00
8	SAT09 reference solver: gnovelty+2	2009-03-22	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
9	SAT09 reference solver: TNM	2009-03-22	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
10	sparrow2011	sparrow2011_ubcsat1.2_2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
11	EagleUP	1.565.350	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
12	csls	pnorm-8cores	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
13	sattime2011	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
14	adaptg2wsat2011	2011-03-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

SAT 2011 Competition: main competition (phase 2): ranking of solvers max CPU time for sequential solvers=5000 s max CPU time for parallel solvers=40000 s max WC time for any solver=5000 s