CSP 2008 Competition: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "binary constraints in extension" (2-ARY-EXT)

Rank	Solver	Version	Number of solved instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Virtual Best Solver (VBS)			622	100%	29521.74	47.46
1	cpHydra	k_10	574	92%	35840.55	62.44
2	cpHydra	k_40	567	91%	58439.64	103.07
3	MDG-probe	2008-06-27	561	90%	38254.42	68.19
4	Mistral-option	1.314	559	90%	39615.82	70.87
5	MDG-noprobe	2008-06-27	558	90%	42913.77	76.91
6	Mistral-prime	1.313	556	89%	39037.26	70.21
7	choco2_dwdeg	2008-06-26	554	89%	56994.46	102.88
8	choco2_impwdeg	2008-06-26	550	88%	52681.12	95.78
9	Abscon 112v4	ESAC	546	88%	49367.61	90.42
10	Abscon 112v4	AC	546	88%	50920.72	93.26
11	Concrete + CPS4J	2008-05-30	498	80%	44900.63	90.16
12	bpsolver	2008-06-27	495	80%	38285.38	77.34
13	spider	2008-06-14	472	76%	72506.25	153.61
14	Sugar	v1.13+minisat	470	76%	76715.74	163.22
15	Sugar	v1.13+picosat	443	71%	52199.00	117.83
16	casper	zito	433	70%	85080.54	196.49
17	SAT4J CSP	2008-06-13	426	68%	53951.41	126.65
18	galac	E	425	68%	37221.02	87.58
19	galac	M	422	68%	46813.60	110.93
20	casper	zao	415	67%	91280.48	219.95
21	Minion/Tailor	2008-07-04	414	67%	61078.43	147.53
22	Concrete + CSP4J - Tabu Engine	2008-05-30	348	56%	43040.67	123.68

Category "binary constraints in intension" (2-ARY-INT)

Rank	Solver	Version	Number of solved instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Virtual Best Solver (VBS)			634	100%	19274.75	30.40
1	cpHydra	k_40	597	94%	59662.02	99.94
2	cpHydra	k_10	596	94%	42539.33	71.37
3	choco2_dwdeg	2008-06-26	523	82%	32147.08	61.47
4	Mistral-prime	1.313	522	82%	30397.71	58.23
5	choco2_impwdeg	2008-06-26	521	82%	29120.02	55.89
6	Abscon 112v4	ESAC	520	82%	19623.97	37.74
7	MDG-probe	2008-06-27	515	81%	52706.41	102.34
8	Abscon 112v4	AC	513	81%	22262.25	43.40
9	MDG-noprobe	2008-06-27	499	79%	36447.71	73.04
10	Sugar	v1.13+picosat	486	77%	32676.12	67.23
11	Sugar	v1.13+minisat	484	76%	32217.43	66.56
12	Mistral-option	1.314	451	71%	29183.67	64.71
13	Concrete + CPS4J	2008-05-30	415	65%	45186.63	108.88
14	bpsolver	2008-06-27	375	59%	35707.06	95.22
15	SAT4J CSP	2008-06-13	309	49%	17304.33	56.00
16	Concrete + CSP4J - WMC Engine	2008-05-30	222	35%	12640.97	56.94
17	Concrete + CSP4J - Tabu Engine	2008-05-30	183	29%	28950.90	158.20

Category "global constraints" (GLOBAL)

Rank	Solver	Version	Number of solved instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Virtual Best Solver (VBS)			501	100%	12233.86	24.42
1	Sugar	v1.13+picosat	424	85%	23575.77	55.60
2	cpHydra	k_40	420	84%	80832.22	192.46
3	cpHydra	k_10	419	84%	33879.89	80.86
4	Sugar	v1.13+minisat	405	81%	30617.59	75.60
5	Mistral-prime	1.313	403	80%	22849.50	56.70
6	casper	zito	397	79%	32937.25	82.97
7	casper	zao	390	78%	24804.15	63.60
8	Mistral-option	1.314	383	76%	37615.36	98.21
9	choco2_dwdeg	2008-06-26	358	71%	26103.18	72.91
10	MDG-noprobe	2008-06-27	353	70%	16548.16	46.88
11	choco2_impwdeg	2008-06-26	347	69%	28393.77	81.83
12	bpsolver	2008-06-27	347	69%	48728.37	140.43
13	MDG-probe	2008-06-27	337	67%	17668.90	52.43
14	Minion/Tailor	2008-07-04	216	43%	14889.77	68.93
15	Abscon 112v4	AC	184	37%	31394.49	170.62
16	Abscon 112v4	ESAC	173	35%	33623.73	194.36
17	SAT4J CSP	2008-06-13	63	13%	8092.34	128.45

Category "n-ary constraints in extension" (N-ARY-EXT)

Rank	Solver	Version	Number of solved instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Virtual Best Solver (VBS)			607	100%	41354.16	68.13
1	cpHydra	k_40	587	97%	53549.56	91.23
2	cpHydra	k_10	586	97%	45823.86	78.20
3	mddc-solv	2008-06-09	576	95%	47121.10	81.81
4	MDG-noprobe	2008-06-27	570	94%	48959.10	85.89
5	Mistral-prime	1.313	569	94%	47608.93	83.67
6	Mistral-option	1.314	565	93%	50550.95	89.47
7	Abscon 112v4	AC	545	90%	43650.74	80.09
8	Concrete + CPS4J	2008-05-30	544	90%	70154.84	128.96
9	Abscon 112v4	ESAC	541	89%	45027.23	83.23
10	choco2_dwdeg	2008-06-26	477	79%	77921.22	163.36
11	choco2_impwdeg	2008-06-26	441	73%	83391.49	189.10
12	Minion/Tailor	2008-07-04	406	67%	42112.68	103.73
13	casper	zito	391	64%	67482.76	172.59
14	casper	zao	386	64%	66833.33	173.14
15	Sugar	v1.13+minisat	370	61%	63823.88	172.50
16	Sugar	v1.13+picosat	347	57%	58679.96	169.11
17	galac	M	326	54%	40481.99	124.18
18	galac	E	323	53%	38360.21	118.76
19	bpsolver	2008-06-27	322	53%	53278.41	165.46
20	SAT4J CSP	2008-06-13	218	36%	28100.83	128.90
21	Concrete + CSP4J - Tabu Engine	2008-05-30	210	35%	36253.41	172.64
22	Concrete + CSP4J - WMC Engine	2008-05-30	135	22%	20996.34	155.53

Category "n-ary constraints in intension" (N-ARY-INT)

Rank	Solver	Version	Number of solved instances	% of VBS	Cumulated CPU time on solved instances	Average CPU time per solved instance
Virtual Best Solver (VBS)			660	100%	20689.09	31.35
1	cpHydra	k_10	569	86%	31124.32	54.70
2	cpHydra	k_40	569	86%	49899.25	87.70
3	casper	zao	562	85%	31205.61	55.53
4	Mistral-prime	1.313	560	85%	17929.82	32.02
5	MDG-probe	2008-06-27	560	85%	20146.70	35.98
6	casper	zito	555	84%	31629.81	56.99
7	Mistral-option	1.314	554	84%	43364.56	78.28
8	MDG-noprobe	2008-06-27	530	80%	18115.71	34.18
9	choco2_impwdeg	2008-06-26	518	78%	25745.10	49.70
10	Abscon 112v4	ESAC	488	74%	36101.57	73.98
11	Abscon 112v4	AC	486	74%	25860.43	53.21
12	Sugar	v1.13+minisat	486	74%	29306.62	60.30
13	Sugar	v1.13+picosat	481	73%	26580.81	55.26
14	bpsolver	2008-06-27	459	70%	77397.38	168.62
15	Concrete + CPS4J	2008-05-30	456	69%	91862.96	201.45
16	Concrete + CSP4J - Tabu Engine	2008-05-30	199	30%	36056.08	181.19
17	SAT4J CSP	2008-06-13	174	26%	19628.31	112.81
18	Concrete + CSP4J - WMC Engine	2008-05-30	174	26%	23628.78	135.80