CSP 2008 Competition: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "binary constraints in extension" (2-ARY-EXT)

Rank SolverVersionNumber of
solved instances
% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Virtual Best Solver (VBS)622100%29521.7447.46
1cpHydrak_1057492%35840.5562.44
2cpHydrak_4056791%58439.64103.07
3MDG-probe2008-06-2756190%38254.4268.19
4Mistral-option1.31455990%39615.8270.87
5MDG-noprobe2008-06-2755890%42913.7776.91
6Mistral-prime1.31355689%39037.2670.21
7choco2_dwdeg2008-06-2655489%56994.46102.88
8choco2_impwdeg2008-06-2655088%52681.1295.78
9Abscon 112v4ESAC54688%49367.6190.42
10Abscon 112v4AC54688%50920.7293.26
11Concrete + CPS4J2008-05-3049880%44900.6390.16
12bpsolver2008-06-2749580%38285.3877.34
13spider2008-06-1447276%72506.25153.61
14Sugarv1.13+minisat47076%76715.74163.22
15Sugarv1.13+picosat44371%52199.00117.83
16casperzito43370%85080.54196.49
17SAT4J CSP2008-06-1342668%53951.41126.65
18galacE42568%37221.0287.58
19galacM42268%46813.60110.93
20casperzao41567%91280.48219.95
21Minion/Tailor2008-07-0441467%61078.43147.53
22Concrete + CSP4J - Tabu Engine2008-05-3034856%43040.67123.68

Category "binary constraints in intension" (2-ARY-INT)

Rank SolverVersionNumber of
solved instances
% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Virtual Best Solver (VBS)634100%19274.7530.40
1cpHydrak_4059794%59662.0299.94
2cpHydrak_1059694%42539.3371.37
3choco2_dwdeg2008-06-2652382%32147.0861.47
4Mistral-prime1.31352282%30397.7158.23
5choco2_impwdeg2008-06-2652182%29120.0255.89
6Abscon 112v4ESAC52082%19623.9737.74
7MDG-probe2008-06-2751581%52706.41102.34
8Abscon 112v4AC51381%22262.2543.40
9MDG-noprobe2008-06-2749979%36447.7173.04
10Sugarv1.13+picosat48677%32676.1267.23
11Sugarv1.13+minisat48476%32217.4366.56
12Mistral-option1.31445171%29183.6764.71
13Concrete + CPS4J2008-05-3041565%45186.63108.88
14bpsolver2008-06-2737559%35707.0695.22
15SAT4J CSP2008-06-1330949%17304.3356.00
16Concrete + CSP4J - WMC Engine2008-05-3022235%12640.9756.94
17Concrete + CSP4J - Tabu Engine2008-05-3018329%28950.90158.20

Category "global constraints" (GLOBAL)

Rank SolverVersionNumber of
solved instances
% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Virtual Best Solver (VBS)501100%12233.8624.42
1Sugarv1.13+picosat42485%23575.7755.60
2cpHydrak_4042084%80832.22192.46
3cpHydrak_1041984%33879.8980.86
4Sugarv1.13+minisat40581%30617.5975.60
5Mistral-prime1.31340380%22849.5056.70
6casperzito39779%32937.2582.97
7casperzao39078%24804.1563.60
8Mistral-option1.31438376%37615.3698.21
9choco2_dwdeg2008-06-2635871%26103.1872.91
10MDG-noprobe2008-06-2735370%16548.1646.88
11choco2_impwdeg2008-06-2634769%28393.7781.83
12bpsolver2008-06-2734769%48728.37140.43
13MDG-probe2008-06-2733767%17668.9052.43
14Minion/Tailor2008-07-0421643%14889.7768.93
15Abscon 112v4AC18437%31394.49170.62
16Abscon 112v4ESAC17335%33623.73194.36
17SAT4J CSP2008-06-136313%8092.34128.45

Category "n-ary constraints in extension" (N-ARY-EXT)

Rank SolverVersionNumber of
solved instances
% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Virtual Best Solver (VBS)607100%41354.1668.13
1cpHydrak_4058797%53549.5691.23
2cpHydrak_1058697%45823.8678.20
3mddc-solv2008-06-0957695%47121.1081.81
4MDG-noprobe2008-06-2757094%48959.1085.89
5Mistral-prime1.31356994%47608.9383.67
6Mistral-option1.31456593%50550.9589.47
7Abscon 112v4AC54590%43650.7480.09
8Concrete + CPS4J2008-05-3054490%70154.84128.96
9Abscon 112v4ESAC54189%45027.2383.23
10choco2_dwdeg2008-06-2647779%77921.22163.36
11choco2_impwdeg2008-06-2644173%83391.49189.10
12Minion/Tailor2008-07-0440667%42112.68103.73
13casperzito39164%67482.76172.59
14casperzao38664%66833.33173.14
15Sugarv1.13+minisat37061%63823.88172.50
16Sugarv1.13+picosat34757%58679.96169.11
17galacM32654%40481.99124.18
18galacE32353%38360.21118.76
19bpsolver2008-06-2732253%53278.41165.46
20SAT4J CSP2008-06-1321836%28100.83128.90
21Concrete + CSP4J - Tabu Engine2008-05-3021035%36253.41172.64
22Concrete + CSP4J - WMC Engine2008-05-3013522%20996.34155.53

Category "n-ary constraints in intension" (N-ARY-INT)

Rank SolverVersionNumber of
solved instances
% of VBSCumulated CPU time
on solved instances
Average CPU time
per solved instance
Virtual Best Solver (VBS)660100%20689.0931.35
1cpHydrak_1056986%31124.3254.70
2cpHydrak_4056986%49899.2587.70
3casperzao56285%31205.6155.53
4Mistral-prime1.31356085%17929.8232.02
5MDG-probe2008-06-2756085%20146.7035.98
6casperzito55584%31629.8156.99
7Mistral-option1.31455484%43364.5678.28
8MDG-noprobe2008-06-2753080%18115.7134.18
9choco2_impwdeg2008-06-2651878%25745.1049.70
10Abscon 112v4ESAC48874%36101.5773.98
11Abscon 112v4AC48674%25860.4353.21
12Sugarv1.13+minisat48674%29306.6260.30
13Sugarv1.13+picosat48173%26580.8155.26
14bpsolver2008-06-2745970%77397.38168.62
15Concrete + CPS4J2008-05-3045669%91862.96201.45
16Concrete + CSP4J - Tabu Engine2008-05-3019930%36056.08181.19
17SAT4J CSP2008-06-1317426%19628.31112.81
18Concrete + CSP4J - WMC Engine2008-05-3017426%23628.78135.80