CSP 2009 Competition: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "binary constraints in extension" (2-ARY-EXT): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 635
Virtual Best Solver (VBS)609359 SAT, 250 UNSAT96%100%37089.2060.90
1Mistral1.545570326 SAT, 244 UNSAT90%94%46856.8282.20
2Choco2.1.1b2009-07-16556314 SAT, 242 UNSAT88%91%55414.5499.67
3Abscon 112v4AC551314 SAT, 237 UNSAT87%90%58656.18106.45
4Abscon 112v4ESAC547310 SAT, 237 UNSAT86%90%50388.3192.12
5Choco2.1.12009-06-10547309 SAT, 238 UNSAT86%90%57385.70104.91
6Concrete DC2009-07-14504297 SAT, 207 UNSAT79%83%60964.15120.96
7Concrete2009-07-14503297 SAT, 206 UNSAT79%83%49380.8998.17
8Sugarv1.14.6+minisat466274 SAT, 192 UNSAT73%77%71234.79152.86
9Sugarv1.14.6+picosat438270 SAT, 168 UNSAT69%72%53442.74122.02
10SAT4J CSP2.1.1421259 SAT, 162 UNSAT66%69%51843.43123.14
11bpsolver09416253 SAT, 163 UNSAT66%68%56052.25134.74
12pcs-restart0.3.2394241 SAT, 153 UNSAT62%65%46361.44117.67
13pcs0.3.2393238 SAT, 155 UNSAT62%65%56915.05144.82

Category "binary constraints in extension" (2-ARY-EXT): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 635
Virtual Best Solver (VBS)35957%100%25119.0769.97
1Mistral1.54532651%91%33080.85101.48
2Choco2.1.1b2009-07-1631449%87%32299.32102.86
3Abscon 112v4AC31449%87%36620.13116.62
4Abscon 112v4ESAC31049%86%30136.9397.22
5Choco2.1.12009-06-1030949%86%36256.88117.34
6Concrete2009-07-1429747%83%32837.24110.56
7Concrete DC2009-07-1429747%83%36288.72122.18
8Sugarv1.14.6+minisat27443%76%30781.06112.34
9Sugarv1.14.6+picosat27043%75%30604.69113.35
10SAT4J CSP2.1.125941%72%27148.08104.82
11bpsolver0925340%70%29092.79114.99
12pcs-restart0.3.224138%67%20370.2484.52
13pcs0.3.223837%66%27327.34114.82

Category "binary constraints in extension" (2-ARY-EXT): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 635
Virtual Best Solver (VBS)25039%100%11970.1347.88
1Mistral1.54524438%98%13775.9756.46
2Choco2.1.1b2009-07-1624238%97%23115.2295.52
3Choco2.1.12009-06-1023837%95%21128.8188.78
4Abscon 112v4ESAC23737%95%20251.3885.45
5Abscon 112v4AC23737%95%22036.0592.98
6Concrete DC2009-07-1420733%83%24675.43119.20
7Concrete2009-07-1420632%82%16543.6580.31
8Sugarv1.14.6+minisat19230%77%40453.72210.70
9Sugarv1.14.6+picosat16826%67%22838.05135.94
10bpsolver0916326%65%26959.46165.40
11SAT4J CSP2.1.116226%65%24695.35152.44
12pcs0.3.215524%62%29587.72190.89
13pcs-restart0.3.215324%61%25991.20169.88

Category "binary constraints in intension" (2-ARY-INT): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 686
Virtual Best Solver (VBS)617302 SAT, 315 UNSAT90%100%18536.7330.04
1Abscon 112v4ESAC517259 SAT, 258 UNSAT75%84%17096.3233.07
2Abscon 112v4AC513258 SAT, 255 UNSAT75%83%21759.0642.42
3Mistral1.545511242 SAT, 269 UNSAT74%83%27455.4053.73
4Choco2.1.1b2009-07-16510240 SAT, 270 UNSAT74%83%35843.6270.28
5Choco2.1.12009-06-10508236 SAT, 272 UNSAT74%82%32542.7864.06
6Sugarv1.14.6+picosat479263 SAT, 216 UNSAT70%78%36242.9975.66
7Sugarv1.14.6+minisat470254 SAT, 216 UNSAT69%76%27807.6159.17
8Concrete2009-07-14428237 SAT, 191 UNSAT62%69%34105.0179.68
9pcs-restart0.3.2419211 SAT, 208 UNSAT61%68%21837.8652.12
10pcs0.3.2419210 SAT, 209 UNSAT61%68%23992.6757.26
11Concrete DC2009-07-14372179 SAT, 193 UNSAT54%60%66687.23179.27
12bpsolver09349178 SAT, 171 UNSAT51%57%36187.51103.69
13SAT4J CSP2.1.1306177 SAT, 129 UNSAT45%50%17934.9058.61

Category "binary constraints in intension" (2-ARY-INT): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 686
Virtual Best Solver (VBS)30244%100%10730.9935.53
1Sugarv1.14.6+picosat26338%87%16185.3661.54
2Abscon 112v4ESAC25938%86%8065.2931.14
3Abscon 112v4AC25838%85%11146.0843.20
4Sugarv1.14.6+minisat25437%84%13007.3851.21
5Mistral1.54524235%80%9125.6337.71
6Choco2.1.1b2009-07-1624035%79%16530.9968.88
7Concrete2009-07-1423735%78%8145.4534.37
8Choco2.1.12009-06-1023634%78%14391.6760.98
9pcs-restart0.3.221131%70%6735.0631.92
10pcs0.3.221031%70%11557.4055.04
11Concrete DC2009-07-1417926%59%33876.94189.26
12bpsolver0917826%59%24178.71135.84
13SAT4J CSP2.1.117726%59%9198.4051.97

Category "binary constraints in intension" (2-ARY-INT): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 686
Virtual Best Solver (VBS)31546%100%7805.7424.78
1Choco2.1.12009-06-1027240%86%18151.1266.73
2Choco2.1.1b2009-07-1627039%86%19312.6371.53
3Mistral1.54526939%85%18329.7768.14
4Abscon 112v4ESAC25838%82%9031.0335.00
5Abscon 112v4AC25537%81%10612.9841.62
6Sugarv1.14.6+minisat21631%69%14800.2468.52
7Sugarv1.14.6+picosat21631%69%20057.6292.86
8pcs0.3.220930%66%12435.2859.50
9pcs-restart0.3.220830%66%15102.8072.61
10Concrete DC2009-07-1419328%61%32810.29170.00
11Concrete2009-07-1419128%61%25959.56135.91
12bpsolver0917125%54%12008.8070.23
13SAT4J CSP2.1.112919%41%8736.5067.72

Category "alldiff" (Alldiff): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 118
Virtual Best Solver (VBS)10780 SAT, 27 UNSAT91%100%2919.1827.28
1Sugarv1.14.6+picosat10477 SAT, 27 UNSAT88%97%6050.3858.18
2Mistral1.5459875 SAT, 23 UNSAT83%92%5337.8354.47
3Sugarv1.14.6+minisat8861 SAT, 27 UNSAT75%82%11507.92130.77
4Abscon 112v4ESAC7854 SAT, 24 UNSAT66%73%10820.98138.73
5Abscon 112v4AC7753 SAT, 24 UNSAT65%72%8404.10109.14
6Concrete2009-07-147350 SAT, 23 UNSAT62%68%5389.2973.83
7Choco2.1.12009-06-107249 SAT, 23 UNSAT61%67%6341.3788.07
8Choco2.1.1b2009-07-167148 SAT, 23 UNSAT60%66%3690.1351.97
9Concrete DC2009-07-146845 SAT, 23 UNSAT58%64%9825.10144.49
10SAT4J CSP2.1.16553 SAT, 12 UNSAT55%61%11822.21181.88
11bpsolver096037 SAT, 23 UNSAT51%56%1767.5329.46

Category "alldiff" (Alldiff): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 118
Virtual Best Solver (VBS)8068%100%2914.0736.43
1Sugarv1.14.6+picosat7765%96%6032.6578.35
2Mistral1.5457564%94%5337.6271.17
3Sugarv1.14.6+minisat6152%76%11491.91188.39
4Abscon 112v4ESAC5446%68%10078.53186.64
5Abscon 112v4AC5345%66%7713.08145.53
6SAT4J CSP2.1.15345%66%10619.05200.36
7Concrete2009-07-145042%63%5370.06107.40
8Choco2.1.12009-06-104942%61%6332.42129.23
9Choco2.1.1b2009-07-164841%60%3681.0576.69
10Concrete DC2009-07-144538%56%9805.11217.89
11bpsolver093731%46%1766.8047.75

Category "alldiff" (Alldiff): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 118
Virtual Best Solver (VBS)2723%100%5.120.19
1Sugarv1.14.6+minisat2723%100%16.010.59
2Sugarv1.14.6+picosat2723%100%17.730.66
3Abscon 112v4AC2420%89%691.0328.79
4Abscon 112v4ESAC2420%89%742.4530.94
5Mistral1.5452319%85%0.200.01
6bpsolver092319%85%0.740.03
7Choco2.1.12009-06-102319%85%8.960.39
8Choco2.1.1b2009-07-162319%85%9.080.39
9Concrete2009-07-142319%85%19.230.84
10Concrete DC2009-07-142319%85%19.990.87
11SAT4J CSP2.1.11210%44%1203.16100.26

Category "alldiff,cumulative,element,weightedsum" (Alldiff+Cumul+Elt+WSum): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 162
Virtual Best Solver (VBS)14474 SAT, 70 UNSAT89%100%2414.9816.77
1Sugarv1.14.6+minisat13674 SAT, 62 UNSAT84%94%4196.2330.85
2Sugarv1.14.6+picosat13573 SAT, 62 UNSAT83%94%3035.8122.49
3Choco2.1.12009-06-1012559 SAT, 66 UNSAT77%87%10938.9487.51
4Choco2.1.1b2009-07-1612257 SAT, 65 UNSAT75%85%7647.7062.69
5Mistral1.54512062 SAT, 58 UNSAT74%83%2549.0421.24
6bpsolver0911158 SAT, 53 UNSAT69%77%27888.36251.25

Category "alldiff,cumulative,element,weightedsum" (Alldiff+Cumul+Elt+WSum): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 162
Virtual Best Solver (VBS)7446%100%2279.6530.81
1Sugarv1.14.6+minisat7446%100%3627.3549.02
2Sugarv1.14.6+picosat7345%99%2572.7435.24
3Mistral1.5456238%84%922.2614.88
4Choco2.1.12009-06-105936%80%5714.1196.85
5bpsolver095836%78%12252.15211.24
6Choco2.1.1b2009-07-165735%77%2786.2448.88

Category "alldiff,cumulative,element,weightedsum" (Alldiff+Cumul+Elt+WSum): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 162
Virtual Best Solver (VBS)7043%100%135.321.93
1Choco2.1.12009-06-106641%94%5224.8379.16
2Choco2.1.1b2009-07-166540%93%4861.4674.79
3Sugarv1.14.6+picosat6238%89%463.077.47
4Sugarv1.14.6+minisat6238%89%568.899.18
5Mistral1.5455836%83%1626.7828.05
6bpsolver095333%76%15636.21295.02

Category "alldiff,element,weightedsum" (Alldiff+Elt+WSum): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 276
Virtual Best Solver (VBS)249163 SAT, 86 UNSAT90%100%5072.8320.37
1Sugarv1.14.6+picosat229146 SAT, 83 UNSAT83%92%10863.4447.44
2Sugarv1.14.6+minisat222137 SAT, 85 UNSAT80%89%13704.7461.73
3Mistral1.545217148 SAT, 69 UNSAT79%87%10112.0646.60
4Choco2.1.12009-06-10194129 SAT, 65 UNSAT70%78%10375.6953.48
5Choco2.1.1b2009-07-16193129 SAT, 64 UNSAT70%78%18774.4997.28
6bpsolver09186109 SAT, 77 UNSAT67%75%20642.14110.98
7Abscon 112v4AC11369 SAT, 44 UNSAT41%45%31535.22279.07
8Abscon 112v4ESAC9554 SAT, 41 UNSAT34%38%27897.30293.66

Category "alldiff,element,weightedsum" (Alldiff+Elt+WSum): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 276
Virtual Best Solver (VBS)16359%100%2038.5812.51
1Mistral1.54514854%91%3627.5824.51
2Sugarv1.14.6+picosat14653%90%4229.2828.97
3Sugarv1.14.6+minisat13750%84%6821.6149.79
4Choco2.1.12009-06-1012947%79%7307.9256.65
5Choco2.1.1b2009-07-1612947%79%11304.6787.63
6bpsolver0910939%67%18868.41173.10
7Abscon 112v4AC6925%42%8556.36124.01
8Abscon 112v4ESAC5420%33%11109.64205.73

Category "alldiff,element,weightedsum" (Alldiff+Elt+WSum): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 276
Virtual Best Solver (VBS)8631%100%3034.2535.28
1Sugarv1.14.6+minisat8531%99%6883.1380.98
2Sugarv1.14.6+picosat8330%97%6634.1679.93
3bpsolver097728%90%1773.7323.04
4Mistral1.5456925%80%6484.4893.98
5Choco2.1.12009-06-106524%76%3067.7747.20
6Choco2.1.1b2009-07-166423%74%7469.81116.72
7Abscon 112v4AC4416%51%22978.86522.25
8Abscon 112v4ESAC4115%48%16787.66409.46

Category "n-ary constraints in extension" (N-ARY-EXT): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 699
Virtual Best Solver (VBS)612304 SAT, 308 UNSAT88%100%45432.1374.24
1Mistral1.545585293 SAT, 292 UNSAT84%96%50114.0385.67
2Abscon 112v4AC545250 SAT, 295 UNSAT78%89%41302.3775.78
3Concrete2009-07-14544267 SAT, 277 UNSAT78%89%114700.63210.85
4Abscon 112v4ESAC543249 SAT, 294 UNSAT78%89%42093.6077.52
5Conquer2009-07-10532261 SAT, 271 UNSAT76%87%46955.7088.26
6Choco2.1.12009-06-10532267 SAT, 265 UNSAT76%87%84300.32158.46
7Concrete DC2009-07-14532265 SAT, 267 UNSAT76%87%169470.05318.55
8Choco2.1.1b2009-07-16528262 SAT, 266 UNSAT76%86%75926.12143.80
9pcs0.3.2497225 SAT, 272 UNSAT71%81%82946.74166.89
10pcs-restart0.3.2496223 SAT, 273 UNSAT71%81%81860.66165.04
11bpsolver09394196 SAT, 198 UNSAT56%64%42707.44108.39
12Sugarv1.14.6+minisat374184 SAT, 190 UNSAT54%61%72646.69194.24
13Sugarv1.14.6+picosat350183 SAT, 167 UNSAT50%57%60281.82172.23
14SAT4J CSP2.1.1209119 SAT, 90 UNSAT30%34%30160.19144.31

Category "n-ary constraints in extension" (N-ARY-EXT): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 699
Virtual Best Solver (VBS)30443%100%25562.8384.09
1Mistral1.54529342%96%30645.62104.59
2Choco2.1.12009-06-1026738%88%42854.33160.50
3Concrete2009-07-1426738%88%50797.72190.25
4Concrete DC2009-07-1426538%87%76701.79289.44
5Choco2.1.1b2009-07-1626237%86%31810.09121.41
6Conquer2009-07-1026137%86%25338.9697.08
7Abscon 112v4AC25036%82%21187.4884.75
8Abscon 112v4ESAC24936%82%20722.7483.22
9pcs0.3.222532%74%44769.59198.98
10pcs-restart0.3.222332%73%41961.80188.17
11bpsolver0919628%64%26437.12134.88
12Sugarv1.14.6+minisat18426%61%34611.34188.11
13Sugarv1.14.6+picosat18326%60%32983.92180.24
14SAT4J CSP2.1.111917%39%26367.03221.57

Category "n-ary constraints in extension" (N-ARY-EXT): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 699
Virtual Best Solver (VBS)30844%100%19869.2964.51
1Abscon 112v4AC29542%96%20114.8968.19
2Abscon 112v4ESAC29442%95%21370.8672.69
3Mistral1.54529242%95%19468.4166.67
4Concrete2009-07-1427740%90%63902.91230.70
5pcs-restart0.3.227339%89%39898.85146.15
6pcs0.3.227239%88%38177.14140.36
7Conquer2009-07-1027139%88%21616.7479.77
8Concrete DC2009-07-1426738%87%92768.27347.45
9Choco2.1.1b2009-07-1626638%86%44116.03165.85
10Choco2.1.12009-06-1026538%86%41445.99156.40
11bpsolver0919828%64%16270.3282.17
12Sugarv1.14.6+minisat19027%62%38035.35200.19
13Sugarv1.14.6+picosat16724%54%27297.90163.46
14SAT4J CSP2.1.19013%29%3793.1642.15

Category "n-ary constraints in intension" (N-ARY-INT): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 709
Virtual Best Solver (VBS)630532 SAT, 98 UNSAT89%100%17013.7827.01
1Mistral1.545572494 SAT, 78 UNSAT81%91%17837.8131.18
2Choco2.1.12009-06-10560482 SAT, 78 UNSAT79%89%27734.8649.53
3Choco2.1.1b2009-07-16548469 SAT, 79 UNSAT77%87%31917.2658.24
4pcs-restart0.3.2546461 SAT, 85 UNSAT77%87%22451.1941.12
5pcs0.3.2542457 SAT, 85 UNSAT76%86%21390.6839.47
6bpsolver09513442 SAT, 71 UNSAT72%81%92578.86180.47
7Abscon 112v4ESAC489426 SAT, 63 UNSAT69%78%38474.8978.68
8Abscon 112v4AC481418 SAT, 63 UNSAT68%76%28646.8259.56
9Sugarv1.14.6+minisat481397 SAT, 84 UNSAT68%76%33430.4669.50
10Sugarv1.14.6+picosat478396 SAT, 82 UNSAT67%76%24531.1451.32
11Concrete2009-07-14439379 SAT, 60 UNSAT62%70%82222.21187.29
12Concrete DC2009-07-14342290 SAT, 52 UNSAT48%54%85478.60249.94
13SAT4J CSP2.1.1171133 SAT, 38 UNSAT24%27%18993.60111.07

Category "n-ary constraints in intension" (N-ARY-INT): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 709
Virtual Best Solver (VBS)53275%100%10905.1120.50
1Mistral1.54549470%93%14912.3530.19
2Choco2.1.12009-06-1048268%91%23247.4748.23
3Choco2.1.1b2009-07-1646966%88%26038.6555.52
4pcs-restart0.3.246165%87%17796.5038.60
5pcs0.3.245764%86%18531.1840.55
6bpsolver0944262%83%81740.76184.93
7Abscon 112v4ESAC42660%80%31930.7874.95
8Abscon 112v4AC41859%79%21544.1351.54
9Sugarv1.14.6+minisat39756%75%28456.3771.68
10Sugarv1.14.6+picosat39656%74%21609.9554.57
11Concrete2009-07-1437953%71%67194.16177.29
12Concrete DC2009-07-1429041%55%72487.14249.96
13SAT4J CSP2.1.113319%25%13792.25103.70

Category "n-ary constraints in intension" (N-ARY-INT): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 709
Virtual Best Solver (VBS)9814%100%6108.6762.33
1pcs0.3.28512%87%2859.4933.64
2pcs-restart0.3.28512%87%4654.7054.76
3Sugarv1.14.6+minisat8412%86%4974.0959.22
4Sugarv1.14.6+picosat8212%84%2921.1935.62
5Choco2.1.1b2009-07-167911%81%5878.6174.41
6Mistral1.5457811%80%2925.4637.51
7Choco2.1.12009-06-107811%80%4487.3957.53
8bpsolver097110%72%10838.10152.65
9Abscon 112v4ESAC639%64%6544.11103.87
10Abscon 112v4AC639%64%7102.69112.74
11Concrete2009-07-14608%61%15028.05250.47
12Concrete DC2009-07-14527%53%12991.46249.84
13SAT4J CSP2.1.1385%39%5201.35136.88