2017 XCSP3 competition: main track (CSP and COP, sequential and parallel solvers): ranking of solvers
max WC time for any solver=2400 s

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "optimization problem" (COP): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances
Detail % of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 438
Virtual Best Solver (VBS)160158 OPT, 2 UNSAT37%100%77455.762.19484.10----
1choco-solver4.0.5 par (2017-08-18)145143 OPT, 2 UNSAT33%91%124214.1917.06856.6516181.163.00111.597.68
2OscaR - Parallel with EPS2017-08-22135133 OPT, 2 UNSAT31%84%55362.8217.44410.0910743.354.8779.585.15
3Mistral-2.02017-07-28133131 OPT, 2 UNSAT30%83%17412.460.80130.9217425.170.89131.021.00
4OscaR - Conflict Ordering2017-07-26125123 OPT, 2 UNSAT29%78%8132.194.8865.067787.432.2162.301.04
5AbsCon-basic2017-06-11117115 OPT, 2 UNSAT27%73%14901.593.96127.3614670.591.99125.391.02
6choco-solver4.0.5 seq (2017-08-18)115113 OPT, 2 UNSAT26%72%12607.604.97109.6312273.092.49106.721.03
7OscaR - Hybrid2017-07-26109107 OPT, 2 UNSAT25%68%27940.083.60256.3327119.101.73248.801.03
8choco-solver5a (2017-08-18)106104 OPT, 2 UNSAT24%66%17076.338.35161.1016211.863.25152.941.05
9Concrete3.4103101 OPT, 2 UNSAT24%64%14878.9515.50144.4613669.117.33132.711.09
10cosoco1.129896 OPT, 2 UNSAT22%61%12003.190.98122.4812010.841.08122.561.00
11cosoco-sat1.129896 OPT, 2 UNSAT22%61%12628.791.12128.8712633.371.15128.911.00
12sat4j-CSP2017-07-057775 OPT, 2 UNSAT18%48%17799.4955.04231.1617151.7137.72222.751.04
13OscaR - ALNS2017-07-2622 UNSAT0%1%4.342.172.172.751.371.371.58

Category "optimization problem" (COP): OPT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 438
Virtual Best Solver (VBS)15836%100%77455.452.61490.22----
1choco-solver4.0.5 par (2017-08-18)14333%91%124205.2518.88868.5716179.123.02113.147.68
2OscaR - Parallel with EPS2017-08-2213330%84%55351.7117.71416.1810740.204.9780.755.15
3Mistral-2.02017-07-2813130%83%17348.310.80132.4317360.900.89132.531.00
4OscaR - Conflict Ordering2017-07-2612328%78%8128.054.8866.087785.372.3363.301.04
5AbsCon-basic2017-06-1111526%73%14898.754.02129.5514668.922.04127.561.02
6choco-solver4.0.5 seq (2017-08-18)11326%72%12602.865.11111.5312270.812.53108.591.03
7OscaR - Hybrid2017-07-2610724%68%27935.953.70261.0827117.041.77253.431.03
8choco-solver5a (2017-08-18)10424%66%17067.379.07164.1116208.355.24155.851.05
9Concrete3.410123%64%14861.1215.92147.1413660.007.47135.251.09
10cosoco1.129622%61%12002.881.10125.0312010.391.18125.111.00
11cosoco-sat1.129622%61%12628.451.24131.5512632.891.25131.591.00
12sat4j-CSP2017-07-057517%47%17795.3956.91237.2717149.0339.82228.651.04
13OscaR - ALNS2017-07-2600%0%0.000.00-0.000.00--

Category "optimization problem" (COP): UNSAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 438
Virtual Best Solver (VBS)20%100%0.310.150.15----
1cosoco1.1220%100%0.310.150.150.450.220.220.69
2cosoco-sat1.1220%100%0.340.170.170.480.240.240.72
3AbsCon-basic2017-06-1120%100%2.841.421.421.670.830.831.70
4sat4j-CSP2017-07-0520%100%4.102.052.052.671.341.341.53
5OscaR - Hybrid2017-07-2620%100%4.132.062.062.061.031.032.00
6OscaR - Conflict Ordering2017-07-2620%100%4.142.072.072.071.031.032.00
7OscaR - ALNS2017-07-2620%100%4.342.172.172.751.371.371.58
8choco-solver4.0.5 seq (2017-08-18)20%100%4.742.372.372.281.141.142.08
9choco-solver4.0.5 par (2017-08-18)20%100%8.954.474.472.041.021.024.39
10choco-solver5a (2017-08-18)20%100%8.964.484.483.511.751.752.55
11OscaR - Parallel with EPS2017-08-2220%100%11.115.565.563.161.581.583.52
12Concrete3.420%100%17.838.928.929.114.564.561.96
13Mistral-2.02017-07-2820%100%64.1532.0732.0764.2732.1432.141.00

Category "decision problem" (CSP): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances
Detail % of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 510
Virtual Best Solver (VBS)420297 SAT, 123 UNSAT82%100%45134.660.59107.46----
1choco-solver4.0.5 par (2017-08-18)395286 SAT, 109 UNSAT77%94%96219.0311.49243.5917561.112.7444.465.48
2choco-solver4.0.5 seq (2017-08-18)372264 SAT, 108 UNSAT73%89%16902.963.5745.4415988.971.7042.981.06
3choco-solver5a (2017-08-18)371262 SAT, 109 UNSAT73%88%31159.545.0483.9929250.562.3178.841.07
4AbsCon-basic2017-06-11368266 SAT, 102 UNSAT72%88%12834.583.2934.8812058.741.5132.771.06
5Mistral-2.02017-07-28364273 SAT, 91 UNSAT71%87%22447.190.8761.6722480.490.9761.761.00
6OscaR - Conflict Ordering2017-07-26336234 SAT, 102 UNSAT66%80%22237.893.3766.1820926.891.7362.281.06
7cosoco1.12335238 SAT, 97 UNSAT66%80%21275.490.6763.5121316.720.8863.631.00
8Concrete3.4331238 SAT, 93 UNSAT65%79%26725.1113.8380.7423039.256.2969.601.16
9OscaR - Parallel with EPS2017-08-22327226 SAT, 101 UNSAT64%78%72016.0211.85220.2316354.233.7250.014.40
10BTD2017-08-10241159 SAT, 82 UNSAT47%57%15196.251.0563.0515224.151.1763.171.00
11sat4j-CSP2017-07-05235168 SAT, 67 UNSAT46%56%45876.367.99195.2243974.744.51187.131.04

Category "decision problem" (CSP): SAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 510
Virtual Best Solver (VBS)29758%100%36127.210.46121.64----
1choco-solver4.0.5 par (2017-08-18)28656%96%66071.469.34231.0211666.862.0940.795.66
2Mistral-2.02017-07-2827354%92%11042.400.4940.4511067.500.5740.541.00
3AbsCon-basic2017-06-1126652%90%7065.193.0526.566515.631.3924.491.08
4choco-solver4.0.5 seq (2017-08-18)26452%89%11827.542.7044.8011178.281.2842.341.06
5choco-solver5a (2017-08-18)26251%88%21077.564.2480.4519809.561.8475.611.06
6cosoco1.1223847%80%11239.300.4147.2211271.840.5547.361.00
7Concrete3.423847%80%17406.7912.7273.1414962.835.7362.871.16
8OscaR - Conflict Ordering2017-07-2623446%79%11574.632.9049.4610755.191.5245.961.08
9OscaR - Parallel with EPS2017-08-2222644%76%46307.5010.99204.9010385.003.5145.954.46
10sat4j-CSP2017-07-0516833%57%31639.507.60188.3330671.524.24182.571.03
11BTD2017-08-1015931%54%10633.320.8666.8810650.810.9766.991.00

Category "decision problem" (CSP): UNSAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 510
Virtual Best Solver (VBS)12324%100%9007.451.2473.23----
1choco-solver5a (2017-08-18)10921%89%10081.987.7192.509441.003.5586.611.07
2choco-solver4.0.5 par (2017-08-18)10921%89%30147.5723.03276.585894.256.5354.085.11
3choco-solver4.0.5 seq (2017-08-18)10821%88%5075.425.8346.994810.693.0744.541.06
4AbsCon-basic2017-06-1110220%83%5769.393.7256.565543.111.9454.341.04
5OscaR - Conflict Ordering2017-07-2610220%83%10663.264.23104.5410171.712.5699.721.05
6OscaR - Parallel with EPS2017-08-2210120%82%25708.5114.12254.545969.234.3359.104.31
7cosoco1.129719%79%10036.181.45103.4710044.881.52103.561.00
8Concrete3.49318%76%9318.3115.44100.208076.426.9286.841.15
9Mistral-2.02017-07-289118%74%11404.792.46125.3311412.992.55125.421.00
10BTD2017-08-108216%67%4562.931.1755.654573.341.5155.771.00
11sat4j-CSP2017-07-056713%54%14236.869.92212.4913303.215.20198.561.07