2019 XCSP3 competition: main track (CSP and COP, sequential and parallel solvers): ranking of solvers
max CPU time for any solver=2400 s

Ranking on final answers (complete solver point of view)

This section presents the ranking of the different solvers. Solvers are ranked on the number of solved instances, i.e. where a final answer was given. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "optimization problem" (COP): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances
Detail % of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)234232 OPT, 2 UNSAT78%100%16407.7822.6170.12----
1PicatSAT2019-09-12221219 OPT, 2 UNSAT74%94%52947.4192.98239.5852951.2793.00239.601.00
2choco-solver2019-09-24 parallel210208 OPT, 2 UNSAT70%90%60065.29114.35286.038349.1415.9639.767.19
3AbsCon2019-07-23191189 OPT, 2 UNSAT64%82%35022.3727.52183.3634307.5119.69179.621.02
4choco-solver2019-09-24180179 OPT, 1 UNSAT60%77%71474.3979.97397.0870372.7075.86390.961.02
5Concrete3.12.3170169 OPT, 1 UNSAT57%73%89290.65146.66525.2486506.72129.67508.861.03
6cosoco2.0 parallel136135 OPT, 1 UNSAT45%58%46182.3060.66339.585827.188.4742.857.93
7cosoco2.0117116 OPT, 1 UNSAT39%50%24630.1230.87210.5124649.7030.88210.681.00

Category "optimization problem" (COP): OPT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)23277%100%16374.8422.6170.58----
1PicatSAT2019-09-1221973%94%52481.3592.98239.6452485.1593.00239.661.00
2choco-solver2019-09-24 parallel20869%90%59218.47114.35284.708241.3815.9639.627.19
3AbsCon2019-07-2318963%81%34987.3627.52185.1234276.0819.69181.351.02
4choco-solver2019-09-2417960%77%71381.6079.10398.7870284.3775.36392.651.02
5Concrete3.12.316956%73%89118.53142.40527.3386341.28123.56510.901.03
6cosoco2.0 parallel13545%58%44813.6557.16331.955655.818.3141.897.92
7cosoco2.011639%50%24208.6530.80208.7024228.1430.80208.861.00

Category "optimization problem" (COP): UNSAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)21%100%32.9416.4716.47----
1AbsCon2019-07-2321%100%35.0017.5017.5031.4315.7115.711.11
2PicatSAT2019-09-1221%100%466.06233.03233.03466.11233.06233.061.00
3choco-solver2019-09-24 parallel21%100%846.81423.41423.41107.7653.8853.887.86
4choco-solver2019-09-2410%50%92.7992.7992.7988.3388.3388.331.05
5Concrete3.12.310%50%172.12172.12172.12165.44165.44165.441.04
6cosoco2.010%50%421.47421.47421.47421.56421.56421.561.00
7cosoco2.0 parallel10%50%1368.651368.651368.65171.38171.38171.387.99

Category "decision problem" (CSP): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances
Detail % of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)272177 SAT, 95 UNSAT91%100%29590.7817.15108.79----
1PicatSAT2019-09-12245163 SAT, 82 UNSAT82%90%60332.73101.04246.2660337.09101.05246.271.00
2Fun-sCOPhybrid+CryptoMiniSat (2019-06-15)209132 SAT, 77 UNSAT70%77%55923.12108.68267.5749861.1392.49238.571.12
3Fun-sCOPhybrid+ManyGlucose (2019-09-22)198121 SAT, 77 UNSAT66%73%90109.30167.24455.1035194.7743.72177.752.56
4Fun-sCOPorder+ManyGlucose (2019-09-22)192122 SAT, 70 UNSAT64%71%93357.33204.78486.2435169.1252.21183.172.65
5Fun-sCOPorder+GlueMiniSat (2019-06-15)190122 SAT, 68 UNSAT63%70%56070.24114.72295.1149887.2599.42262.561.12
6AbsCon2019-07-23167114 SAT, 53 UNSAT56%61%22081.3418.61132.2221440.8014.21128.391.03
7Concrete3.12.3156106 SAT, 50 UNSAT52%57%40676.7042.17260.7538331.8932.82245.721.06
8choco-solver2019-09-24 parallel153113 SAT, 40 UNSAT51%56%49254.5678.91321.937847.9212.5951.296.28
9choco-solver2019-09-24149101 SAT, 48 UNSAT50%55%28234.2822.39189.4927471.9718.51184.381.03
10BTD19.07.0113584 SAT, 51 UNSAT45%50%25087.6314.47185.8325092.3914.47185.871.00
11cosoco2.012682 SAT, 44 UNSAT42%46%29242.8212.27232.0929251.7212.27232.161.00
12cosoco2.0 parallel12186 SAT, 35 UNSAT40%44%30919.5816.37255.535190.833.2442.905.96

Category "decision problem" (CSP): SAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)17759%100%22507.1113.50127.16----
1PicatSAT2019-09-1216354%92%37887.7183.95232.4437890.8883.97232.461.00
2Fun-sCOPhybrid+CryptoMiniSat (2019-06-15)13244%75%39141.34125.06296.5336261.81108.38274.711.08
3Fun-sCOPorder+GlueMiniSat (2019-06-15)12241%69%37305.10116.56305.7834221.6199.42280.511.09
4Fun-sCOPorder+ManyGlucose (2019-09-22)12241%69%58991.64169.61483.5422613.5250.48185.362.61
5Fun-sCOPhybrid+ManyGlucose (2019-09-22)12140%68%57077.22167.68471.7123897.3446.38197.502.39
6AbsCon2019-07-2311438%64%12692.5314.63111.3412338.3011.22108.231.03
7choco-solver2019-09-24 parallel11338%64%35443.5856.99313.665496.459.7148.646.45
8Concrete3.12.310635%60%17520.2730.88165.2915984.1816.02150.791.10
9choco-solver2019-09-2410134%57%17071.5715.10169.0316563.3812.10163.991.03
10cosoco2.0 parallel8629%49%11545.979.63134.262174.991.9825.295.31
11BTD19.07.018428%47%12867.256.65153.1812870.116.65153.221.00
12cosoco2.08227%46%13989.003.65170.6013996.864.22170.691.00

Category "decision problem" (CSP): UNSAT answers

Rank SolverVersion Number of
solved instances
% of all instances% of VBSCumulated CPU time
on solved instances
Median CPU timeAverage CPU time
per solved instance
Cumulated WC time
on solved instances
Median WC timeAverage WC time
per solved instance
CPU/WC
Total number of instances in the category: 300
Virtual Best Solver (VBS)9532%100%7083.6722.3974.56----
1PicatSAT2019-09-128227%86%22445.02121.79273.7222446.21121.81273.731.00
2Fun-sCOPhybrid+CryptoMiniSat (2019-06-15)7726%81%16781.7773.80217.9513599.3255.20176.611.23
3Fun-sCOPhybrid+ManyGlucose (2019-09-22)7726%81%33032.08157.05428.9911297.4342.74146.722.92
4Fun-sCOPorder+ManyGlucose (2019-09-22)7023%74%34365.70260.88490.9412555.6055.83179.372.74
5Fun-sCOPorder+GlueMiniSat (2019-06-15)6823%72%18765.14109.56275.9615665.6497.88230.381.20
6AbsCon2019-07-235318%56%9388.8132.26177.159102.4927.87171.751.03
7BTD19.07.015117%54%12220.3820.96239.6212222.2820.96239.651.00
8Concrete3.12.35017%53%23156.44209.71463.1322347.71195.82446.951.04
9choco-solver2019-09-244816%51%11162.7129.82232.5610908.5921.77227.261.02
10cosoco2.04415%46%15253.82127.08346.6815254.86127.11346.701.00
11choco-solver2019-09-24 parallel4013%42%13810.98164.39345.272351.4735.5558.795.87
12cosoco2.0 parallel3512%37%19373.61139.67553.533015.8424.2386.176.42

Ranking on number of best answers found (incomplete solver point of view)

Number of best answers found with no further time limit

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within the global time limit. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "optimization problem" (COP): answers

Rank SolverVersion Number of best solutions found% of all instances% of VBSCumulated time
Total number of instances in the category: 300
Virtual Best Solver (VBS)27090%100%
1choco-solver2019-09-24 parallel24080%89%194328.0529
2PicatSAT2019-09-1222174%82%45523.67277
3AbsCon2019-07-2320468%76%29664.94308
4choco-solver2019-09-2419766%73%46089.8538
5Concrete3.12.318060%67%20772.236
6cosoco2.0 parallel17157%63%83022.17
7cosoco2.012341%46%7485.41

Number of best answers found within 2 times the VBS time

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 2 times the VBS time. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "optimization problem" (COP): answers

Rank SolverVersion Number of best solutions found% of all instances% of VBSCumulated time
Total number of instances in the category: 300
Virtual Best Solver (VBS)27090%100%
1choco-solver2019-09-248428%31%24661.54
2cosoco2.06823%25%418.73
3cosoco2.0 parallel5619%21%899.62
4AbsCon2019-07-235117%19%3458.29
5choco-solver2019-09-24 parallel4415%16%133915.64
6PicatSAT2019-09-12269%10%6230.26
7Concrete3.12.3269%10%2817.71

Number of best answers found within 4 times the VBS time

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 4 times the VBS time. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "optimization problem" (COP): answers

Rank SolverVersion Number of best solutions found% of all instances% of VBSCumulated time
Total number of instances in the category: 300
Virtual Best Solver (VBS)27090%100%
1choco-solver2019-09-2410936%40%25152.93
2choco-solver2019-09-24 parallel10134%37%142780.37
3cosoco2.08027%30%648.36
4cosoco2.0 parallel7926%29%1454.22
5AbsCon2019-07-237525%28%3968.26
6Concrete3.12.34515%17%3514.63
7PicatSAT2019-09-124114%15%7725.25

Number of best answers found within 8 times the VBS time

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 8 times the VBS time. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "optimization problem" (COP): answers

Rank SolverVersion Number of best solutions found% of all instances% of VBSCumulated time
Total number of instances in the category: 300
Virtual Best Solver (VBS)27090%100%
1choco-solver2019-09-24 parallel14147%52%146580.6
2choco-solver2019-09-2412943%48%25920.85
3AbsCon2019-07-2311037%41%4969.52
4cosoco2.0 parallel10234%38%2132.28
5cosoco2.09632%36%1274.09
6Concrete3.12.37525%28%4620.12
7PicatSAT2019-09-126120%23%10117.42