PB'09 competition: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 371
Virtual Best Solver (VBS)363145 SAT, 218 UNSAT98%100%8197.1222.58
1bsolo3.1325137 SAT, 188 UNSAT88%90%50919.32156.67
2SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)310119 SAT, 191 UNSAT84%85%63911.94206.17
3wbo1.0309137 SAT, 172 UNSAT83%85%34877.81112.87
4SCIPclpSCIP 1.1.0.7 with CLP 1.8.2308119 SAT, 189 UNSAT83%85%58055.30188.49
5pbclasp2009-04-24300126 SAT, 174 UNSAT81%83%8955.1029.85
6bsolo3.1 cl290136 SAT, 154 UNSAT78%80%61088.55210.65
7BoolVar2009-04-26288111 SAT, 177 UNSAT78%79%12493.1643.38
8bsolo3.1 pb284136 SAT, 148 UNSAT77%78%21755.3376.60
9SAT4J Pseudo Resolution2.1.1278106 SAT, 172 UNSAT75%77%13686.4949.23
10SAT4J Pseudo CP2.1.116472 SAT, 92 UNSAT44%45%15164.2192.46

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 371
Virtual Best Solver (VBS)14539%100%3130.5221.59
1wbo1.013737%94%3059.2222.33
2bsolo3.113737%94%5232.2638.19
3bsolo3.1 cl13637%94%6464.5547.53
4bsolo3.1 pb13637%94%14300.56105.15
5pbclasp2009-04-2412634%87%4271.4933.90
6SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)11932%82%6304.3052.98
7SCIPclpSCIP 1.1.0.7 with CLP 1.8.211932%82%6798.7257.13
8BoolVar2009-04-2611130%77%2420.1721.80
9SAT4J Pseudo Resolution2.1.110629%73%4511.9842.57
10SAT4J Pseudo CP2.1.17219%50%8656.30120.23

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 371
Virtual Best Solver (VBS)21859%100%5066.6023.24
1SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)19151%88%57607.63301.61
2SCIPclpSCIP 1.1.0.7 with CLP 1.8.218951%87%51256.57271.20
3bsolo3.118851%86%45687.06243.02
4BoolVar2009-04-2617748%81%10072.9956.91
5pbclasp2009-04-2417447%80%4683.6126.92
6SAT4J Pseudo Resolution2.1.117246%79%9174.5253.34
7wbo1.017246%79%31818.59184.99
8bsolo3.1 cl15442%71%54624.00354.70
9bsolo3.1 pb14840%68%7454.7750.37
10SAT4J Pseudo CP2.1.19225%42%6507.9270.74

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 100
Virtual Best Solver (VBS)7555 SAT, 20 UNSAT75%100%4018.1253.57
1SCIPclpSCIP 1.1.0.7 with CLP 1.8.27454 SAT, 20 UNSAT74%99%5204.7670.33
2SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)7454 SAT, 20 UNSAT74%99%7112.7596.12
3pbclasp2009-04-246550 SAT, 15 UNSAT65%87%198.543.05
4SAT4J Pseudo CP2.1.16550 SAT, 15 UNSAT65%87%4930.8975.86
5bsolo3.1 pb6449 SAT, 15 UNSAT64%85%1874.9829.30
6SAT4J Pseudo Resolution2.1.16451 SAT, 13 UNSAT64%85%7091.12110.80
7bsolo3.16146 SAT, 15 UNSAT61%81%4465.7373.21
8wbo1.05944 SAT, 15 UNSAT59%79%5271.6089.35
9BoolVar2009-04-265742 SAT, 15 UNSAT57%76%6718.29117.86
10bsolo3.1 cl2616 SAT, 10 UNSAT26%35%4954.23190.55

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 100
Virtual Best Solver (VBS)5555%100%2955.0053.73
1SCIPclpSCIP 1.1.0.7 with CLP 1.8.25454%98%3781.3170.02
2SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)5454%98%5965.81110.48
3SAT4J Pseudo Resolution2.1.15151%93%4346.0585.22
4pbclasp2009-04-245050%91%34.450.69
5SAT4J Pseudo CP2.1.15050%91%4190.1883.80
6bsolo3.1 pb4949%89%1743.6435.58
7bsolo3.14646%84%1856.6740.36
8wbo1.04444%80%1591.1936.16
9BoolVar2009-04-264242%76%6271.30149.32
10bsolo3.1 cl1616%29%4954.19309.64

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 100
Virtual Best Solver (VBS)2020%100%1063.1253.16
1SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)2020%100%1146.9457.35
2SCIPclpSCIP 1.1.0.7 with CLP 1.8.22020%100%1423.4571.17
3bsolo3.1 pb1515%75%131.348.76
4pbclasp2009-04-241515%75%164.0910.94
5BoolVar2009-04-261515%75%446.9929.80
6SAT4J Pseudo CP2.1.11515%75%740.7049.38
7bsolo3.11515%75%2609.06173.94
8wbo1.01515%75%3680.41245.36
9SAT4J Pseudo Resolution2.1.11313%65%2745.07211.16
10bsolo3.1 cl1010%50%0.040.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 452
Virtual Best Solver (VBS)146104 OPT, 42 UNSAT32%100%8693.3159.54
1SAT4J Pseudo Resolution2.1.113393 OPT, 40 UNSAT29%91%7971.4959.94
2SAT4J Pseudo CP2.1.112484 OPT, 40 UNSAT27%85%4173.0833.65

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 452
Virtual Best Solver (VBS)10423%100%7340.9670.59
1SAT4J Pseudo Resolution2.1.19321%89%6643.3871.43
2SAT4J Pseudo CP2.1.18419%81%3367.4940.09

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 452
Virtual Best Solver (VBS)429%100%1352.3432.20
1SAT4J Pseudo CP2.1.1409%95%805.5920.14
2SAT4J Pseudo Resolution2.1.1409%95%1328.1133.20

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 554
Virtual Best Solver (VBS)343330 OPT, 13 UNSAT62%100%24432.5771.23
1SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)295282 OPT, 13 UNSAT53%86%44703.12151.54
2bsolo3.1 pb231218 OPT, 13 UNSAT42%67%20591.1989.14
3bsolo3.1 cl230217 OPT, 13 UNSAT42%67%24836.00107.98
4pbclasp2009-04-24188175 OPT, 13 UNSAT34%55%13425.0971.41
5SAT4J Pseudo Resolution2.1.1169158 OPT, 11 UNSAT31%49%12972.9976.76
6SAT4J Pseudo CP2.1.1140131 OPT, 9 UNSAT25%41%9139.7065.28
7wbo1.0136124 OPT, 12 UNSAT25%40%17359.81127.65

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 554
Virtual Best Solver (VBS)33060%100%24370.5973.85
1SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)28251%85%41908.18148.61
2bsolo3.1 pb21839%66%20206.4792.69
3bsolo3.1 cl21739%66%23789.72109.63
4pbclasp2009-04-2417532%53%13362.6176.36
5SAT4J Pseudo Resolution2.1.115829%48%10620.0367.22
6SAT4J Pseudo CP2.1.113124%40%9121.3669.63
7wbo1.012422%38%14591.00117.67

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 554
Virtual Best Solver (VBS)132%100%61.984.77
1pbclasp2009-04-24132%100%62.484.81
2bsolo3.1 pb132%100%384.7229.59
3bsolo3.1 cl132%100%1046.2880.48
4SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)132%100%2794.94215.00
5wbo1.0122%92%2768.80230.73
6SAT4J Pseudo Resolution2.1.1112%85%2352.97213.91
7SAT4J Pseudo CP2.1.192%69%18.342.04

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 405
Virtual Best Solver (VBS)290290 OPT72%100%11423.0839.39
1SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)289289 OPT71%100%14549.0050.34
2SCIPclpSCIP 1.1.0.7 with CLP 1.8.2286286 OPT71%99%19685.7468.83
3pbclasp2009-04-24263263 OPT65%91%31540.80119.93
4bsolo3.1 cl232232 OPT57%80%31511.36135.82
5SAT4J Pseudo Resolution2.1.1226226 OPT56%78%25283.64111.87
6wbo1.0223223 OPT55%77%22415.57100.52
7SAT4J Pseudo CP2.1.1118118 OPT29%41%19970.56169.24

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 405
Virtual Best Solver (VBS)29072%100%11423.0839.39
1SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)28971%100%14549.0050.34
2SCIPclpSCIP 1.1.0.7 with CLP 1.8.228671%99%19685.7468.83
3pbclasp2009-04-2426365%91%31540.80119.93
4bsolo3.1 cl23257%80%31511.36135.82
5SAT4J Pseudo Resolution2.1.122656%78%25283.64111.87
6wbo1.022355%77%22415.57100.52
7SAT4J Pseudo CP2.1.111829%41%19970.56169.24

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 405
Virtual Best Solver (VBS)00%-0.00-
1SAT4J Pseudo CP2.1.100%-0.00-
2SAT4J Pseudo Resolution2.1.100%-0.00-
3SCIPspxSCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)00%-0.00-
4SCIPclpSCIP 1.1.0.7 with CLP 1.8.200%-0.00-
5pbclasp2009-04-2400%-0.00-
6wbo1.000%-0.00-
7bsolo3.1 cl00%-0.00-