This section presents the ranking of the different solvers. Solvers are ranked on the number of solved instances, i.e. where a final answer was given. Ties are broken by considering the minimum total solution time.
Solvers which gave a wrong answer in a category are excluded from the ranking in that category.
The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.
| Rank | Solver | Version | Number of solved instances | Detail | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | |||||||||||||
| Virtual Best Solver (VBS) | 234 | 232 OPT, 2 UNSAT | 78% | 100% | 16407.78 | 22.61 | 70.12 | - | - | - | - | ||
| 1 | PicatSAT | 2019-09-12 | 221 | 219 OPT, 2 UNSAT | 74% | 94% | 52947.41 | 92.98 | 239.58 | 52951.27 | 93.00 | 239.60 | 1.00 |
| 2 | choco-solver | 2019-09-24 parallel | 210 | 208 OPT, 2 UNSAT | 70% | 90% | 60065.29 | 114.35 | 286.03 | 8349.14 | 15.96 | 39.76 | 7.19 |
| 3 | AbsCon | 2019-07-23 | 191 | 189 OPT, 2 UNSAT | 64% | 82% | 35022.37 | 27.52 | 183.36 | 34307.51 | 19.69 | 179.62 | 1.02 |
| 4 | choco-solver | 2019-09-24 | 180 | 179 OPT, 1 UNSAT | 60% | 77% | 71474.39 | 79.97 | 397.08 | 70372.70 | 75.86 | 390.96 | 1.02 |
| 5 | Concrete | 3.12.3 | 170 | 169 OPT, 1 UNSAT | 57% | 73% | 89290.65 | 146.66 | 525.24 | 86506.72 | 129.67 | 508.86 | 1.03 |
| 6 | cosoco | 2.0 parallel | 136 | 135 OPT, 1 UNSAT | 45% | 58% | 46182.30 | 60.66 | 339.58 | 5827.18 | 8.47 | 42.85 | 7.93 |
| 7 | cosoco | 2.0 | 117 | 116 OPT, 1 UNSAT | 39% | 50% | 24630.12 | 30.87 | 210.51 | 24649.70 | 30.88 | 210.68 | 1.00 |
| Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||||||||
| Virtual Best Solver (VBS) | 232 | 77% | 100% | 16374.84 | 22.61 | 70.58 | - | - | - | - | ||
| 1 | PicatSAT | 2019-09-12 | 219 | 73% | 94% | 52481.35 | 92.98 | 239.64 | 52485.15 | 93.00 | 239.66 | 1.00 |
| 2 | choco-solver | 2019-09-24 parallel | 208 | 69% | 90% | 59218.47 | 114.35 | 284.70 | 8241.38 | 15.96 | 39.62 | 7.19 |
| 3 | AbsCon | 2019-07-23 | 189 | 63% | 81% | 34987.36 | 27.52 | 185.12 | 34276.08 | 19.69 | 181.35 | 1.02 |
| 4 | choco-solver | 2019-09-24 | 179 | 60% | 77% | 71381.60 | 79.10 | 398.78 | 70284.37 | 75.36 | 392.65 | 1.02 |
| 5 | Concrete | 3.12.3 | 169 | 56% | 73% | 89118.53 | 142.40 | 527.33 | 86341.28 | 123.56 | 510.90 | 1.03 |
| 6 | cosoco | 2.0 parallel | 135 | 45% | 58% | 44813.65 | 57.16 | 331.95 | 5655.81 | 8.31 | 41.89 | 7.92 |
| 7 | cosoco | 2.0 | 116 | 39% | 50% | 24208.65 | 30.80 | 208.70 | 24228.14 | 30.80 | 208.86 | 1.00 |
| Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||||||||
| Virtual Best Solver (VBS) | 2 | 1% | 100% | 32.94 | 16.47 | 16.47 | - | - | - | - | ||
| 1 | AbsCon | 2019-07-23 | 2 | 1% | 100% | 35.00 | 17.50 | 17.50 | 31.43 | 15.71 | 15.71 | 1.11 |
| 2 | PicatSAT | 2019-09-12 | 2 | 1% | 100% | 466.06 | 233.03 | 233.03 | 466.11 | 233.06 | 233.06 | 1.00 |
| 3 | choco-solver | 2019-09-24 parallel | 2 | 1% | 100% | 846.81 | 423.41 | 423.41 | 107.76 | 53.88 | 53.88 | 7.86 |
| 4 | choco-solver | 2019-09-24 | 1 | 0% | 50% | 92.79 | 92.79 | 92.79 | 88.33 | 88.33 | 88.33 | 1.05 |
| 5 | Concrete | 3.12.3 | 1 | 0% | 50% | 172.12 | 172.12 | 172.12 | 165.44 | 165.44 | 165.44 | 1.04 |
| 6 | cosoco | 2.0 | 1 | 0% | 50% | 421.47 | 421.47 | 421.47 | 421.56 | 421.56 | 421.56 | 1.00 |
| 7 | cosoco | 2.0 parallel | 1 | 0% | 50% | 1368.65 | 1368.65 | 1368.65 | 171.38 | 171.38 | 171.38 | 7.99 |
| Rank | Solver | Version | Number of solved instances | Detail | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | |||||||||||||
| Virtual Best Solver (VBS) | 272 | 177 SAT, 95 UNSAT | 91% | 100% | 29590.78 | 17.15 | 108.79 | - | - | - | - | ||
| 1 | PicatSAT | 2019-09-12 | 245 | 163 SAT, 82 UNSAT | 82% | 90% | 60332.73 | 101.04 | 246.26 | 60337.09 | 101.05 | 246.27 | 1.00 |
| 2 | Fun-sCOP | hybrid+CryptoMiniSat (2019-06-15) | 209 | 132 SAT, 77 UNSAT | 70% | 77% | 55923.12 | 108.68 | 267.57 | 49861.13 | 92.49 | 238.57 | 1.12 |
| 3 | Fun-sCOP | hybrid+ManyGlucose (2019-09-22) | 198 | 121 SAT, 77 UNSAT | 66% | 73% | 90109.30 | 167.24 | 455.10 | 35194.77 | 43.72 | 177.75 | 2.56 |
| 4 | Fun-sCOP | order+ManyGlucose (2019-09-22) | 192 | 122 SAT, 70 UNSAT | 64% | 71% | 93357.33 | 204.78 | 486.24 | 35169.12 | 52.21 | 183.17 | 2.65 |
| 5 | Fun-sCOP | order+GlueMiniSat (2019-06-15) | 190 | 122 SAT, 68 UNSAT | 63% | 70% | 56070.24 | 114.72 | 295.11 | 49887.25 | 99.42 | 262.56 | 1.12 |
| 6 | AbsCon | 2019-07-23 | 167 | 114 SAT, 53 UNSAT | 56% | 61% | 22081.34 | 18.61 | 132.22 | 21440.80 | 14.21 | 128.39 | 1.03 |
| 7 | Concrete | 3.12.3 | 156 | 106 SAT, 50 UNSAT | 52% | 57% | 40676.70 | 42.17 | 260.75 | 38331.89 | 32.82 | 245.72 | 1.06 |
| 8 | choco-solver | 2019-09-24 parallel | 153 | 113 SAT, 40 UNSAT | 51% | 56% | 49254.56 | 78.91 | 321.93 | 7847.92 | 12.59 | 51.29 | 6.28 |
| 9 | choco-solver | 2019-09-24 | 149 | 101 SAT, 48 UNSAT | 50% | 55% | 28234.28 | 22.39 | 189.49 | 27471.97 | 18.51 | 184.38 | 1.03 |
| 10 | BTD | 19.07.01 | 135 | 84 SAT, 51 UNSAT | 45% | 50% | 25087.63 | 14.47 | 185.83 | 25092.39 | 14.47 | 185.87 | 1.00 |
| 11 | cosoco | 2.0 | 126 | 82 SAT, 44 UNSAT | 42% | 46% | 29242.82 | 12.27 | 232.09 | 29251.72 | 12.27 | 232.16 | 1.00 |
| 12 | cosoco | 2.0 parallel | 121 | 86 SAT, 35 UNSAT | 40% | 44% | 30919.58 | 16.37 | 255.53 | 5190.83 | 3.24 | 42.90 | 5.96 |
| Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||||||||
| Virtual Best Solver (VBS) | 177 | 59% | 100% | 22507.11 | 13.50 | 127.16 | - | - | - | - | ||
| 1 | PicatSAT | 2019-09-12 | 163 | 54% | 92% | 37887.71 | 83.95 | 232.44 | 37890.88 | 83.97 | 232.46 | 1.00 |
| 2 | Fun-sCOP | hybrid+CryptoMiniSat (2019-06-15) | 132 | 44% | 75% | 39141.34 | 125.06 | 296.53 | 36261.81 | 108.38 | 274.71 | 1.08 |
| 3 | Fun-sCOP | order+GlueMiniSat (2019-06-15) | 122 | 41% | 69% | 37305.10 | 116.56 | 305.78 | 34221.61 | 99.42 | 280.51 | 1.09 |
| 4 | Fun-sCOP | order+ManyGlucose (2019-09-22) | 122 | 41% | 69% | 58991.64 | 169.61 | 483.54 | 22613.52 | 50.48 | 185.36 | 2.61 |
| 5 | Fun-sCOP | hybrid+ManyGlucose (2019-09-22) | 121 | 40% | 68% | 57077.22 | 167.68 | 471.71 | 23897.34 | 46.38 | 197.50 | 2.39 |
| 6 | AbsCon | 2019-07-23 | 114 | 38% | 64% | 12692.53 | 14.63 | 111.34 | 12338.30 | 11.22 | 108.23 | 1.03 |
| 7 | choco-solver | 2019-09-24 parallel | 113 | 38% | 64% | 35443.58 | 56.99 | 313.66 | 5496.45 | 9.71 | 48.64 | 6.45 |
| 8 | Concrete | 3.12.3 | 106 | 35% | 60% | 17520.27 | 30.88 | 165.29 | 15984.18 | 16.02 | 150.79 | 1.10 |
| 9 | choco-solver | 2019-09-24 | 101 | 34% | 57% | 17071.57 | 15.10 | 169.03 | 16563.38 | 12.10 | 163.99 | 1.03 |
| 10 | cosoco | 2.0 parallel | 86 | 29% | 49% | 11545.97 | 9.63 | 134.26 | 2174.99 | 1.98 | 25.29 | 5.31 |
| 11 | BTD | 19.07.01 | 84 | 28% | 47% | 12867.25 | 6.65 | 153.18 | 12870.11 | 6.65 | 153.22 | 1.00 |
| 12 | cosoco | 2.0 | 82 | 27% | 46% | 13989.00 | 3.65 | 170.60 | 13996.86 | 4.22 | 170.69 | 1.00 |
| Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||||||||
| Virtual Best Solver (VBS) | 95 | 32% | 100% | 7083.67 | 22.39 | 74.56 | - | - | - | - | ||
| 1 | PicatSAT | 2019-09-12 | 82 | 27% | 86% | 22445.02 | 121.79 | 273.72 | 22446.21 | 121.81 | 273.73 | 1.00 |
| 2 | Fun-sCOP | hybrid+CryptoMiniSat (2019-06-15) | 77 | 26% | 81% | 16781.77 | 73.80 | 217.95 | 13599.32 | 55.20 | 176.61 | 1.23 |
| 3 | Fun-sCOP | hybrid+ManyGlucose (2019-09-22) | 77 | 26% | 81% | 33032.08 | 157.05 | 428.99 | 11297.43 | 42.74 | 146.72 | 2.92 |
| 4 | Fun-sCOP | order+ManyGlucose (2019-09-22) | 70 | 23% | 74% | 34365.70 | 260.88 | 490.94 | 12555.60 | 55.83 | 179.37 | 2.74 |
| 5 | Fun-sCOP | order+GlueMiniSat (2019-06-15) | 68 | 23% | 72% | 18765.14 | 109.56 | 275.96 | 15665.64 | 97.88 | 230.38 | 1.20 |
| 6 | AbsCon | 2019-07-23 | 53 | 18% | 56% | 9388.81 | 32.26 | 177.15 | 9102.49 | 27.87 | 171.75 | 1.03 |
| 7 | BTD | 19.07.01 | 51 | 17% | 54% | 12220.38 | 20.96 | 239.62 | 12222.28 | 20.96 | 239.65 | 1.00 |
| 8 | Concrete | 3.12.3 | 50 | 17% | 53% | 23156.44 | 209.71 | 463.13 | 22347.71 | 195.82 | 446.95 | 1.04 |
| 9 | choco-solver | 2019-09-24 | 48 | 16% | 51% | 11162.71 | 29.82 | 232.56 | 10908.59 | 21.77 | 227.26 | 1.02 |
| 10 | cosoco | 2.0 | 44 | 15% | 46% | 15253.82 | 127.08 | 346.68 | 15254.86 | 127.11 | 346.70 | 1.00 |
| 11 | choco-solver | 2019-09-24 parallel | 40 | 13% | 42% | 13810.98 | 164.39 | 345.27 | 2351.47 | 35.55 | 58.79 | 5.87 |
| 12 | cosoco | 2.0 parallel | 35 | 12% | 37% | 19373.61 | 139.67 | 553.53 | 3015.84 | 24.23 | 86.17 | 6.42 |
This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within the global time limit. Ties are broken using the cumulated T1 time.
Solvers which gave a wrong answer in a category are excluded from the ranking in that category.
The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.
| Rank | Solver | Version | Number of best solutions found | % of all instances | % of VBS | Cumulated time |
|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||
| Virtual Best Solver (VBS) | 270 | 90% | 100% | |||
| 1 | choco-solver | 2019-09-24 parallel | 240 | 80% | 89% | 194328.0529 |
| 2 | PicatSAT | 2019-09-12 | 221 | 74% | 82% | 45523.67277 |
| 3 | AbsCon | 2019-07-23 | 204 | 68% | 76% | 29664.94308 |
| 4 | choco-solver | 2019-09-24 | 197 | 66% | 73% | 46089.8538 |
| 5 | Concrete | 3.12.3 | 180 | 60% | 67% | 20772.236 |
| 6 | cosoco | 2.0 parallel | 171 | 57% | 63% | 83022.17 |
| 7 | cosoco | 2.0 | 123 | 41% | 46% | 7485.41 |
This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 2 times the VBS time. Ties are broken using the cumulated T1 time.
Solvers which gave a wrong answer in a category are excluded from the ranking in that category.
The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.
| Rank | Solver | Version | Number of best solutions found | % of all instances | % of VBS | Cumulated time |
|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||
| Virtual Best Solver (VBS) | 270 | 90% | 100% | |||
| 1 | choco-solver | 2019-09-24 | 84 | 28% | 31% | 24661.54 |
| 2 | cosoco | 2.0 | 68 | 23% | 25% | 418.73 |
| 3 | cosoco | 2.0 parallel | 56 | 19% | 21% | 899.62 |
| 4 | AbsCon | 2019-07-23 | 51 | 17% | 19% | 3458.29 |
| 5 | choco-solver | 2019-09-24 parallel | 44 | 15% | 16% | 133915.64 |
| 6 | PicatSAT | 2019-09-12 | 26 | 9% | 10% | 6230.26 |
| 7 | Concrete | 3.12.3 | 26 | 9% | 10% | 2817.71 |
This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 4 times the VBS time. Ties are broken using the cumulated T1 time.
Solvers which gave a wrong answer in a category are excluded from the ranking in that category.
The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.
| Rank | Solver | Version | Number of best solutions found | % of all instances | % of VBS | Cumulated time |
|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||
| Virtual Best Solver (VBS) | 270 | 90% | 100% | |||
| 1 | choco-solver | 2019-09-24 | 109 | 36% | 40% | 25152.93 |
| 2 | choco-solver | 2019-09-24 parallel | 101 | 34% | 37% | 142780.37 |
| 3 | cosoco | 2.0 | 80 | 27% | 30% | 648.36 |
| 4 | cosoco | 2.0 parallel | 79 | 26% | 29% | 1454.22 |
| 5 | AbsCon | 2019-07-23 | 75 | 25% | 28% | 3968.26 |
| 6 | Concrete | 3.12.3 | 45 | 15% | 17% | 3514.63 |
| 7 | PicatSAT | 2019-09-12 | 41 | 14% | 15% | 7725.25 |
This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 8 times the VBS time. Ties are broken using the cumulated T1 time.
Solvers which gave a wrong answer in a category are excluded from the ranking in that category.
The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.
| Rank | Solver | Version | Number of best solutions found | % of all instances | % of VBS | Cumulated time |
|---|---|---|---|---|---|---|
| Total number of instances in the category: 300 | ||||||
| Virtual Best Solver (VBS) | 270 | 90% | 100% | |||
| 1 | choco-solver | 2019-09-24 parallel | 141 | 47% | 52% | 146580.6 |
| 2 | choco-solver | 2019-09-24 | 129 | 43% | 48% | 25920.85 |
| 3 | AbsCon | 2019-07-23 | 110 | 37% | 41% | 4969.52 |
| 4 | cosoco | 2.0 parallel | 102 | 34% | 38% | 2132.28 |
| 5 | cosoco | 2.0 | 96 | 32% | 36% | 1274.09 |
| 6 | Concrete | 3.12.3 | 75 | 25% | 28% | 4620.12 |
| 7 | PicatSAT | 2019-09-12 | 61 | 20% | 23% | 10117.42 |