This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.
Solvers which gave a wrong answer in a category are excluded from the ranking in that category.
The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.
Rank | Solver | Version | Number of solved instances | Detail | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 481 | |||||||||||||
Virtual Best Solver (VBS) | 448 | 191 SAT, 257 UNSAT | 93% | 100% | 12868.51 | 0.12 | 28.72 | - | - | - | - | ||
1 | borg | pb-dec-11.04.03 | 431 | 183 SAT, 248 UNSAT | 90% | 96% | 10281.04 | 1.93 | 23.85 | 10957.42 | 3.08 | 25.42 | 0.94 |
2 | Sat4j Res//CP | 2.3.0 | 420 | 183 SAT, 237 UNSAT | 87% | 94% | 51588.82 | 5.47 | 122.83 | 27980.84 | 2.63 | 66.62 | 1.84 |
3 | bsolo | 3.2 | 416 | 179 SAT, 237 UNSAT | 86% | 93% | 58921.79 | 8.33 | 141.64 | 58922.02 | 8.33 | 141.64 | 1.00 |
4 | wbo | 1.6 | 394 | 180 SAT, 214 UNSAT | 82% | 88% | 17024.67 | 1.45 | 43.21 | 17025.89 | 1.45 | 43.21 | 1.00 |
5 | Sat4j Resolution | 2.3.0 | 392 | 184 SAT, 208 UNSAT | 81% | 88% | 33537.21 | 4.07 | 85.55 | 33199.22 | 3.42 | 84.69 | 1.01 |
6 | SCIP spx E_2 | 2011-06-10 (fixed) | 384 | 149 SAT, 235 UNSAT | 80% | 86% | 46478.58 | 1.28 | 121.04 | 46478.01 | 1.28 | 121.04 | 1.00 |
7 | SCIP spx 2 | 2011-06-10 (fixed) | 383 | 148 SAT, 235 UNSAT | 80% | 85% | 44742.45 | 1.34 | 116.82 | 44741.93 | 1.35 | 116.82 | 1.00 |
8 | clasp | 2.0-R4191 | 380 | 168 SAT, 212 UNSAT | 79% | 85% | 12375.19 | 0.63 | 32.57 | 12376.43 | 0.63 | 32.57 | 1.00 |
9 | MinisatID | 2.5.2 (fixed) | 368 | 169 SAT, 199 UNSAT | 77% | 82% | 19281.13 | 5.20 | 52.39 | 19281.13 | 5.20 | 52.39 | 1.00 |
10 | MinisatID | 2.5.2-gmp (fixed) | 362 | 165 SAT, 197 UNSAT | 75% | 81% | 37474.45 | 20.60 | 103.52 | 37474.12 | 20.60 | 103.52 | 1.00 |
11 | Sat4j CuttingPlanes | 2.3.0 | 242 | 114 SAT, 128 UNSAT | 50% | 54% | 22807.02 | 2.22 | 94.24 | 22539.31 | 1.04 | 93.14 | 1.01 |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 481 | ||||||||||||
Virtual Best Solver (VBS) | 191 | 40% | 100% | 5863.78 | 0.09 | 30.70 | - | - | - | - | ||
1 | Sat4j Resolution | 2.3.0 | 184 | 38% | 96% | 5265.59 | 1.89 | 28.62 | 5145.79 | 1.21 | 27.97 | 1.02 |
2 | borg | pb-dec-11.04.03 | 183 | 38% | 96% | 3819.10 | 1.52 | 20.87 | 4257.53 | 1.91 | 23.27 | 0.90 |
3 | Sat4j Res//CP | 2.3.0 | 183 | 38% | 96% | 6865.66 | 3.90 | 37.52 | 3513.53 | 1.67 | 19.20 | 1.95 |
4 | wbo | 1.6 | 180 | 37% | 94% | 2504.41 | 0.32 | 13.91 | 2505.05 | 0.32 | 13.92 | 1.00 |
5 | bsolo | 3.2 | 179 | 37% | 94% | 7297.97 | 2.52 | 40.77 | 7298.52 | 2.52 | 40.77 | 1.00 |
6 | MinisatID | 2.5.2 (fixed) | 169 | 35% | 88% | 5498.95 | 1.27 | 32.54 | 5498.95 | 1.27 | 32.54 | 1.00 |
7 | clasp | 2.0-R4191 | 168 | 35% | 88% | 7648.76 | 0.15 | 45.53 | 7649.40 | 0.15 | 45.53 | 1.00 |
8 | MinisatID | 2.5.2-gmp (fixed) | 165 | 34% | 86% | 3892.70 | 5.28 | 23.59 | 3892.90 | 5.28 | 23.59 | 1.00 |
9 | SCIP spx E_2 | 2011-06-10 (fixed) | 149 | 31% | 78% | 19631.44 | 3.43 | 131.75 | 19631.19 | 3.44 | 131.75 | 1.00 |
10 | SCIP spx 2 | 2011-06-10 (fixed) | 148 | 31% | 77% | 17871.17 | 3.19 | 120.75 | 17871.13 | 3.19 | 120.75 | 1.00 |
11 | Sat4j CuttingPlanes | 2.3.0 | 114 | 24% | 60% | 12424.65 | 5.11 | 108.99 | 12260.76 | 3.40 | 107.55 | 1.01 |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 481 | ||||||||||||
Virtual Best Solver (VBS) | 257 | 53% | 100% | 7004.74 | 0.17 | 27.26 | - | - | - | - | ||
1 | borg | pb-dec-11.04.03 | 248 | 52% | 96% | 6461.93 | 5.10 | 26.06 | 6699.88 | 5.81 | 27.02 | 0.96 |
2 | Sat4j Res//CP | 2.3.0 | 237 | 49% | 92% | 44723.16 | 11.94 | 188.71 | 24467.31 | 5.60 | 103.24 | 1.83 |
3 | bsolo | 3.2 | 237 | 49% | 92% | 51623.82 | 24.96 | 217.82 | 51623.50 | 24.96 | 217.82 | 1.00 |
4 | SCIP spx E_2 | 2011-06-10 (fixed) | 235 | 49% | 91% | 26847.14 | 0.59 | 114.24 | 26846.82 | 0.59 | 114.24 | 1.00 |
5 | SCIP spx 2 | 2011-06-10 (fixed) | 235 | 49% | 91% | 26871.28 | 0.64 | 114.35 | 26870.80 | 0.64 | 114.34 | 1.00 |
6 | wbo | 1.6 | 214 | 44% | 83% | 14520.26 | 2.71 | 67.85 | 14520.84 | 2.71 | 67.85 | 1.00 |
7 | clasp | 2.0-R4191 | 212 | 44% | 82% | 4726.43 | 0.90 | 22.29 | 4727.03 | 0.90 | 22.30 | 1.00 |
8 | Sat4j Resolution | 2.3.0 | 208 | 43% | 81% | 28271.62 | 9.08 | 135.92 | 28053.43 | 8.37 | 134.87 | 1.01 |
9 | MinisatID | 2.5.2 (fixed) | 199 | 41% | 77% | 13782.18 | 9.62 | 69.26 | 13782.17 | 9.62 | 69.26 | 1.00 |
10 | MinisatID | 2.5.2-gmp (fixed) | 197 | 41% | 77% | 33581.75 | 32.70 | 170.47 | 33581.22 | 32.70 | 170.46 | 1.00 |
11 | Sat4j CuttingPlanes | 2.3.0 | 128 | 27% | 50% | 10382.37 | 1.00 | 81.11 | 10278.55 | 0.53 | 80.30 | 1.01 |
Rank | Solver | Version | Number of solved instances | Detail | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 100 | |||||||||||||
Virtual Best Solver (VBS) | 76 | 55 SAT, 21 UNSAT | 76% | 100% | 2063.40 | 0.22 | 27.15 | - | - | - | - | ||
1 | SCIP spx E_2 | 2011-06-10 (fixed) | 75 | 55 SAT, 20 UNSAT | 75% | 99% | 8178.76 | 13.76 | 109.05 | 8178.67 | 13.76 | 109.05 | 1.00 |
2 | SCIP spx 2 | 2011-06-10 (fixed) | 74 | 54 SAT, 20 UNSAT | 74% | 97% | 6518.32 | 6.42 | 88.09 | 6518.47 | 6.42 | 88.09 | 1.00 |
3 | borg | pb-dec-11.04.03 | 73 | 53 SAT, 20 UNSAT | 73% | 96% | 10603.89 | 0.88 | 145.26 | 10569.77 | 1.19 | 144.79 | 1.00 |
4 | Sat4j CuttingPlanes | 2.3.0 | 65 | 50 SAT, 15 UNSAT | 65% | 86% | 446.59 | 1.87 | 6.87 | 388.60 | 0.86 | 5.98 | 1.15 |
5 | Sat4j Res//CP | 2.3.0 | 65 | 50 SAT, 15 UNSAT | 65% | 86% | 792.28 | 2.82 | 12.19 | 396.23 | 1.34 | 6.10 | 2.00 |
6 | clasp | 2.0-R4191 | 64 | 49 SAT, 15 UNSAT | 64% | 84% | 3517.38 | 0.09 | 54.96 | 3517.66 | 0.09 | 54.96 | 1.00 |
7 | bsolo | 3.2 | 62 | 47 SAT, 15 UNSAT | 62% | 82% | 3615.09 | 3.03 | 58.31 | 3615.27 | 3.03 | 58.31 | 1.00 |
8 | Sat4j Resolution | 2.3.0 | 27 | 12 SAT, 15 UNSAT | 27% | 36% | 3673.64 | 5.41 | 136.06 | 3656.90 | 4.65 | 135.44 | 1.00 |
9 | MinisatID | 2.5.2 (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - | |
10 | MinisatID | 2.5.2-gmp (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 100 | ||||||||||||
Virtual Best Solver (VBS) | 55 | 55% | 100% | 234.89 | 0.21 | 4.27 | - | - | - | - | ||
1 | SCIP spx E_2 | 2011-06-10 (fixed) | 55 | 55% | 100% | 8084.85 | 26.62 | 147.00 | 8084.75 | 26.62 | 147.00 | 1.00 |
2 | SCIP spx 2 | 2011-06-10 (fixed) | 54 | 54% | 98% | 4656.35 | 6.42 | 86.23 | 4656.52 | 6.42 | 86.23 | 1.00 |
3 | borg | pb-dec-11.04.03 | 53 | 53% | 96% | 4211.29 | 0.88 | 79.46 | 4202.82 | 1.17 | 79.30 | 1.00 |
4 | Sat4j CuttingPlanes | 2.3.0 | 50 | 50% | 91% | 270.45 | 2.02 | 5.41 | 220.69 | 0.91 | 4.41 | 1.23 |
5 | Sat4j Res//CP | 2.3.0 | 50 | 50% | 91% | 463.95 | 2.86 | 9.28 | 224.98 | 1.35 | 4.50 | 2.06 |
6 | clasp | 2.0-R4191 | 49 | 49% | 89% | 3461.65 | 0.13 | 70.65 | 3461.77 | 0.13 | 70.65 | 1.00 |
7 | bsolo | 3.2 | 47 | 47% | 85% | 2476.44 | 3.64 | 52.69 | 2476.50 | 3.64 | 52.69 | 1.00 |
8 | Sat4j Resolution | 2.3.0 | 12 | 12% | 22% | 1877.13 | 10.98 | 156.43 | 1868.66 | 10.50 | 155.72 | 1.00 |
9 | MinisatID | 2.5.2 (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
10 | MinisatID | 2.5.2-gmp (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 100 | ||||||||||||
Virtual Best Solver (VBS) | 21 | 21% | 100% | 1828.51 | 0.57 | 87.07 | - | - | - | - | ||
1 | SCIP spx E_2 | 2011-06-10 (fixed) | 20 | 20% | 95% | 93.91 | 0.30 | 4.70 | 93.92 | 0.30 | 4.70 | 1.00 |
2 | SCIP spx 2 | 2011-06-10 (fixed) | 20 | 20% | 95% | 1861.98 | 10.22 | 93.10 | 1861.95 | 10.22 | 93.10 | 1.00 |
3 | borg | pb-dec-11.04.03 | 20 | 20% | 95% | 6392.60 | 6.27 | 319.63 | 6366.96 | 7.83 | 318.35 | 1.00 |
4 | clasp | 2.0-R4191 | 15 | 15% | 71% | 55.73 | 0.00 | 3.72 | 55.89 | 0.02 | 3.73 | 1.00 |
5 | Sat4j CuttingPlanes | 2.3.0 | 15 | 15% | 71% | 176.13 | 0.28 | 11.74 | 167.91 | 0.22 | 11.19 | 1.05 |
6 | Sat4j Res//CP | 2.3.0 | 15 | 15% | 71% | 328.33 | 0.58 | 21.89 | 171.25 | 0.65 | 11.42 | 1.92 |
7 | bsolo | 3.2 | 15 | 15% | 71% | 1138.65 | 0.00 | 75.91 | 1138.77 | 0.02 | 75.92 | 1.00 |
8 | Sat4j Resolution | 2.3.0 | 15 | 15% | 71% | 1796.51 | 0.31 | 119.77 | 1788.23 | 0.22 | 119.22 | 1.00 |
9 | MinisatID | 2.5.2 (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
10 | MinisatID | 2.5.2-gmp (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
Rank | Solver | Version | Number of solved instances | Detail | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 557 | |||||||||||||
Virtual Best Solver (VBS) | 213 | 154 OPT, 59 UNSAT | 38% | 100% | 9573.55 | 2.48 | 44.95 | - | - | - | - | ||
1 | Sat4j Res//CP | 2.3.0 | 208 | 149 OPT, 59 UNSAT | 37% | 98% | 13907.35 | 3.52 | 66.86 | 8297.48 | 2.93 | 39.89 | 1.68 |
2 | Sat4j Resolution | 2.3.0 | 201 | 144 OPT, 57 UNSAT | 36% | 94% | 10070.77 | 2.63 | 50.10 | 9894.98 | 1.75 | 49.23 | 1.02 |
3 | Sat4j CuttingPlanes | 2.3.0 | 175 | 116 OPT, 59 UNSAT | 31% | 82% | 10253.32 | 2.30 | 58.59 | 10077.46 | 1.38 | 57.59 | 1.02 |
4 | MinisatID | 2.5.2-gmp (fixed) | 117 | 60 OPT, 57 UNSAT | 21% | 55% | 23089.88 | 3.58 | 197.35 | 23089.54 | 3.58 | 197.35 | 1.00 |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 557 | ||||||||||||
Virtual Best Solver (VBS) | 154 | 28% | 100% | 9460.02 | 4.25 | 61.43 | - | - | - | - | ||
1 | Sat4j Res//CP | 2.3.0 | 149 | 27% | 97% | 13719.20 | 7.33 | 92.08 | 8167.71 | 6.07 | 54.82 | 1.68 |
2 | Sat4j Resolution | 2.3.0 | 144 | 26% | 94% | 9962.32 | 5.90 | 69.18 | 9814.40 | 4.84 | 68.16 | 1.02 |
3 | Sat4j CuttingPlanes | 2.3.0 | 116 | 21% | 75% | 8943.45 | 4.00 | 77.10 | 8806.65 | 2.91 | 75.92 | 1.02 |
4 | MinisatID | 2.5.2-gmp (fixed) | 60 | 11% | 39% | 20756.40 | 111.05 | 345.94 | 20756.07 | 111.14 | 345.93 | 1.00 |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 557 | ||||||||||||
Virtual Best Solver (VBS) | 59 | 11% | 100% | 113.53 | 1.01 | 1.92 | - | - | - | - | ||
1 | Sat4j Res//CP | 2.3.0 | 59 | 11% | 100% | 188.15 | 1.73 | 3.19 | 129.77 | 1.20 | 2.20 | 1.45 |
2 | Sat4j CuttingPlanes | 2.3.0 | 59 | 11% | 100% | 1309.87 | 1.36 | 22.20 | 1270.81 | 0.68 | 21.54 | 1.03 |
3 | Sat4j Resolution | 2.3.0 | 57 | 10% | 97% | 108.44 | 1.07 | 1.90 | 80.58 | 0.58 | 1.41 | 1.35 |
4 | MinisatID | 2.5.2-gmp (fixed) | 57 | 10% | 97% | 2333.48 | 1.10 | 40.94 | 2333.46 | 1.10 | 40.94 | 1.00 |
Rank | Solver | Version | Number of solved instances | Detail | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 731 | |||||||||||||
Virtual Best Solver (VBS) | 494 | 459 OPT, 35 UNSAT | 68% | 100% | 27524.70 | 0.54 | 55.72 | - | - | - | - | ||
1 | SCIP spx E_2 | 2011-06-10 (fixed) | 409 | 374 OPT, 35 UNSAT | 56% | 83% | 51061.22 | 4.28 | 124.84 | 51055.05 | 4.28 | 124.83 | 1.00 |
2 | SCIP spx 2 | 2011-06-10 (fixed) | 409 | 374 OPT, 35 UNSAT | 56% | 83% | 52208.54 | 4.23 | 127.65 | 52208.03 | 4.23 | 127.65 | 1.00 |
3 | pwbo | 1.1 | 383 | 350 OPT, 33 UNSAT | 52% | 78% | 25328.59 | 1.07 | 66.13 | 12714.50 | 0.69 | 33.20 | 1.99 |
4 | bsolo | 3.2 | 350 | 316 OPT, 34 UNSAT | 48% | 71% | 35382.65 | 2.88 | 101.09 | 35383.54 | 2.88 | 101.10 | 1.00 |
5 | Sat4j Res//CP | 2.3.0 | 329 | 295 OPT, 34 UNSAT | 45% | 67% | 21642.59 | 3.24 | 65.78 | 13140.56 | 2.73 | 39.94 | 1.65 |
6 | clasp | 2.0-R4191 | 320 | 286 OPT, 34 UNSAT | 44% | 65% | 11896.81 | 0.40 | 37.18 | 11898.45 | 0.40 | 37.18 | 1.00 |
7 | Sat4j Resolution | 2.3.0 | 316 | 282 OPT, 34 UNSAT | 43% | 64% | 14889.11 | 2.13 | 47.12 | 14623.56 | 1.18 | 46.28 | 1.02 |
8 | MinisatID | 2.5.2 (fixed) | 288 | 256 OPT, 32 UNSAT | 39% | 58% | 31859.77 | 5.74 | 110.62 | 31859.28 | 5.74 | 110.62 | 1.00 |
9 | Sat4j CuttingPlanes | 2.3.0 | 270 | 240 OPT, 30 UNSAT | 37% | 55% | 24809.58 | 2.14 | 91.89 | 24492.17 | 1.03 | 90.71 | 1.01 |
10 | wbo | 1.6 | 269 | 236 OPT, 33 UNSAT | 37% | 54% | 18212.75 | 0.88 | 67.71 | 18213.68 | 0.88 | 67.71 | 1.00 |
11 | MinisatID | 2.5.2-gmp (fixed) | 262 | 232 OPT, 30 UNSAT | 36% | 53% | 36343.88 | 9.04 | 138.72 | 36345.50 | 9.04 | 138.72 | 1.00 |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 731 | ||||||||||||
Virtual Best Solver (VBS) | 459 | 63% | 100% | 27261.29 | 0.72 | 59.39 | - | - | - | - | ||
1 | SCIP spx E_2 | 2011-06-10 (fixed) | 374 | 51% | 81% | 50199.41 | 4.89 | 134.22 | 50193.23 | 4.90 | 134.21 | 1.00 |
2 | SCIP spx 2 | 2011-06-10 (fixed) | 374 | 51% | 81% | 51348.81 | 4.95 | 137.30 | 51348.30 | 4.95 | 137.29 | 1.00 |
3 | pwbo | 1.1 | 350 | 48% | 76% | 24711.03 | 1.32 | 70.60 | 12404.91 | 0.90 | 35.44 | 1.99 |
4 | bsolo | 3.2 | 316 | 43% | 69% | 34517.45 | 6.60 | 109.23 | 34518.17 | 6.60 | 109.23 | 1.00 |
5 | Sat4j Res//CP | 2.3.0 | 295 | 40% | 64% | 20982.25 | 3.59 | 71.13 | 12798.43 | 2.93 | 43.38 | 1.64 |
6 | clasp | 2.0-R4191 | 286 | 39% | 62% | 11784.44 | 0.51 | 41.20 | 11785.82 | 0.52 | 41.21 | 1.00 |
7 | Sat4j Resolution | 2.3.0 | 282 | 39% | 61% | 14566.30 | 2.38 | 51.65 | 14317.92 | 1.36 | 50.77 | 1.02 |
8 | MinisatID | 2.5.2 (fixed) | 256 | 35% | 56% | 29555.22 | 8.20 | 115.45 | 29554.78 | 8.20 | 115.45 | 1.00 |
9 | Sat4j CuttingPlanes | 2.3.0 | 240 | 33% | 52% | 24787.01 | 2.57 | 103.28 | 24479.56 | 1.49 | 102.00 | 1.01 |
10 | wbo | 1.6 | 236 | 32% | 51% | 17500.77 | 1.10 | 74.16 | 17501.55 | 1.11 | 74.16 | 1.00 |
11 | MinisatID | 2.5.2-gmp (fixed) | 232 | 32% | 51% | 36301.53 | 20.21 | 156.47 | 36303.13 | 20.21 | 156.48 | 1.00 |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 731 | ||||||||||||
Virtual Best Solver (VBS) | 35 | 5% | 100% | 263.40 | 0.01 | 7.53 | - | - | - | - | ||
1 | SCIP spx 2 | 2011-06-10 (fixed) | 35 | 5% | 100% | 859.73 | 0.34 | 24.56 | 859.73 | 0.34 | 24.56 | 1.00 |
2 | SCIP spx E_2 | 2011-06-10 (fixed) | 35 | 5% | 100% | 861.81 | 0.31 | 24.62 | 861.82 | 0.31 | 24.62 | 1.00 |
3 | clasp | 2.0-R4191 | 34 | 5% | 97% | 112.38 | 0.01 | 3.31 | 112.63 | 0.02 | 3.31 | 1.00 |
4 | Sat4j Resolution | 2.3.0 | 34 | 5% | 97% | 322.81 | 0.82 | 9.49 | 305.64 | 0.45 | 8.99 | 1.06 |
5 | Sat4j Res//CP | 2.3.0 | 34 | 5% | 97% | 660.34 | 0.98 | 19.42 | 342.13 | 0.86 | 10.06 | 1.93 |
6 | bsolo | 3.2 | 34 | 5% | 97% | 865.19 | 0.06 | 25.45 | 865.37 | 0.06 | 25.45 | 1.00 |
7 | pwbo | 1.1 | 33 | 5% | 94% | 617.56 | 0.03 | 18.71 | 309.60 | 0.03 | 9.38 | 1.99 |
8 | wbo | 1.6 | 33 | 5% | 94% | 711.98 | 0.06 | 21.58 | 712.14 | 0.07 | 21.58 | 1.00 |
9 | MinisatID | 2.5.2 (fixed) | 32 | 4% | 91% | 2304.56 | 0.25 | 72.02 | 2304.51 | 0.25 | 72.02 | 1.00 |
10 | Sat4j CuttingPlanes | 2.3.0 | 30 | 4% | 86% | 22.57 | 0.85 | 0.75 | 12.60 | 0.46 | 0.42 | 1.79 |
11 | MinisatID | 2.5.2-gmp (fixed) | 30 | 4% | 86% | 42.35 | 0.38 | 1.41 | 42.38 | 0.38 | 1.41 | 1.00 |
Rank | Solver | Version | Number of solved instances | Detail | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 409 | |||||||||||||
Virtual Best Solver (VBS) | 298 | 298 OPT | 73% | 100% | 5387.76 | 0.09 | 18.08 | - | - | - | - | ||
1 | SCIP spx E_2 | 2011-06-10 (fixed) | 297 | 297 OPT | 73% | 100% | 6442.90 | 0.13 | 21.69 | 6442.97 | 0.13 | 21.69 | 1.00 |
2 | SCIP spx 2 | 2011-06-10 (fixed) | 294 | 294 OPT | 72% | 99% | 5240.65 | 0.14 | 17.83 | 5240.81 | 0.14 | 17.83 | 1.00 |
3 | clasp | 2.0-R4191-patched (fixed) | 280 | 280 OPT | 68% | 94% | 9225.64 | 1.02 | 32.95 | 9225.77 | 1.02 | 32.95 | 1.00 |
4 | Sat4j Resolution | 2.3.0 | 277 | 277 OPT | 68% | 93% | 18696.59 | 2.95 | 67.50 | 18465.85 | 2.30 | 66.66 | 1.01 |
5 | Sat4j Res//CP | 2.3.0 | 274 | 274 OPT | 67% | 92% | 34407.38 | 7.48 | 125.57 | 19080.46 | 3.77 | 69.64 | 1.80 |
6 | bsolo | 3.2 | 234 | 234 OPT | 57% | 79% | 29076.06 | 2.58 | 124.26 | 29076.57 | 2.58 | 124.26 | 1.00 |
7 | Sat4j CuttingPlanes | 2.3.0 | 120 | 120 OPT | 29% | 40% | 9357.40 | 4.11 | 77.98 | 9218.19 | 2.82 | 76.82 | 1.02 |
8 | MinisatID | 2.5.2 (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - | |
9 | MinisatID | 2.5.2-gmp (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 409 | ||||||||||||
Virtual Best Solver (VBS) | 298 | 73% | 100% | 5387.76 | 0.09 | 18.08 | - | - | - | - | ||
1 | SCIP spx E_2 | 2011-06-10 (fixed) | 297 | 73% | 100% | 6442.90 | 0.13 | 21.69 | 6442.97 | 0.13 | 21.69 | 1.00 |
2 | SCIP spx 2 | 2011-06-10 (fixed) | 294 | 72% | 99% | 5240.65 | 0.14 | 17.83 | 5240.81 | 0.14 | 17.83 | 1.00 |
3 | clasp | 2.0-R4191-patched (fixed) | 280 | 68% | 94% | 9225.64 | 1.02 | 32.95 | 9225.77 | 1.02 | 32.95 | 1.00 |
4 | Sat4j Resolution | 2.3.0 | 277 | 68% | 93% | 18696.59 | 2.95 | 67.50 | 18465.85 | 2.30 | 66.66 | 1.01 |
5 | Sat4j Res//CP | 2.3.0 | 274 | 67% | 92% | 34407.38 | 7.48 | 125.57 | 19080.46 | 3.77 | 69.64 | 1.80 |
6 | bsolo | 3.2 | 234 | 57% | 79% | 29076.06 | 2.58 | 124.26 | 29076.57 | 2.58 | 124.26 | 1.00 |
7 | Sat4j CuttingPlanes | 2.3.0 | 120 | 29% | 40% | 9357.40 | 4.11 | 77.98 | 9218.19 | 2.82 | 76.82 | 1.02 |
8 | MinisatID | 2.5.2 (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
9 | MinisatID | 2.5.2-gmp (fixed) | 0 | 0% | 0% | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
Rank | Solver | Version | Number of solved instances | % of all instances | % of VBS | Cumulated CPU time on solved instances | Median CPU time | Average CPU time per solved instance | Cumulated WC time on solved instances | Median WC time | Average WC time per solved instance | CPU/WC |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Total number of instances in the category: 409 | ||||||||||||
Virtual Best Solver (VBS) | 0 | 0% | - | 0.00 | 0.00 | - | - | - | - | - | ||
1 | Sat4j Res//CP | 2.3.0 | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
2 | Sat4j CuttingPlanes | 2.3.0 | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
3 | Sat4j Resolution | 2.3.0 | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
4 | bsolo | 3.2 | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
5 | SCIP spx 2 | 2011-06-10 (fixed) | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
6 | SCIP spx E_2 | 2011-06-10 (fixed) | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
7 | MinisatID | 2.5.2 (fixed) | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
8 | clasp | 2.0-R4191-patched (fixed) | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |
9 | MinisatID | 2.5.2-gmp (fixed) | 0 | 0% | - | 0.00 | 0.00 | - | 0.00 | 0.00 | - | - |