PB'09 competition: solvers page (tabular format, summary)

Some explanations

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
BoolVar
2009-04-26
(complete)		 371288
(77.63%)		 177
(47.71%)		 111
(29.92%)		 0
(0%)		 83
(22.37%)		 0
(0%)
bsolo
3.1
(complete)		 371325
(87.6%)		 188
(50.67%)		 137
(36.93%)		 0
(0%)		 46
(12.4%)		 0
(0%)
bsolo
3.1 cl
(complete)		 371290
(78.17%)		 154
(41.51%)		 136
(36.66%)		 0
(0%)		 81
(21.83%)		 0
(0%)
bsolo
3.1 pb
(complete)		 371284
(76.55%)		 148
(39.89%)		 136
(36.66%)		 0
(0%)		 87
(23.45%)		 0
(0%)
pbclasp
2009-04-24
(complete)		 371300
(80.86%)		 174
(46.9%)		 126
(33.96%)		 0
(0%)		 71
(19.14%)		 0
(0%)
SAT4J Pseudo CP
2.1.1
(complete)		 371164
(44.2%)		 92
(24.8%)		 72
(19.41%)		 0
(0%)		 207
(55.8%)		 0
(0%)
SAT4J Pseudo Resolution
2.1.1
(complete)		 371278
(74.93%)		 172
(46.36%)		 106
(28.57%)		 0
(0%)		 93
(25.07%)		 0
(0%)
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)		 371308
(83.02%)		 189
(50.94%)		 119
(32.08%)		 0
(0%)		 63
(16.98%)		 0
(0%)
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)		 371310
(83.56%)		 191
(51.48%)		 119
(32.08%)		 0
(0%)		 61
(16.44%)		 0
(0%)
wbo
1.0
(complete)		 371309
(83.29%)		 172
(46.36%)		 137
(36.93%)		 0
(0%)		 62
(16.71%)		 0
(0%)

Solver NameProgress
BoolVar
2009-04-26
(complete)
done
371
UNSAT
177		OPT/SAT
111		?
83
bsolo
3.1
(complete)
done
371
UNSAT
188		OPT/SAT
137		?
46
bsolo
3.1 cl
(complete)
done
371
UNSAT
154		OPT/SAT
136		?
81
bsolo
3.1 pb
(complete)
done
371
UNSAT
148		OPT/SAT
136		?
87
pbclasp
2009-04-24
(complete)
done
371
UNSAT
174		OPT/SAT
126		?
71
SAT4J Pseudo CP
2.1.1
(complete)
done
371
UNSAT
92		OPT/SAT
72		?
207
SAT4J Pseudo Resolution
2.1.1
(complete)
done
371
UNSAT
172		OPT/SAT
106		?
93
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)
done
371
UNSAT
189		OPT/SAT
119		?
63
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)
done
371
UNSAT
191		OPT/SAT
119		?
61
wbo
1.0
(complete)
done
371
UNSAT
172		OPT/SAT
137		?
62

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
BoolVar
2009-04-26
(complete)		 10057
(57%)		 15
(15%)		 42
(42%)		 0
(0%)		 43
(43%)		 0
(0%)
bsolo
3.1
(complete)		 10061
(61%)		 15
(15%)		 46
(46%)		 0
(0%)		 39
(39%)		 0
(0%)
bsolo
3.1 cl
(complete)		 10026
(26%)		 10
(10%)		 16
(16%)		 0
(0%)		 74
(74%)		 0
(0%)
bsolo
3.1 pb
(complete)		 10064
(64%)		 15
(15%)		 49
(49%)		 0
(0%)		 36
(36%)		 0
(0%)
pbclasp
2009-04-24
(complete)		 10065
(65%)		 15
(15%)		 50
(50%)		 0
(0%)		 35
(35%)		 0
(0%)
SAT4J Pseudo CP
2.1.1
(complete)		 10065
(65%)		 15
(15%)		 50
(50%)		 0
(0%)		 35
(35%)		 0
(0%)
SAT4J Pseudo Resolution
2.1.1
(complete)		 10064
(64%)		 13
(13%)		 51
(51%)		 0
(0%)		 36
(36%)		 0
(0%)
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)		 10074
(74%)		 20
(20%)		 54
(54%)		 0
(0%)		 26
(26%)		 0
(0%)
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)		 10074
(74%)		 20
(20%)		 54
(54%)		 0
(0%)		 26
(26%)		 0
(0%)
wbo
1.0
(complete)		 10059
(59%)		 15
(15%)		 44
(44%)		 0
(0%)		 41
(41%)		 0
(0%)

Solver NameProgress
BoolVar
2009-04-26
(complete)
done
100
UNSAT
15		OPT/SAT
42		?
43
bsolo
3.1
(complete)
done
100
UNSAT
15		OPT/SAT
46		?
39
bsolo
3.1 cl
(complete)
done
100
UNSAT
10		OPT/SAT
16		?
74
bsolo
3.1 pb
(complete)
done
100
UNSAT
15		OPT/SAT
49		?
36
pbclasp
2009-04-24
(complete)
done
100
UNSAT
15		OPT/SAT
50		?
35
SAT4J Pseudo CP
2.1.1
(complete)
done
100
UNSAT
15		OPT/SAT
50		?
35
SAT4J Pseudo Resolution
2.1.1
(complete)
done
100
UNSAT
13		OPT/SAT
51		?
36
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)
done
100
UNSAT
20		OPT/SAT
54		?
26
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)
done
100
UNSAT
20		OPT/SAT
54		?
26
wbo
1.0
(complete)
done
100
UNSAT
15		OPT/SAT
44		?
41

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
SAT4J Pseudo CP
2.1.1
(complete)		 452124
(27.43%)		 40
(8.85%)		 84
(18.58%)		 119
(26.33%)		 209
(46.24%)		 0
(0%)
SAT4J Pseudo Resolution
2.1.1
(complete)		 452133
(29.42%)		 40
(8.85%)		 93
(20.58%)		 138
(30.53%)		 181
(40.04%)		 0
(0%)

Solver NameProgress
SAT4J Pseudo CP
2.1.1
(complete)
done
452
UNSAT
40		OPT/SAT
84		SAT (TO/MO)
119		?
209
SAT4J Pseudo Resolution
2.1.1
(complete)
done
452
UNSAT
40		OPT/SAT
93		SAT (TO/MO)
138		?
181

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
bsolo
3.1
(complete)		 554234
(42.24%)		 13
(2.35%)		 221
(39.89%)		 274
(49.46%)		 45
(8.12%)		 1
(0.18%)
bsolo
3.1 cl
(complete)		 554230
(41.52%)		 13
(2.35%)		 217
(39.17%)		 278
(50.18%)		 46
(8.3%)		 0
(0%)
bsolo
3.1 pb
(complete)		 554231
(41.7%)		 13
(2.35%)		 218
(39.35%)		 276
(49.82%)		 47
(8.48%)		 0
(0%)
pbclasp
2009-04-24
(complete)		 554188
(33.94%)		 13
(2.35%)		 175
(31.59%)		 303
(54.69%)		 60
(10.83%)		 3
(0.54%)
SAT4J Pseudo CP
2.1.1
(complete)		 554140
(25.27%)		 9
(1.62%)		 131
(23.65%)		 339
(61.19%)		 75
(13.54%)		 0
(0%)
SAT4J Pseudo Resolution
2.1.1
(complete)		 554169
(30.51%)		 11
(1.99%)		 158
(28.52%)		 315
(56.86%)		 70
(12.64%)		 0
(0%)
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)		 554299
(53.97%)		 13
(2.35%)		 286
(51.62%)		 150
(27.08%)		 104
(18.77%)		 1
(0.18%)
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)		 554295
(53.25%)		 13
(2.35%)		 282
(50.9%)		 149
(26.9%)		 110
(19.86%)		 0
(0%)
wbo
1.0
(complete)		 554136
(24.55%)		 12
(2.17%)		 124
(22.38%)		 0
(0%)		 418
(75.45%)		 0
(0%)

Solver NameProgress
bsolo
3.1
(complete)
done
554
UNSAT
13		OPT/SAT
221		SAT (TO/MO)
274		?
45		ERR
1
bsolo
3.1 cl
(complete)
done
554
UNSAT
13		OPT/SAT
217		SAT (TO/MO)
278		?
46
bsolo
3.1 pb
(complete)
done
554
UNSAT
13		OPT/SAT
218		SAT (TO/MO)
276		?
47
pbclasp
2009-04-24
(complete)
done
554
UNSAT
13		OPT/SAT
175		SAT (TO/MO)
303		?
60		ERR
3
SAT4J Pseudo CP
2.1.1
(complete)
done
554
UNSAT
9		OPT/SAT
131		SAT (TO/MO)
339		?
75
SAT4J Pseudo Resolution
2.1.1
(complete)
done
554
UNSAT
11		OPT/SAT
158		SAT (TO/MO)
315		?
70
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)
done
554
UNSAT
13		OPT/SAT
286		SAT (TO/MO)
150		?
104		ERR
1
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)
done
554
UNSAT
13		OPT/SAT
282		SAT (TO/MO)
149		?
110
wbo
1.0
(complete)
done
554
UNSAT
12		OPT/SAT
124		?
418

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
bsolo
3.1
(complete)		 405182
(44.94%)		 0
(0%)		 182
(44.94%)		 139
(34.32%)		 2
(0.49%)		 82
(20.25%)
bsolo
3.1 cl
(complete)		 405232
(57.28%)		 0
(0%)		 232
(57.28%)		 162
(40%)		 11
(2.72%)		 0
(0%)
bsolo
3.1 pb
(complete)		 405193
(47.65%)		 0
(0%)		 193
(47.65%)		 166
(40.99%)		 35
(8.64%)		 11
(2.72%)
pbclasp
2009-04-24
(complete)		 405263
(64.94%)		 0
(0%)		 263
(64.94%)		 138
(34.07%)		 4
(0.99%)		 0
(0%)
SAT4J Pseudo CP
2.1.1
(complete)		 405118
(29.14%)		 0
(0%)		 118
(29.14%)		 228
(56.3%)		 59
(14.57%)		 0
(0%)
SAT4J Pseudo Resolution
2.1.1
(complete)		 405226
(55.8%)		 0
(0%)		 226
(55.8%)		 177
(43.7%)		 2
(0.49%)		 0
(0%)
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)		 405286
(70.62%)		 0
(0%)		 286
(70.62%)		 105
(25.93%)		 14
(3.46%)		 0
(0%)
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)		 405289
(71.36%)		 0
(0%)		 289
(71.36%)		 99
(24.44%)		 17
(4.2%)		 0
(0%)
wbo
1.0
(complete)		 405223
(55.06%)		 0
(0%)		 223
(55.06%)		 0
(0%)		 182
(44.94%)		 0
(0%)

Solver NameProgress
bsolo
3.1
(complete)
done
405
OPT/SAT
182		SAT (TO/MO)
139		?
2		ERR
82
bsolo
3.1 cl
(complete)
done
405
OPT/SAT
232		SAT (TO/MO)
162		?
11
bsolo
3.1 pb
(complete)
done
405
OPT/SAT
193		SAT (TO/MO)
166		?
35		ERR
11
pbclasp
2009-04-24
(complete)
done
405
OPT/SAT
263		SAT (TO/MO)
138		?
4
SAT4J Pseudo CP
2.1.1
(complete)
done
405
OPT/SAT
118		SAT (TO/MO)
228		?
59
SAT4J Pseudo Resolution
2.1.1
(complete)
done
405
OPT/SAT
226		SAT (TO/MO)
177		?
2
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)
done
405
OPT/SAT
286		SAT (TO/MO)
105		?
14
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)
done
405
OPT/SAT
289		SAT (TO/MO)
99		?
17
wbo
1.0
(complete)
done
405
OPT/SAT
223		?
182

Category "instances containing only clauses" (PURE-SAT)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
BoolVar
2009-04-26
(complete)		 189150
(79.37%)		 64
(33.86%)		 86
(45.5%)		 0
(0%)		 35
(18.52%)		 4
(2.12%)
bsolo
3.1
(complete)		 189107
(56.61%)		 39
(20.63%)		 68
(35.98%)		 0
(0%)		 82
(43.39%)		 0
(0%)
bsolo
3.1 cl
(complete)		 189115
(60.85%)		 46
(24.34%)		 69
(36.51%)		 0
(0%)		 74
(39.15%)		 0
(0%)
bsolo
3.1 pb
(complete)		 189104
(55.03%)		 34
(17.99%)		 70
(37.04%)		 0
(0%)		 85
(44.97%)		 0
(0%)
pbclasp
2009-04-24
(complete)		 189163
(86.24%)		 66
(34.92%)		 97
(51.32%)		 0
(0%)		 26
(13.76%)		 0
(0%)
SAT4J Pseudo CP
2.1.1
(complete)		 189124
(65.61%)		 45
(23.81%)		 79
(41.8%)		 0
(0%)		 65
(34.39%)		 0
(0%)
SAT4J Pseudo Resolution
2.1.1
(complete)		 189144
(76.19%)		 60
(31.75%)		 84
(44.44%)		 0
(0%)		 45
(23.81%)		 0
(0%)
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)		 18943
(22.75%)		 15
(7.94%)		 28
(14.81%)		 0
(0%)		 146
(77.25%)		 0
(0%)
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)		 18948
(25.4%)		 16
(8.47%)		 32
(16.93%)		 0
(0%)		 141
(74.6%)		 0
(0%)
wbo
1.0
(complete)		 189131
(69.31%)		 50
(26.46%)		 81
(42.86%)		 0
(0%)		 58
(30.69%)		 0
(0%)

Solver NameProgress
BoolVar
2009-04-26
(complete)
done
189
UNSAT
64		OPT/SAT
86		?
35		ERR
4
bsolo
3.1
(complete)
done
189
UNSAT
39		OPT/SAT
68		?
82
bsolo
3.1 cl
(complete)
done
189
UNSAT
46		OPT/SAT
69		?
74
bsolo
3.1 pb
(complete)
done
189
UNSAT
34		OPT/SAT
70		?
85
pbclasp
2009-04-24
(complete)
done
189
UNSAT
66		OPT/SAT
97		?
26
SAT4J Pseudo CP
2.1.1
(complete)
done
189
UNSAT
45		OPT/SAT
79		?
65
SAT4J Pseudo Resolution
2.1.1
(complete)
done
189
UNSAT
60		OPT/SAT
84		?
45
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)
done
189
UNSAT
15		OPT/SAT
28		?
146
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)
done
189
UNSAT
16		OPT/SAT
32		?
141
wbo
1.0
(complete)
done
189
UNSAT
50		OPT/SAT
81		?
58

All categories for solvers registered in all categories

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
BoolVar
2009-04-26
(complete)		 660495
(75%)		 256
(38.79%)		 239
(36.21%)		 0
(0%)		 161
(24.39%)		 4
(0.61%)
bsolo
3.1
(complete)		 1619909
(56.15%)		 255
(15.75%)		 654
(40.4%)		 413
(25.51%)		 214
(13.22%)		 83
(5.13%)
bsolo
3.1 cl
(complete)		 1619893
(55.16%)		 223
(13.77%)		 670
(41.38%)		 440
(27.18%)		 286
(17.67%)		 0
(0%)
bsolo
3.1 pb
(complete)		 1619876
(54.11%)		 210
(12.97%)		 666
(41.14%)		 442
(27.3%)		 290
(17.91%)		 11
(0.68%)
pbclasp
2009-04-24
(complete)		 1619979
(60.47%)		 268
(16.55%)		 711
(43.92%)		 441
(27.24%)		 196
(12.11%)		 3
(0.19%)
SAT4J Pseudo CP
2.1.1
(complete)		 2071735
(35.49%)		 201
(9.71%)		 534
(25.78%)		 686
(33.12%)		 650
(31.39%)		 0
(0%)
SAT4J Pseudo Resolution
2.1.1
(complete)		 20711014
(48.96%)		 296
(14.29%)		 718
(34.67%)		 630
(30.42%)		 427
(20.62%)		 0
(0%)
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)		 16191010
(62.38%)		 237
(14.64%)		 773
(47.75%)		 255
(15.75%)		 353
(21.8%)		 1
(0.06%)
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)		 16191016
(62.75%)		 240
(14.82%)		 776
(47.93%)		 248
(15.32%)		 355
(21.93%)		 0
(0%)
wbo
1.0
(complete)		 1619858
(53%)		 249
(15.38%)		 609
(37.62%)		 0
(0%)		 761
(47%)		 0
(0%)

Solver NameProgress
BoolVar
2009-04-26
(complete)
done
660
UNSAT
256		OPT/SAT
239		?
161		ERR
4
bsolo
3.1
(complete)
done
1619
UNSAT
255		OPT/SAT
654		SAT (TO/MO)
413		?
214		ERR
83
bsolo
3.1 cl
(complete)
done
1619
UNSAT
223		OPT/SAT
670		SAT (TO/MO)
440		?
286
bsolo
3.1 pb
(complete)
done
1619
UNSAT
210		OPT/SAT
666		SAT (TO/MO)
442		?
290		ERR
11
pbclasp
2009-04-24
(complete)
done
1619
UNSAT
268		OPT/SAT
711		SAT (TO/MO)
441		?
196		ERR
3
SAT4J Pseudo CP
2.1.1
(complete)
done
2071
UNSAT
201		OPT/SAT
534		SAT (TO/MO)
686		?
650
SAT4J Pseudo Resolution
2.1.1
(complete)
done
2071
UNSAT
296		OPT/SAT
718		SAT (TO/MO)
630		?
427
SCIPclp
SCIP 1.1.0.7 with CLP 1.8.2
(complete)
done
1619
UNSAT
237		OPT/SAT
773		SAT (TO/MO)
255		?
353		ERR
1
SCIPspx
SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)
(complete)
done
1619
UNSAT
240		OPT/SAT
776		SAT (TO/MO)
248		?
355
wbo
1.0
(complete)
done
1619
UNSAT
249		OPT/SAT
609		?
761