PB'06 evaluation: solvers page (tabular format, summary)

Category "optimisation, big integers" (OPT-BIGINT)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
bsolo
2006/05		 35544
(12.39%)		 30
(8.45%)		 14
(3.94%)		 69
(19.44%)		 242
(68.17%)		 0
(0%)		 0
minisat+
1.14		 35571
(20%)		 38
(10.7%)		 33
(9.3%)		 52
(14.65%)		 232
(65.35%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 35546
(12.96%)		 39
(10.99%)		 7
(1.97%)		 77
(21.69%)		 231
(65.07%)		 1
(0.28%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 35594
(26.48%)		 37
(10.42%)		 57
(16.06%)		 77
(21.69%)		 184
(51.83%)		 0
(0%)		 0

Solver NameProgress
bsolo
2006/05
done
355
UNSAT
30		OPT/SAT
14		SAT (TO/MO)
69		?
242
minisat+
1.14
done
355
UNSAT
38		OPT/SAT
33		SAT (TO/MO)
52		?
232
SAT4JPSEUDO
2006.2
done
355
UNSAT
39		OPT/SAT
7		SAT (TO/MO)
77		?
231		ERR
1
SAT4JPSEUDO
2006.2 Heuristics
done
355
UNSAT
37		OPT/SAT
57		SAT (TO/MO)
77		?
184

Category "optimisation, medium integers" (OPT-MEDINT)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
absconPseudo
1		 255
(20%)		 0
(0%)		 5
(20%)		 2
(8%)		 18
(72%)		 0
(0%)		 0
bsolo
2006/05		 256
(24%)		 0
(0%)		 6
(24%)		 7
(28%)		 12
(48%)		 0
(0%)		 0
glpPB
0.2		 259
(36%)		 0
(0%)		 9
(36%)		 0
(0%)		 16
(64%)		 0
(0%)		 0
minisat+
1.14		 253
(12%)		 0
(0%)		 3
(12%)		 9
(36%)		 13
(52%)		 0
(0%)		 0
PBS
4.1L		 255
(20%)		 0
(0%)		 5
(20%)		 0
(0%)		 20
(80%)		 0
(0%)		 0
Pueblo
1.3		 254
(16%)		 0
(0%)		 4
(16%)		 14
(56%)		 7
(28%)		 0
(0%)		 0
Pueblo
1.4		 254
(16%)		 0
(0%)		 4
(16%)		 15
(60%)		 6
(24%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 253
(12%)		 0
(0%)		 3
(12%)		 8
(32%)		 14
(56%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 254
(16%)		 0
(0%)		 4
(16%)		 9
(36%)		 12
(48%)		 0
(0%)		 0

Solver NameProgress
absconPseudo
1
done
25
OPT/SAT
5		SAT (TO/MO)
2		?
18
bsolo
2006/05
done
25
OPT/SAT
6		SAT (TO/MO)
7		?
12
glpPB
0.2
done
25
OPT/SAT
9		?
16
minisat+
1.14
done
25
OPT/SAT
3		SAT (TO/MO)
9		?
13
PBS
4.1L
done
25
OPT/SAT
5		?
20
Pueblo
1.3
done
25
OPT/SAT
4		SAT (TO/MO)
14		?
7
Pueblo
1.4
done
25
OPT/SAT
4		SAT (TO/MO)
15		?
6
SAT4JPSEUDO
2006.2
done
25
OPT/SAT
3		SAT (TO/MO)
8		?
14
SAT4JPSEUDO
2006.2 Heuristics
done
25
OPT/SAT
4		SAT (TO/MO)
9		?
12

Category "optimisation, small integers" (OPT-SMALLINT)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
absconPseudo
1		 954321
(33.65%)		 14
(1.47%)		 307
(32.18%)		 204
(21.38%)		 429
(44.97%)		 0
(0%)		 0
bsolo
2006/05		 954449
(47.06%)		 40
(4.19%)		 409
(42.87%)		 280
(29.35%)		 225
(23.58%)		 0
(0%)		 0
glpPB
0.2		 954424
(44.44%)		 46
(4.82%)		 378
(39.62%)		 0
(0%)		 530
(55.56%)		 0
(0%)		 0
minisat+
1.14		 954448
(46.96%)		 43
(4.51%)		 405
(42.45%)		 250
(26.21%)		 256
(26.83%)		 0
(0%)		 0
PBS
4.1L		 954381
(39.94%)		 29
(3.04%)		 352
(36.9%)		 0
(0%)		 573
(60.06%)		 0
(0%)		 0
PB-smodels
1.28		 954379
(39.73%)		 34
(3.56%)		 345
(36.16%)		 240
(25.16%)		 335
(35.12%)		 0
(0%)		 0
PB-smodels
1.31		 954381
(39.94%)		 34
(3.56%)		 347
(36.37%)		 266
(27.88%)		 307
(32.18%)		 0
(0%)		 0
Pueblo
1.3		 954405
(42.45%)		 37
(3.88%)		 368
(38.57%)		 341
(35.74%)		 208
(21.8%)		 0
(0%)		 0
Pueblo
1.4		 954422
(44.23%)		 37
(3.88%)		 385
(40.36%)		 323
(33.86%)		 209
(21.91%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 954409
(42.87%)		 53
(5.56%)		 356
(37.32%)		 319
(33.44%)		 223
(23.38%)		 3
(0.31%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 954411
(43.08%)		 54
(5.66%)		 357
(37.42%)		 300
(31.45%)		 240
(25.16%)		 3
(0.31%)		 0
wildcat-rnp
0.8.0		 9541
(0.1%)		 0
(0%)		 1
(0.1%)		 482
(50.52%)		 471
(49.37%)		 0
(0%)		 0
wildcat-skc
0.8.0		 9542
(0.21%)		 0
(0%)		 2
(0.21%)		 499
(52.31%)		 453
(47.48%)		 0
(0%)		 0

Solver NameProgress
absconPseudo
1
done
954
UNSAT
14		OPT/SAT
307		SAT (TO/MO)
204		?
429
bsolo
2006/05
done
954
UNSAT
40		OPT/SAT
409		SAT (TO/MO)
280		?
225
glpPB
0.2
done
954
UNSAT
46		OPT/SAT
378		?
530
minisat+
1.14
done
954
UNSAT
43		OPT/SAT
405		SAT (TO/MO)
250		?
256
PBS
4.1L
done
954
UNSAT
29		OPT/SAT
352		?
573
PB-smodels
1.28
done
954
UNSAT
34		OPT/SAT
345		SAT (TO/MO)
240		?
335
PB-smodels
1.31
done
954
UNSAT
34		OPT/SAT
347		SAT (TO/MO)
266		?
307
Pueblo
1.3
done
954
UNSAT
37		OPT/SAT
368		SAT (TO/MO)
341		?
208
Pueblo
1.4
done
954
UNSAT
37		OPT/SAT
385		SAT (TO/MO)
323		?
209
SAT4JPSEUDO
2006.2
done
954
UNSAT
53		OPT/SAT
356		SAT (TO/MO)
319		?
223		ERR
3
SAT4JPSEUDO
2006.2 Heuristics
done
954
UNSAT
54		OPT/SAT
357		SAT (TO/MO)
300		?
240		ERR
3
wildcat-rnp
0.8.0
done
954
OPT/SAT
1		SAT (TO/MO)
482		?
471
wildcat-skc
0.8.0
done
954
OPT/SAT
2		SAT (TO/MO)
499		?
453

Category "no optimisation, big integers" (SATUNSAT-BIGINT)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
bsolo
2006/05		 21
(50%)		 0
(0%)		 1
(50%)		 0
(0%)		 1
(50%)		 0
(0%)		 0
minisat+
1.14		 21
(50%)		 0
(0%)		 1
(50%)		 0
(0%)		 1
(50%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 20
(0%)		 0
(0%)		 0
(0%)		 0
(0%)		 2
(100%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 20
(0%)		 0
(0%)		 0
(0%)		 0
(0%)		 2
(100%)		 0
(0%)		 0

Solver NameProgress
bsolo
2006/05
done
2
OPT/SAT
1		?
1
minisat+
1.14
done
2
OPT/SAT
1		?
1
SAT4JPSEUDO
2006.2
done
2
?
2
SAT4JPSEUDO
2006.2 Heuristics
done
2
?
2

Category "no optimisation, medium integers" (SATUNSAT-MEDINT)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
absconPseudo
1		 120
(0%)		 0
(0%)		 0
(0%)		 0
(0%)		 12
(100%)		 0
(0%)		 0
bsolo
2006/05		 122
(16.67%)		 0
(0%)		 2
(16.67%)		 0
(0%)		 10
(83.33%)		 0
(0%)		 0
glpPB
0.2		 120
(0%)		 0
(0%)		 0
(0%)		 0
(0%)		 12
(100%)		 0
(0%)		 0
minisat+
1.14		 1210
(83.33%)		 0
(0%)		 10
(83.33%)		 0
(0%)		 2
(16.67%)		 0
(0%)		 0
PBS
4.1L		 123
(25%)		 0
(0%)		 3
(25%)		 0
(0%)		 9
(75%)		 0
(0%)		 0
Pueblo
1.3		 120
(0%)		 0
(0%)		 0
(0%)		 0
(0%)		 12
(100%)		 0
(0%)		 0
Pueblo
1.4		 120
(0%)		 0
(0%)		 0
(0%)		 0
(0%)		 12
(100%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 122
(16.67%)		 0
(0%)		 2
(16.67%)		 0
(0%)		 10
(83.33%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 122
(16.67%)		 0
(0%)		 2
(16.67%)		 0
(0%)		 10
(83.33%)		 0
(0%)		 0

Solver NameProgress
absconPseudo
1
done
12
?
12
bsolo
2006/05
done
12
OPT/SAT
2		?
10
glpPB
0.2
done
12
?
12
minisat+
1.14
done
12
OPT/SAT
10		?
2
PBS
4.1L
done
12
OPT/SAT
3		?
9
Pueblo
1.3
done
12
?
12
Pueblo
1.4
done
12
?
12
SAT4JPSEUDO
2006.2
done
12
OPT/SAT
2		?
10
SAT4JPSEUDO
2006.2 Heuristics
done
12
OPT/SAT
2		?
10

Category "no optimisation, small integers" (SATUNSAT-SMALLINT)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
absconPseudo
1		 405176
(43.46%)		 80
(19.75%)		 96
(23.7%)		 0
(0%)		 229
(56.54%)		 0
(0%)		 0
bsolo
2006/05		 405229
(56.54%)		 111
(27.41%)		 118
(29.14%)		 0
(0%)		 176
(43.46%)		 0
(0%)		 0
glpPB
0.2		 405213
(52.59%)		 121
(29.88%)		 92
(22.72%)		 0
(0%)		 192
(47.41%)		 0
(0%)		 0
minisat+
1.14		 405320
(79.01%)		 172
(42.47%)		 148
(36.54%)		 0
(0%)		 85
(20.99%)		 0
(0%)		 0
PBS
4.1L		 405343
(84.69%)		 199
(49.14%)		 144
(35.56%)		 0
(0%)		 62
(15.31%)		 0
(0%)		 0
PB-smodels
1.28		 405270
(66.67%)		 147
(36.3%)		 123
(30.37%)		 0
(0%)		 135
(33.33%)		 0
(0%)		 0
PB-smodels
1.31		 405285
(70.37%)		 148
(36.54%)		 137
(33.83%)		 0
(0%)		 120
(29.63%)		 0
(0%)		 0
Pueblo
1.3		 405315
(77.78%)		 173
(42.72%)		 142
(35.06%)		 0
(0%)		 90
(22.22%)		 0
(0%)		 0
Pueblo
1.4		 405357
(88.15%)		 204
(50.37%)		 153
(37.78%)		 0
(0%)		 48
(11.85%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 405171
(42.22%)		 79
(19.51%)		 92
(22.72%)		 0
(0%)		 234
(57.78%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 405171
(42.22%)		 79
(19.51%)		 92
(22.72%)		 0
(0%)		 234
(57.78%)		 0
(0%)		 0
wildcat-rnp
0.8.0		 405156
(38.52%)		 0
(0%)		 156
(38.52%)		 0
(0%)		 249
(61.48%)		 0
(0%)		 0
wildcat-skc
0.8.0		 405163
(40.25%)		 0
(0%)		 163
(40.25%)		 0
(0%)		 242
(59.75%)		 0
(0%)		 0

Solver NameProgress
absconPseudo
1
done
405
UNSAT
80		OPT/SAT
96		?
229
bsolo
2006/05
done
405
UNSAT
111		OPT/SAT
118		?
176
glpPB
0.2
done
405
UNSAT
121		OPT/SAT
92		?
192
minisat+
1.14
done
405
UNSAT
172		OPT/SAT
148		?
85
PBS
4.1L
done
405
UNSAT
199		OPT/SAT
144		?
62
PB-smodels
1.28
done
405
UNSAT
147		OPT/SAT
123		?
135
PB-smodels
1.31
done
405
UNSAT
148		OPT/SAT
137		?
120
Pueblo
1.3
done
405
UNSAT
173		OPT/SAT
142		?
90
Pueblo
1.4
done
405
UNSAT
204		OPT/SAT
153		?
48
SAT4JPSEUDO
2006.2
done
405
UNSAT
79		OPT/SAT
92		?
234
SAT4JPSEUDO
2006.2 Heuristics
done
405
UNSAT
79		OPT/SAT
92		?
234
wildcat-rnp
0.8.0
done
405
OPT/SAT
156		?
249
wildcat-skc
0.8.0
done
405
OPT/SAT
163		?
242

All categories for solvers registered in all categories

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
bsolo
2006/05		 1753731
(41.7%)		 181
(10.33%)		 550
(31.37%)		 356
(20.31%)		 666
(37.99%)		 0
(0%)		 0
minisat+
1.14		 1753853
(48.66%)		 253
(14.43%)		 600
(34.23%)		 311
(17.74%)		 589
(33.6%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 1753631
(36%)		 171
(9.75%)		 460
(26.24%)		 404
(23.05%)		 714
(40.73%)		 4
(0.23%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 1753682
(38.9%)		 170
(9.7%)		 512
(29.21%)		 386
(22.02%)		 682
(38.9%)		 3
(0.17%)		 0

Solver NameProgress
bsolo
2006/05
done
1753
UNSAT
181		OPT/SAT
550		SAT (TO/MO)
356		?
666
minisat+
1.14
done
1753
UNSAT
253		OPT/SAT
600		SAT (TO/MO)
311		?
589
SAT4JPSEUDO
2006.2
done
1753
UNSAT
171		OPT/SAT
460		SAT (TO/MO)
404		?
714		ERR
4
SAT4JPSEUDO
2006.2 Heuristics
done
1753
UNSAT
170		OPT/SAT
512		SAT (TO/MO)
386		?
682		ERR
3

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad AnswersMisc.
absconPseudo
1		 1396502
(35.96%)		 94
(6.73%)		 408
(29.23%)		 206
(14.76%)		 688
(49.28%)		 0
(0%)		 0
bsolo
2006/05		 1753731
(41.7%)		 181
(10.33%)		 550
(31.37%)		 356
(20.31%)		 666
(37.99%)		 0
(0%)		 0
glpPB
0.2		 1396646
(46.28%)		 167
(11.96%)		 479
(34.31%)		 0
(0%)		 750
(53.72%)		 0
(0%)		 0
minisat+
1.14		 1753853
(48.66%)		 253
(14.43%)		 600
(34.23%)		 311
(17.74%)		 589
(33.6%)		 0
(0%)		 0
PBS
4.1L		 1396732
(52.44%)		 228
(16.33%)		 504
(36.1%)		 0
(0%)		 664
(47.56%)		 0
(0%)		 0
PB-smodels
1.28		 1359649
(47.76%)		 181
(13.32%)		 468
(34.44%)		 240
(17.66%)		 470
(34.58%)		 0
(0%)		 0
PB-smodels
1.31		 1359666
(49.01%)		 182
(13.39%)		 484
(35.61%)		 266
(19.57%)		 427
(31.42%)		 0
(0%)		 0
Pueblo
1.3		 1396724
(51.86%)		 210
(15.04%)		 514
(36.82%)		 355
(25.43%)		 317
(22.71%)		 0
(0%)		 0
Pueblo
1.4		 1396783
(56.09%)		 241
(17.26%)		 542
(38.83%)		 338
(24.21%)		 275
(19.7%)		 0
(0%)		 0
SAT4JPSEUDO
2006.2		 1753631
(36%)		 171
(9.75%)		 460
(26.24%)		 404
(23.05%)		 714
(40.73%)		 4
(0.23%)		 0
SAT4JPSEUDO
2006.2 Heuristics		 1753682
(38.9%)		 170
(9.7%)		 512
(29.21%)		 386
(22.02%)		 682
(38.9%)		 3
(0.17%)		 0
wildcat-rnp
0.8.0		 1359157
(11.55%)		 0
(0%)		 157
(11.55%)		 482
(35.47%)		 720
(52.98%)		 0
(0%)		 0
wildcat-skc
0.8.0		 1359165
(12.14%)		 0
(0%)		 165
(12.14%)		 499
(36.72%)		 695
(51.14%)		 0
(0%)		 0

Solver NameProgress
absconPseudo
1
done
1396
UNSAT
94		OPT/SAT
408		SAT (TO/MO)
206		?
688
bsolo
2006/05
done
1753
UNSAT
181		OPT/SAT
550		SAT (TO/MO)
356		?
666
glpPB
0.2
done
1396
UNSAT
167		OPT/SAT
479		?
750
minisat+
1.14
done
1753
UNSAT
253		OPT/SAT
600		SAT (TO/MO)
311		?
589
PBS
4.1L
done
1396
UNSAT
228		OPT/SAT
504		?
664
PB-smodels
1.28
done
1359
UNSAT
181		OPT/SAT
468		SAT (TO/MO)
240		?
470
PB-smodels
1.31
done
1359
UNSAT
182		OPT/SAT
484		SAT (TO/MO)
266		?
427
Pueblo
1.3
done
1396
UNSAT
210		OPT/SAT
514		SAT (TO/MO)
355		?
317
Pueblo
1.4
done
1396
UNSAT
241		OPT/SAT
542		SAT (TO/MO)
338		?
275
SAT4JPSEUDO
2006.2
done
1753
UNSAT
171		OPT/SAT
460		SAT (TO/MO)
404		?
714		ERR
4
SAT4JPSEUDO
2006.2 Heuristics
done
1753
UNSAT
170		OPT/SAT
512		SAT (TO/MO)
386		?
682		ERR
3
wildcat-rnp
0.8.0
done
1359
OPT/SAT
157		SAT (TO/MO)
482		?
720
wildcat-skc
0.8.0
done
1359
OPT/SAT
165		SAT (TO/MO)
499		?
695

Some explanations