PB'16 competition: satisfaction and optimization track: solvers page (tabular format, summary)

Some explanations

Category "no optimisation, big integers, linear constraints" (DEC-BIGINT-LIN)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
minisatp
2012-10-02 git-d91742b
(complete)		 3710
(27.03%)		 1
(2.7%)		 9
(24.32%)		 0
(0%)		 27
(72.97%)		 0
(0%)
NaPS
1.02
(complete)		 372
(5.41%)		 0
(0%)		 2
(5.41%)		 0
(0%)		 35
(94.59%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 3721
(56.76%)		 0
(0%)		 21
(56.76%)		 0
(0%)		 16
(43.24%)		 0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 3722
(59.46%)		 0
(0%)		 22
(59.46%)		 0
(0%)		 15
(40.54%)		 0
(0%)
toysat
2016-05-02
(complete)		 3713
(35.14%)		 0
(0%)		 13
(35.14%)		 0
(0%)		 24
(64.86%)		 0
(0%)

Solver NameProgress
minisatp
2012-10-02 git-d91742b
(complete)
done
37
UNSAT
1		OPT/SAT
9		?
27
NaPS
1.02
(complete)
done
37
OPT/SAT
2		?
35
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
37
OPT/SAT
21		?
16
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
37
OPT/SAT
22		?
15
toysat
2016-05-02
(complete)
done
37
OPT/SAT
13		?
24

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
cdcl-cuttingplanes DEC
2016-05-01
(complete)		 17831395
(78.24%)		 1092
(61.25%)		 303
(16.99%)		 0
(0%)		 388
(21.76%)		 0
(0%)
minisatp
2012-10-02 git-d91742b
(complete)		 17831319
(73.98%)		 935
(52.44%)		 384
(21.54%)		 0
(0%)		 464
(26.02%)		 0
(0%)
NaPS
1.02
(complete)		 17831361
(76.33%)		 1023
(57.38%)		 338
(18.96%)		 0
(0%)		 422
(23.67%)		 0
(0%)
Open-WBO
PB16
(complete)		 17831378
(77.29%)		 1049
(58.83%)		 329
(18.45%)		 0
(0%)		 405
(22.71%)		 0
(0%)
Open-WBO-LSU
PB16
(complete)		 17831378
(77.29%)		 1048
(58.78%)		 330
(18.51%)		 0
(0%)		 405
(22.71%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 17831367
(76.67%)		 1052
(59%)		 315
(17.67%)		 0
(0%)		 416
(23.33%)		 0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 17831240
(69.55%)		 898
(50.36%)		 342
(19.18%)		 0
(0%)		 543
(30.45%)		 0
(0%)
toysat
2016-05-02
(complete)		 17831164
(65.28%)		 841
(47.17%)		 323
(18.12%)		 0
(0%)		 619
(34.72%)		 0
(0%)

Solver NameProgress
cdcl-cuttingplanes DEC
2016-05-01
(complete)
done
1783
UNSAT
1092		OPT/SAT
303		?
388
minisatp
2012-10-02 git-d91742b
(complete)
done
1783
UNSAT
935		OPT/SAT
384		?
464
NaPS
1.02
(complete)
done
1783
UNSAT
1023		OPT/SAT
338		?
422
Open-WBO
PB16
(complete)
done
1783
UNSAT
1049		OPT/SAT
329		?
405
Open-WBO-LSU
PB16
(complete)
done
1783
UNSAT
1048		OPT/SAT
330		?
405
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
1783
UNSAT
1052		OPT/SAT
315		?
416
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
1783
UNSAT
898		OPT/SAT
342		?
543
toysat
2016-05-02
(complete)
done
1783
UNSAT
841		OPT/SAT
323		?
619

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
minisatp
2012-10-02 git-d91742b
(complete)		 10067
(67%)		 15
(15%)		 52
(52%)		 0
(0%)		 33
(33%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 10065
(65%)		 15
(15%)		 50
(50%)		 0
(0%)		 35
(35%)		 0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 10026
(26%)		 15
(15%)		 11
(11%)		 0
(0%)		 74
(74%)		 0
(0%)
toysat
2016-05-02
(complete)		 10065
(65%)		 15
(15%)		 50
(50%)		 0
(0%)		 35
(35%)		 0
(0%)

Solver NameProgress
minisatp
2012-10-02 git-d91742b
(complete)
done
100
UNSAT
15		OPT/SAT
52		?
33
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
100
UNSAT
15		OPT/SAT
50		?
35
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
100
UNSAT
15		OPT/SAT
11		?
74
toysat
2016-05-02
(complete)
done
100
UNSAT
15		OPT/SAT
50		?
35

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
minisatp
2012-10-02 git-d91742b
(complete)		 1109236
(21.28%)		 70
(6.31%)		 166
(14.97%)		 514
(46.35%)		 359
(32.37%)		 0
(0%)
NaPS
1.02
(complete)		 1109374
(33.72%)		 69
(6.22%)		 305
(27.5%)		 361
(32.55%)		 373
(33.63%)		 1
(0.09%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 1109266
(23.99%)		 70
(6.31%)		 196
(17.67%)		 638
(57.53%)		 203
(18.3%)		 2
(0.18%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 1109243
(21.91%)		 68
(6.13%)		 175
(15.78%)		 510
(45.99%)		 231
(20.83%)		 125
(11.27%)
toysat
2016-05-02
(complete)		 110976
(6.85%)		 38
(3.43%)		 38
(3.43%)		 0
(0%)		 1033
(93.15%)		 0
(0%)

Solver NameProgress
minisatp
2012-10-02 git-d91742b
(complete)
done
1109
UNSAT
70		OPT/SAT
166		SAT (TO/MO)
514		?
359
NaPS
1.02
(complete)
done
1109
UNSAT
69		OPT/SAT
305		SAT (TO/MO)
361		?
373		ERR
1
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
1109
UNSAT
70		OPT/SAT
196		SAT (TO/MO)
638		?
203		ERR
2
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
1109
UNSAT
68		OPT/SAT
175		SAT (TO/MO)
510		?
231		ERR
125
toysat
2016-05-02
(complete)
done
1109
UNSAT
38		OPT/SAT
38		?
1033

Category "optimisation, big integers, non linear constraints" (OPT-BIGINT-NLC)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
minisatp
2012-10-02 git-d91742b
(complete)		 413
(7.32%)		 0
(0%)		 3
(7.32%)		 37
(90.24%)		 1
(2.44%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 413
(7.32%)		 0
(0%)		 3
(7.32%)		 38
(92.68%)		 0
(0%)		 0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 413
(7.32%)		 0
(0%)		 3
(7.32%)		 38
(92.68%)		 0
(0%)		 0
(0%)
toysat
2016-05-02
(complete)		 412
(4.88%)		 0
(0%)		 2
(4.88%)		 0
(0%)		 39
(95.12%)		 0
(0%)

Solver NameProgress
minisatp
2012-10-02 git-d91742b
(complete)
done
41
OPT/SAT
3		SAT (TO/MO)
37		?
1
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
41
OPT/SAT
3		SAT (TO/MO)
38
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
41
OPT/SAT
3		SAT (TO/MO)
38
toysat
2016-05-02
(complete)
done
41
OPT/SAT
2		?
39

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
cdcl-cuttingplanes OPT binary search
2016-05-01
(complete)		 1600774
(48.38%)		 89
(5.56%)		 685
(42.81%)		 0
(0%)		 826
(51.63%)		 0
(0%)
cdcl-cuttingplanes OPT linear search
2016-05-01
(complete)		 1600749
(46.81%)		 89
(5.56%)		 660
(41.25%)		 0
(0%)		 851
(53.19%)		 0
(0%)
minisatp
2012-10-02 git-d91742b
(complete)		 1600789
(49.31%)		 76
(4.75%)		 713
(44.56%)		 552
(34.5%)		 259
(16.19%)		 0
(0%)
NaPS
1.02
(complete)		 1600887
(55.44%)		 85
(5.31%)		 802
(50.13%)		 456
(28.5%)		 257
(16.06%)		 0
(0%)
Open-WBO
PB16
(complete)		 1600826
(51.63%)		 45
(2.81%)		 781
(48.81%)		 435
(27.19%)		 339
(21.19%)		 0
(0%)
Open-WBO-LSU
PB16
(complete)		 1600812
(50.75%)		 45
(2.81%)		 767
(47.94%)		 326
(20.38%)		 462
(28.88%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 1600761
(47.56%)		 89
(5.56%)		 672
(42%)		 656
(41%)		 183
(11.44%)		 0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 1600716
(44.75%)		 67
(4.19%)		 649
(40.56%)		 567
(35.44%)		 245
(15.31%)		 72
(4.5%)
toysat
2016-05-02
(complete)		 1600733
(45.81%)		 63
(3.94%)		 670
(41.88%)		 0
(0%)		 867
(54.19%)		 0
(0%)

Solver NameProgress
cdcl-cuttingplanes OPT binary search
2016-05-01
(complete)
done
1600
UNSAT
89		OPT/SAT
685		?
826
cdcl-cuttingplanes OPT linear search
2016-05-01
(complete)
done
1600
UNSAT
89		OPT/SAT
660		?
851
minisatp
2012-10-02 git-d91742b
(complete)
done
1600
UNSAT
76		OPT/SAT
713		SAT (TO/MO)
552		?
259
NaPS
1.02
(complete)
done
1600
UNSAT
85		OPT/SAT
802		SAT (TO/MO)
456		?
257
Open-WBO
PB16
(complete)
done
1600
UNSAT
45		OPT/SAT
781		SAT (TO/MO)
435		?
339
Open-WBO-LSU
PB16
(complete)
done
1600
UNSAT
45		OPT/SAT
767		SAT (TO/MO)
326		?
462
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
1600
UNSAT
89		OPT/SAT
672		SAT (TO/MO)
656		?
183
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
1600
UNSAT
67		OPT/SAT
649		SAT (TO/MO)
567		?
245		ERR
72
toysat
2016-05-02
(complete)
done
1600
UNSAT
63		OPT/SAT
670		?
867

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC)

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
minisatp
2012-10-02 git-d91742b
(complete)		 506306
(60.47%)		 0
(0%)		 306
(60.47%)		 200
(39.53%)		 0
(0%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 506285
(56.32%)		 0
(0%)		 285
(56.32%)		 221
(43.68%)		 0
(0%)		 0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 506288
(56.92%)		 0
(0%)		 288
(56.92%)		 211
(41.7%)		 7
(1.38%)		 0
(0%)
toysat
2016-05-02
(complete)		 506207
(40.91%)		 0
(0%)		 207
(40.91%)		 0
(0%)		 299
(59.09%)		 0
(0%)

Solver NameProgress
minisatp
2012-10-02 git-d91742b
(complete)
done
506
OPT/SAT
306		SAT (TO/MO)
200
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
506
OPT/SAT
285		SAT (TO/MO)
221
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
506
OPT/SAT
288		SAT (TO/MO)
211		?
7
toysat
2016-05-02
(complete)
done
506
OPT/SAT
207		?
299

All categories for solvers registered in all categories

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
minisatp
2012-10-02 git-d91742b
(complete)		 51762730
(52.74%)		 1097
(21.19%)		 1633
(31.55%)		 1303
(25.17%)		 1143
(22.08%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 51762768
(53.48%)		 1226
(23.69%)		 1542
(29.79%)		 1553
(30%)		 853
(16.48%)		 2
(0.04%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 51762538
(49.03%)		 1048
(20.25%)		 1490
(28.79%)		 1326
(25.62%)		 1115
(21.54%)		 197
(3.81%)

Solver NameProgress
minisatp
2012-10-02 git-d91742b
(complete)
done
5176
UNSAT
1097		OPT/SAT
1633		SAT (TO/MO)
1303		?
1143
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
5176
UNSAT
1226		OPT/SAT
1542		SAT (TO/MO)
1553		?
853		ERR
2
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
5176
UNSAT
1048		OPT/SAT
1490		SAT (TO/MO)
1326		?
1115		ERR
197

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name#benchsSolvedUNSATOPT/SATSAT (TO/MO)UNKNOWNBad Answers
cdcl-cuttingplanes DEC
2016-05-01
(complete)		 17831395
(78.24%)		 1092
(61.25%)		 303
(16.99%)		 0
(0%)		 388
(21.76%)		 0
(0%)
cdcl-cuttingplanes OPT binary search
2016-05-01
(complete)		 1600774
(48.38%)		 89
(5.56%)		 685
(42.81%)		 0
(0%)		 826
(51.63%)		 0
(0%)
cdcl-cuttingplanes OPT linear search
2016-05-01
(complete)		 1600749
(46.81%)		 89
(5.56%)		 660
(41.25%)		 0
(0%)		 851
(53.19%)		 0
(0%)
minisatp
2012-10-02 git-d91742b
(complete)		 51762730
(52.74%)		 1097
(21.19%)		 1633
(31.55%)		 1303
(25.17%)		 1143
(22.08%)		 0
(0%)
NaPS
1.02
(complete)		 45292624
(57.94%)		 1177
(25.99%)		 1447
(31.95%)		 817
(18.04%)		 1087
(24%)		 1
(0.02%)
Open-WBO
PB16
(complete)		 33832204
(65.15%)		 1094
(32.34%)		 1110
(32.81%)		 435
(12.86%)		 744
(21.99%)		 0
(0%)
Open-WBO-LSU
PB16
(complete)		 33832190
(64.74%)		 1093
(32.31%)		 1097
(32.43%)		 326
(9.64%)		 867
(25.63%)		 0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)		 51762768
(53.48%)		 1226
(23.69%)		 1542
(29.79%)		 1553
(30%)		 853
(16.48%)		 2
(0.04%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)		 51762538
(49.03%)		 1048
(20.25%)		 1490
(28.79%)		 1326
(25.62%)		 1115
(21.54%)		 197
(3.81%)
toysat
2016-05-02
(complete)		 51762260
(43.66%)		 957
(18.49%)		 1303
(25.17%)		 0
(0%)		 2916
(56.34%)		 0
(0%)

Solver NameProgress
cdcl-cuttingplanes DEC
2016-05-01
(complete)
done
1783
UNSAT
1092		OPT/SAT
303		?
388
cdcl-cuttingplanes OPT binary search
2016-05-01
(complete)
done
1600
UNSAT
89		OPT/SAT
685		?
826
cdcl-cuttingplanes OPT linear search
2016-05-01
(complete)
done
1600
UNSAT
89		OPT/SAT
660		?
851
minisatp
2012-10-02 git-d91742b
(complete)
done
5176
UNSAT
1097		OPT/SAT
1633		SAT (TO/MO)
1303		?
1143
NaPS
1.02
(complete)
done
4529
UNSAT
1177		OPT/SAT
1447		SAT (TO/MO)
817		?
1087		ERR
1
Open-WBO
PB16
(complete)
done
3383
UNSAT
1094		OPT/SAT
1110		SAT (TO/MO)
435		?
744
Open-WBO-LSU
PB16
(complete)
done
3383
UNSAT
1093		OPT/SAT
1097		SAT (TO/MO)
326		?
867
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
5176
UNSAT
1226		OPT/SAT
1542		SAT (TO/MO)
1553		?
853		ERR
2
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
5176
UNSAT
1048		OPT/SAT
1490		SAT (TO/MO)
1326		?
1115		ERR
197
toysat
2016-05-02
(complete)
done
5176
UNSAT
957		OPT/SAT
1303		?
2916