PB'16 competition: WBO track: solvers page (tabular format, summary)

Some explanations

#Benchs: Number of benchs for this solver
Solved: Total number of solved instances (this is the sum of the SAT and UNSAT columns)
UNSAT: solver proved unsatitisfiability ("s UNSATISFIABLE" was output)
MOPT: Total number of OPT answers
SAT: Total number of SAT answers
MSAT: Total number of MSAT answers
UNKNOWN: Total number of UNKNOWN answers
Incomplete Answers: Total number of incomplete answers
Bad Answers: Total number of bad answers

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN)

Solver Name	#benchs	Solved	UNSAT	MOPT	SAT	MSAT	UNKNOWN	Incomplete Answers	Bad Answers
NaPS 1.02 (complete)	263	114 (43.35%)	0 (0%)	114 (43.35%)	0 (0%)	97 (36.88%)	52 (19.77%)	0 (0%)	0 (0%)
Sat4j PB 2.3.6 Res+CP PB16 (complete)	263	121 (46.01%)	0 (0%)	121 (46.01%)	0 (0%)	122 (46.39%)	3 (1.14%)	2 (0.76%)	15 (5.7%)
Sat4j PB 2.3.6 Resolution PB16 (complete)	263	134 (50.95%)	0 (0%)	134 (50.95%)	0 (0%)	112 (42.59%)	0 (0%)	0 (0%)	17 (6.46%)
toysat 2016-05-02 (complete)	263	122 (46.39%)	0 (0%)	122 (46.39%)	0 (0%)	0 (0%)	141 (53.61%)	0 (0%)	0 (0%)

Solver Name

Progress

NaPS
1.02
(complete)

done
263

MOPT
114

MSAT
97

?
52

Sat4j PB 2.3.6 Res+CP
PB16
(complete)

done
263

MOPT
121

MSAT
122

?
3

Inc. Answer
2

ERR
15

Sat4j PB 2.3.6 Resolution
PB16
(complete)

done
263

MOPT
134

MSAT
112

ERR
17

toysat
2016-05-02
(complete)

done
263

MOPT
122

?
141

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN)

Solver Name	#benchs	Solved	UNSAT	MOPT	SAT	MSAT	UNKNOWN	Incomplete Answers	Bad Answers
NaPS 1.02 (complete)	549	470 (85.61%)	1 (0.18%)	469 (85.43%)	0 (0%)	79 (14.39%)	0 (0%)	0 (0%)	0 (0%)
Sat4j PB 2.3.6 Res+CP PB16 (complete)	549	456 (83.06%)	0 (0%)	456 (83.06%)	0 (0%)	91 (16.58%)	0 (0%)	0 (0%)	2 (0.36%)
Sat4j PB 2.3.6 Resolution PB16 (complete)	549	458 (83.42%)	0 (0%)	458 (83.42%)	0 (0%)	87 (15.85%)	0 (0%)	0 (0%)	4 (0.73%)
toysat 2016-05-02 (complete)	549	454 (82.7%)	1 (0.18%)	453 (82.51%)	0 (0%)	0 (0%)	95 (17.3%)	0 (0%)	0 (0%)

Solver Name

Progress

NaPS
1.02
(complete)

done
549

UNSAT
1

MOPT
469

MSAT
79

Sat4j PB 2.3.6 Res+CP
PB16
(complete)

done
549

MOPT
456

MSAT
91

ERR
2

Sat4j PB 2.3.6 Resolution
PB16
(complete)

done
549

MOPT
458

MSAT
87

ERR
4

toysat
2016-05-02
(complete)

done
549

UNSAT
1

MOPT
453

?
95

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN)

Solver Name	#benchs	Solved	UNSAT	MOPT	SAT	MSAT	UNKNOWN	Incomplete Answers	Bad Answers
NaPS 1.02 (complete)	46	6 (13.04%)	0 (0%)	6 (13.04%)	0 (0%)	37 (80.43%)	3 (6.52%)	0 (0%)	0 (0%)
Sat4j PB 2.3.6 Res+CP PB16 (complete)	46	33 (71.74%)	0 (0%)	33 (71.74%)	0 (0%)	13 (28.26%)	0 (0%)	0 (0%)	0 (0%)
Sat4j PB 2.3.6 Resolution PB16 (complete)	46	41 (89.13%)	0 (0%)	41 (89.13%)	0 (0%)	5 (10.87%)	0 (0%)	0 (0%)	0 (0%)
toysat 2016-05-02 (complete)	46	34 (73.91%)	0 (0%)	34 (73.91%)	0 (0%)	0 (0%)	12 (26.09%)	0 (0%)	0 (0%)

Solver Name

Progress

NaPS
1.02
(complete)

done
46

MOPT
6

MSAT
37

?
3

Sat4j PB 2.3.6 Res+CP
PB16
(complete)

done
46

MOPT
33

MSAT
13

Sat4j PB 2.3.6 Resolution
PB16
(complete)

done
46

MOPT
41

MSAT
5

toysat
2016-05-02
(complete)

done
46

MOPT
34

?
12

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN)

Solver Name	#benchs	Solved	UNSAT	MOPT	SAT	MSAT	UNKNOWN	Incomplete Answers	Bad Answers
NaPS 1.02 (complete)	201	164 (81.59%)	0 (0%)	164 (81.59%)	0 (0%)	37 (18.41%)	0 (0%)	0 (0%)	0 (0%)
Sat4j PB 2.3.6 Res+CP PB16 (complete)	201	127 (63.18%)	0 (0%)	127 (63.18%)	0 (0%)	74 (36.82%)	0 (0%)	0 (0%)	0 (0%)
Sat4j PB 2.3.6 Resolution PB16 (complete)	201	163 (81.09%)	0 (0%)	163 (81.09%)	0 (0%)	38 (18.91%)	0 (0%)	0 (0%)	0 (0%)
toysat 2016-05-02 (complete)	201	160 (79.6%)	0 (0%)	160 (79.6%)	0 (0%)	0 (0%)	41 (20.4%)	0 (0%)	0 (0%)

Solver Name

Progress

NaPS
1.02
(complete)

done
201

MOPT
164

MSAT
37

Sat4j PB 2.3.6 Res+CP
PB16
(complete)

done
201

MOPT
127

MSAT
74

Sat4j PB 2.3.6 Resolution
PB16
(complete)

done
201

MOPT
163

MSAT
38

toysat
2016-05-02
(complete)

done
201

MOPT
160

?
41

All categories for solvers registered in all categories

Solver Name	#benchs	Solved	UNSAT	MOPT	SAT	MSAT	UNKNOWN	Incomplete Answers	Bad Answers
Sat4j PB 2.3.6 Res+CP PB16 (complete)	1059	737 (69.59%)	0 (0%)	737 (69.59%)	0 (0%)	300 (28.33%)	3 (0.28%)	2 (0.19%)	17 (1.61%)
Sat4j PB 2.3.6 Resolution PB16 (complete)	1059	796 (75.17%)	0 (0%)	796 (75.17%)	0 (0%)	242 (22.85%)	0 (0%)	0 (0%)	21 (1.98%)
toysat 2016-05-02 (complete)	1059	770 (72.71%)	1 (0.09%)	769 (72.62%)	0 (0%)	0 (0%)	289 (27.29%)	0 (0%)	0 (0%)

Solver Name

Progress

Sat4j PB 2.3.6 Res+CP
PB16
(complete)

done
1059

MOPT
737

MSAT
300

?
3

Inc. Answer
2

ERR
17

Sat4j PB 2.3.6 Resolution
PB16
(complete)

done
1059

MOPT
796

MSAT
242

ERR
21

toysat
2016-05-02
(complete)

done
1059

UNSAT
1

MOPT
769

?
289

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name	#benchs	Solved	UNSAT	MOPT	SAT	MSAT	UNKNOWN	Incomplete Answers	Bad Answers
NaPS 1.02 (complete)	1059	754 (71.2%)	1 (0.09%)	753 (71.1%)	0 (0%)	250 (23.61%)	55 (5.19%)	0 (0%)	0 (0%)
Sat4j PB 2.3.6 Res+CP PB16 (complete)	1059	737 (69.59%)	0 (0%)	737 (69.59%)	0 (0%)	300 (28.33%)	3 (0.28%)	2 (0.19%)	17 (1.61%)
Sat4j PB 2.3.6 Resolution PB16 (complete)	1059	796 (75.17%)	0 (0%)	796 (75.17%)	0 (0%)	242 (22.85%)	0 (0%)	0 (0%)	21 (1.98%)
toysat 2016-05-02 (complete)	1059	770 (72.71%)	1 (0.09%)	769 (72.62%)	0 (0%)	0 (0%)	289 (27.29%)	0 (0%)	0 (0%)

Solver Name

Progress

NaPS
1.02
(complete)

done
1059

UNSAT
1

MOPT
753

MSAT
250

?
55

Sat4j PB 2.3.6 Res+CP
PB16
(complete)

done
1059

MOPT
737

MSAT
300

?
3

Inc. Answer
2

ERR
17

Sat4j PB 2.3.6 Resolution
PB16
(complete)

done
1059

MOPT
796

MSAT
242

ERR
21

toysat
2016-05-02
(complete)

done
1059

UNSAT
1

MOPT
769

?
289