PB'25 competition WBO: solvers page (tabular format, summary)

Some explanations

#Benchs: Number of benchs for this solver
Solved: Total number of solved instances (this is the sum of the SAT and UNSAT columns)
UNSAT: solver proved unsatitisfiability ("s UNSATISFIABLE" was output)
MOPT: Total number of OPT answers
SAT: Total number of SAT answers
MSAT: Total number of MSAT answers
UNKNOWN: Total number of UNKNOWN answers
Incomplete Answers: Total number of incomplete answers
Bad Answers: Total number of bad answers

Status of a solver

solver a "light red" background denotes a solver which was disqualified because it gave an incorrect answer, and which will not appear in the ranking
solver a "light yellow" background denotes a solver which was replaced by another version, and which will not appear in the ranking

Status of a job

done: the job is done
running: the solver is running
analyzing: the solver answer is being analyzed
fail: the job has failed for some reason (probably an unexpected change of checksum)
abort: the job was canceled
postanalysis: the solver answer is scheduled for analysis
unscheck: an unsat proof is being checked
postverif: the checking of an unsat proof is in the queue, waiting for execution
queued: the job is in the queue, waiting for execution
new: the job is not yet scheduled for execution

Category "WBO, both soft and hard constraints, linear constraints" (PARTIAL-LIN)

Solver Name	#benchs	Solved	UNSAT	MOPT	MSAT	UNKNOWN	Bad Answers
Exact 2025-06-02 (complete)	208	158 (75.96%)	1 (0.48%)	157 (75.48%)	43 (20.67%)	7 (3.37%)	0 (0%)
Exact_no_dombreaking 2025-06-16 (complete)	208	157 (75.48%)	1 (0.48%)	156 (75%)	42 (20.19%)	9 (4.33%)	0 (0%)
NaPS naps-1.03a6 (complete)	208	146 (70.19%)	1 (0.48%)	145 (69.71%)	47 (22.6%)	15 (7.21%)	0 (0%)
OR-Tools CP-SAT 9.14 (complete)	208	172 (82.69%)	0 (0%)	172 (82.69%)	27 (12.98%)	1 (0.48%)	8 (3.85%)
OR-Tools CP-SAT 9.14.2 (complete)	208	172 (82.69%)	0 (0%)	172 (82.69%)	28 (13.46%)	8 (3.85%)	0 (0%)
PRINTEMPS 2025-05-28 (incomplete)	208	0 (0%)	0 (0%)	0 (0%)	173 (83.17%)	35 (16.83%)	0 (0%)
Sat4j CP 2025-06-06 (complete)	208	115 (55.29%)	0 (0%)	115 (55.29%)	63 (30.29%)	30 (14.42%)	0 (0%)
Sat4j Resolution 2025-06-06 (complete)	208	133 (63.94%)	1 (0.48%)	132 (63.46%)	63 (30.29%)	12 (5.77%)	0 (0%)
SCIP 2024-07-07 (complete)	208	159 (76.44%)	1 (0.48%)	158 (75.96%)	18 (8.65%)	31 (14.9%)	0 (0%)
toulbar2 2025-06-13 (complete)	208	144 (69.23%)	1 (0.48%)	143 (68.75%)	60 (28.85%)	4 (1.92%)	0 (0%)

Solver Name

Progress

Exact
2025-06-02
(complete)

done
208

UNSAT
1

MOPT
157

MSAT
43

?
7

Exact_no_dombreaking
2025-06-16
(complete)

done
208

UNSAT
1

MOPT
156

MSAT
42

?
9

NaPS
naps-1.03a6
(complete)

done
208

UNSAT
1

MOPT
145

MSAT
47

?
15

OR-Tools CP-SAT
9.14
(complete)

done
208

MOPT
172

MSAT
27

?
1

ERR
8

OR-Tools CP-SAT
9.14.2
(complete)

done
208

MOPT
172

MSAT
28

?
8

PRINTEMPS
2025-05-28
(incomplete)

done
208

MSAT
173

?
35

Sat4j CP
2025-06-06
(complete)

done
208

MOPT
115

MSAT
63

?
30

Sat4j Resolution
2025-06-06
(complete)

done
208

UNSAT
1

MOPT
132

MSAT
63

?
12

SCIP
2024-07-07
(complete)

done
208

UNSAT
1

MOPT
158

MSAT
18

?
31

toulbar2
2025-06-13
(complete)

done
208

UNSAT
1

MOPT
143

MSAT
60

?
4

Category "WBO, only soft constraints, linear constraints" (SOFT-LIN)

Solver Name	#benchs	Solved	MOPT	MSAT	UNKNOWN
Exact 2025-06-02 (complete)	60	58 (96.67%)	58 (96.67%)	2 (3.33%)	0 (0%)
Exact_no_dombreaking 2025-06-16 (complete)	60	59 (98.33%)	59 (98.33%)	1 (1.67%)	0 (0%)
NaPS naps-1.03a6 (complete)	60	35 (58.33%)	35 (58.33%)	20 (33.33%)	5 (8.33%)
OR-Tools CP-SAT 9.14 (complete)	60	58 (96.67%)	58 (96.67%)	2 (3.33%)	0 (0%)
OR-Tools CP-SAT 9.14.2 (complete)	60	58 (96.67%)	58 (96.67%)	2 (3.33%)	0 (0%)
PRINTEMPS 2025-05-28 (incomplete)	60	0 (0%)	0 (0%)	60 (100%)	0 (0%)
Sat4j CP 2025-06-06 (complete)	60	32 (53.33%)	32 (53.33%)	28 (46.67%)	0 (0%)
Sat4j Resolution 2025-06-06 (complete)	60	41 (68.33%)	41 (68.33%)	19 (31.67%)	0 (0%)
SCIP 2024-07-07 (complete)	60	49 (81.67%)	49 (81.67%)	10 (16.67%)	1 (1.67%)
toulbar2 2025-06-13 (complete)	60	15 (25%)	15 (25%)	45 (75%)	0 (0%)

Solver Name

Progress

Exact
2025-06-02
(complete)

done
60

MOPT
58

MSAT
2

Exact_no_dombreaking
2025-06-16
(complete)

done
60

MOPT
59

MSAT
1

NaPS
naps-1.03a6
(complete)

done
60

MOPT
35

MSAT
20

?
5

OR-Tools CP-SAT
9.14
(complete)

done
60

MOPT
58

MSAT
2

OR-Tools CP-SAT
9.14.2
(complete)

done
60

MOPT
58

MSAT
2

PRINTEMPS
2025-05-28
(incomplete)

done
60

MSAT
60

Sat4j CP
2025-06-06
(complete)

done
60

MOPT
32

MSAT
28

Sat4j Resolution
2025-06-06
(complete)

done
60

MOPT
41

MSAT
19

SCIP
2024-07-07
(complete)

done
60

MOPT
49

MSAT
10

?
1

toulbar2
2025-06-13
(complete)

done
60

MOPT
15

MSAT
45

All categories for solvers registered in all categories

Solver Name	#benchs	Solved	UNSAT	MOPT	MSAT	UNKNOWN	Bad Answers
Exact 2025-06-02 (complete)	268	216 (80.6%)	1 (0.37%)	215 (80.22%)	45 (16.79%)	7 (2.61%)	0 (0%)
Exact_no_dombreaking 2025-06-16 (complete)	268	216 (80.6%)	1 (0.37%)	215 (80.22%)	43 (16.04%)	9 (3.36%)	0 (0%)
NaPS naps-1.03a6 (complete)	268	181 (67.54%)	1 (0.37%)	180 (67.16%)	67 (25%)	20 (7.46%)	0 (0%)
OR-Tools CP-SAT 9.14 (complete)	268	230 (85.82%)	0 (0%)	230 (85.82%)	29 (10.82%)	1 (0.37%)	8 (2.99%)
OR-Tools CP-SAT 9.14.2 (complete)	268	230 (85.82%)	0 (0%)	230 (85.82%)	30 (11.19%)	8 (2.99%)	0 (0%)
PRINTEMPS 2025-05-28 (incomplete)	268	0 (0%)	0 (0%)	0 (0%)	233 (86.94%)	35 (13.06%)	0 (0%)
Sat4j CP 2025-06-06 (complete)	268	147 (54.85%)	0 (0%)	147 (54.85%)	91 (33.96%)	30 (11.19%)	0 (0%)
Sat4j Resolution 2025-06-06 (complete)	268	174 (64.93%)	1 (0.37%)	173 (64.55%)	82 (30.6%)	12 (4.48%)	0 (0%)
SCIP 2024-07-07 (complete)	268	208 (77.61%)	1 (0.37%)	207 (77.24%)	28 (10.45%)	32 (11.94%)	0 (0%)
toulbar2 2025-06-13 (complete)	268	159 (59.33%)	1 (0.37%)	158 (58.96%)	105 (39.18%)	4 (1.49%)	0 (0%)

Solver Name

Progress

Exact
2025-06-02
(complete)

done
268

UNSAT
1

MOPT
215

MSAT
45

?
7

Exact_no_dombreaking
2025-06-16
(complete)

done
268

UNSAT
1

MOPT
215

MSAT
43

?
9

NaPS
naps-1.03a6
(complete)

done
268

UNSAT
1

MOPT
180

MSAT
67

?
20

OR-Tools CP-SAT
9.14
(complete)

done
268

MOPT
230

MSAT
29

?
1

ERR
8

OR-Tools CP-SAT
9.14.2
(complete)

done
268

MOPT
230

MSAT
30

?
8

PRINTEMPS
2025-05-28
(incomplete)

done
268

MSAT
233

?
35

Sat4j CP
2025-06-06
(complete)

done
268

MOPT
147

MSAT
91

?
30

Sat4j Resolution
2025-06-06
(complete)

done
268

UNSAT
1

MOPT
173

MSAT
82

?
12

SCIP
2024-07-07
(complete)

done
268

UNSAT
1

MOPT
207

MSAT
28

?
32

toulbar2
2025-06-13
(complete)

done
268

UNSAT
1

MOPT
158

MSAT
105

?
4

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name	#benchs	Solved	UNSAT	MOPT	MSAT	UNKNOWN	Bad Answers
Exact 2025-06-02 (complete)	268	216 (80.6%)	1 (0.37%)	215 (80.22%)	45 (16.79%)	7 (2.61%)	0 (0%)
Exact_no_dombreaking 2025-06-16 (complete)	268	216 (80.6%)	1 (0.37%)	215 (80.22%)	43 (16.04%)	9 (3.36%)	0 (0%)
NaPS naps-1.03a6 (complete)	268	181 (67.54%)	1 (0.37%)	180 (67.16%)	67 (25%)	20 (7.46%)	0 (0%)
OR-Tools CP-SAT 9.14 (complete)	268	230 (85.82%)	0 (0%)	230 (85.82%)	29 (10.82%)	1 (0.37%)	8 (2.99%)
OR-Tools CP-SAT 9.14.2 (complete)	268	230 (85.82%)	0 (0%)	230 (85.82%)	30 (11.19%)	8 (2.99%)	0 (0%)
PRINTEMPS 2025-05-28 (incomplete)	268	0 (0%)	0 (0%)	0 (0%)	233 (86.94%)	35 (13.06%)	0 (0%)
Sat4j CP 2025-06-06 (complete)	268	147 (54.85%)	0 (0%)	147 (54.85%)	91 (33.96%)	30 (11.19%)	0 (0%)
Sat4j Resolution 2025-06-06 (complete)	268	174 (64.93%)	1 (0.37%)	173 (64.55%)	82 (30.6%)	12 (4.48%)	0 (0%)
SCIP 2024-07-07 (complete)	268	208 (77.61%)	1 (0.37%)	207 (77.24%)	28 (10.45%)	32 (11.94%)	0 (0%)
toulbar2 2025-06-13 (complete)	268	159 (59.33%)	1 (0.37%)	158 (58.96%)	105 (39.18%)	4 (1.49%)	0 (0%)