2019 XCSP3 competition: mini-solver track (sequential and parallel solvers): solvers page (tabular format, summary)

Some explanations

#Benchs: Number of benchs for this solver
Solved: Total number of solved instances (this is the sum of the SAT and UNSAT columns)
UNSAT: Total number of UNSAT answers
OPT/SAT: solver found an optimum solution to an optimization problem (the provided implicant satisfies each constraint and no solver gave a better solution) or solver found a solution to a SAT/UNSAT instance
SAT (TO/MO): Solver found a solution but this may not be the optimal solution
UNKNOWN: Total number of UNKNOWN answers
Bad Answers: Total number of bad answers
solver a "light red" background denotes a solver which was disqualified because it gave an incorrect answer, and which will not appear in the ranking
solver a "light yellow" background denotes a solver which was replaced by another version, and which will not appear in the ranking

Category "optimization problem" (COP)

Solver Name	#benchs	Solved	UNSAT	OPT/SAT	SAT (TO/MO)	UNKNOWN
cosoco 2 (complete)	180	100 (55.56%)	1 (0.56%)	99 (55%)	75 (41.67%)	5 (2.78%)
cosoco 2.0 (complete)	180	100 (55.56%)	1 (0.56%)	99 (55%)	75 (41.67%)	5 (2.78%)
cosoco 2.0 (complete)	180	100 (55.56%)	1 (0.56%)	99 (55%)	75 (41.67%)	5 (2.78%)
(reference) PicatSAT 2019-09-12 (complete)	180	136 (75.56%)	1 (0.56%)	135 (75%)	0 (0%)	44 (24.44%)

Solver Name

Progress

cosoco
2
(complete)

done
180

UNSAT
1

OPT/SAT
99

SAT (TO/MO)
75

?
5

cosoco
2.0
(complete)

done
180

UNSAT
1

OPT/SAT
99

SAT (TO/MO)
75

?
5

cosoco
2.0
(complete)

done
180

UNSAT
1

OPT/SAT
99

SAT (TO/MO)
75

?
5

(reference) PicatSAT
2019-09-12
(complete)

done
180

UNSAT
1

OPT/SAT
135

?
44

Category "decision problem" (CSP)

Solver Name	#benchs	Solved	UNSAT	OPT/SAT	UNKNOWN
cosoco 2 (complete)	200	127 (63.5%)	42 (21%)	85 (42.5%)	73 (36.5%)
cosoco 2.0 (complete)	200	127 (63.5%)	42 (21%)	85 (42.5%)	73 (36.5%)
cosoco 2.0 (complete)	200	127 (63.5%)	42 (21%)	85 (42.5%)	73 (36.5%)
miniBTD 19.06.16 (complete)	200	133 (66.5%)	44 (22%)	89 (44.5%)	67 (33.5%)
NACRE 1.0.5 (complete)	200	116 (58%)	35 (17.5%)	81 (40.5%)	84 (42%)
NACRE 1.0.5-Hybrid (complete)	200	135 (67.5%)	44 (22%)	91 (45.5%)	65 (32.5%)
(reference) PicatSAT 2019-09-12 (complete)	200	148 (74%)	51 (25.5%)	97 (48.5%)	52 (26%)

Solver Name

Progress

cosoco
2
(complete)

done
200

UNSAT
42

OPT/SAT
85

?
73

cosoco
2.0
(complete)

done
200

UNSAT
42

OPT/SAT
85

?
73

cosoco
2.0
(complete)

done
200

UNSAT
42

OPT/SAT
85

?
73

miniBTD
19.06.16
(complete)

done
200

UNSAT
44

OPT/SAT
89

?
67

NACRE
1.0.5
(complete)

done
200

UNSAT
35

OPT/SAT
81

?
84

NACRE
1.0.5-Hybrid
(complete)

done
200

UNSAT
44

OPT/SAT
91

?
65

(reference) PicatSAT
2019-09-12
(complete)

done
200

UNSAT
51

OPT/SAT
97

?
52

All categories for solvers registered in all categories

Solver Name	#benchs	Solved	UNSAT	OPT/SAT	SAT (TO/MO)	UNKNOWN
cosoco 2 (complete)	380	227 (59.74%)	43 (11.32%)	184 (48.42%)	75 (19.74%)	78 (20.53%)
cosoco 2.0 (complete)	380	227 (59.74%)	43 (11.32%)	184 (48.42%)	75 (19.74%)	78 (20.53%)
cosoco 2.0 (complete)	380	227 (59.74%)	43 (11.32%)	184 (48.42%)	75 (19.74%)	78 (20.53%)
(reference) PicatSAT 2019-09-12 (complete)	380	284 (74.74%)	52 (13.68%)	232 (61.05%)	0 (0%)	96 (25.26%)

Solver Name

Progress

cosoco
2
(complete)

done
380

UNSAT
43

OPT/SAT
184

SAT (TO/MO)
75

?
78

cosoco
2.0
(complete)

done
380

UNSAT
43

OPT/SAT
184

SAT (TO/MO)
75

?
78

cosoco
2.0
(complete)

done
380

UNSAT
43

OPT/SAT
184

SAT (TO/MO)
75

?
78

(reference) PicatSAT
2019-09-12
(complete)

done
380

UNSAT
52

OPT/SAT
232

?
96

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name	#benchs	Solved	UNSAT	OPT/SAT	SAT (TO/MO)	UNKNOWN
cosoco 2 (complete)	380	227 (59.74%)	43 (11.32%)	184 (48.42%)	75 (19.74%)	78 (20.53%)
cosoco 2.0 (complete)	380	227 (59.74%)	43 (11.32%)	184 (48.42%)	75 (19.74%)	78 (20.53%)
cosoco 2.0 (complete)	380	227 (59.74%)	43 (11.32%)	184 (48.42%)	75 (19.74%)	78 (20.53%)
miniBTD 19.06.16 (complete)	200	133 (66.5%)	44 (22%)	89 (44.5%)	0 (0%)	67 (33.5%)
NACRE 1.0.5 (complete)	200	116 (58%)	35 (17.5%)	81 (40.5%)	0 (0%)	84 (42%)
NACRE 1.0.5-Hybrid (complete)	200	135 (67.5%)	44 (22%)	91 (45.5%)	0 (0%)	65 (32.5%)
(reference) PicatSAT 2019-09-12 (complete)	380	284 (74.74%)	52 (13.68%)	232 (61.05%)	0 (0%)	96 (25.26%)