PB'16 competition: WBO track: solvers page (tabular format, detailed results)

Some explanations

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN)

Solver Name#benchsUNSAT.MOPT.SAT.MSAT.MSAT (timeout)MSAT (out of mem.)UNKNOWNUNKNOWN (timeout)UNKNOWN (out of mem.)UNKNOWN (exit code)UNKNOWN (conv. script)UNKNOWN (problem)Sig. CaughtNO CERT.WRONG UNSAT.WRONG OPT.WRONG CERT.
NaPS
1.02
(complete)
2630
(0%)
114
(43.35%)
0
(0%)
32
(12.17%)
48
(18.25%)
17
(6.46%)
17
(6.46%)
22
(8.37%)
13
(4.94%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
2630
(0%)
121
(46.01%)
0
(0%)
0
(0%)
122
(46.39%)
0
(0%)
0
(0%)
3
(1.14%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
2
(0.76%)
0
(0%)
0
(0%)
15
(5.7%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)
2630
(0%)
134
(50.95%)
0
(0%)
0
(0%)
112
(42.59%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
17
(6.46%)
toysat
2016-05-02
(complete)
2630
(0%)
122
(46.39%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
139
(52.85%)
2
(0.76%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)

Solver NameProgress
NaPS
1.02
(complete)
done
263
MOPT.
114
MSAT.
32
MSAT (TO)
48
MSAT (MO)
17
?
17
? (TO)
22
? (MO)
13
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
263
MOPT.
121
MSAT (TO)
122
? (TO)
3
NO CERT.
2
ERR CERT
15
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
263
MOPT.
134
MSAT (TO)
112
ERR CERT
17
toysat
2016-05-02
(complete)
done
263
MOPT.
122
? (TO)
139
? (MO)
2

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN)

Solver Name#benchsUNSAT.MOPT.SAT.MSAT.MSAT (timeout)MSAT (out of mem.)UNKNOWNUNKNOWN (timeout)UNKNOWN (out of mem.)UNKNOWN (exit code)UNKNOWN (conv. script)UNKNOWN (problem)Sig. CaughtNO CERT.WRONG UNSAT.WRONG OPT.WRONG CERT.
NaPS
1.02
(complete)
5491
(0.18%)
469
(85.43%)
0
(0%)
2
(0.36%)
76
(13.84%)
1
(0.18%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
5490
(0%)
456
(83.06%)
0
(0%)
0
(0%)
91
(16.58%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
2
(0.36%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)
5490
(0%)
458
(83.42%)
0
(0%)
0
(0%)
87
(15.85%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
4
(0.73%)
toysat
2016-05-02
(complete)
5491
(0.18%)
453
(82.51%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
95
(17.3%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)

Solver NameProgress
NaPS
1.02
(complete)
done
549
UNSAT.
1
MOPT.
469
MSAT.
2
MSAT (TO)
76
MSAT (MO)
1
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
549
MOPT.
456
MSAT (TO)
91
ERR CERT
2
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
549
MOPT.
458
MSAT (TO)
87
ERR CERT
4
toysat
2016-05-02
(complete)
done
549
UNSAT.
1
MOPT.
453
? (TO)
95

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN)

Solver Name#benchsUNSAT.MOPT.SAT.MSAT.MSAT (timeout)MSAT (out of mem.)UNKNOWNUNKNOWN (timeout)UNKNOWN (out of mem.)UNKNOWN (exit code)UNKNOWN (conv. script)UNKNOWN (problem)Sig. CaughtNO CERT.WRONG UNSAT.WRONG OPT.WRONG CERT.
NaPS
1.02
(complete)
460
(0%)
6
(13.04%)
0
(0%)
21
(45.65%)
0
(0%)
16
(34.78%)
0
(0%)
0
(0%)
3
(6.52%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
460
(0%)
33
(71.74%)
0
(0%)
0
(0%)
13
(28.26%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)
460
(0%)
41
(89.13%)
0
(0%)
0
(0%)
5
(10.87%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
toysat
2016-05-02
(complete)
460
(0%)
34
(73.91%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
12
(26.09%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)

Solver NameProgress
NaPS
1.02
(complete)
done
46
MOPT.
6
MSAT.
21
MSAT (MO)
16
? (MO)
3
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
46
MOPT.
33
MSAT (TO)
13
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
46
MOPT.
41
MSAT (TO)
5
toysat
2016-05-02
(complete)
done
46
MOPT.
34
? (TO)
12

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN)

Solver Name#benchsUNSAT.MOPT.SAT.MSAT.MSAT (timeout)MSAT (out of mem.)UNKNOWNUNKNOWN (timeout)UNKNOWN (out of mem.)UNKNOWN (exit code)UNKNOWN (conv. script)UNKNOWN (problem)Sig. CaughtNO CERT.WRONG UNSAT.WRONG OPT.WRONG CERT.
NaPS
1.02
(complete)
2010
(0%)
164
(81.59%)
0
(0%)
2
(1%)
35
(17.41%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
2010
(0%)
127
(63.18%)
0
(0%)
0
(0%)
74
(36.82%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)
2010
(0%)
163
(81.09%)
0
(0%)
0
(0%)
38
(18.91%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
toysat
2016-05-02
(complete)
2010
(0%)
160
(79.6%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
41
(20.4%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)

Solver NameProgress
NaPS
1.02
(complete)
done
201
MOPT.
164
MSAT.
2
MSAT (TO)
35
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
201
MOPT.
127
MSAT (TO)
74
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
201
MOPT.
163
MSAT (TO)
38
toysat
2016-05-02
(complete)
done
201
MOPT.
160
? (TO)
41

All categories for solvers registered in all categories

Solver Name#benchsUNSAT.MOPT.SAT.MSAT.MSAT (timeout)MSAT (out of mem.)UNKNOWNUNKNOWN (timeout)UNKNOWN (out of mem.)UNKNOWN (exit code)UNKNOWN (conv. script)UNKNOWN (problem)Sig. CaughtNO CERT.WRONG UNSAT.WRONG OPT.WRONG CERT.
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
10590
(0%)
737
(69.59%)
0
(0%)
0
(0%)
300
(28.33%)
0
(0%)
0
(0%)
3
(0.28%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
2
(0.19%)
0
(0%)
0
(0%)
17
(1.61%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)
10590
(0%)
796
(75.17%)
0
(0%)
0
(0%)
242
(22.85%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
21
(1.98%)
toysat
2016-05-02
(complete)
10591
(0.09%)
769
(72.62%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
287
(27.1%)
2
(0.19%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)

Solver NameProgress
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
1059
MOPT.
737
MSAT (TO)
300
? (TO)
3
NO CERT.
2
ERR CERT
17
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
1059
MOPT.
796
MSAT (TO)
242
ERR CERT
21
toysat
2016-05-02
(complete)
done
1059
UNSAT.
1
MOPT.
769
? (TO)
287
? (MO)
2

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Solver Name#benchsUNSAT.MOPT.SAT.MSAT.MSAT (timeout)MSAT (out of mem.)UNKNOWNUNKNOWN (timeout)UNKNOWN (out of mem.)UNKNOWN (exit code)UNKNOWN (conv. script)UNKNOWN (problem)Sig. CaughtNO CERT.WRONG UNSAT.WRONG OPT.WRONG CERT.
NaPS
1.02
(complete)
10591
(0.09%)
753
(71.1%)
0
(0%)
57
(5.38%)
159
(15.01%)
34
(3.21%)
17
(1.61%)
22
(2.08%)
16
(1.51%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
10590
(0%)
737
(69.59%)
0
(0%)
0
(0%)
300
(28.33%)
0
(0%)
0
(0%)
3
(0.28%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
2
(0.19%)
0
(0%)
0
(0%)
17
(1.61%)
Sat4j PB 2.3.6 Resolution
PB16
(complete)
10590
(0%)
796
(75.17%)
0
(0%)
0
(0%)
242
(22.85%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
21
(1.98%)
toysat
2016-05-02
(complete)
10591
(0.09%)
769
(72.62%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
287
(27.1%)
2
(0.19%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)
0
(0%)

Solver NameProgress
NaPS
1.02
(complete)
done
1059
UNSAT.
1
MOPT.
753
MSAT.
57
MSAT (TO)
159
MSAT (MO)
34
?
17
? (TO)
22
? (MO)
16
Sat4j PB 2.3.6 Res+CP
PB16
(complete)
done
1059
MOPT.
737
MSAT (TO)
300
? (TO)
3
NO CERT.
2
ERR CERT
17
Sat4j PB 2.3.6 Resolution
PB16
(complete)
done
1059
MOPT.
796
MSAT (TO)
242
ERR CERT
21
toysat
2016-05-02
(complete)
done
1059
UNSAT.
1
MOPT.
769
? (TO)
287
? (MO)
2