PB'06 evaluation: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

Category "optimisation, big integers" (OPT-BIGINT)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	SAT4JPSEUDO	2006.2 Heuristics	94	11383.9	121.11
2	minisat+	1.14	71	6683.03	94.13
3	SAT4JPSEUDO	2006.2	46	1556.75	33.84
4	bsolo	2006/05	44	2829.62	64.31

Category "optimisation, medium integers" (OPT-MEDINT)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	glpPB	0.2	9	410.5	45.61
2	bsolo	2006/05	6	481.87	80.31
3	absconPseudo	1	5	40.9	8.18
4	PBS	4.1L	5	206.92	41.38
5	Pueblo	1.4	4	32.73	8.18
6	Pueblo	1.3	4	87.84	21.96
7	SAT4JPSEUDO	2006.2 Heuristics	4	292.82	73.2
8	SAT4JPSEUDO	2006.2	3	894.95	298.32
9	minisat+	1.14	3	1355.38	451.79

Category "optimisation, small integers" (OPT-SMALLINT)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	bsolo	2006/05	449	24813.54	55.26
2	minisat+	1.14	448	22052.85	49.23
3	glpPB	0.2	424	21528.21	50.77
4	Pueblo	1.4	422	17097.08	40.51
5	SAT4JPSEUDO	2006.2 Heuristics	411	23060.63	56.11
6	SAT4JPSEUDO	2006.2	409	32074.36	78.42
7	Pueblo	1.3	405	18899.55	46.67
8	PBS	4.1L	381	11619.61	30.5
9	PB-smodels	1.31	381	13114.01	34.42
10	PB-smodels	1.28	379	10981.95	28.98
11	absconPseudo	1	321	19628.41	61.15
12	wildcat-skc	0.8.0	2	1751.48	875.74
13	wildcat-rnp	0.8.0	1	0.03	0.03

Category "no optimisation, big integers" (SATUNSAT-BIGINT)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	minisat+	1.14	1	15.31	15.31
2	bsolo	2006/05	1	146.45	146.45
3	SAT4JPSEUDO	2006.2 Heuristics	0	0	-
4	SAT4JPSEUDO	2006.2	0	0	-

Category "no optimisation, medium integers" (SATUNSAT-MEDINT)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	minisat+	1.14	10	5246.46	524.65
2	PBS	4.1L	3	47.46	15.82
3	bsolo	2006/05	2	15.42	7.71
4	SAT4JPSEUDO	2006.2	2	45.96	22.98
5	SAT4JPSEUDO	2006.2 Heuristics	2	46.42	23.21
6	Pueblo	1.4	0	0	-
7	Pueblo	1.3	0	0	-
8	absconPseudo	1	0	0	-
9	glpPB	0.2	0	0	-

Category "no optimisation, small integers" (SATUNSAT-SMALLINT)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	Pueblo	1.4	357	25631.16	71.8
2	PBS	4.1L	343	46967.85	136.93
3	minisat+	1.14	320	63615.68	198.8
4	Pueblo	1.3	315	23739.45	75.36
5	PB-smodels	1.31	285	34078.24	119.57
6	PB-smodels	1.28	270	43124.46	159.72
7	bsolo	2006/05	229	21777.01	95.1
8	glpPB	0.2	213	15286.63	71.77
9	absconPseudo	1	176	9400.63	53.41
10	SAT4JPSEUDO	2006.2	171	25232.57	147.56
11	SAT4JPSEUDO	2006.2 Heuristics	171	25643.19	149.96
12	wildcat-skc	0.8.0	163	3538.05	21.71
13	wildcat-rnp	0.8.0	156	1003.43	6.43