PB'07 evaluation: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	SAT4JPseudoResolution	2007-03-23	118	7816.35	66.24
2	sat4jPseudoCP	2007-03-23	98	1856.23	18.94
3	sat4jPseudoCPClause	2007-03-23	93	1250.33	13.44
4	minisat+	1.14	74	8108.22	109.57
5	bsolo	3.0.17	54	3322.41	61.53
6	oree	0.1.2 alpha	48	1462.76	30.47

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	bsolo	3.0.17	270	22410.87	83
2	glpPB	0.2	252	31993.3	126.96
3	minisat+	1.14	243	24323.18	100.1
4	Pueblo	1.4	211	19640.49	93.08
5	sat4jPseudoCP	2007-03-23	194	16980.56	87.53
6	sat4jPseudoCPClause	2007-03-23	191	15655.3	81.96
7	SAT4JPseudoResolution	2007-03-23	191	16254.01	85.1
8	PB-clasp	2007-04-10	181	18902.59	104.43
9	PBS4_v2	2007-03-23	164	12746.43	77.72
10	PBS4	2007-03-23	164	12836.18	78.27
11	PB-clasp	2007-03-23	154	27125.66	176.14
12	absconPseudo	102	99	19643.64	198.42
13	oree	0.1.2 alpha	92	19795.07	215.16
14	wildcat-rnp	2007-03-21	1	0.05	0.05
15	wildcat-skc	2007-03-21	1	0.09	0.09

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	minisat+	1.14	275	2360.19	8.58
2	Pueblo	1.4	248	33686.45	135.83
3	bsolo	3.0.17	239	32240.49	134.9
4	SAT4JPseudoResolution	2007-03-23	216	27351.99	126.63
5	PBS4	2007-03-23	171	17131.05	100.18
6	PBS4_v2	2007-03-23	171	17169.4	100.41
7	oree	0.1.2 alpha	99	23282.83	235.18
8	absconPseudo	102	88	9232.02	104.91
9	sat4jPseudoCP	2007-03-23	78	12837.68	164.59
10	sat4jPseudoCPClause	2007-03-23	76	14325.18	188.49
11	glpPB	0.2	67	12939.01	193.12
12	wildcat-skc	2007-03-21	0	0	-
13	wildcat-rnp	2007-03-21	0	0	-

Category "instances containing only clauses" (PURE-SAT)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	PB-clasp	2007-04-10	125	41516.66	332.13
2	minisat+	1.14	115	38171.02	331.92
3	PB-clasp	2007-03-23	102	47287.67	463.6
4	SAT4JPseudoResolution	2007-03-23	98	42743.49	436.16
5	Pueblo	1.4	85	49194.58	578.76
6	bsolo	3.0.17	63	21354.04	338.95
7	bsolo	3.0.16	60	20841.35	347.36
8	sat4jPseudoCP	2007-03-23	47	23900.47	508.52
9	sat4jPseudoCPClause	2007-03-23	47	23969.11	509.98
10	PBS4_v2	2007-03-23	35	16432.94	469.51
11	PBS4	2007-03-23	35	16467.55	470.5
12	oree	0.1.2 alpha	13	3367.73	259.06
13	absconPseudo	102	9	1543.97	171.55
14	wildcat-skc	2007-03-21	4	1634.58	408.64
15	wildcat-rnp	2007-03-21	1	1216.7	1216.7
16	glpPB	0.2	1	1695.89	1695.89

Category "no optimisation, big integers, linear constraints" (SATUNSAT-BIGINT-LIN)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	minisat+	1.14	11	5290.2	480.93
2	bsolo	3.0.17	7	330.57	47.22
3	bsolo	3.0.16	7	332.65	47.52
4	SAT4JPseudoResolution	2007-03-23	4	44.15	11.04
5	sat4jPseudoCP	2007-03-23	3	43.12	14.37
6	sat4jPseudoCPClause	2007-03-23	2	14.2	7.1
7	oree	0.1.2 alpha	1	0.33	0.33

Category "no optimisation, small integers, linear constraints" (SATUNSAT-SMALLINT-LIN)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	Pueblo	1.4	341	25484.8	74.74
2	PBS4_v2	2007-03-23	327	46370.9	141.81
3	PBS4	2007-03-23	327	46472.93	142.12
4	PB-clasp	2007-04-10	312	10831.51	34.72
5	minisat+	1.14	305	65244.53	213.92
6	PB-clasp	2007-03-23	283	36851.46	130.22
7	SAT4JPseudoResolution	2007-03-23	273	26839.49	98.31
8	bsolo	3.0.16	271	46125.59	170.21
9	bsolo	3.0.17	271	46453.68	171.42
10	glpPB	0.2	200	15272.9	76.36
11	absconPseudo	102	160	7052.37	44.08
12	sat4jPseudoCPClause	2007-03-23	151	18590.59	123.12
13	wildcat-skc	2007-03-21	148	925.34	6.25
14	wildcat-rnp	2007-03-21	142	3889.71	27.39
15	sat4jPseudoCP	2007-03-23	141	14773.5	104.78
16	oree	0.1.2 alpha	98	30278.35	308.96

Category "no optimisation, small integers, non linear constraints" (SATUNSAT-SMALLINT-NLC)

Rank	Solver	Version	Number of solved instances	Cumulated CPU time on solved instances	Average CPU time per solved instance
1	minisat+	1.14	65	491.22	7.56
2	PBS4	2007-03-23	65	2893.79	44.52
3	PBS4_v2	2007-03-23	65	2931.96	45.11
4	glpPB	0.2	65	6722.23	103.42
5	PB-clasp	2007-04-10	64	795.65	12.43
6	Pueblo	1.4	64	1744.99	27.27
7	PB-clasp	2007-03-23	62	19412.42	313.1
8	wildcat-rnp	2007-03-21	60	208.02	3.47
9	wildcat-skc	2007-03-21	60	5828.95	97.15
10	oree	0.1.2 alpha	55	10403.27	189.15
11	absconPseudo	102	50	2830.14	56.6
12	bsolo	3.0.17	28	5819.32	207.83
13	bsolo	3.0.16	27	4173.24	154.56
14	sat4jPseudoCPClause	2007-03-23	0	0	-
15	sat4jPseudoCP	2007-03-23	0	0	-
16	SAT4JPseudoResolution	2007-03-23	0	0	-