PB'16 competition: WBO track: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 263
Virtual Best Solver (VBS)			142	142 MOPT	54%	100%	9663.06	0.00	68.05	-	-	-	-
1	toysat	2016-05-02	122	122 MOPT	46%	86%	10236.42	6.17	83.91	10238.83	6.18	83.92	1.00
2	NaPS	1.02	114	114 MOPT	43%	80%	6665.99	3.40	58.47	6667.33	3.40	58.49	1.00

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 263
Virtual Best Solver (VBS)			142	54%	100%	9663.06	0.00	68.05	-	-	-	-
1	toysat	2016-05-02	122	46%	86%	10236.42	6.17	83.91	10238.83	6.18	83.92	1.00
2	NaPS	1.02	114	43%	80%	6665.99	3.40	58.47	6667.33	3.40	58.49	1.00

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 263
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	toysat	2016-05-02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	NaPS	1.02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 549
Virtual Best Solver (VBS)			472	471 MOPT, 1 UNSAT	86%	100%	14021.01	16.70	29.71	-	-	-	-
1	NaPS	1.02	470	469 MOPT, 1 UNSAT	86%	100%	20994.41	14.64	44.67	20998.81	14.64	44.68	1.00
2	toysat	2016-05-02	454	453 MOPT, 1 UNSAT	83%	96%	30558.16	24.08	67.31	30565.78	24.09	67.33	1.00

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 549
Virtual Best Solver (VBS)			471	86%	100%	14004.31	0.00	29.73	-	-	-	-
1	NaPS	1.02	469	85%	100%	20977.71	14.63	44.73	20982.10	14.63	44.74	1.00
2	toysat	2016-05-02	453	83%	96%	29725.32	24.01	65.62	29732.75	24.01	65.64	1.00

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 549
Virtual Best Solver (VBS)			1	0%	100%	16.70	16.70	16.70	-	-	-	-
1	NaPS	1.02	1	0%	100%	16.70	16.70	16.70	16.71	16.71	16.71	1.00
2	toysat	2016-05-02	1	0%	100%	832.84	832.84	832.84	833.03	833.03	833.03	1.00

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 46
Virtual Best Solver (VBS)			41	41 MOPT	89%	100%	4495.51	0.00	109.65	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	41	41 MOPT	89%	100%	4857.34	4.52	118.47	4748.26	2.47	115.81	1.02
2	toysat	2016-05-02	34	34 MOPT	74%	83%	3847.33	18.41	113.16	3848.14	18.42	113.18	1.00
3	Sat4j PB 2.3.6 Res+CP	PB16	33	33 MOPT	72%	80%	3041.95	5.42	92.18	2410.60	2.70	73.05	1.26
4	NaPS	1.02	6	6 MOPT	13%	15%	1192.64	185.48	198.77	1192.88	185.52	198.81	1.00

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 46
Virtual Best Solver (VBS)			41	89%	100%	4495.51	0.00	109.65	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	41	89%	100%	4857.34	4.52	118.47	4748.26	2.47	115.81	1.02
2	toysat	2016-05-02	34	74%	83%	3847.33	18.41	113.16	3848.14	18.42	113.18	1.00
3	Sat4j PB 2.3.6 Res+CP	PB16	33	72%	80%	3041.95	5.42	92.18	2410.60	2.70	73.05	1.26
4	NaPS	1.02	6	13%	15%	1192.64	185.48	198.77	1192.88	185.52	198.81	1.00

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 46
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	Sat4j PB 2.3.6 Res+CP	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	toysat	2016-05-02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	NaPS	1.02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 201
Virtual Best Solver (VBS)			195	195 MOPT	97%	100%	3793.31	0.00	19.45	-	-	-	-
1	NaPS	1.02	164	164 MOPT	82%	84%	8293.44	28.81	50.57	8295.35	28.83	50.58	1.00
2	Sat4j PB 2.3.6 Resolution	PB16	163	163 MOPT	81%	84%	6398.42	14.65	39.25	6167.18	13.04	37.84	1.04
3	toysat	2016-05-02	160	160 MOPT	80%	82%	20541.99	33.47	128.39	20546.69	33.48	128.42	1.00
4	Sat4j PB 2.3.6 Res+CP	PB16	127	127 MOPT	63%	65%	5074.70	18.50	39.96	5543.10	21.81	43.65	0.92

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 201
Virtual Best Solver (VBS)			195	97%	100%	3793.31	0.00	19.45	-	-	-	-
1	NaPS	1.02	164	82%	84%	8293.44	28.81	50.57	8295.35	28.83	50.58	1.00
2	Sat4j PB 2.3.6 Resolution	PB16	163	81%	84%	6398.42	14.65	39.25	6167.18	13.04	37.84	1.04
3	toysat	2016-05-02	160	80%	82%	20541.99	33.47	128.39	20546.69	33.48	128.42	1.00
4	Sat4j PB 2.3.6 Res+CP	PB16	127	63%	65%	5074.70	18.50	39.96	5543.10	21.81	43.65	0.92

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 201
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	Sat4j PB 2.3.6 Res+CP	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	toysat	2016-05-02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	NaPS	1.02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-