PB'16 competition: satisfaction and optimization track: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "no optimisation, big integers, linear constraints" (DEC-BIGINT-LIN): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 37
Virtual Best Solver (VBS)			28	27 SAT, 1 UNSAT	76%	100%	2364.64	21.66	84.45	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	22	22 SAT	59%	79%	2788.62	43.77	126.76	2680.57	38.45	121.84	1.04
2	Sat4j PB 2.3.6 Res+CP	PB16	21	21 SAT	57%	75%	4973.40	69.32	236.83	3215.26	37.60	153.11	1.55
3	toysat	2016-05-02	13	13 SAT	35%	46%	1319.20	62.77	101.48	1319.47	62.78	101.50	1.00
4	minisatp	2012-10-02 git-d91742b	10	9 SAT, 1 UNSAT	27%	36%	3225.70	13.52	322.57	3226.19	13.52	322.62	1.00
5	NaPS	1.02	2	2 SAT	5%	7%	240.68	120.34	120.34	240.72	120.36	120.36	1.00

Category "no optimisation, big integers, linear constraints" (DEC-BIGINT-LIN): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 37
Virtual Best Solver (VBS)			27	73%	100%	2362.49	23.60	87.50	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	22	59%	81%	2788.62	43.77	126.76	2680.57	38.45	121.84	1.04
2	Sat4j PB 2.3.6 Res+CP	PB16	21	57%	78%	4973.40	69.32	236.83	3215.26	37.60	153.11	1.55
3	toysat	2016-05-02	13	35%	48%	1319.20	62.77	101.48	1319.47	62.78	101.50	1.00
4	minisatp	2012-10-02 git-d91742b	9	24%	33%	3223.55	16.24	358.17	3224.04	16.24	358.23	1.00
5	NaPS	1.02	2	5%	7%	240.68	120.34	120.34	240.72	120.36	120.36	1.00

Category "no optimisation, big integers, linear constraints" (DEC-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 37
Virtual Best Solver (VBS)			1	3%	100%	2.15	2.15	2.15	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	1	3%	100%	2.15	2.15	2.15	2.15	2.15	2.15	1.00
2	toysat	2016-05-02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
3	Sat4j PB 2.3.6 Res+CP	PB16	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
4	NaPS	1.02	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
5	Sat4j PB 2.3.6 Resolution	PB16	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1783
Virtual Best Solver (VBS)			1560	402 SAT, 1158 UNSAT	87%	100%	67837.04	0.17	43.49	-	-	-	-
1	cdcl-cuttingplanes DEC	2016-05-01	1395	303 SAT, 1092 UNSAT	78%	89%	63054.00	0.19	45.20	63065.91	0.19	45.21	1.00
2	Open-WBO-LSU	PB16	1378	330 SAT, 1048 UNSAT	77%	88%	101082.56	1.12	73.35	101102.17	1.12	73.37	1.00
3	Open-WBO	PB16	1378	329 SAT, 1049 UNSAT	77%	88%	101310.34	1.10	73.52	101329.71	1.10	73.53	1.00
4	Sat4j PB 2.3.6 Res+CP	PB16	1367	315 SAT, 1052 UNSAT	77%	88%	88812.73	4.05	64.97	43931.26	1.54	32.14	2.02
5	NaPS	1.02	1361	338 SAT, 1023 UNSAT	76%	87%	93920.56	1.12	69.01	93938.57	1.12	69.02	1.00
6	minisatp	2012-10-02 git-d91742b	1319	384 SAT, 935 UNSAT	74%	85%	122687.51	3.49	93.02	122709.72	3.49	93.03	1.00
7	Sat4j PB 2.3.6 Resolution	PB16	1240	342 SAT, 898 UNSAT	70%	79%	179551.27	3.22	144.80	178393.26	2.27	143.87	1.01
8	toysat	2016-05-02	1164	323 SAT, 841 UNSAT	65%	75%	129809.61	3.85	111.52	129848.27	3.85	111.55	1.00

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1783
Virtual Best Solver (VBS)			402	23%	100%	45033.79	0.44	112.02	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	384	22%	96%	44011.99	4.73	114.61	44020.52	4.73	114.64	1.00
2	Sat4j PB 2.3.6 Resolution	PB16	342	19%	85%	45821.76	2.26	133.98	45466.47	1.38	132.94	1.01
3	NaPS	1.02	338	19%	84%	35623.16	1.05	105.39	35629.80	1.05	105.41	1.00
4	Open-WBO-LSU	PB16	330	19%	82%	47905.06	1.11	145.17	47913.64	1.11	145.19	1.00
5	Open-WBO	PB16	329	18%	82%	46537.67	1.08	141.45	46546.04	1.08	141.48	1.00
6	toysat	2016-05-02	323	18%	80%	31180.96	2.76	96.54	31190.44	2.76	96.56	1.00
7	Sat4j PB 2.3.6 Res+CP	PB16	315	18%	78%	26303.61	5.38	83.50	13160.85	1.90	41.78	2.00
8	cdcl-cuttingplanes DEC	2016-05-01	303	17%	75%	18992.09	0.47	62.68	18995.54	0.48	62.69	1.00

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1783
Virtual Best Solver (VBS)			1158	65%	100%	22803.25	0.10	19.69	-	-	-	-
1	cdcl-cuttingplanes DEC	2016-05-01	1092	61%	94%	44061.91	0.13	40.35	44070.38	0.13	40.36	1.00
2	Sat4j PB 2.3.6 Res+CP	PB16	1052	59%	91%	62509.12	3.28	59.42	30770.40	1.20	29.25	2.03
3	Open-WBO	PB16	1049	59%	91%	54772.66	1.12	52.21	54783.67	1.12	52.22	1.00
4	Open-WBO-LSU	PB16	1048	59%	91%	53177.51	1.12	50.74	53188.53	1.12	50.75	1.00
5	NaPS	1.02	1023	57%	88%	58297.39	1.23	56.99	58308.77	1.24	57.00	1.00
6	minisatp	2012-10-02 git-d91742b	935	52%	81%	78675.52	2.69	84.14	78689.21	2.69	84.16	1.00
7	Sat4j PB 2.3.6 Resolution	PB16	898	50%	78%	133729.51	3.82	148.92	132926.79	2.84	148.03	1.01
8	toysat	2016-05-02	841	47%	73%	98628.65	4.27	117.28	98657.83	4.27	117.31	1.00

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 100
Virtual Best Solver (VBS)			67	52 SAT, 15 UNSAT	67%	100%	2207.72	0.24	32.95	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	67	52 SAT, 15 UNSAT	67%	100%	2270.80	0.41	33.89	2271.31	0.41	33.90	1.00
2	Sat4j PB 2.3.6 Res+CP	PB16	65	50 SAT, 15 UNSAT	65%	97%	621.97	4.65	9.57	281.61	1.73	4.33	2.21
3	toysat	2016-05-02	65	50 SAT, 15 UNSAT	65%	97%	1298.85	0.25	19.98	1299.22	0.25	19.99	1.00
4	Sat4j PB 2.3.6 Resolution	PB16	26	11 SAT, 15 UNSAT	26%	39%	808.39	0.88	31.09	794.18	0.49	30.55	1.02

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 100
Virtual Best Solver (VBS)			52	52%	100%	2174.51	0.24	41.82	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	52	52%	100%	2237.58	0.63	43.03	2238.09	0.63	43.04	1.00
2	toysat	2016-05-02	50	50%	96%	18.32	0.25	0.37	18.45	0.25	0.37	0.99
3	Sat4j PB 2.3.6 Res+CP	PB16	50	50%	96%	393.16	4.82	7.86	168.18	1.74	3.36	2.34
4	Sat4j PB 2.3.6 Resolution	PB16	11	11%	21%	620.55	0.94	56.41	614.11	0.56	55.83	1.01

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 100
Virtual Best Solver (VBS)			15	15%	100%	33.22	0.01	2.21	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	15	15%	100%	33.22	0.01	2.21	33.23	0.01	2.22	1.00
2	Sat4j PB 2.3.6 Resolution	PB16	15	15%	100%	187.84	0.30	12.52	180.07	0.21	12.00	1.04
3	Sat4j PB 2.3.6 Res+CP	PB16	15	15%	100%	228.81	0.59	15.25	113.43	0.66	7.56	2.02
4	toysat	2016-05-02	15	15%	100%	1280.53	0.03	85.37	1280.77	0.03	85.38	1.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1109
Virtual Best Solver (VBS)			470	397 OPT, 73 UNSAT	42%	100%	41026.72	6.64	87.29	-	-	-	-
1	NaPS	1.02	374	305 OPT, 69 UNSAT	34%	80%	53552.24	29.89	143.19	53561.64	29.89	143.21	1.00
2	Sat4j PB 2.3.6 Res+CP	PB16	266	196 OPT, 70 UNSAT	24%	57%	35558.58	4.68	133.68	19107.48	3.02	71.83	1.86
3	Sat4j PB 2.3.6 Resolution	PB16	243	175 OPT, 68 UNSAT	22%	52%	15990.70	2.36	65.81	15709.22	1.32	64.65	1.02
4	minisatp	2012-10-02 git-d91742b	236	166 OPT, 70 UNSAT	21%	50%	55316.73	48.65	234.39	55327.09	48.66	234.44	1.00
5	toysat	2016-05-02	76	38 OPT, 38 UNSAT	7%	16%	10635.53	0.95	139.94	10637.77	0.95	139.97	1.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1109
Virtual Best Solver (VBS)			397	36%	100%	40908.06	17.09	103.04	-	-	-	-
1	NaPS	1.02	305	28%	77%	51443.29	39.57	168.67	51452.28	39.58	168.70	1.00
2	Sat4j PB 2.3.6 Res+CP	PB16	196	18%	49%	35318.59	6.88	180.20	18964.23	4.73	96.76	1.86
3	Sat4j PB 2.3.6 Resolution	PB16	175	16%	44%	15846.26	3.39	90.55	15608.24	2.23	89.19	1.02
4	minisatp	2012-10-02 git-d91742b	166	15%	42%	55082.13	217.67	331.82	55092.38	217.73	331.88	1.00
5	toysat	2016-05-02	38	3%	10%	10608.48	51.20	279.17	10610.61	51.21	279.23	1.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1109
Virtual Best Solver (VBS)			73	7%	100%	118.66	0.47	1.63	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	70	6%	96%	234.60	0.50	3.35	234.71	0.51	3.35	1.00
2	Sat4j PB 2.3.6 Res+CP	PB16	70	6%	96%	239.99	2.25	3.43	143.25	1.23	2.05	1.68
3	NaPS	1.02	69	6%	95%	2108.95	0.68	30.56	2109.35	0.68	30.57	1.00
4	Sat4j PB 2.3.6 Resolution	PB16	68	6%	93%	144.44	1.01	2.12	100.98	0.53	1.49	1.43
5	toysat	2016-05-02	38	3%	52%	27.04	0.60	0.71	27.17	0.60	0.71	1.00

Category "optimisation, big integers, non linear constraints" (OPT-BIGINT-NLC): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 41
Virtual Best Solver (VBS)			4	4 OPT	10%	100%	1184.66	74.09	296.16	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	3	3 OPT	7%	75%	148.87	10.94	49.62	146.41	9.83	48.80	1.02
2	Sat4j PB 2.3.6 Res+CP	PB16	3	3 OPT	7%	75%	417.34	25.65	139.11	227.54	18.32	75.85	1.83
3	minisatp	2012-10-02 git-d91742b	3	3 OPT	7%	75%	2152.99	788.32	717.66	2153.34	788.45	717.78	1.00
4	toysat	2016-05-02	2	2 OPT	5%	50%	132.51	66.26	66.26	132.54	66.27	66.27	1.00

Category "optimisation, big integers, non linear constraints" (OPT-BIGINT-NLC): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 41
Virtual Best Solver (VBS)			4	10%	100%	1184.66	74.09	296.16	-	-	-	-
1	Sat4j PB 2.3.6 Resolution	PB16	3	7%	75%	148.87	10.94	49.62	146.41	9.83	48.80	1.02
2	Sat4j PB 2.3.6 Res+CP	PB16	3	7%	75%	417.34	25.65	139.11	227.54	18.32	75.85	1.83
3	minisatp	2012-10-02 git-d91742b	3	7%	75%	2152.99	788.32	717.66	2153.34	788.45	717.78	1.00
4	toysat	2016-05-02	2	5%	50%	132.51	66.26	66.26	132.54	66.27	66.27	1.00

Category "optimisation, big integers, non linear constraints" (OPT-BIGINT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 41
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	toysat	2016-05-02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	Sat4j PB 2.3.6 Res+CP	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	Sat4j PB 2.3.6 Resolution	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	minisatp	2012-10-02 git-d91742b	0	0%	-	0.00	0.00	-	0.00	0.00	-	-

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1600
Virtual Best Solver (VBS)			983	884 OPT, 99 UNSAT	61%	100%	35856.49	0.16	36.48	-	-	-	-
1	NaPS	1.02	887	802 OPT, 85 UNSAT	55%	90%	44347.58	2.54	50.00	44358.38	2.54	50.01	1.00
2	Open-WBO	PB16	826	781 OPT, 45 UNSAT	52%	84%	27437.51	1.40	33.22	27444.52	1.40	33.23	1.00
3	Open-WBO-LSU	PB16	812	767 OPT, 45 UNSAT	51%	83%	30621.68	2.01	37.71	30631.42	2.01	37.72	1.00
4	minisatp	2012-10-02 git-d91742b	789	713 OPT, 76 UNSAT	49%	80%	63267.83	0.36	80.19	63280.61	0.36	80.20	1.00
5	cdcl-cuttingplanes OPT binary search	2016-05-01	774	685 OPT, 89 UNSAT	48%	79%	32583.84	0.33	42.10	32590.45	0.33	42.11	1.00
6	Sat4j PB 2.3.6 Res+CP	PB16	761	672 OPT, 89 UNSAT	48%	77%	55922.84	2.51	73.49	30629.38	1.80	40.25	1.83
7	cdcl-cuttingplanes OPT linear search	2016-05-01	749	660 OPT, 89 UNSAT	47%	76%	37825.90	0.23	50.50	37833.36	0.23	50.51	1.00
8	toysat	2016-05-02	733	670 OPT, 63 UNSAT	46%	75%	48127.45	1.08	65.66	48139.90	1.08	65.68	1.00
9	Sat4j PB 2.3.6 Resolution	PB16	716	649 OPT, 67 UNSAT	45%	73%	30883.90	0.76	43.13	30376.46	0.43	42.43	1.02

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1600
Virtual Best Solver (VBS)			884	55%	100%	30634.23	0.18	34.65	-	-	-	-
1	NaPS	1.02	802	50%	91%	33102.13	2.54	41.27	33110.58	2.54	41.29	1.00
2	Open-WBO	PB16	781	49%	88%	23078.56	1.53	29.55	23084.75	1.53	29.56	1.00
3	Open-WBO-LSU	PB16	767	48%	87%	26281.33	2.25	34.27	26289.94	2.25	34.28	1.00
4	minisatp	2012-10-02 git-d91742b	713	45%	81%	55616.14	0.25	78.00	55627.52	0.25	78.02	1.00
5	cdcl-cuttingplanes OPT binary search	2016-05-01	685	43%	77%	26178.49	0.37	38.22	26183.98	0.38	38.22	1.00
6	Sat4j PB 2.3.6 Res+CP	PB16	672	42%	76%	49522.47	2.03	73.69	27453.43	2.13	40.85	1.80
7	toysat	2016-05-02	670	42%	76%	45491.92	1.13	67.90	45503.73	1.13	67.92	1.00
8	cdcl-cuttingplanes OPT linear search	2016-05-01	660	41%	75%	31207.06	0.25	47.28	31213.38	0.25	47.29	1.00
9	Sat4j PB 2.3.6 Resolution	PB16	649	41%	73%	26503.31	0.64	40.84	26065.14	0.35	40.16	1.02

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 1600
Virtual Best Solver (VBS)			99	6%	100%	5222.26	0.11	52.75	-	-	-	-
1	Sat4j PB 2.3.6 Res+CP	PB16	89	6%	90%	6400.37	3.03	71.91	3175.95	1.37	35.68	2.02
2	cdcl-cuttingplanes OPT binary search	2016-05-01	89	6%	90%	6405.35	0.13	71.97	6406.47	0.13	71.98	1.00
3	cdcl-cuttingplanes OPT linear search	2016-05-01	89	6%	90%	6618.84	0.11	74.37	6619.98	0.11	74.38	1.00
4	NaPS	1.02	85	5%	86%	11245.46	4.12	132.30	11247.80	4.12	132.33	1.00
5	minisatp	2012-10-02 git-d91742b	76	5%	77%	7651.69	0.99	100.68	7653.09	1.01	100.70	1.00
6	Sat4j PB 2.3.6 Resolution	PB16	67	4%	68%	4380.60	1.15	65.38	4311.32	0.59	64.35	1.02
7	toysat	2016-05-02	63	4%	64%	2635.53	0.65	41.83	2636.17	0.65	41.84	1.00
8	Open-WBO-LSU	PB16	45	3%	45%	4340.36	0.11	96.45	4341.48	0.11	96.48	1.00
9	Open-WBO	PB16	45	3%	45%	4358.95	0.11	96.87	4359.78	0.11	96.88	1.00

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 506
Virtual Best Solver (VBS)			309	309 OPT	61%	100%	6466.61	0.06	20.93	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	306	306 OPT	60%	99%	7264.70	0.06	23.74	7266.16	0.06	23.75	1.00
2	Sat4j PB 2.3.6 Resolution	PB16	288	288 OPT	57%	93%	23792.71	2.81	82.61	23536.10	1.84	81.72	1.01
3	Sat4j PB 2.3.6 Res+CP	PB16	285	285 OPT	56%	92%	53038.66	7.78	186.10	26828.90	4.05	94.14	1.98
4	toysat	2016-05-02	207	207 OPT	41%	67%	27455.59	5.62	132.64	27462.76	5.63	132.67	1.00

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 506
Virtual Best Solver (VBS)			309	61%	100%	6466.61	0.06	20.93	-	-	-	-
1	minisatp	2012-10-02 git-d91742b	306	60%	99%	7264.70	0.06	23.74	7266.16	0.06	23.75	1.00
2	Sat4j PB 2.3.6 Resolution	PB16	288	57%	93%	23792.71	2.81	82.61	23536.10	1.84	81.72	1.01
3	Sat4j PB 2.3.6 Res+CP	PB16	285	56%	92%	53038.66	7.78	186.10	26828.90	4.05	94.14	1.98
4	toysat	2016-05-02	207	41%	67%	27455.59	5.62	132.64	27462.76	5.63	132.67	1.00

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 506
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	toysat	2016-05-02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	Sat4j PB 2.3.6 Res+CP	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	Sat4j PB 2.3.6 Resolution	PB16	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	minisatp	2012-10-02 git-d91742b	0	0%	-	0.00	0.00	-	0.00	0.00	-	-