2017 XCSP3 competition: main track (CSP and COP, sequential and parallel solvers): ranking of solvers
max CPU time for any solver=2400 s

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "optimization problem" (COP): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 438
Virtual Best Solver (VBS)			160	158 OPT, 2 UNSAT	37%	100%	77455.76	2.19	484.10	-	-	-	-
1	Mistral-2.0	2017-07-28	133	131 OPT, 2 UNSAT	30%	83%	17412.46	0.80	130.92	17425.17	0.89	131.02	1.00
2	choco-solver	4.0.5 par (2017-08-18)	132	130 OPT, 2 UNSAT	30%	83%	19951.90	15.34	151.15	2984.41	2.53	22.61	6.69
3	OscaR - Parallel with EPS	2017-08-22	127	125 OPT, 2 UNSAT	29%	79%	12621.04	15.22	99.38	3487.25	4.35	27.46	3.62
4	OscaR - Conflict Ordering	2017-07-26	125	123 OPT, 2 UNSAT	29%	78%	8132.19	4.88	65.06	7787.43	2.21	62.30	1.04
5	AbsCon-basic	2017-06-11	117	115 OPT, 2 UNSAT	27%	73%	14901.59	3.96	127.36	14670.59	1.99	125.39	1.02
6	choco-solver	4.0.5 seq (2017-08-18)	115	113 OPT, 2 UNSAT	26%	72%	12607.60	4.97	109.63	12273.09	2.49	106.72	1.03
7	OscaR - Hybrid	2017-07-26	109	107 OPT, 2 UNSAT	25%	68%	27940.08	3.60	256.33	27119.10	1.73	248.80	1.03
8	choco-solver	5a (2017-08-18)	106	104 OPT, 2 UNSAT	24%	66%	17076.33	8.35	161.10	16211.86	3.25	152.94	1.05
9	Concrete	3.4	103	101 OPT, 2 UNSAT	24%	64%	14878.95	15.50	144.46	13669.11	7.33	132.71	1.09
10	cosoco	1.12	98	96 OPT, 2 UNSAT	22%	61%	12003.19	0.98	122.48	12010.84	1.08	122.56	1.00
11	cosoco-sat	1.12	98	96 OPT, 2 UNSAT	22%	61%	12628.79	1.12	128.87	12633.37	1.15	128.91	1.00
12	sat4j-CSP	2017-07-05	77	75 OPT, 2 UNSAT	18%	48%	17799.49	55.04	231.16	17151.71	37.72	222.75	1.04
13	OscaR - ALNS	2017-07-26	2	2 UNSAT	0%	1%	4.34	2.17	2.17	2.75	1.37	1.37	1.58

Category "optimization problem" (COP): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 438
Virtual Best Solver (VBS)			158	36%	100%	77455.45	2.61	490.22	-	-	-	-
1	Mistral-2.0	2017-07-28	131	30%	83%	17348.31	0.80	132.43	17360.90	0.89	132.53	1.00
2	choco-solver	4.0.5 par (2017-08-18)	130	30%	82%	19942.96	15.46	153.41	2982.37	2.64	22.94	6.69
3	OscaR - Parallel with EPS	2017-08-22	125	29%	79%	12609.93	15.50	100.88	3484.09	4.52	27.87	3.62
4	OscaR - Conflict Ordering	2017-07-26	123	28%	78%	8128.05	4.88	66.08	7785.37	2.33	63.30	1.04
5	AbsCon-basic	2017-06-11	115	26%	73%	14898.75	4.02	129.55	14668.92	2.04	127.56	1.02
6	choco-solver	4.0.5 seq (2017-08-18)	113	26%	72%	12602.86	5.11	111.53	12270.81	2.53	108.59	1.03
7	OscaR - Hybrid	2017-07-26	107	24%	68%	27935.95	3.70	261.08	27117.04	1.77	253.43	1.03
8	choco-solver	5a (2017-08-18)	104	24%	66%	17067.37	9.07	164.11	16208.35	5.24	155.85	1.05
9	Concrete	3.4	101	23%	64%	14861.12	15.92	147.14	13660.00	7.47	135.25	1.09
10	cosoco	1.12	96	22%	61%	12002.88	1.10	125.03	12010.39	1.18	125.11	1.00
11	cosoco-sat	1.12	96	22%	61%	12628.45	1.24	131.55	12632.89	1.25	131.59	1.00
12	sat4j-CSP	2017-07-05	75	17%	47%	17795.39	56.91	237.27	17149.03	39.82	228.65	1.04
13	OscaR - ALNS	2017-07-26	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimization problem" (COP): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 438
Virtual Best Solver (VBS)			2	0%	100%	0.31	0.15	0.15	-	-	-	-
1	cosoco	1.12	2	0%	100%	0.31	0.15	0.15	0.45	0.22	0.22	0.69
2	cosoco-sat	1.12	2	0%	100%	0.34	0.17	0.17	0.48	0.24	0.24	0.72
3	AbsCon-basic	2017-06-11	2	0%	100%	2.84	1.42	1.42	1.67	0.83	0.83	1.70
4	sat4j-CSP	2017-07-05	2	0%	100%	4.10	2.05	2.05	2.67	1.34	1.34	1.53
5	OscaR - Hybrid	2017-07-26	2	0%	100%	4.13	2.06	2.06	2.06	1.03	1.03	2.00
6	OscaR - Conflict Ordering	2017-07-26	2	0%	100%	4.14	2.07	2.07	2.07	1.03	1.03	2.00
7	OscaR - ALNS	2017-07-26	2	0%	100%	4.34	2.17	2.17	2.75	1.37	1.37	1.58
8	choco-solver	4.0.5 seq (2017-08-18)	2	0%	100%	4.74	2.37	2.37	2.28	1.14	1.14	2.08
9	choco-solver	4.0.5 par (2017-08-18)	2	0%	100%	8.95	4.47	4.47	2.04	1.02	1.02	4.39
10	choco-solver	5a (2017-08-18)	2	0%	100%	8.96	4.48	4.48	3.51	1.75	1.75	2.55
11	OscaR - Parallel with EPS	2017-08-22	2	0%	100%	11.11	5.56	5.56	3.16	1.58	1.58	3.52
12	Concrete	3.4	2	0%	100%	17.83	8.92	8.92	9.11	4.56	4.56	1.96
13	Mistral-2.0	2017-07-28	2	0%	100%	64.15	32.07	32.07	64.27	32.14	32.14	1.00

Category "decision problem" (CSP): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 510
Virtual Best Solver (VBS)			420	297 SAT, 123 UNSAT	82%	100%	45134.66	0.59	107.46	-	-	-	-
1	choco-solver	4.0.5 par (2017-08-18)	385	280 SAT, 105 UNSAT	75%	92%	41306.84	10.80	107.29	10359.64	2.49	26.91	3.99
2	choco-solver	4.0.5 seq (2017-08-18)	372	264 SAT, 108 UNSAT	73%	89%	16902.96	3.57	45.44	15988.97	1.70	42.98	1.06
3	choco-solver	5a (2017-08-18)	371	262 SAT, 109 UNSAT	73%	88%	31159.54	5.04	83.99	29250.56	2.31	78.84	1.07
4	AbsCon-basic	2017-06-11	368	266 SAT, 102 UNSAT	72%	88%	12834.58	3.29	34.88	12058.74	1.51	32.77	1.06
5	Mistral-2.0	2017-07-28	364	273 SAT, 91 UNSAT	71%	87%	22447.19	0.87	61.67	22480.49	0.97	61.76	1.00
6	OscaR - Conflict Ordering	2017-07-26	336	234 SAT, 102 UNSAT	66%	80%	22237.89	3.37	66.18	20926.89	1.73	62.28	1.06
7	cosoco	1.12	335	238 SAT, 97 UNSAT	66%	80%	21275.49	0.67	63.51	21316.72	0.88	63.63	1.00
8	Concrete	3.4	331	238 SAT, 93 UNSAT	65%	79%	26725.11	13.83	80.74	23039.25	6.29	69.60	1.16
9	OscaR - Parallel with EPS	2017-08-22	319	220 SAT, 99 UNSAT	63%	76%	31777.91	11.42	99.62	11230.40	3.62	35.21	2.83
10	BTD	2017-08-10	241	159 SAT, 82 UNSAT	47%	57%	15196.25	1.05	63.05	15224.15	1.17	63.17	1.00
11	sat4j-CSP	2017-07-05	235	168 SAT, 67 UNSAT	46%	56%	45876.36	7.99	195.22	43974.74	4.51	187.13	1.04

Category "decision problem" (CSP): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 510
Virtual Best Solver (VBS)			297	58%	100%	36127.21	0.46	121.64	-	-	-	-
1	choco-solver	4.0.5 par (2017-08-18)	280	55%	94%	23640.82	9.15	84.43	6045.01	1.99	21.59	3.91
2	Mistral-2.0	2017-07-28	273	54%	92%	11042.40	0.49	40.45	11067.50	0.57	40.54	1.00
3	AbsCon-basic	2017-06-11	266	52%	90%	7065.19	3.05	26.56	6515.63	1.39	24.49	1.08
4	choco-solver	4.0.5 seq (2017-08-18)	264	52%	89%	11827.54	2.70	44.80	11178.28	1.28	42.34	1.06
5	choco-solver	5a (2017-08-18)	262	51%	88%	21077.56	4.24	80.45	19809.56	1.84	75.61	1.06
6	cosoco	1.12	238	47%	80%	11239.30	0.41	47.22	11271.84	0.55	47.36	1.00
7	Concrete	3.4	238	47%	80%	17406.79	12.72	73.14	14962.83	5.73	62.87	1.16
8	OscaR - Conflict Ordering	2017-07-26	234	46%	79%	11574.63	2.90	49.46	10755.19	1.52	45.96	1.08
9	OscaR - Parallel with EPS	2017-08-22	220	43%	74%	16621.41	10.27	75.55	6610.50	3.41	30.05	2.51
10	sat4j-CSP	2017-07-05	168	33%	57%	31639.50	7.60	188.33	30671.52	4.24	182.57	1.03
11	BTD	2017-08-10	159	31%	54%	10633.32	0.86	66.88	10650.81	0.97	66.99	1.00

Category "decision problem" (CSP): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 510
Virtual Best Solver (VBS)			123	24%	100%	9007.45	1.24	73.23	-	-	-	-
1	choco-solver	5a (2017-08-18)	109	21%	89%	10081.98	7.71	92.50	9441.00	3.55	86.61	1.07
2	choco-solver	4.0.5 seq (2017-08-18)	108	21%	88%	5075.42	5.83	46.99	4810.69	3.07	44.54	1.06
3	choco-solver	4.0.5 par (2017-08-18)	105	21%	85%	17666.02	22.23	168.25	4314.63	6.51	41.09	4.09
4	AbsCon-basic	2017-06-11	102	20%	83%	5769.39	3.72	56.56	5543.11	1.94	54.34	1.04
5	OscaR - Conflict Ordering	2017-07-26	102	20%	83%	10663.26	4.23	104.54	10171.71	2.56	99.72	1.05
6	OscaR - Parallel with EPS	2017-08-22	99	19%	80%	15156.49	14.00	153.10	4619.90	4.11	46.67	3.28
7	cosoco	1.12	97	19%	79%	10036.18	1.45	103.47	10044.88	1.52	103.56	1.00
8	Concrete	3.4	93	18%	76%	9318.31	15.44	100.20	8076.42	6.92	86.84	1.15
9	Mistral-2.0	2017-07-28	91	18%	74%	11404.79	2.46	125.33	11412.99	2.55	125.42	1.00
10	BTD	2017-08-10	82	16%	67%	4562.93	1.17	55.65	4573.34	1.51	55.77	1.00
11	sat4j-CSP	2017-07-05	67	13%	54%	14236.86	9.92	212.49	13303.21	5.20	198.56	1.07