CSP 2009 Competition: results by benchmark

Results by benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory problems with patterns

This page displays the results of the different solvers for each benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory problems with patterns

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC (complete)
Abscon 112v4 ESAC (complete)
bpsolver 09 (complete)
Choco2.1.1 2009-06-10 (complete)
Choco2.1.1b 2009-07-16 (complete)
Concrete 2009-07-14 (complete)
Concrete DC 2009-07-14 (complete)
Conquer 2009-07-10 (complete)
Mistral 1.545 (complete)
pcs 0.3.2 (complete)
pcs-restart 0.3.2 (complete)
SAT4J CSP 2.1.1 (complete)
Sugar v1.14.6+minisat (complete)
Sugar v1.14.6+picosat (complete)

Bench name	Best results on this instance	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/travellingSalesman-20/ normalized-tsp-20-142_ext.xml	SAT TT=10.303	SAT TT=10.303	SAT TT=22.035	SAT TT=117.308	SAT TT=249.163	SAT TT=101.385	SAT TT=19.998	SAT TT=183.233	SAT TT=21.236	SAT TT=37.492	SAT TT=1037.89	? (TO) TT=1800.05	? TT=891.293	SAT TT=198.94	SAT TT=476.921
csp/travellingSalesman-20/ normalized-tsp-20-190_ext.xml	SAT TT=1.836	SAT TT=3.423	SAT TT=4.163	SAT TT=16.366	SAT TT=12.326	SAT TT=11.027	SAT TT=3.136	SAT TT=84.753	SAT TT=1.836	SAT TT=2.952	SAT TT=1369.56	? (TO) TT=1800.06	? (TO) TT=1800.29	SAT TT=75.083	SAT TT=92.687
csp/travellingSalesman-20/ normalized-tsp-20-193_ext.xml	SAT TT=2.499	SAT TT=58.703	SAT TT=2.499	SAT TT=12.53	SAT TT=161.239	SAT TT=87.758	SAT TT=14.377	SAT TT=599.454	SAT TT=17.476	SAT TT=4.911	SAT TT=68.769	SAT TT=338.663	? TT=389.14	SAT TT=201.082	SAT TT=214.174
csp/travellingSalesman-20/ normalized-tsp-20-1_ext.xml	SAT TT=0.179	SAT TT=1.201	SAT TT=1.695	SAT TT=2.16	SAT TT=6.333	SAT TT=6.391	SAT TT=1.605	SAT TT=30.248	SAT TT=1.727	SAT TT=0.179	SAT TT=28.693	SAT TT=60.103	? (TO) TT=1802.14	SAT TT=48.785	SAT TT=80.494
csp/travellingSalesman-20/ normalized-tsp-20-29_ext.xml	SAT TT=0.083	SAT TT=1.052	SAT TT=1.358	SAT TT=0.774	SAT TT=3.685	SAT TT=4.038	SAT TT=1.427	SAT TT=15.117	SAT TT=1.164	SAT TT=0.083	SAT TT=6.016	SAT TT=6.51	? (TO) TT=1800.34	SAT TT=29.714	SAT TT=48.829
csp/travellingSalesman-20/ normalized-tsp-20-2_ext.xml	SAT TT=1.276	SAT TT=2.671	SAT TT=4.923	SAT TT=10.38	SAT TT=22.399	SAT TT=49.318	SAT TT=3.477	SAT TT=96.733	SAT TT=3.879	SAT TT=1.276	SAT TT=840.96	SAT TT=1369.8	? TT=391.214	SAT TT=73.816	SAT TT=172.067
csp/travellingSalesman-20/ normalized-tsp-20-30_ext.xml	SAT TT=0.135	SAT TT=1.046	SAT TT=1.475	SAT TT=1.169	SAT TT=11.319	SAT TT=11.311	SAT TT=1.493	SAT TT=81.193	SAT TT=1.458	SAT TT=0.135	SAT TT=17.995	SAT TT=0.992	? (TO) TT=1800.18	SAT TT=43.716	SAT TT=71.253
csp/travellingSalesman-20/ normalized-tsp-20-366_ext.xml	SAT TT=3.291	SAT TT=14.996	SAT TT=11.385	SAT TT=92.343	SAT TT=47.488	SAT TT=20.059	SAT TT=17.641	SAT TT=131.122	SAT TT=17.229	SAT TT=3.291	SAT TT=756.075	? (TO) TT=1800.12	? TT=364.68	SAT TT=87.116	SAT TT=128.892
csp/travellingSalesman-20/ normalized-tsp-20-453_ext.xml	SAT TT=4.482	SAT TT=26.009	SAT TT=12.868	SAT TT=9.741	SAT TT=145.947	SAT TT=132.119	SAT TT=26.072	SAT TT=454.054	SAT TT=15.469	SAT TT=4.482	? (TO) TT=1800.03	SAT TT=1680.56	? TT=806.18	SAT TT=104.888	SAT TT=413.55
csp/travellingSalesman-20/ normalized-tsp-20-68_ext.xml	SAT TT=0.069	SAT TT=0.912	SAT TT=0.893	SAT TT=0.4	SAT TT=0.916	SAT TT=0.923	SAT TT=1.264	SAT TT=4.907	SAT TT=0.945	SAT TT=0.069	SAT TT=0.175	SAT TT=0.212	SAT TT=610.146	SAT TT=16.564	SAT TT=20.209
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/travellingSalesman-20/ normalized-tsp-20-727_ext.xml	SAT TT=1.571	SAT TT=1.571	SAT TT=1.575	SAT TT=23.886	SAT TT=20.562	SAT TT=17.751	SAT TT=4.361	SAT TT=141.909	SAT TT=3.368	SAT TT=2.603	SAT TT=377.647	SAT TT=504.243	? TT=402.033	SAT TT=152.25	SAT TT=297.915
csp/travellingSalesman-20/ normalized-tsp-20-75_ext.xml	SAT TT=3.662	SAT TT=3.662	SAT TT=4.507	SAT TT=74.611	SAT TT=44.502	SAT TT=113.714	SAT TT=11.637	SAT TT=169.674	SAT TT=5.846	SAT TT=9.682	SAT TT=108.906	? (TO) TT=1800.03	? TT=384.073	SAT TT=224.018	SAT TT=257.289
csp/travellingSalesman-20/ normalized-tsp-20-76_ext.xml	SAT TT=2.379	SAT TT=2.664	SAT TT=2.379	SAT TT=7.997	SAT TT=34.875	SAT TT=36.212	SAT TT=7.124	SAT TT=171.934	SAT TT=12.293	SAT TT=5.192	? (TO) TT=1800.1	SAT TT=1745.23	? TT=382.392	SAT TT=91.385	SAT TT=438.906
csp/travellingSalesman-20/ normalized-tsp-20-8_ext.xml	SAT TT=0.084	SAT TT=1.187	SAT TT=1.314	SAT TT=0.655	SAT TT=5.831	SAT TT=6.063	SAT TT=1.375	SAT TT=60.879	SAT TT=1.215	SAT TT=0.084	SAT TT=4.903	SAT TT=5.873	? (TO) TT=1800.42	SAT TT=30.96	SAT TT=57.651
csp/travellingSalesman-20/ normalized-tsp-20-901_ext.xml	SAT TT=6.576	SAT TT=6.576	SAT TT=6.879	SAT TT=35.726	SAT TT=32.97	SAT TT=54.284	SAT TT=10.465	SAT TT=185.894	SAT TT=6.761	SAT TT=7.512	SAT TT=687.948	SAT TT=1630.06	? TT=390.408	SAT TT=145.767	SAT TT=169.978
csp/travellingSalesman-25/ normalized-tsp-25-13_ext.xml	SAT TT=1.294	SAT TT=2.15	SAT TT=2.719	SAT TT=2.551	SAT TT=10.437	SAT TT=10.073	SAT TT=3.012	SAT TT=128.114	SAT TT=3.63	SAT TT=1.294	SAT TT=118.36	SAT TT=89.098	? (TO) TT=1801.56	SAT TT=88.51	SAT TT=202.446
csp/travellingSalesman-25/ normalized-tsp-25-163_ext.xml	SAT TT=83.444	SAT TT=222.517	SAT TT=233.95	? (TO) TT=1800.09	SAT TT=577.974	SAT TT=1439.36	SAT TT=141.237	SAT TT=441.068	SAT TT=83.444	SAT TT=262.921	? (TO) TT=1800.1	? (TO) TT=1800.1	? TT=380.62	SAT TT=1055.84	SAT TT=693.341
csp/travellingSalesman-25/ normalized-tsp-25-190_ext.xml	SAT TT=54.107	SAT TT=85.065	SAT TT=84.323	SAT TT=690.61	SAT TT=959.021	SAT TT=316.185	SAT TT=148.847	? (TO) TT=1800.02	SAT TT=99.125	SAT TT=54.107	? (TO) TT=1800.08	? (TO) TT=1800.1	? TT=335.172	SAT TT=1135.03	SAT TT=954.815
csp/travellingSalesman-25/ normalized-tsp-25-312_ext.xml	SAT TT=28.593	SAT TT=33.431	SAT TT=28.593	SAT TT=86.414	SAT TT=377.018	SAT TT=549.706	SAT TT=130.622	SAT TT=562.987	SAT TT=72.81	SAT TT=35.13	? (TO) TT=1800.04	? (TO) TT=1800.03	? TT=864.151	SAT TT=934.008	SAT TT=578.865
csp/travellingSalesman-25/ normalized-tsp-25-38_ext.xml	SAT TT=1.122	SAT TT=4.693	SAT TT=2.006	SAT TT=10.724	SAT TT=25.318	SAT TT=32.378	SAT TT=4.682	SAT TT=159.391	SAT TT=4.288	SAT TT=1.122	SAT TT=947.458	SAT TT=448.913	? TT=647.558	SAT TT=105.132	SAT TT=104.501
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/travellingSalesman-25/ normalized-tsp-25-3_ext.xml	SAT TT=0.534	SAT TT=2.81	SAT TT=2.914	SAT TT=34.938	SAT TT=21.911	SAT TT=23.186	SAT TT=2.328	SAT TT=168.801	SAT TT=7.899	SAT TT=0.534	SAT TT=1566.14	SAT TT=1194.75	? TT=873.846	SAT TT=104.296	SAT TT=66.584
csp/travellingSalesman-25/ normalized-tsp-25-456_ext.xml	SAT TT=11.577	SAT TT=487.122	SAT TT=119.786	SAT TT=20.299	SAT TT=134.962	SAT TT=261.594	SAT TT=354.96	SAT TT=1437.31	SAT TT=800.802	SAT TT=11.577	? (TO) TT=1800.09	? (TO) TT=1800.09	? TT=347.314	SAT TT=800.13	SAT TT=486.561
csp/travellingSalesman-25/ normalized-tsp-25-48_ext.xml	SAT TT=96.186	SAT TT=141.565	SAT TT=302.107	SAT TT=384.29	? (TO) TT=1800.14	SAT TT=525.874	SAT TT=140.364	SAT TT=1090.97	SAT TT=243.675	SAT TT=96.186	? (TO) TT=1800.03	? (TO) TT=1800.05	? TT=915.068	SAT TT=536.363	SAT TT=798.392
csp/travellingSalesman-25/ normalized-tsp-25-523_ext.xml	SAT TT=39.909	SAT TT=44.849	SAT TT=114.491	SAT TT=136.964	SAT TT=809.003	SAT TT=97.356	SAT TT=39.909	SAT TT=937.808	SAT TT=118.17	SAT TT=42.419	? (TO) TT=1800.09	? (TO) TT=1800.12	? (TO) TT=1802.14	SAT TT=1735.42	SAT TT=894.903
csp/travellingSalesman-25/ normalized-tsp-25-54_ext.xml	SAT TT=0.55	SAT TT=1.475	SAT TT=1.796	SAT TT=4.624	SAT TT=7.499	SAT TT=7.596	SAT TT=1.966	SAT TT=34.915	SAT TT=1.56	SAT TT=0.55	SAT TT=127.305	SAT TT=76.73	? TT=447.108	SAT TT=68.277	SAT TT=71.185
csp/travellingSalesman-25/ normalized-tsp-25-561_ext.xml	SAT TT=4.018	SAT TT=4.018	SAT TT=22.821	SAT TT=5.797	SAT TT=97.346	SAT TT=410.937	SAT TT=6.669	SAT TT=405.147	SAT TT=9.101	SAT TT=19.689	SAT TT=464.375	SAT TT=496.897	? TT=372.06	SAT TT=323.232	SAT TT=226.942
csp/travellingSalesman-25/ normalized-tsp-25-66_ext.xml	SAT TT=0.527	SAT TT=1.933	SAT TT=1.399	SAT TT=8.091	SAT TT=15.692	SAT TT=12.167	SAT TT=2.757	SAT TT=176.369	SAT TT=1.282	SAT TT=0.527	SAT TT=39.758	SAT TT=13.788	? (TO) TT=1800.33	SAT TT=34.1	SAT TT=64.065
csp/travellingSalesman-25/ normalized-tsp-25-681_ext.xml	SAT TT=14.696	SAT TT=44.267	SAT TT=79.212	SAT TT=159.916	SAT TT=456.188	SAT TT=242.859	SAT TT=25.59	SAT TT=593.01	SAT TT=14.696	SAT TT=31.019	? (TO) TT=1800.12	? (TO) TT=1800.02	? (TO) TT=1802.14	SAT TT=512.835	SAT TT=1318.49
csp/travellingSalesman-25/ normalized-tsp-25-715_ext.xml	SAT TT=141.74	SAT TT=442.018	SAT TT=272.574	SAT TT=649.608	SAT TT=1476.57	? (TO) TT=1800.07	SAT TT=655.785	SAT TT=662.135	SAT TT=1296.39	SAT TT=141.74	? (TO) TT=1800.12	? (TO) TT=1800.12	? TT=381.407	SAT TT=1107.35	SAT TT=509.352
csp/travellingSalesman-25/ normalized-tsp-25-843_ext.xml	SAT TT=6.134	SAT TT=28.844	SAT TT=33.697	SAT TT=47.926	SAT TT=72.452	SAT TT=382.678	SAT TT=73.793	SAT TT=385.041	SAT TT=6.134	SAT TT=15.3	? (TO) TT=1800.05	? (TO) TT=1800.05	? TT=354.737	SAT TT=204.632	SAT TT=131.031

Statistics		Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
Number of times the solver is able to give the best known answer		30	30	29	29	29	30	29	30	30	19	17	1	30	30
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		5	3	0	0	0	1	0	4	17	0	0	0	0	0