CSP 2009 Competition: results by benchmark

Results by benchmark for category alldiff (Alldiff), subcategory real world problems

This page displays the results of the different solvers for each benchmark for category alldiff (Alldiff), subcategory real world problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC (complete)
Abscon 112v4 ESAC (complete)
bpsolver 09 (complete)
Choco2.1.1 2009-06-10 (complete)
Choco2.1.1b 2009-07-16 (complete)
Concrete 2009-07-14 (complete)
Concrete DC 2009-07-14 (complete)
Mistral 1.545 (complete)
SAT4J CSP 2.1.1 (complete)
Sugar v1.14.6+minisat (complete)
Sugar v1.14.6+picosat (complete)

Bench name	Best results on this instance	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/compet02/ normalized-patat-02-comp-10.xml	SAT TT=125.758	? (TO) TT=1800.14	? (TO) TT=1800.16	? (TO) TT=1800.08	? (TO) TT=1800.15	? (TO) TT=1800.14	? (TO) TT=1800.05	? (TO) TT=1800.02	? (TO) TT=1800.1	? (TO) TT=1800.12	? (TO) TT=1800.17	SAT TT=125.758
csp/compet02/ normalized-patat-02-comp-11.xml	SAT TT=65.374	? (TO) TT=1800.17	? (TO) TT=1800.11	? (TO) TT=1800.08	? (TO) TT=1800.16	? (TO) TT=1800.15	? (TO) TT=1800.03	? (TO) TT=1800.09	SAT TT=65.374	? (TO) TT=1800.29	? (TO) TT=1800.08	SAT TT=76.826
csp/compet02/ normalized-patat-02-comp-12.xml	SAT TT=43.561	? (TO) TT=1800.13	? (TO) TT=1800.23	? (TO) TT=1800.05	? (TO) TT=1800.15	? (TO) TT=1800.11	? (TO) TT=1800.04	? (TO) TT=1800.06	SAT TT=43.561	? (TO) TT=1800.33	? (TO) TT=1800.07	SAT TT=77.566
csp/compet02/ normalized-patat-02-comp-13.xml	SAT TT=83.048	SAT TT=1295.63	SAT TT=1450.94	? (TO) TT=1800.08	? (TO) TT=1800.14	? (TO) TT=1800.18	? (TO) TT=1800.05	? (TO) TT=1800.09	SAT TT=99.489	? (TO) TT=1800.36	? (TO) TT=1800.11	SAT TT=83.048
csp/compet02/ normalized-patat-02-comp-14.xml	SAT TT=104.91	? (TO) TT=1800.11	SAT TT=1601.53	? (TO) TT=1800.09	? (TO) TT=1800.17	? (TO) TT=1800.17	? (TO) TT=1800.08	? (TO) TT=1800.08	SAT TT=155.555	SAT TT=1757.28	SAT TT=464.562	SAT TT=104.91
csp/compet02/ normalized-patat-02-comp-15.xml	SAT TT=80.284	SAT TT=1112.88	SAT TT=1271.99	? (TO) TT=1800.09	? (TO) TT=1800.08	? (TO) TT=1800.18	? (TO) TT=1800.01	? (TO) TT=1800.08	SAT TT=91.885	SAT TT=719.119	SAT TT=377.563	SAT TT=80.284
csp/compet02/ normalized-patat-02-comp-16.xml	SAT TT=98.933	? (TO) TT=1800.17	? (TO) TT=1800.21	? (TO) TT=1800.07	? (TO) TT=1800.18	? (TO) TT=1800.16	? (TO) TT=1800.09	? (TO) TT=1800.01	SAT TT=207.516	? (TO) TT=1800.18	? (TO) TT=1800.12	SAT TT=98.933
csp/compet02/ normalized-patat-02-comp-17.xml	SAT TT=0.598	SAT TT=4.197	SAT TT=7.305	? (TO) TT=1800.06	SAT TT=118.625	SAT TT=101.647	? (TO) TT=1800.1	? (TO) TT=1800	SAT TT=0.598	SAT TT=54.785	SAT TT=21.468	SAT TT=54.657
csp/compet02/ normalized-patat-02-comp-18.xml	SAT TT=37.567	SAT TT=632.064	? (MO) TT=5.71	? (TO) TT=1800.08	? (TO) TT=1800.12	? (TO) TT=1800.14	? (TO) TT=1800.08	? (TO) TT=1800.09	SAT TT=37.567	? (TO) TT=1800.37	? (TO) TT=1800.01	SAT TT=56.474
csp/compet02/ normalized-patat-02-comp-19.xml	SAT TT=149.487	? (TO) TT=1800.14	? (TO) TT=1800.14	? (TO) TT=1800.07	? (TO) TT=1800.17	? (TO) TT=1800.15	? (TO) TT=1800.08	? (TO) TT=1800	SAT TT=297.758	? (TO) TT=1800.13	? (MO) TT=1661.21	SAT TT=149.487
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/compet02/ normalized-patat-02-comp-1.xml	SAT TT=68.035	? (TO) TT=1800.21	? (TO) TT=1800.09	? (TO) TT=1800.09	? (TO) TT=1800.14	? (TO) TT=1800.14	? (TO) TT=1800.04	? (TO) TT=1800.03	SAT TT=68.035	? (TO) TT=1800.34	SAT TT=908.51	SAT TT=81.832
csp/compet02/ normalized-patat-02-comp-20.xml	SAT TT=109.014	? (TO) TT=1800.16	? (TO) TT=1800.22	? (TO) TT=1800.08	? (TO) TT=1800.19	? (TO) TT=1800.18	? (TO) TT=1800.09	? (TO) TT=1800	SAT TT=141.142	? (TO) TT=1800.16	SAT TT=1074.25	SAT TT=109.014
csp/compet02/ normalized-patat-02-comp-2.xml	SAT TT=63.816	SAT TT=742.12	SAT TT=509.76	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.13	? (TO) TT=1800.01	? (TO) TT=1800.04	SAT TT=186.724	? (TO) TT=1800.34	SAT TT=660.532	SAT TT=63.816
csp/compet02/ normalized-patat-02-comp-3.xml	SAT TT=78.075	? (TO) TT=1800.23	? (TO) TT=1800.13	? (TO) TT=1800.05	? (TO) TT=1800.15	? (TO) TT=1800.09	? (TO) TT=1800.02	? (TO) TT=1800.06	SAT TT=135.683	? (TO) TT=1800.17	? (TO) TT=1800.07	SAT TT=78.075
csp/compet02/ normalized-patat-02-comp-4.xml	SAT TT=58.62	? (TO) TT=1800.21	? (TO) TT=1800.13	? (TO) TT=1800.06	? (TO) TT=1800.14	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800.08	SAT TT=58.62	? (TO) TT=1800.33	SAT TT=965.11	SAT TT=102.104
csp/compet02/ normalized-patat-02-comp-5.xml	SAT TT=10.612	SAT TT=466.811	SAT TT=507.381	? (TO) TT=1800.09	? (TO) TT=1800.15	? (TO) TT=1800.13	SAT TT=654.831	? (TO) TT=1800.08	SAT TT=10.612	SAT TT=374.735	SAT TT=24.266	SAT TT=61.587
csp/compet02/ normalized-patat-02-comp-6.xml	SAT TT=53.119	? (TO) TT=1800.15	? (TO) TT=1800.17	? (TO) TT=1800.07	? (TO) TT=1800.13	? (TO) TT=1800.13	? (TO) TT=1800.05	? (TO) TT=1800.1	SAT TT=53.119	? (TO) TT=1800.14	? (TO) TT=1800.21	SAT TT=111.098
csp/compet02/ normalized-patat-02-comp-7.xml	SAT TT=71.4	? (TO) TT=1800.13	? (TO) TT=1800.15	? (TO) TT=1800.1	? (TO) TT=1800.22	? (TO) TT=1800.08	? (TO) TT=1800.08	? (TO) TT=1800.02	SAT TT=71.4	SAT TT=1538.06	SAT TT=1546.22	SAT TT=102.008
csp/compet02/ normalized-patat-02-comp-8.xml	SAT TT=91.414	? (TO) TT=1800.13	? (TO) TT=1800.17	? (TO) TT=1800.05	? (TO) TT=1800.16	? (TO) TT=1800.21	? (TO) TT=1800.04	? (TO) TT=1800.01	SAT TT=91.414	? (TO) TT=1800.17	? (TO) TT=1800.19	SAT TT=108.54
csp/compet02/ normalized-patat-02-comp-9.xml	SAT TT=97.3	? (TO) TT=1800.16	? (TO) TT=1800.15	? (TO) TT=1800.03	? (TO) TT=1800.16	? (TO) TT=1800.18	? (TO) TT=1800.07	? (TO) TT=1800.07	SAT TT=97.3	? (TO) TT=1800.29	? (TO) TT=1800.11	SAT TT=98.371
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/compet08/ normalized-patat-08-comp-10.xml	SAT TT=1.078	SAT TT=38.603	SAT TT=11.772	? (TO) TT=1800.02	SAT TT=254.593	SAT TT=264.167	SAT TT=91.012	SAT TT=429.196	SAT TT=1.078	SAT TT=374.759	SAT TT=89.568	SAT TT=111.072
csp/compet08/ normalized-patat-08-comp-11.xml	SAT TT=7.884	SAT TT=31.213	SAT TT=33.79	? (TO) TT=1800.07	SAT TT=579.458	SAT TT=335.232	SAT TT=144.843	SAT TT=588.942	SAT TT=7.884	SAT TT=486.878	SAT TT=82.837	SAT TT=145.865
csp/compet08/ normalized-patat-08-comp-12.xml	SAT TT=23.135	SAT TT=347.934	SAT TT=244.406	? (TO) TT=1800.08	SAT TT=1492.89	? (TO) TT=1800.1	SAT TT=1735.75	? (TO) TT=1800	SAT TT=23.135	? (TO) TT=1800.15	SAT TT=303.713	SAT TT=142.876
csp/compet08/ normalized-patat-08-comp-13.xml	SAT TT=8.715	SAT TT=107.267	SAT TT=205.104	? (TO) TT=1800.11	SAT TT=817.547	SAT TT=1318.24	? (TO) TT=1800.05	? (TO) TT=1800.05	SAT TT=8.715	SAT TT=1644.39	SAT TT=91.398	SAT TT=131.588
csp/compet08/ normalized-patat-08-comp-14.xml	SAT TT=105.254	? (TO) TT=1800.19	? (TO) TT=1800.14	? (TO) TT=1800.05	? (TO) TT=1800.16	? (TO) TT=1800.17	? (TO) TT=1800.08	SAT TT=1754.57	SAT TT=478.394	SAT TT=840.116	? (TO) TT=1800.04	SAT TT=105.254
csp/compet08/ normalized-patat-08-comp-15.xml	SAT TT=0.407	SAT TT=4.357	SAT TT=10.12	? (TO) TT=1800.07	SAT TT=91.962	SAT TT=102.957	SAT TT=9.442	SAT TT=113.367	SAT TT=0.407	SAT TT=15.837	SAT TT=38.365	SAT TT=45.498
csp/compet08/ normalized-patat-08-comp-16.xml		? (TO) TT=1800.13	? (TO) TT=1800.1	? (TO) TT=1800.08	? (TO) TT=1800.21	? (TO) TT=1800.23	? (TO) TT=1800.07	? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.27	? TT=142.055	? TT=161.274
csp/compet08/ normalized-patat-08-comp-1.xml		? (TO) TT=1800.2	? (TO) TT=1800.16	? (TO) TT=1800.08	? (TO) TT=1800.12	? (TO) TT=1800.16	? (TO) TT=1800.06	? (TO) TT=1800.06	? (TO) TT=1800.1	? (TO) TT=1800.44	? TT=241.74	? TT=236.727
csp/compet08/ normalized-patat-08-comp-2.xml		? (TO) TT=1800.17	? (TO) TT=1800.19	? (TO) TT=1800.04	? (TO) TT=1800.16	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800.04	? (TO) TT=1800.04	? (TO) TT=1800.2	? (TO) TT=1800.05	? (TO) TT=1800.09
csp/compet08/ normalized-patat-08-comp-3.xml	SAT TT=7.293	? (TO) TT=1800.15	SAT TT=166.783	? (TO) TT=1800.03	SAT TT=923.378	SAT TT=361.278	SAT TT=150.379	SAT TT=450.495	SAT TT=7.293	SAT TT=150.531	SAT TT=416.45	SAT TT=138.968
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/compet08/ normalized-patat-08-comp-4.xml	SAT TT=132.468	? (TO) TT=1800.07	? (TO) TT=1800.12	? (TO) TT=1800.06	? (TO) TT=1800.11	SAT TT=478.49	SAT TT=248.053	SAT TT=805.396	SAT TT=132.468	? (TO) TT=1800.47	SAT TT=1765.38	SAT TT=1045.62
csp/compet08/ normalized-patat-08-comp-5.xml	SAT TT=328.151	? (TO) TT=1800.14	? (TO) TT=1800.18	? (TO) TT=1800.11	? (TO) TT=1800.18	? (TO) TT=1800.11	? (TO) TT=1800.03	? (TO) TT=1800.01	? (TO) TT=1800.1	? (TO) TT=1800.14	SAT TT=421.811	SAT TT=328.151
csp/compet08/ normalized-patat-08-comp-6.xml	SAT TT=116.03	? (TO) TT=1800.18	SAT TT=1318.21	? (TO) TT=1800.05	? (TO) TT=1800.1	? (TO) TT=1800.16	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800.12	? (TO) TT=1800.07	SAT TT=116.03	SAT TT=156.986
csp/compet08/ normalized-patat-08-comp-7.xml	SAT TT=5.033	SAT TT=26.91	SAT TT=272.302	? (TO) TT=1800.12	SAT TT=105.058	SAT TT=176.512	SAT TT=50.691	SAT TT=219.603	SAT TT=5.033	SAT TT=25.811	SAT TT=117.002	SAT TT=61.795
csp/compet08/ normalized-patat-08-comp-8.xml	SAT TT=0.495	SAT TT=6.005	SAT TT=5.315	? (TO) TT=1800.1	SAT TT=98.134	SAT TT=100.276	SAT TT=25.388	SAT TT=123.351	SAT TT=0.495	SAT TT=19.142	SAT TT=36.632	SAT TT=47.439
csp/compet08/ normalized-patat-08-comp-9.xml		? (TO) TT=1800.23	? (TO) TT=1800.15	? (TO) TT=1800.02	? (TO) TT=1800.12	? (TO) TT=1800.12	? (TO) TT=1800.09	? (TO) TT=1800.07	? (TO) TT=1800.05	? (TO) TT=1800.35	? TT=552.91	? TT=563.859
csp/medium/ normalized-patat-02-medium-1.xml	SAT TT=111.921	? (TO) TT=1800.16	? (TO) TT=1800.16	? (TO) TT=1800.04	? (TO) TT=1800.14	? (TO) TT=1800.19	? (TO) TT=1800.07	? (TO) TT=1800.05	SAT TT=111.921	? (TO) TT=1800.17	? (TO) TT=1800.15	SAT TT=118.562
csp/medium/ normalized-patat-02-medium-2.xml	SAT TT=130.19	? (TO) TT=1800.24	? (TO) TT=1800.15	? (TO) TT=1800.1	? (TO) TT=1800.16	? (TO) TT=1800.23	? (TO) TT=1800.05	? (TO) TT=1800.08	SAT TT=149.475	? (TO) TT=1800.15	? (TO) TT=1800.21	SAT TT=130.19
csp/medium/ normalized-patat-02-medium-3.xml	SAT TT=129.188	? (TO) TT=1800.18	? (TO) TT=1800.25	? (TO) TT=1800.09	? (TO) TT=1800.17	? (TO) TT=1800.23	? (TO) TT=1800.02	? (TO) TT=1800.06	SAT TT=330.821	? (TO) TT=1800.2	? (TO) TT=1800.26	SAT TT=129.188
csp/medium/ normalized-patat-02-medium-4.xml	SAT TT=85.367	? (TO) TT=1800.14	? (TO) TT=1800.18	? (TO) TT=1800.09	? (TO) TT=1800.2	? (TO) TT=1800.15	? (TO) TT=1800.06	? (TO) TT=1800.04	SAT TT=85.367	? (TO) TT=1800.14	? (TO) TT=1800.13	SAT TT=130.727
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/medium/ normalized-patat-02-medium-5.xml	SAT TT=96.766	? (TO) TT=1800.18	? (TO) TT=1800.19	? (TO) TT=1800.04	? (TO) TT=1800.16	? (TO) TT=1800.15	? (TO) TT=1800.01	? (TO) TT=1800.08	SAT TT=298.172	? (TO) TT=1800.21	? (TO) TT=1800.17	SAT TT=96.766
csp/small/ normalized-patat-02-small-1.xml	SAT TT=0.179	SAT TT=2.262	SAT TT=3.877	? (TO) TT=1800.09	SAT TT=23.178	SAT TT=25.585	SAT TT=53.199	SAT TT=767.024	SAT TT=0.179	SAT TT=39.981	SAT TT=3.347	SAT TT=7.111
csp/small/ normalized-patat-02-small-2.xml	SAT TT=0.128	SAT TT=1.984	SAT TT=2.116	SAT TT=0.425	SAT TT=30.319	SAT TT=26.348	SAT TT=4.937	SAT TT=80.303	SAT TT=0.128	SAT TT=31.86	SAT TT=3.609	SAT TT=8.232
csp/small/ normalized-patat-02-small-3.xml	SAT TT=0.183	SAT TT=2.644	SAT TT=3.444	? (TO) TT=1800.11	SAT TT=28.429	SAT TT=29.245	SAT TT=141.491	SAT TT=1609.8	SAT TT=0.183	SAT TT=65.186	SAT TT=3.839	SAT TT=7.708
csp/small/ normalized-patat-02-small-4.xml	SAT TT=0.243	? (TO) TT=1800.09	? (TO) TT=1800.13	? (TO) TT=1800.09	SAT TT=13.032	SAT TT=13.465	SAT TT=1180.73	SAT TT=710.101	SAT TT=0.243	SAT TT=131.706	SAT TT=4.316	SAT TT=7.553
csp/small/ normalized-patat-02-small-5.xml	SAT TT=0.214	SAT TT=25.976	SAT TT=27.13	? (TO) TT=1800.07	SAT TT=18.573	SAT TT=17.764	SAT TT=70.285	SAT TT=752.991	SAT TT=0.214	SAT TT=79.314	SAT TT=4.276	SAT TT=10.188

Statistics		Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
Number of times the solver is able to give the best known answer		17	19	1	14	14	14	13	39	18	25	42
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		0	0	0	0	0	0	0	27	0	1	14