CSP 2009 Competition: results by benchmark

Results by benchmark for category binary constraints in intension (2-ARY-INT), subcategory academic problems

This page displays the results of the different solvers for each benchmark for category binary constraints in intension (2-ARY-INT), subcategory academic problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC (complete)
Abscon 112v4 ESAC (complete)
bpsolver 09 (complete)
Choco2.1.1 2009-06-10 (complete)
Choco2.1.1b 2009-07-16 (complete)
Concrete 2009-07-14 (complete)
Concrete DC 2009-07-14 (complete)
Mistral 1.545 (complete)
pcs 0.3.2 (complete)
pcs-restart 0.3.2 (complete)
SAT4J CSP 2.1.1 (complete)
Sugar v1.14.6+minisat (complete)
Sugar v1.14.6+picosat (complete)

Bench name	Best results on this instance	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/domino/ normalized-domino-1000-1000.xml	SAT TT=0.046	SAT TT=15.337	SAT TT=15.394	SAT TT=0.27	SAT TT=1.104	SAT TT=1.033	SAT TT=98.697	SAT TT=98.234	SAT TT=0.046	SAT TT=7.696	SAT TT=7.793	SAT TT=837.365	SAT TT=11.786	SAT TT=27.419
csp/domino/ normalized-domino-1000-100.xml	SAT TT=0.044	SAT TT=1.868	SAT TT=2.034	SAT TT=0.264	SAT TT=1.343	SAT TT=1.113	SAT TT=7.294	SAT TT=7.459	SAT TT=0.044	SAT TT=0.832	SAT TT=0.258	SAT TT=13.278	SAT TT=2.141	SAT TT=4.293
csp/domino/ normalized-domino-1000-500.xml	SAT TT=0.047	SAT TT=5.831	SAT TT=5.49	SAT TT=0.249	SAT TT=1.097	SAT TT=1.28	SAT TT=28.784	SAT TT=28.982	SAT TT=0.047	SAT TT=30.859	SAT TT=31.76	SAT TT=197.243	SAT TT=5.991	SAT TT=15.333
csp/domino/ normalized-domino-100-100.xml	SAT TT=0.008	SAT TT=0.934	SAT TT=0.844	SAT TT=0.041	SAT TT=0.625	SAT TT=0.627	SAT TT=1.47	SAT TT=1.439	SAT TT=0.008	SAT TT=0.158	SAT TT=0.163	SAT TT=3.514	SAT TT=0.855	SAT TT=0.991
csp/domino/ normalized-domino-100-200.xml	SAT TT=0.023	SAT TT=0.943	SAT TT=0.956	SAT TT=0.043	SAT TT=0.632	SAT TT=0.654	SAT TT=1.761	SAT TT=1.747	SAT TT=0.023	SAT TT=0.435	SAT TT=0.431	SAT TT=7.224	SAT TT=1.033	SAT TT=1.235
csp/domino/ normalized-domino-100-300.xml	SAT TT=0.009	SAT TT=1.07	SAT TT=1.108	SAT TT=0.043	SAT TT=0.569	SAT TT=0.596	SAT TT=2.116	SAT TT=2.194	SAT TT=0.009	SAT TT=0.859	SAT TT=0.85	SAT TT=10.971	SAT TT=0.969	SAT TT=1.78
csp/domino/ normalized-domino-2000-2000.xml	SAT TT=0.09	SAT TT=97.57	SAT TT=97.049	SAT TT=0.583	SAT TT=1.516	SAT TT=1.397	? (exit code) TT=812.163	? (exit code) TT=824.877	SAT TT=0.09	SAT TT=914.083	SAT TT=7.832	? TT=69.55	SAT TT=44.267	SAT TT=110.18
csp/domino/ normalized-domino-300-100.xml	SAT TT=0.017	SAT TT=1.077	SAT TT=1.091	SAT TT=0.086	SAT TT=0.858	SAT TT=0.834	SAT TT=2.188	SAT TT=2.195	SAT TT=0.017	SAT TT=0.091	SAT TT=0.09	SAT TT=5.398	SAT TT=1.246	SAT TT=1.874
csp/domino/ normalized-domino-300-200.xml	SAT TT=0.018	SAT TT=1.316	SAT TT=1.325	SAT TT=0.087	SAT TT=0.915	SAT TT=0.837	SAT TT=2.773	SAT TT=3.038	SAT TT=0.018	SAT TT=1.459	SAT TT=1.451	SAT TT=13.162	SAT TT=1.716	SAT TT=2.884
csp/domino/ normalized-domino-500-300.xml	SAT TT=0.028	SAT TT=2.35	SAT TT=2.368	SAT TT=0.136	SAT TT=1.087	SAT TT=0.919	SAT TT=6.758	SAT TT=6.916	SAT TT=0.028	SAT TT=0.039	SAT TT=6.391	SAT TT=39.707	SAT TT=2.641	SAT TT=5.195
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/knights/ normalized-knights-100-9.xml	UNSAT TT=84.919	? (TO) TT=1800.1	? (TO) TT=1800.05	? (TO) TT=1800.06	? (TO) TT=1800.13	? (TO) TT=1800.1	? (TO) TT=1800.02	UNSAT TT=1707.42	UNSAT TT=84.919	? (TO) TT=1800.12	? (TO) TT=1800.12	? (exit code) TT=1227.81	? (TO) TT=1800	? (TO) TT=1800.02
csp/knights/ normalized-knights-12-5.xml	UNSAT TT=0.781	UNSAT TT=0.839	UNSAT TT=0.867	? (problem) TT=0.03	UNSAT TT=0.899	UNSAT TT=0.921	UNSAT TT=1.211	UNSAT TT=1.185	UNSAT TT=2.576	UNSAT TT=1666.95	UNSAT TT=1722.34	UNSAT TT=4.408	UNSAT TT=0.805	UNSAT TT=0.781
csp/knights/ normalized-knights-12-9.xml	UNSAT TT=0.946	UNSAT TT=0.946	UNSAT TT=1.009	? (problem) TT=0.034	UNSAT TT=2.188	UNSAT TT=1.979	UNSAT TT=1.357	UNSAT TT=1.332	UNSAT TT=10.083	? (TO) TT=1800.14	? (TO) TT=1800.13	UNSAT TT=6.451	UNSAT TT=59.194	UNSAT TT=167.171
csp/knights/ normalized-knights-15-5.xml	UNSAT TT=0.73	UNSAT TT=0.984	UNSAT TT=1.078	? (problem) TT=0.027	UNSAT TT=1.211	UNSAT TT=1.337	UNSAT TT=1.25	UNSAT TT=1.185	UNSAT TT=9.812	? (TO) TT=1800.1	? (TO) TT=1800.16	UNSAT TT=7.013	UNSAT TT=0.73	UNSAT TT=0.919
csp/knights/ normalized-knights-20-5.xml	UNSAT TT=0.788	UNSAT TT=1.298	UNSAT TT=1.375	? (problem) TT=0.03	UNSAT TT=61.098	UNSAT TT=58.336	UNSAT TT=1.608	UNSAT TT=1.518	UNSAT TT=57.275	? (TO) TT=1800.15	? (TO) TT=1800.1	UNSAT TT=16.276	UNSAT TT=0.788	UNSAT TT=1.07
csp/knights/ normalized-knights-25-9.xml	UNSAT TT=2.35	UNSAT TT=2.35	UNSAT TT=2.756	? (problem) TT=0.034	? (TO) TT=1800.13	? (TO) TT=1800.09	UNSAT TT=3.499	UNSAT TT=2.983	UNSAT TT=734.08	? (TO) TT=1800.06	? (TO) TT=1800.1	UNSAT TT=91.469	UNSAT TT=607.788	? (TO) TT=1800.08
csp/knights/ normalized-knights-50-25.xml	UNSAT TT=70.099	UNSAT TT=75.675	UNSAT TT=70.099	? (problem) TT=0.069	? (TO) TT=1800.1	? (TO) TT=1800.09	UNSAT TT=202.918	UNSAT TT=140.563	? (TO) TT=1800.04	? (TO) TT=1800.04	? (TO) TT=1800.06	? TT=196.6	? (TO) TT=1800.03	? (TO) TT=1800.08
csp/knights/ normalized-knights-50-5.xml	UNSAT TT=2.3	UNSAT TT=20.212	UNSAT TT=26.071	? (problem) TT=0.035	? (TO) TT=1800.05	? (TO) TT=1800.14	UNSAT TT=23.731	UNSAT TT=21.065	? (TO) TT=1800.08	? (TO) TT=1800.08	? (TO) TT=1800.12	UNSAT TT=810.552	UNSAT TT=2.3	UNSAT TT=4.647
csp/knights/ normalized-knights-50-9.xml	UNSAT TT=30.272	UNSAT TT=30.272	UNSAT TT=38.822	? (problem) TT=0.036	? (TO) TT=1800.06	? (TO) TT=1800.1	UNSAT TT=53.376	UNSAT TT=37.154	? (TO) TT=1800.08	? (TO) TT=1800.1	? (TO) TT=1800.09	? (exit code) TT=337.537	? (TO) TT=1800	? (TO) TT=1800.07
csp/knights/ normalized-knights-80-9.xml	UNSAT TT=22.184	? (TO) TT=1800.14	? (TO) TT=1800.06	? (TO) TT=1800.02	? (TO) TT=1800.05	? (TO) TT=1800.14	UNSAT TT=610.195	UNSAT TT=520.722	UNSAT TT=22.184	? (TO) TT=1800.06	? (TO) TT=1800.08	? TT=524.917	? (TO) TT=1800.06	? (TO) TT=1800.08
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/langford2/ normalized-langford-2-10.xml	UNSAT TT=0.888	UNSAT TT=7.294	UNSAT TT=7.241	UNSAT TT=0.888	UNSAT TT=5.859	UNSAT TT=7.242	UNSAT TT=8.664	UNSAT TT=5.89	UNSAT TT=1.284	UNSAT TT=44.531	UNSAT TT=40.959	UNSAT TT=399.757	UNSAT TT=12.052	UNSAT TT=19.481
csp/langford2/ normalized-langford-2-12.xml	SAT TT=0.015	SAT TT=1.107	SAT TT=0.969	SAT TT=0.064	SAT TT=0.875	SAT TT=0.862	SAT TT=1.573	SAT TT=1.932	SAT TT=0.015	SAT TT=0.031	SAT TT=0.028	SAT TT=3.303	SAT TT=0.915	SAT TT=0.964
csp/langford2/ normalized-langford-2-13.xml	UNSAT TT=268.404	? (TO) TT=1800.12	? (TO) TT=1800.12	UNSAT TT=268.404	UNSAT TT=1617.3	UNSAT TT=1520.66	? (TO) TT=1800.1	? (TO) TT=1800.06	UNSAT TT=402.502	? (MO) TT=411.742	? (MO) TT=424.013	? (TO) TT=1800.15	? (TO) TT=1800.07	? (TO) TT=1800.1
csp/langford2/ normalized-langford-2-14.xml		? (TO) TT=1800.13	? (TO) TT=1800.13	? (TO) TT=1800.08	? (TO) TT=1800.09	? (TO) TT=1800.05	? (TO) TT=1800.03	? (TO) TT=1800.07	? (TO) TT=1800.08	? (MO) TT=434.355	? (MO) TT=433.439	? (TO) TT=1800.19	? (TO) TT=1800.12	? (TO) TT=1800.15
csp/langford2/ normalized-langford-2-15.xml	SAT TT=0.021	SAT TT=0.989	SAT TT=0.977	SAT TT=0.094	SAT TT=1.076	SAT TT=0.993	SAT TT=1.785	SAT TT=2.324	SAT TT=0.021	SAT TT=0.039	SAT TT=0.046	SAT TT=3.55	SAT TT=1.123	SAT TT=1.155
csp/langford2/ normalized-langford-2-18.xml		? (TO) TT=1800.05	? (TO) TT=1800.15	? (TO) TT=1800.04	? (TO) TT=1800.13	? (TO) TT=1800.14	? (TO) TT=1800.03	? (TO) TT=1800.08	? (TO) TT=1800.07	? (MO) TT=431.57	? (MO) TT=437.024	? (TO) TT=1800.28	? (TO) TT=1800.05	? (TO) TT=1800.1
csp/langford2/ normalized-langford-2-20.xml	SAT TT=0.035	SAT TT=1.315	SAT TT=1.341	SAT TT=0.16	SAT TT=1.498	SAT TT=1.294	SAT TT=2.168	SAT TT=4.339	SAT TT=0.035	SAT TT=0.137	SAT TT=0.051	SAT TT=6.207	SAT TT=1.246	SAT TT=1.582
csp/langford2/ normalized-langford-2-21.xml		? (TO) TT=1800.11	? (TO) TT=1800.06	? (TO) TT=1800.08	? (TO) TT=1800.06	? (TO) TT=1800.1	? (TO) TT=1800.01	? (TO) TT=1800.05	? (TO) TT=1800.06	? (MO) TT=463.358	? (MO) TT=488.811	? (TO) TT=1800.36	? (TO) TT=1800.03	? (TO) TT=1800.12
csp/langford2/ normalized-langford-2-30.xml		? (TO) TT=1800.1	? (TO) TT=1800.08	? (TO) TT=1800.07	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.06	? (TO) TT=1800.09	? (TO) TT=1800.03	? (MO) TT=553.514	? (MO) TT=555.13	? (TO) TT=1800.19	? (TO) TT=1800.06	? (TO) TT=1800.11
csp/langford2/ normalized-langford-2-3.xml	SAT TT=0.017	SAT TT=0.746	SAT TT=0.702	SAT TT=0.028	SAT TT=0.491	SAT TT=0.505	SAT TT=1.085	SAT TT=1.117	SAT TT=0.02	SAT TT=0.018	SAT TT=0.017	SAT TT=0.804	SAT TT=0.543	SAT TT=0.534
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/langford2/ normalized-langford-2-40.xml	SAT TT=0.125	SAT TT=1.879	SAT TT=2.06	SAT TT=0.971	SAT TT=2.742	SAT TT=2.9	SAT TT=4.402	SAT TT=142.196	SAT TT=0.125	SAT TT=0.196	SAT TT=0.533	SAT TT=34.505	SAT TT=3.167	SAT TT=5.723
csp/langford2/ normalized-langford-2-5.xml	UNSAT TT=0.007	UNSAT TT=0.776	UNSAT TT=0.736	UNSAT TT=0.032	UNSAT TT=0.617	UNSAT TT=0.644	UNSAT TT=1.239	UNSAT TT=1.203	UNSAT TT=0.007	UNSAT TT=0.029	UNSAT TT=0.027	UNSAT TT=1.575	UNSAT TT=0.585	UNSAT TT=0.603
csp/langford2/ normalized-langford-2-7.xml	SAT TT=0.01	SAT TT=0.742	SAT TT=0.77	SAT TT=0.041	SAT TT=0.678	SAT TT=0.682	SAT TT=1.338	SAT TT=1.428	SAT TT=0.01	SAT TT=0.018	SAT TT=0.022	SAT TT=1.847	SAT TT=0.646	SAT TT=0.675
csp/langford2/ normalized-langford-2-8.xml	SAT TT=0.01	SAT TT=0.862	SAT TT=0.825	SAT TT=0.039	SAT TT=0.721	SAT TT=0.713	SAT TT=1.466	SAT TT=1.655	SAT TT=0.01	SAT TT=0.027	SAT TT=0.024	SAT TT=2.161	SAT TT=0.776	SAT TT=0.732
csp/langford2/ normalized-langford-2-9.xml	UNSAT TT=0.2	UNSAT TT=2.18	UNSAT TT=2.032	UNSAT TT=0.2	UNSAT TT=1.689	UNSAT TT=1.772	UNSAT TT=2.673	UNSAT TT=2.154	UNSAT TT=0.244	UNSAT TT=6.976	UNSAT TT=6.271	UNSAT TT=16.046	UNSAT TT=2.075	UNSAT TT=2.357
csp/langford3/ normalized-langford-3-11.xml	UNSAT TT=0.757	UNSAT TT=6.685	UNSAT TT=6.788	UNSAT TT=0.855	UNSAT TT=3.79	UNSAT TT=4.18	UNSAT TT=12.182	UNSAT TT=6.281	UNSAT TT=0.757	UNSAT TT=21.804	UNSAT TT=22.725	UNSAT TT=171.361	UNSAT TT=11.268	UNSAT TT=12.506
csp/langford3/ normalized-langford-3-12.xml	UNSAT TT=3.428	UNSAT TT=27.763	UNSAT TT=30.064	UNSAT TT=3.459	UNSAT TT=10.646	UNSAT TT=11.644	UNSAT TT=50.125	UNSAT TT=21.985	UNSAT TT=3.428	UNSAT TT=107.362	UNSAT TT=105.212	UNSAT TT=1452.6	UNSAT TT=84.334	UNSAT TT=82.02
csp/langford3/ normalized-langford-3-13.xml	UNSAT TT=14.958	UNSAT TT=155.967	UNSAT TT=142.522	UNSAT TT=14.958	UNSAT TT=52.957	UNSAT TT=62.828	UNSAT TT=261.026	UNSAT TT=97.676	UNSAT TT=16.27	? (MO) TT=426.551	? (MO) TT=455.149	? (TO) TT=1800.24	UNSAT TT=743.724	UNSAT TT=1118.92
csp/langford3/ normalized-langford-3-16.xml		? (TO) TT=1800.15	? (TO) TT=1800.12	? (TO) TT=1800.07	? (TO) TT=1800.11	? (TO) TT=1800.12	? (TO) TT=1800.07	? (TO) TT=1800.06	? (TO) TT=1800.07	? (MO) TT=465.999	? (MO) TT=434.977	? (TO) TT=1800.3	? (TO) TT=1800.06	? (TO) TT=1800.12
csp/langford3/ normalized-langford-3-17.xml	SAT TT=0.976	SAT TT=10.333	SAT TT=10.908	SAT TT=1.388	SAT TT=4.032	SAT TT=3.154	SAT TT=2.932	SAT TT=8.097	SAT TT=0.976	SAT TT=21.278	SAT TT=4.582	SAT TT=912.11	SAT TT=13.799	SAT TT=38.869
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/langford3/ normalized-langford-3-19.xml	SAT TT=2.882	SAT TT=58.73	SAT TT=61.722	SAT TT=13.92	SAT TT=5.369	SAT TT=3.653	SAT TT=9.33	SAT TT=23.999	SAT TT=3.334	SAT TT=16.151	SAT TT=2.882	SAT TT=26.92	SAT TT=7.446	SAT TT=43.782
csp/langford3/ normalized-langford-3-20.xml		? (TO) TT=1800.14	? (TO) TT=1800.17	? (TO) TT=1800.08	? (TO) TT=1800.18	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.08	? (TO) TT=1800.1	? (MO) TT=445.518	? (MO) TT=419.45	? (TO) TT=1800.32	? (TO) TT=1800.11	? (TO) TT=1800.12
csp/langford3/ normalized-langford-3-21.xml		? (TO) TT=1800.09	? (TO) TT=1800.11	? (TO) TT=1800.07	? (TO) TT=1800.09	? (TO) TT=1800.05	? (TO) TT=1800.07	? (TO) TT=1800.02	? (TO) TT=1800.05	? (MO) TT=420.955	? (MO) TT=440.999	? (TO) TT=1800.22	? (TO) TT=1800	? (TO) TT=1800.06
csp/langford3/ normalized-langford-3-25.xml		? (TO) TT=1800.1	? (TO) TT=1800.1	? (TO) TT=1800.11	? (TO) TT=1800.05	? (TO) TT=1800.16	? (TO) TT=1800.09	? (TO) TT=1800	? (TO) TT=1800.06	? (MO) TT=415.439	? (MO) TT=395.948	? (TO) TT=1800.25	? (TO) TT=1800.09	? (TO) TT=1800.11
csp/langford3/ normalized-langford-3-40.xml		? (TO) TT=1800.13	? (TO) TT=1800.1	? (TO) TT=1800.05	? (TO) TT=1800.18	? (TO) TT=1800.11	? (TO) TT=1800.04	? (TO) TT=1800.06	? (TO) TT=1800.05	? (MO) TT=420.263	? (MO) TT=430.841	? TT=676.785	? (TO) TT=1800.1	? (TO) TT=1800.09
csp/langford3/ normalized-langford-3-4.xml	UNSAT TT=0.009	UNSAT TT=0.716	UNSAT TT=0.722	UNSAT TT=0.034	UNSAT TT=0.578	UNSAT TT=0.574	UNSAT TT=1.245	UNSAT TT=1.248	UNSAT TT=0.009	UNSAT TT=0.02	UNSAT TT=0.02	UNSAT TT=1.739	UNSAT TT=0.617	UNSAT TT=0.624
csp/langford3/ normalized-langford-3-5.xml	UNSAT TT=0.008	UNSAT TT=0.877	UNSAT TT=0.824	UNSAT TT=0.037	UNSAT TT=0.62	UNSAT TT=0.711	UNSAT TT=1.38	UNSAT TT=1.291	UNSAT TT=0.008	UNSAT TT=0.021	UNSAT TT=0.023	UNSAT TT=1.989	UNSAT TT=0.686	UNSAT TT=0.697
csp/langford3/ normalized-langford-3-9.xml	SAT TT=0.031	SAT TT=1.123	SAT TT=1.092	SAT TT=0.088	SAT TT=1.057	SAT TT=1.157	SAT TT=1.859	SAT TT=2.158	SAT TT=0.031	SAT TT=0.637	SAT TT=0.175	SAT TT=3.465	SAT TT=1.05	SAT TT=1.139
csp/langford4/ normalized-langford-4-11.xml	UNSAT TT=0.218	UNSAT TT=3.034	UNSAT TT=3.345	UNSAT TT=0.446	UNSAT TT=2.175	UNSAT TT=2.291	UNSAT TT=5.075	UNSAT TT=5.849	UNSAT TT=0.218	UNSAT TT=4.724	UNSAT TT=5.721	UNSAT TT=49.462	UNSAT TT=3.527	UNSAT TT=3.528
csp/langford4/ normalized-langford-4-12.xml	UNSAT TT=0.605	UNSAT TT=8.43	UNSAT TT=7.732	UNSAT TT=1.081	UNSAT TT=3.614	UNSAT TT=4.093	UNSAT TT=12.742	? TT=11.64	UNSAT TT=0.605	UNSAT TT=17.146	UNSAT TT=19.014	UNSAT TT=187.856	UNSAT TT=11.25	UNSAT TT=10.721
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/langford4/ normalized-langford-4-13.xml	UNSAT TT=2.286	UNSAT TT=29.346	UNSAT TT=29.363	UNSAT TT=3.152	UNSAT TT=8.305	UNSAT TT=10.152	UNSAT TT=46.041	UNSAT TT=25.999	UNSAT TT=2.286	UNSAT TT=63.799	UNSAT TT=75.786	UNSAT TT=980.411	UNSAT TT=65.899	UNSAT TT=52.483
csp/langford4/ normalized-langford-4-14.xml	UNSAT TT=8.24	UNSAT TT=109.574	UNSAT TT=111.833	UNSAT TT=9.853	UNSAT TT=23.531	UNSAT TT=31.26	UNSAT TT=146.349	UNSAT TT=88.236	UNSAT TT=8.24	UNSAT TT=255.046	UNSAT TT=263.637	? (TO) TT=1800.18	UNSAT TT=507.789	UNSAT TT=384.745
csp/langford4/ normalized-langford-4-15.xml	UNSAT TT=32.199	UNSAT TT=511.049	UNSAT TT=509.209	UNSAT TT=42.16	UNSAT TT=109.428	UNSAT TT=137.861	UNSAT TT=755.533	UNSAT TT=378.347	UNSAT TT=32.199	? (MO) TT=476.123	? (MO) TT=472.31	? (TO) TT=1800.27	? (TO) TT=1800.09	? (TO) TT=1800.09
csp/langford4/ normalized-langford-4-16.xml	UNSAT TT=156.906	? (TO) TT=1800.09	? (TO) TT=1800.05	UNSAT TT=170.303	UNSAT TT=448.368	UNSAT TT=580.272	? (TO) TT=1800.05	UNSAT TT=1676.83	UNSAT TT=156.906	? (MO) TT=462.46	? (MO) TT=465.504	? (TO) TT=1800.31	? (TO) TT=1800.14	? (TO) TT=1800.1
csp/langford4/ normalized-langford-4-17.xml	UNSAT TT=651.328	? (TO) TT=1800.11	? (TO) TT=1800.17	UNSAT TT=714.45	? (TO) TT=1800.13	? (TO) TT=1800.11	? (TO) TT=1800.01	? (TO) TT=1800.06	UNSAT TT=651.328	? (MO) TT=429.852	? (MO) TT=447.725	? (TO) TT=1800.36	? (TO) TT=1800.04	? (TO) TT=1800.14
csp/langford4/ normalized-langford-4-20.xml		? (TO) TT=1800.16	? (TO) TT=1800.17	? (TO) TT=1800.03	? (TO) TT=1800.17	? (TO) TT=1800.07	? (TO) TT=1800	? (TO) TT=1800.09	? (TO) TT=1800.08	? (MO) TT=381.335	? (MO) TT=421.848	? (TO) TT=1800.32	? (TO) TT=1800.08	? (TO) TT=1800.13
csp/langford4/ normalized-langford-4-21.xml		? (TO) TT=1800.14	? (TO) TT=1800.14	? (TO) TT=1800.09	? (TO) TT=1800.12	? (TO) TT=1800.11	? (TO) TT=1800.05	? (TO) TT=1800.03	? (TO) TT=1800.08	? (MO) TT=398.411	? (MO) TT=386.725	? (TO) TT=1800.26	? (TO) TT=1800.09	? (TO) TT=1800.14
csp/langford4/ normalized-langford-4-25.xml		? (TO) TT=1800.1	? (TO) TT=1800.1	? (TO) TT=1800.07	? (TO) TT=1800.11	? (TO) TT=1800.09	? (TO) TT=1800.02	? (TO) TT=1800.07	? (TO) TT=1800.09	? (MO) TT=391.736	? (MO) TT=384.882	? (TO) TT=1800.27	? (TO) TT=1800.05	? (TO) TT=1800.13
csp/langford4/ normalized-langford-4-30.xml		? (TO) TT=1800.14	? (TO) TT=1800.1	? (TO) TT=1800.05	? (TO) TT=1800.13	? (TO) TT=1800.05	? (TO) TT=1800.03	? (TO) TT=1800.09	? (TO) TT=1800.1	? (MO) TT=358.939	? (MO) TT=363.828	? TT=698.094	? (TO) TT=1800.06	? (TO) TT=1800.07
csp/langford4/ normalized-langford-4-3.xml	UNSAT TT=0.009	UNSAT TT=0.756	UNSAT TT=0.722	UNSAT TT=0.035	UNSAT TT=0.611	UNSAT TT=0.57	UNSAT TT=1.166	UNSAT TT=1.18	UNSAT TT=0.009	UNSAT TT=0.019	UNSAT TT=0.019	UNSAT TT=1.586	UNSAT TT=0.462	UNSAT TT=0.481
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/langford4/ normalized-langford-4-4.xml	UNSAT TT=0.009	UNSAT TT=0.851	UNSAT TT=0.81	UNSAT TT=0.041	UNSAT TT=0.623	UNSAT TT=0.689	UNSAT TT=1.368	UNSAT TT=1.365	UNSAT TT=0.009	UNSAT TT=0.022	UNSAT TT=0.022	UNSAT TT=2.056	UNSAT TT=0.71	UNSAT TT=0.707
csp/langford4/ normalized-langford-4-6.xml	UNSAT TT=0.015	UNSAT TT=0.946	UNSAT TT=0.947	UNSAT TT=0.066	UNSAT TT=0.802	UNSAT TT=0.801	UNSAT TT=1.64	UNSAT TT=1.636	UNSAT TT=0.015	UNSAT TT=0.039	UNSAT TT=0.038	UNSAT TT=2.906	UNSAT TT=1	UNSAT TT=0.959
csp/langford4/ normalized-langford-4-8.xml	UNSAT TT=0.034	UNSAT TT=1.139	UNSAT TT=1.1	UNSAT TT=0.112	UNSAT TT=1.139	UNSAT TT=1.164	UNSAT TT=1.938	UNSAT TT=1.903	UNSAT TT=0.034	UNSAT TT=0.144	UNSAT TT=0.145	UNSAT TT=4.824	UNSAT TT=1.021	UNSAT TT=1.167
csp/largeQueens/ normalized-queens-1000.xml		? (MO) TT=184.006	? (MO) TT=194.309	? (problem) TT=104.077	? (problem) TT=606.464	? (problem) TT=743.441	? (TO) TT=1800.09	? (TO) TT=1800.07	? (TO) TT=1800.13	? TT=65.984	? TT=67.324	? TT=247.353	? (MO) TT=25.846	? (MO) TT=6.381
csp/largeQueens/ normalized-queens-300.xml	SAT TT=47.141	SAT TT=47.141	SAT TT=47.755	SAT TT=197.829	SAT TT=582.25	SAT TT=416.631	SAT TT=369.711	? (TO) TT=1800.07	? (TO) TT=1800.07	? (MO) TT=260.902	? (MO) TT=268.261	? TT=476.673	? (MO) TT=150.435	? (MO) TT=336.939
csp/largeQueens/ normalized-queens-400.xml	SAT TT=1171.75	? (TO) TT=1800.14	? (TO) TT=1800.16	? (MO) TT=501.872	? (TO) TT=1800.19	SAT TT=1171.75	? (exit code) TT=773.336	? TT=694.497	? (TO) TT=1800.06	? TT=141.266	? TT=205.25	? TT=278.367	? TT=253.651	? TT=262.785
csp/largeQueens/ normalized-queens-500.xml		? (TO) TT=1800.26	? (TO) TT=1800.24	? (problem) TT=658.071	? (MO) TT=65.004	? (TO) TT=1800.22	? TT=1529.53	? (TO) TT=1585.15	? (TO) TT=1800.08	? TT=88.895	? TT=150.641	? TT=370.944	? (MO) TT=24.95	? (MO) TT=28.017
csp/largeQueens/ normalized-queens-800.xml		? (MO) TT=59.166	? (MO) TT=59.303	? (TO) TT=1800.11	? (problem) TT=115.487	? (problem) TT=190.087	? (TO) TT=1800.07	? (TO) TT=1800	? (TO) TT=1800.07	? (MO) TT=49.429	? (MO) TT=46.544	? TT=267.793	? (MO) TT=6.302	? (MO) TT=6.29
csp/nengfa/ normalized-color.xml	UNSAT TT=1.399	UNSAT TT=30.758	UNSAT TT=20.035	UNSAT TT=101.963	UNSAT TT=73.407	UNSAT TT=116.932	UNSAT TT=118.402	UNSAT TT=143.714	UNSAT TT=1.477	UNSAT TT=2.195	UNSAT TT=1.399	UNSAT TT=44.178	UNSAT TT=7.764	UNSAT TT=7.63
csp/nengfa/ normalized-queens1.xml	SAT TT=4.626	SAT TT=4.626	SAT TT=4.758	SAT TT=60.851	SAT TT=7.079	SAT TT=6.747	SAT TT=19.267	SAT TT=100.644	SAT TT=5.38	SAT TT=27.028	SAT TT=16.014	? TT=784.553	SAT TT=12.254	SAT TT=36.111
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/nengfa/ normalized-route.xml	SAT TT=2.863	SAT TT=7.866	SAT TT=8.44	SAT TT=224.951	SAT TT=12.351	SAT TT=9.9	SAT TT=21.444	SAT TT=25.769	SAT TT=3.506	SAT TT=2.863	SAT TT=2.974	? (exit code) TT=80.286	SAT TT=6.282	SAT TT=5.978
csp/pigeons/ normalized-pigeons-11.xml	UNSAT TT=0.008	UNSAT TT=0.723	UNSAT TT=0.777	UNSAT TT=0.035	UNSAT TT=0.588	UNSAT TT=0.586	UNSAT TT=79.491	UNSAT TT=70.589	UNSAT TT=0.008	UNSAT TT=0.018	UNSAT TT=0.018	UNSAT TT=576.595	UNSAT TT=3.056	UNSAT TT=10.156
csp/pigeons/ normalized-pigeons-16.xml	UNSAT TT=0.01	UNSAT TT=0.78	UNSAT TT=0.752	UNSAT TT=0.04	UNSAT TT=0.654	UNSAT TT=0.611	? (TO) TT=1800.08	? (TO) TT=1800.06	UNSAT TT=0.01	UNSAT TT=0.021	UNSAT TT=0.02	? (TO) TT=1800.58	? (TO) TT=1800.1	? (TO) TT=1800.12
csp/pigeons/ normalized-pigeons-18.xml	UNSAT TT=0.01	UNSAT TT=0.778	UNSAT TT=0.874	UNSAT TT=0.043	UNSAT TT=0.693	UNSAT TT=0.62	? (TO) TT=1800.02	? (TO) TT=1800.01	UNSAT TT=0.01	UNSAT TT=0.021	UNSAT TT=0.021	? (TO) TT=1800.32	? (TO) TT=1800.12	? (TO) TT=1800.13
csp/pigeons/ normalized-pigeons-20-ord.xml	UNSAT TT=0.017	UNSAT TT=0.99	UNSAT TT=0.896	UNSAT TT=0.068	UNSAT TT=0.795	UNSAT TT=0.833	UNSAT TT=1.653	UNSAT TT=1.623	UNSAT TT=0.017	UNSAT TT=0.029	UNSAT TT=0.028	UNSAT TT=2.688	UNSAT TT=0.997	UNSAT TT=1.052
csp/pigeons/ normalized-pigeons-25.xml	UNSAT TT=0.014	UNSAT TT=0.902	UNSAT TT=0.905	UNSAT TT=0.064	UNSAT TT=0.782	UNSAT TT=0.752	? (TO) TT=1800.08	? (TO) TT=1800.03	UNSAT TT=0.014	UNSAT TT=0.024	UNSAT TT=0.025	? (TO) TT=1800.45	? (TO) TT=1800.06	? (TO) TT=1800.05
csp/pigeons/ normalized-pigeons-30.xml	UNSAT TT=0.022	UNSAT TT=0.979	UNSAT TT=0.951	UNSAT TT=0.088	UNSAT TT=0.806	UNSAT TT=0.858	? (TO) TT=1800.02	? (TO) TT=1800.01	UNSAT TT=0.022	UNSAT TT=0.031	UNSAT TT=0.03	? (TO) TT=1800.32	? (TO) TT=1800.14	? (TO) TT=1800.1
csp/pigeons/ normalized-pigeons-35.xml	UNSAT TT=0.025	UNSAT TT=1.056	UNSAT TT=1.068	UNSAT TT=0.12	UNSAT TT=1.057	UNSAT TT=0.919	? (TO) TT=1800.02	? (TO) TT=1800.04	UNSAT TT=0.025	UNSAT TT=0.034	UNSAT TT=0.034	? (TO) TT=1800.46	? (TO) TT=1800.02	? (TO) TT=1800.11
csp/pigeons/ normalized-pigeons-45.xml	UNSAT TT=0.035	UNSAT TT=1.163	UNSAT TT=1.198	UNSAT TT=0.202	UNSAT TT=1.055	UNSAT TT=1.043	? (TO) TT=1800.07	? (TO) TT=1800	UNSAT TT=0.035	UNSAT TT=0.047	UNSAT TT=0.047	? (TO) TT=1800.34	? (TO) TT=1800.04	? (TO) TT=1800.13
csp/pigeons/ normalized-pigeons-8.xml	UNSAT TT=0.008	UNSAT TT=0.721	UNSAT TT=0.674	UNSAT TT=0.139	UNSAT TT=0.509	UNSAT TT=1.115	UNSAT TT=1.284	UNSAT TT=1.243	UNSAT TT=0.008	UNSAT TT=0.018	UNSAT TT=0.018	UNSAT TT=1.498	UNSAT TT=0.609	UNSAT TT=0.624
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/pigeons/ normalized-pigeons-9.xml	UNSAT TT=0.007	UNSAT TT=0.716	UNSAT TT=0.717	UNSAT TT=0.031	UNSAT TT=0.513	UNSAT TT=0.523	UNSAT TT=1.799	UNSAT TT=1.91	UNSAT TT=0.007	UNSAT TT=0.019	UNSAT TT=0.02	UNSAT TT=3.553	UNSAT TT=0.697	UNSAT TT=0.938
csp/queenAttacking/ normalized-queenAttacking-10.xml		? (TO) TT=1800.05	? (TO) TT=1800.06	? (problem) TT=0.657	? (TO) TT=1800.11	? (TO) TT=1800.1	? (TO) TT=1800.07	? (TO) TT=1800.03	? (TO) TT=1800.02	? (TO) TT=1800.12	? (TO) TT=1800.05	? (TO) TT=1800.22	? (TO) TT=1800.09	? (TO) TT=1800.03
csp/queenAttacking/ normalized-queenAttacking-11.xml		? (TO) TT=1800.13	? (TO) TT=1800.09	? (problem) TT=0.942	? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800	? (TO) TT=1800.09	? (TO) TT=1800.1	? (TO) TT=1800.04	? TT=175.349	? (TO) TT=1800.07	? (TO) TT=1800.03
csp/queenAttacking/ normalized-queenAttacking-12.xml		? (TO) TT=1800.11	? (TO) TT=1800.09	? (problem) TT=1.215	? (TO) TT=1800.1	? (TO) TT=1800.12	? (TO) TT=1800.09	? (TO) TT=1800.02	? (TO) TT=1800.1	? (TO) TT=1800.06	? (TO) TT=1800.1	? TT=480.108	? (TO) TT=1800.1	? (TO) TT=1800.08
csp/queenAttacking/ normalized-queenAttacking-3.xml	UNSAT TT=0.01	UNSAT TT=0.691	UNSAT TT=0.733	? (problem) TT=0.036	UNSAT TT=0.665	UNSAT TT=0.594	UNSAT TT=1.266	UNSAT TT=1.287	UNSAT TT=0.01	UNSAT TT=0.421	UNSAT TT=0.378	UNSAT TT=1.457	UNSAT TT=0.725	UNSAT TT=0.707
csp/queenAttacking/ normalized-queenAttacking-4.xml	UNSAT TT=0.054	UNSAT TT=1.022	UNSAT TT=1.037	? (problem) TT=0.05	UNSAT TT=1.351	UNSAT TT=1.317	UNSAT TT=1.466	UNSAT TT=1.657	UNSAT TT=0.054	UNSAT TT=333.386	UNSAT TT=231.426	UNSAT TT=3.047	UNSAT TT=1.369	UNSAT TT=1.575
csp/queenAttacking/ normalized-queenAttacking-5.xml	SAT TT=0.069	SAT TT=1.53	SAT TT=1.451	? (problem) TT=0.07	SAT TT=2.631	SAT TT=2.616	SAT TT=1.793	SAT TT=2.914	SAT TT=0.069	SAT TT=1133.69	SAT TT=1366.5	SAT TT=7.094	SAT TT=7.007	SAT TT=67.512
csp/queenAttacking/ normalized-queenAttacking-6.xml	SAT TT=8.599	? (TO) TT=1800.12	? (TO) TT=1800.15	? (problem) TT=0.113	SAT TT=105.537	SAT TT=21.735	SAT TT=46.777	SAT TT=8.599	SAT TT=67.143	? (TO) TT=1800.14	? (TO) TT=1800.12	SAT TT=121.127	SAT TT=162.827	SAT TT=157.349
csp/queenAttacking/ normalized-queenAttacking-7.xml	SAT TT=32.263	? (TO) TT=1800.14	? (TO) TT=1800.06	? (problem) TT=0.191	? (TO) TT=1800.14	? (TO) TT=1800.12	SAT TT=32.263	SAT TT=265.493	? (TO) TT=1800.06	? (TO) TT=1800.14	? (TO) TT=1800.07	? (TO) TT=1800.24	? (TO) TT=1800.01	? (TO) TT=1800.11
csp/queenAttacking/ normalized-queenAttacking-8.xml		? (TO) TT=1800.14	? (TO) TT=1800.07	? (problem) TT=0.293	? (TO) TT=1800.09	? (TO) TT=1800.09	? (TO) TT=1800.02	? (TO) TT=1800	? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.09	? (TO) TT=1800.28	? (TO) TT=1800.05	? (TO) TT=1800.09
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/queenAttacking/ normalized-queenAttacking-9.xml		? (TO) TT=1800.12	? (TO) TT=1800.05	? (problem) TT=0.425	? (TO) TT=1800.09	? (TO) TT=1800.11	? (TO) TT=1800.01	? (TO) TT=1800.03	? (TO) TT=1800.09	? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.26	? (TO) TT=1800.06	? (TO) TT=1800
csp/queensKnights/ normalized-queensKnights-100-5-add.xml	UNSAT TT=20.802	UNSAT TT=1051.92	UNSAT TT=1044.24	? (TO) TT=1800.06	? (TO) TT=1800.1	? (TO) TT=1800.14	? (TO) TT=1800.05	UNSAT TT=997.57	UNSAT TT=110.698	? (TO) TT=1800.19	? (TO) TT=1800.18	? TT=1333.97	UNSAT TT=20.802	UNSAT TT=42.62
csp/queensKnights/ normalized-queensKnights-100-5-mul.xml	UNSAT TT=29.523	? (TO) TT=1800.11	? (TO) TT=1800.1	? (TO) TT=1800.03	? (TO) TT=1800.16	? (TO) TT=1800.09	? (TO) TT=1800.02	UNSAT TT=1150	UNSAT TT=207.216	? (TO) TT=1800.15	? (TO) TT=1800.15	? TT=357.269	UNSAT TT=29.523	UNSAT TT=44.794
csp/queensKnights/ normalized-queensKnights-10-5-add.xml	UNSAT TT=0.823	UNSAT TT=0.823	UNSAT TT=0.843	? (problem) TT=0.038	UNSAT TT=0.93	UNSAT TT=0.952	UNSAT TT=1.471	UNSAT TT=1.418	UNSAT TT=14.311	UNSAT TT=785.184	UNSAT TT=676.266	UNSAT TT=4.116	UNSAT TT=0.842	UNSAT TT=0.847
csp/queensKnights/ normalized-queensKnights-10-5-mul.xml	UNSAT TT=0.878	UNSAT TT=0.885	UNSAT TT=0.878	? (problem) TT=0.049	UNSAT TT=0.997	UNSAT TT=1.058	UNSAT TT=1.665	UNSAT TT=1.49	UNSAT TT=18.114	UNSAT TT=877.399	UNSAT TT=625.546	UNSAT TT=4.017	UNSAT TT=1.065	UNSAT TT=1.065
csp/queensKnights/ normalized-queensKnights-15-5-add.xml	UNSAT TT=1.008	UNSAT TT=1.115	UNSAT TT=1.021	? (problem) TT=0.051	UNSAT TT=1.805	UNSAT TT=1.844	UNSAT TT=1.868	UNSAT TT=1.682	UNSAT TT=175.889	? (TO) TT=1800.17	? (TO) TT=1800.16	UNSAT TT=6.895	UNSAT TT=1.008	UNSAT TT=1.46
csp/queensKnights/ normalized-queensKnights-20-5-mul.xml	UNSAT TT=1.42	UNSAT TT=1.553	UNSAT TT=1.599	? (problem) TT=0.094	UNSAT TT=305.3	UNSAT TT=326.165	UNSAT TT=3.242	UNSAT TT=2.446	UNSAT TT=783.978	? (TO) TT=1800.11	? (TO) TT=1800.11	UNSAT TT=20.211	UNSAT TT=1.42	UNSAT TT=1.91
csp/queensKnights/ normalized-queensKnights-50-5-add.xml	UNSAT TT=4.906	UNSAT TT=19.114	UNSAT TT=20.694	? (problem) TT=0.3	? (TO) TT=1800.15	? (TO) TT=1800.13	UNSAT TT=66.726	UNSAT TT=26.29	? (TO) TT=1800.04	? (TO) TT=1800.13	? (TO) TT=1800.05	UNSAT TT=816.428	UNSAT TT=4.906	UNSAT TT=8.949
csp/queensKnights/ normalized-queensKnights-50-5-mul.xml	UNSAT TT=4.188	UNSAT TT=22.321	UNSAT TT=25.389	? (problem) TT=0.37	? (TO) TT=1800.09	? (TO) TT=1800.07	UNSAT TT=72.681	UNSAT TT=34.641	? (TO) TT=1800.08	? (TO) TT=1800.09	? (TO) TT=1800.11	? TT=519.676	UNSAT TT=4.188	UNSAT TT=8.499
csp/queensKnights/ normalized-queensKnights-80-5-mul.xml	UNSAT TT=13.261	UNSAT TT=473.298	UNSAT TT=482.738	? (TO) TT=1800.05	? (TO) TT=1800.06	? (TO) TT=1800.13	? (TO) TT=1800.04	UNSAT TT=346.732	UNSAT TT=346.816	? (TO) TT=1800.08	? (TO) TT=1800.13	? TT=342.508	UNSAT TT=13.261	UNSAT TT=27.415
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/queensKnights/ normalized-queensKnights-8-5-add.xml	UNSAT TT=0.742	UNSAT TT=0.788	UNSAT TT=0.847	? (problem) TT=0.034	UNSAT TT=0.769	UNSAT TT=0.844	UNSAT TT=1.304	UNSAT TT=1.289	UNSAT TT=1.277	UNSAT TT=121.14	UNSAT TT=164.86	UNSAT TT=2.967	UNSAT TT=0.742	UNSAT TT=0.768
csp/queens/ normalized-queens-10.xml	SAT TT=0.011	SAT TT=0.723	SAT TT=0.792	SAT TT=0.036	SAT TT=0.713	SAT TT=0.739	SAT TT=1.239	SAT TT=1.283	SAT TT=0.011	SAT TT=0.027	SAT TT=0.038	SAT TT=1.468	SAT TT=0.645	SAT TT=0.76
csp/queens/ normalized-queens-12.xml	SAT TT=0.014	SAT TT=0.756	SAT TT=0.89	SAT TT=0.039	SAT TT=0.755	SAT TT=0.743	SAT TT=1.295	SAT TT=1.366	SAT TT=0.014	SAT TT=0.034	SAT TT=0.071	SAT TT=1.643	SAT TT=0.934	SAT TT=0.716
csp/queens/ normalized-queens-150.xml	SAT TT=8.577	SAT TT=8.577	SAT TT=10.358	SAT TT=10.532	SAT TT=54.828	SAT TT=60.268	SAT TT=28.477	SAT TT=562.602	SAT TT=89.801	SAT TT=158.265	SAT TT=100.712	? TT=775.49	SAT TT=30.876	SAT TT=106.411
csp/queens/ normalized-queens-15.xml	SAT TT=0.023	SAT TT=0.807	SAT TT=0.839	SAT TT=0.05	SAT TT=0.812	SAT TT=0.827	SAT TT=1.301	SAT TT=1.444	SAT TT=0.023	SAT TT=0.048	SAT TT=0.052	SAT TT=1.961	SAT TT=0.817	SAT TT=0.924
csp/queens/ normalized-queens-180.xml	SAT TT=9.791	SAT TT=11.672	SAT TT=9.791	? (TO) TT=1800.1	SAT TT=118.619	SAT TT=123.849	SAT TT=48.21	SAT TT=1191.51	SAT TT=298.674	? (MO) TT=146.702	? (MO) TT=137.518	? TT=311.162	SAT TT=63.302	SAT TT=170.968
csp/queens/ normalized-queens-200.xml	SAT TT=16.294	SAT TT=16.294	SAT TT=20.249	SAT TT=33.139	SAT TT=173.343	SAT TT=171.749	SAT TT=78.226	? (TO) TT=1800.04	SAT TT=642.154	? (MO) TT=139.975	? (MO) TT=130.169	? TT=259.732	SAT TT=134.617	SAT TT=234.317
csp/queens/ normalized-queens-20.xml	SAT TT=0.027	SAT TT=0.963	SAT TT=0.9	SAT TT=0.068	SAT TT=1.154	SAT TT=1.199	SAT TT=1.567	SAT TT=1.697	SAT TT=0.027	SAT TT=0.074	SAT TT=0.093	SAT TT=2.37	SAT TT=1.043	SAT TT=1.162
csp/queens/ normalized-queens-30.xml	SAT TT=0.129	SAT TT=1.738	SAT TT=1.134	SAT TT=0.129	SAT TT=1.86	SAT TT=1.989	SAT TT=1.699	SAT TT=2.302	SAT TT=0.152	SAT TT=0.204	SAT TT=0.211	SAT TT=4.313	SAT TT=1.412	SAT TT=1.835
csp/queens/ normalized-queens-4.xml	SAT TT=0.005	SAT TT=0.683	SAT TT=0.677	SAT TT=0.025	SAT TT=0.557	SAT TT=0.534	SAT TT=1.079	SAT TT=1.048	SAT TT=0.005	SAT TT=0.017	SAT TT=0.019	SAT TT=0.689	SAT TT=0.534	SAT TT=0.494
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/queens/ normalized-queens-80.xml	SAT TT=1.267	SAT TT=2.472	SAT TT=3.187	SAT TT=1.267	SAT TT=7.175	SAT TT=6.955	SAT TT=5.015	SAT TT=36.995	SAT TT=6.403	SAT TT=20.538	SAT TT=13.506	SAT TT=109.808	SAT TT=7.136	SAT TT=16.629

Statistics		Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
Number of times the solver is able to give the best known answer		81	81	63	76	77	75	77	82	63	63	61	73	72
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		8	3	6	0	1	1	1	54	1	3	0	11	1