CSP 2009 Competition: results by benchmark

Results by benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory quasi random

This page displays the results of the different solvers for each benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory quasi random

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC (complete)
Abscon 112v4 ESAC (complete)
bpsolver 09 (complete)
Choco2.1.1 2009-06-10 (complete)
Choco2.1.1b 2009-07-16 (complete)
Concrete 2009-07-14 (complete)
Concrete DC 2009-07-14 (complete)
Conquer 2009-07-10 (complete)
Mistral 1.545 (complete)
pcs 0.3.2 (complete)
pcs-restart 0.3.2 (complete)
SAT4J CSP 2.1.1 (complete)
Sugar v1.14.6+minisat (complete)
Sugar v1.14.6+picosat (complete)

Bench name	Best results on this instance	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/bddLarge/ normalized-bdd-21-2713-15-79-10_ext.xml	UNSAT TT=27.214	UNSAT TT=72.358	UNSAT TT=82.587	UNSAT TT=98.451	UNSAT TT=540.832	UNSAT TT=510.133	UNSAT TT=914.772	UNSAT TT=1272.13	UNSAT TT=60.805	UNSAT TT=27.214	UNSAT TT=37.407	UNSAT TT=36.217	? TT=110.959	? TT=131.116	? TT=133.852
csp/bddLarge/ normalized-bdd-21-2713-15-79-11_ext.xml	UNSAT TT=34.999	UNSAT TT=73.388	UNSAT TT=76.75	UNSAT TT=68.696	UNSAT TT=625.03	UNSAT TT=522.365	UNSAT TT=1066.48	UNSAT TT=1415.5	UNSAT TT=56.991	UNSAT TT=37.319	UNSAT TT=35.566	UNSAT TT=34.999	? TT=109.981	? TT=146.8	? TT=157.887
csp/bddLarge/ normalized-bdd-21-2713-15-79-12_ext.xml	UNSAT TT=33.628	UNSAT TT=75.262	UNSAT TT=78.213	UNSAT TT=123.876	UNSAT TT=573.132	UNSAT TT=543.151	UNSAT TT=1084.2	UNSAT TT=1457.52	UNSAT TT=63.229	UNSAT TT=33.628	UNSAT TT=63.554	UNSAT TT=60.871	? TT=100.256	? TT=144.198	? TT=136.612
csp/bddLarge/ normalized-bdd-21-2713-15-79-13_ext.xml	SAT TT=0.805	SAT TT=11.401	SAT TT=12.697	SAT TT=70.05	SAT TT=114.458	SAT TT=119.515	SAT TT=937.226	SAT TT=1226.48	SAT TT=2.702	SAT TT=20.747	SAT TT=0.811	SAT TT=0.805	? TT=110.114	? TT=172.334	? TT=156.259
csp/bddLarge/ normalized-bdd-21-2713-15-79-14_ext.xml	UNSAT TT=27.692	UNSAT TT=71.247	UNSAT TT=80.401	UNSAT TT=121.5	UNSAT TT=557.944	UNSAT TT=551.934	UNSAT TT=961.677	UNSAT TT=1276.12	UNSAT TT=59.745	UNSAT TT=27.692	UNSAT TT=42.748	UNSAT TT=46.537	? TT=95.535	? TT=157.669	? TT=137.003
csp/bddLarge/ normalized-bdd-21-2713-15-79-15_ext.xml	UNSAT TT=24.861	UNSAT TT=81.284	UNSAT TT=81.923	UNSAT TT=70.356	UNSAT TT=556.686	UNSAT TT=586.831	UNSAT TT=1013.44	UNSAT TT=1369.42	UNSAT TT=57.506	UNSAT TT=24.861	UNSAT TT=32.782	UNSAT TT=37.127	? TT=109.245	? TT=164.215	? TT=170.961
csp/bddLarge/ normalized-bdd-21-2713-15-79-16_ext.xml	UNSAT TT=28.902	UNSAT TT=90.869	UNSAT TT=86.228	UNSAT TT=102.667	UNSAT TT=545.942	UNSAT TT=568.603	UNSAT TT=967.403	UNSAT TT=1381.5	UNSAT TT=58.594	UNSAT TT=28.902	UNSAT TT=42.166	UNSAT TT=50.189	? TT=107.346	? TT=152.688	? TT=145.12
csp/bddLarge/ normalized-bdd-21-2713-15-79-17_ext.xml	UNSAT TT=31.107	UNSAT TT=96.517	UNSAT TT=79.806	UNSAT TT=79.449	UNSAT TT=586.968	UNSAT TT=596.483	UNSAT TT=1007.58	UNSAT TT=1444.25	UNSAT TT=57.618	UNSAT TT=31.107	UNSAT TT=41.381	UNSAT TT=41.747	? TT=112.084	? TT=163	? TT=151.767
csp/bddLarge/ normalized-bdd-21-2713-15-79-18_ext.xml	UNSAT TT=28.013	UNSAT TT=70.275	UNSAT TT=76.665	UNSAT TT=65.211	UNSAT TT=536.998	UNSAT TT=551.384	UNSAT TT=1014.69	UNSAT TT=1275.56	UNSAT TT=61.625	UNSAT TT=28.013	UNSAT TT=37.274	UNSAT TT=37.358	? TT=92.474	? TT=148.373	? TT=149.009
csp/bddLarge/ normalized-bdd-21-2713-15-79-19_ext.xml	SAT TT=0.921	SAT TT=28.527	SAT TT=26.648	SAT TT=78.082	SAT TT=94.968	SAT TT=150.17	SAT TT=982.321	SAT TT=1343.9	SAT TT=2.523	SAT TT=22.7	SAT TT=0.921	SAT TT=0.942	? TT=101.076	? TT=124.907	? TT=183.058
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/bddLarge/ normalized-bdd-21-2713-15-79-1_ext.xml	UNSAT TT=30.555	UNSAT TT=75.997	UNSAT TT=85.556	UNSAT TT=80.704	UNSAT TT=544.528	UNSAT TT=592.563	UNSAT TT=1005.5	UNSAT TT=1406.83	UNSAT TT=60.637	UNSAT TT=30.555	UNSAT TT=36.78	UNSAT TT=37.239	? TT=87.839	? TT=238.066	? TT=256.91
csp/bddLarge/ normalized-bdd-21-2713-15-79-20_ext.xml	UNSAT TT=24.811	UNSAT TT=88.307	UNSAT TT=88.087	UNSAT TT=86.863	UNSAT TT=518.264	UNSAT TT=591.562	UNSAT TT=994.396	UNSAT TT=1272.8	UNSAT TT=65.345	UNSAT TT=24.811	UNSAT TT=45.482	UNSAT TT=41.002	? TT=102.223	? TT=140.645	? TT=165.345
csp/bddLarge/ normalized-bdd-21-2713-15-79-21_ext.xml	UNSAT TT=28.985	UNSAT TT=66.591	UNSAT TT=71.132	UNSAT TT=86.083	UNSAT TT=556.045	UNSAT TT=610.246	UNSAT TT=950.273	UNSAT TT=1299.94	UNSAT TT=53.911	UNSAT TT=28.985	UNSAT TT=37.191	UNSAT TT=39.122	? TT=95.368	? TT=145.678	? TT=155.74
csp/bddLarge/ normalized-bdd-21-2713-15-79-22_ext.xml	SAT TT=0.316	SAT TT=17.948	SAT TT=7.361	SAT TT=5.945	SAT TT=104.516	SAT TT=19.485	SAT TT=26.405	SAT TT=345.578	SAT TT=2.524	SAT TT=12.155	SAT TT=0.316	SAT TT=0.343	? TT=109.374	? TT=105.702	? TT=97.257
csp/bddLarge/ normalized-bdd-21-2713-15-79-23_ext.xml	UNSAT TT=32.686	UNSAT TT=74.599	UNSAT TT=78.572	UNSAT TT=105.595	UNSAT TT=545.363	UNSAT TT=622.689	UNSAT TT=962.592	UNSAT TT=1298.31	UNSAT TT=59.783	UNSAT TT=32.686	UNSAT TT=46.104	UNSAT TT=52.353	? TT=87.442	? TT=285.133	? TT=277.005
csp/bddLarge/ normalized-bdd-21-2713-15-79-24_ext.xml	SAT TT=0.327	SAT TT=61.791	SAT TT=7.875	SAT TT=6.662	SAT TT=42.362	SAT TT=79.786	SAT TT=29.044	SAT TT=337.361	SAT TT=2.744	SAT TT=9.887	SAT TT=0.327	SAT TT=0.327	? TT=92.523	? TT=86.66	? TT=80.585
csp/bddLarge/ normalized-bdd-21-2713-15-79-25_ext.xml	UNSAT TT=26.153	UNSAT TT=88.053	UNSAT TT=96.561	UNSAT TT=109.743	UNSAT TT=529.805	UNSAT TT=501.866	UNSAT TT=878.589	UNSAT TT=1196.03	UNSAT TT=60.683	UNSAT TT=26.153	UNSAT TT=42.851	UNSAT TT=47.743	? TT=96.724	? TT=152.688	? TT=146.675
csp/bddLarge/ normalized-bdd-21-2713-15-79-26_ext.xml	UNSAT TT=30.371	UNSAT TT=108.486	UNSAT TT=103.372	UNSAT TT=116.755	UNSAT TT=535.092	UNSAT TT=569.111	UNSAT TT=993.949	UNSAT TT=1277.96	UNSAT TT=57.187	UNSAT TT=30.371	UNSAT TT=52.642	UNSAT TT=42.918	? TT=96.889	? TT=163.538	? TT=166.164
csp/bddLarge/ normalized-bdd-21-2713-15-79-27_ext.xml	UNSAT TT=24.72	UNSAT TT=91.679	UNSAT TT=88.922	UNSAT TT=138.909	UNSAT TT=504.387	UNSAT TT=560.473	UNSAT TT=1038.9	UNSAT TT=1331.09	UNSAT TT=60.958	UNSAT TT=24.72	UNSAT TT=60.879	UNSAT TT=55.993	? TT=91.884	? TT=133.073	? TT=143.595
csp/bddLarge/ normalized-bdd-21-2713-15-79-28_ext.xml	UNSAT TT=26.447	UNSAT TT=80.856	UNSAT TT=81.844	UNSAT TT=98.402	UNSAT TT=498.072	UNSAT TT=509.49	UNSAT TT=987.769	UNSAT TT=1239.91	UNSAT TT=54.51	UNSAT TT=26.447	UNSAT TT=48.23	UNSAT TT=46.835	? TT=102.24	? TT=147.379	? TT=140.801
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/bddLarge/ normalized-bdd-21-2713-15-79-29_ext.xml	UNSAT TT=33.572	UNSAT TT=78.186	UNSAT TT=86.375	UNSAT TT=74.174	UNSAT TT=598.276	UNSAT TT=565.093	UNSAT TT=1093.18	UNSAT TT=1347.89	UNSAT TT=62.302	UNSAT TT=33.572	UNSAT TT=44.304	UNSAT TT=40.53	? TT=113.677	? TT=152.45	? TT=159.103
csp/bddLarge/ normalized-bdd-21-2713-15-79-2_ext.xml	UNSAT TT=25.854	UNSAT TT=86.883	UNSAT TT=105.596	UNSAT TT=130.661	UNSAT TT=529.087	UNSAT TT=575.486	UNSAT TT=953.029	UNSAT TT=1247.65	UNSAT TT=58.992	UNSAT TT=25.854	UNSAT TT=41.741	UNSAT TT=38.399	? TT=96.44	? TT=152.206	? TT=135.907
csp/bddLarge/ normalized-bdd-21-2713-15-79-30_ext.xml	UNSAT TT=23.266	UNSAT TT=98.745	UNSAT TT=95.368	UNSAT TT=85.289	UNSAT TT=501.571	UNSAT TT=502.926	UNSAT TT=906.105	UNSAT TT=1244.51	UNSAT TT=63.824	UNSAT TT=23.266	UNSAT TT=37.687	UNSAT TT=38.53	? TT=85.503	? TT=146.787	? TT=145.573
csp/bddLarge/ normalized-bdd-21-2713-15-79-31_ext.xml	SAT TT=0.355	SAT TT=25.171	SAT TT=7.605	SAT TT=6.47	SAT TT=23.113	SAT TT=39.738	SAT TT=29.271	SAT TT=355.695	SAT TT=2.684	SAT TT=11.321	SAT TT=0.355	SAT TT=0.382	? TT=87.085	? TT=193.328	? TT=197.853
csp/bddLarge/ normalized-bdd-21-2713-15-79-32_ext.xml	SAT TT=1.051	SAT TT=11.501	SAT TT=16.942	SAT TT=74.37	SAT TT=37.75	SAT TT=113.085	SAT TT=1027.53	SAT TT=1378.73	SAT TT=2.677	SAT TT=23.785	SAT TT=1.081	SAT TT=1.051	? TT=89.955	? TT=140.275	? TT=145.411
csp/bddLarge/ normalized-bdd-21-2713-15-79-33_ext.xml	UNSAT TT=24.018	UNSAT TT=98.935	UNSAT TT=88.622	UNSAT TT=131.422	UNSAT TT=625.807	UNSAT TT=572.61	UNSAT TT=846.161	UNSAT TT=1193.67	UNSAT TT=62.738	UNSAT TT=24.018	UNSAT TT=44.216	UNSAT TT=54.793	? TT=110.904	? TT=140.029	? TT=138.877
csp/bddLarge/ normalized-bdd-21-2713-15-79-34_ext.xml	SAT TT=1.076	SAT TT=63.068	SAT TT=69.285	SAT TT=93.73	SAT TT=372.899	SAT TT=131.786	SAT TT=1067.86	SAT TT=1330.18	SAT TT=2.718	SAT TT=20.575	SAT TT=1.076	SAT TT=1.16	? TT=95.182	? TT=158.674	? TT=147.942
csp/bddLarge/ normalized-bdd-21-2713-15-79-35_ext.xml	SAT TT=1.047	SAT TT=70.987	SAT TT=75.857	SAT TT=127.509	SAT TT=139.54	SAT TT=164.013	SAT TT=934.436	SAT TT=1343	SAT TT=2.675	SAT TT=15.605	SAT TT=1.047	SAT TT=1.11	? TT=111.132	? TT=145.684	? TT=148.597
csp/bddLarge/ normalized-bdd-21-2713-15-79-3_ext.xml	UNSAT TT=23.062	UNSAT TT=72.567	UNSAT TT=92.845	UNSAT TT=131.669	UNSAT TT=532.305	UNSAT TT=523.276	UNSAT TT=931.214	UNSAT TT=1296.35	UNSAT TT=54.971	UNSAT TT=23.062	UNSAT TT=48.492	UNSAT TT=45.366	? TT=86.838	? TT=166.786	? TT=163.093
csp/bddLarge/ normalized-bdd-21-2713-15-79-4_ext.xml	SAT TT=0.371	SAT TT=20.653	SAT TT=7.321	SAT TT=7.058	SAT TT=47.463	SAT TT=45.588	SAT TT=28.27	SAT TT=358.776	SAT TT=2.744	SAT TT=14.323	SAT TT=0.371	SAT TT=0.377	? TT=109.979	? TT=132.348	? TT=133.251
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/bddLarge/ normalized-bdd-21-2713-15-79-5_ext.xml	SAT TT=0.341	SAT TT=59.375	SAT TT=7.633	SAT TT=6.463	SAT TT=51.069	SAT TT=47.466	SAT TT=26.535	SAT TT=332.11	SAT TT=2.549	SAT TT=9.387	SAT TT=0.341	SAT TT=0.342	? TT=100.656	? TT=134.79	? TT=143.218
csp/bddLarge/ normalized-bdd-21-2713-15-79-6_ext.xml	UNSAT TT=27.186	UNSAT TT=76.754	UNSAT TT=81.354	UNSAT TT=106.275	UNSAT TT=538.081	UNSAT TT=583.967	UNSAT TT=1263.48	UNSAT TT=1634.28	UNSAT TT=58.643	UNSAT TT=27.186	UNSAT TT=42.503	UNSAT TT=45.138	? TT=87.175	? TT=207.296	? TT=210.649
csp/bddLarge/ normalized-bdd-21-2713-15-79-7_ext.xml	UNSAT TT=35.543	UNSAT TT=76.658	UNSAT TT=80.627	UNSAT TT=80.951	UNSAT TT=579.694	UNSAT TT=715.845	UNSAT TT=950.717	UNSAT TT=1284.36	UNSAT TT=64.984	UNSAT TT=35.543	UNSAT TT=39.343	UNSAT TT=47.025	? TT=104.088	? TT=139.036	? TT=134.075
csp/bddLarge/ normalized-bdd-21-2713-15-79-8_ext.xml	SAT TT=0.363	SAT TT=49.214	SAT TT=8.053	SAT TT=7.374	SAT TT=71.849	SAT TT=47.627	SAT TT=27.82	SAT TT=355.711	SAT TT=2.545	SAT TT=12.186	SAT TT=0.369	SAT TT=0.363	? TT=114.469	? TT=133.004	? TT=135.699
csp/bddLarge/ normalized-bdd-21-2713-15-79-9_ext.xml	UNSAT TT=26.461	UNSAT TT=101.25	UNSAT TT=91.419	UNSAT TT=110.948	UNSAT TT=549.548	UNSAT TT=582.24	UNSAT TT=1025.2	UNSAT TT=1418.93	UNSAT TT=62.069	UNSAT TT=26.461	UNSAT TT=47.841	UNSAT TT=49.658	? TT=103.303	? TT=156.283	? TT=139.755
csp/bddSmall/ normalized-bdd-21-133-18-78-10_ext.xml	SAT TT=0.788	SAT TT=29.712	SAT TT=4.337	SAT TT=13.004	SAT TT=44.108	SAT TT=61.146	SAT TT=15.591	SAT TT=321.125	SAT TT=2.789	SAT TT=6.961	SAT TT=0.788	SAT TT=0.794	? TT=80.066	? TT=213.658	? TT=205.816
csp/bddSmall/ normalized-bdd-21-133-18-78-11_ext.xml	SAT TT=0.872	SAT TT=39.618	SAT TT=45.291	SAT TT=67.703	SAT TT=80.339	SAT TT=208.313	SAT TT=713.983	SAT TT=1093.42	SAT TT=3.027	SAT TT=8.539	SAT TT=0.898	SAT TT=0.872	? TT=82.854	? TT=411.401	? TT=399.113
csp/bddSmall/ normalized-bdd-21-133-18-78-12_ext.xml	UNSAT TT=19.362	UNSAT TT=48.836	UNSAT TT=61.407	UNSAT TT=47.32	UNSAT TT=369.802	UNSAT TT=384.073	UNSAT TT=770.029	UNSAT TT=1116.59	UNSAT TT=28.739	UNSAT TT=19.362	UNSAT TT=100.149	UNSAT TT=87.878	? TT=94.555	? TT=288.597	? TT=295.149
csp/bddSmall/ normalized-bdd-21-133-18-78-13_ext.xml	UNSAT TT=14.969	UNSAT TT=78.253	UNSAT TT=82.797	UNSAT TT=147.224	UNSAT TT=370.436	UNSAT TT=391.598	UNSAT TT=723.773	UNSAT TT=1170.63	UNSAT TT=30.595	UNSAT TT=14.969	UNSAT TT=121.722	UNSAT TT=128.943	? TT=81.114	? TT=107.178	? TT=103.046
csp/bddSmall/ normalized-bdd-21-133-18-78-14_ext.xml	UNSAT TT=17.779	UNSAT TT=56.042	UNSAT TT=66.753	UNSAT TT=66.819	UNSAT TT=348.473	UNSAT TT=380.711	UNSAT TT=796.99	UNSAT TT=1200.14	UNSAT TT=29.296	UNSAT TT=17.779	UNSAT TT=91.997	UNSAT TT=85.236	? TT=84.576	? TT=314.508	? TT=336.116
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/bddSmall/ normalized-bdd-21-133-18-78-15_ext.xml	UNSAT TT=18.389	UNSAT TT=58.235	UNSAT TT=64.415	UNSAT TT=56.053	UNSAT TT=366.108	UNSAT TT=353.27	UNSAT TT=723.983	UNSAT TT=1139.26	UNSAT TT=30.348	UNSAT TT=18.389	UNSAT TT=102.377	UNSAT TT=102.579	? TT=110.074	? TT=237.237	? TT=251.927
csp/bddSmall/ normalized-bdd-21-133-18-78-16_ext.xml	UNSAT TT=18.878	UNSAT TT=58.797	UNSAT TT=63.613	UNSAT TT=84.477	UNSAT TT=378.016	UNSAT TT=394.628	UNSAT TT=774.449	UNSAT TT=1144.19	UNSAT TT=29.467	UNSAT TT=18.878	UNSAT TT=111.824	UNSAT TT=131.073	? TT=88.546	? TT=240.078	? TT=232.693
csp/bddSmall/ normalized-bdd-21-133-18-78-17_ext.xml	SAT TT=0.741	SAT TT=33.951	SAT TT=4.001	SAT TT=13.026	SAT TT=66.858	SAT TT=49.974	SAT TT=14.532	SAT TT=362.216	SAT TT=2.823	SAT TT=6.748	SAT TT=0.741	SAT TT=0.741	? TT=94.479	? TT=111.059	? TT=101.663
csp/bddSmall/ normalized-bdd-21-133-18-78-18_ext.xml	SAT TT=0.749	SAT TT=30.162	SAT TT=4.137	SAT TT=12.747	SAT TT=7.152	SAT TT=49.403	SAT TT=14.607	SAT TT=328.639	SAT TT=2.986	SAT TT=6.138	SAT TT=0.749	SAT TT=0.784	? TT=80.323	? TT=224.23	? TT=220.225
csp/bddSmall/ normalized-bdd-21-133-18-78-19_ext.xml	UNSAT TT=18.398	UNSAT TT=53.397	UNSAT TT=60.443	UNSAT TT=53.692	UNSAT TT=363.818	UNSAT TT=382.355	UNSAT TT=765.278	UNSAT TT=1147.65	UNSAT TT=30.062	UNSAT TT=18.398	UNSAT TT=85.305	UNSAT TT=85.871	? TT=91.478	? TT=260.919	? TT=256.218
csp/bddSmall/ normalized-bdd-21-133-18-78-1_ext.xml	SAT TT=0.806	SAT TT=31.708	SAT TT=4.315	SAT TT=13.853	SAT TT=8.83	SAT TT=7.454	SAT TT=15.183	SAT TT=337.177	SAT TT=3.022	SAT TT=7.835	SAT TT=0.807	SAT TT=0.806	? TT=92.047	? TT=226.988	? TT=219.142
csp/bddSmall/ normalized-bdd-21-133-18-78-20_ext.xml	SAT TT=0.738	SAT TT=41.674	SAT TT=4	SAT TT=13.419	SAT TT=12.35	SAT TT=7.433	SAT TT=14.66	SAT TT=324.425	SAT TT=3.004	SAT TT=6.656	SAT TT=0.738	SAT TT=0.741	? TT=81.139	? TT=254.413	? TT=250.34
csp/bddSmall/ normalized-bdd-21-133-18-78-21_ext.xml	UNSAT TT=16.457	UNSAT TT=55.778	UNSAT TT=63.104	UNSAT TT=65.076	UNSAT TT=380.699	UNSAT TT=343.956	UNSAT TT=756.231	UNSAT TT=1164.77	UNSAT TT=30.534	UNSAT TT=16.457	UNSAT TT=111.488	UNSAT TT=122.279	? TT=91.764	? TT=109.071	? TT=114.602
csp/bddSmall/ normalized-bdd-21-133-18-78-22_ext.xml	UNSAT TT=17.624	UNSAT TT=60.392	UNSAT TT=68.713	UNSAT TT=70.044	UNSAT TT=378.81	UNSAT TT=419.642	UNSAT TT=778.041	UNSAT TT=1110.39	UNSAT TT=28.843	UNSAT TT=17.624	UNSAT TT=104.962	UNSAT TT=105.023	? TT=84.657	? TT=222.103	? TT=242.332
csp/bddSmall/ normalized-bdd-21-133-18-78-23_ext.xml	SAT TT=0.781	SAT TT=29.647	SAT TT=4.399	SAT TT=12.665	SAT TT=48.245	SAT TT=32.546	SAT TT=15.611	SAT TT=328.038	SAT TT=2.889	SAT TT=6.668	SAT TT=0.786	SAT TT=0.781	? TT=84.542	? TT=234.759	? TT=215.387
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/bddSmall/ normalized-bdd-21-133-18-78-24_ext.xml	UNSAT TT=15.76	UNSAT TT=59.658	UNSAT TT=62.122	UNSAT TT=67.612	UNSAT TT=343.047	UNSAT TT=363.617	UNSAT TT=780.98	UNSAT TT=1131.93	UNSAT TT=29.913	UNSAT TT=15.76	UNSAT TT=108.763	UNSAT TT=88.435	? TT=89.509	? TT=258.173	? TT=243.598
csp/bddSmall/ normalized-bdd-21-133-18-78-25_ext.xml	SAT TT=0.798	SAT TT=39.351	SAT TT=4.27	SAT TT=13.161	SAT TT=43.475	SAT TT=7.023	SAT TT=15.577	SAT TT=333.01	SAT TT=3.135	SAT TT=4.706	SAT TT=0.823	SAT TT=0.798	? TT=91.623	? TT=252.954	? TT=226.254
csp/bddSmall/ normalized-bdd-21-133-18-78-26_ext.xml	SAT TT=0.794	SAT TT=25.038	SAT TT=4.198	SAT TT=12.716	SAT TT=355.987	SAT TT=33.176	SAT TT=16.075	SAT TT=334.515	SAT TT=2.836	SAT TT=6.688	SAT TT=0.794	SAT TT=0.8	? TT=80.687	? TT=400.142	? TT=516.419
csp/bddSmall/ normalized-bdd-21-133-18-78-27_ext.xml	SAT TT=0.793	SAT TT=24.051	SAT TT=3.969	SAT TT=12.89	SAT TT=7.905	SAT TT=8.512	SAT TT=15.104	SAT TT=331.861	SAT TT=3.108	SAT TT=6.593	SAT TT=0.793	SAT TT=0.799	? TT=88.598	? TT=245.568	? TT=262.855
csp/bddSmall/ normalized-bdd-21-133-18-78-28_ext.xml	UNSAT TT=18.175	UNSAT TT=50.666	UNSAT TT=61.337	UNSAT TT=59.988	UNSAT TT=336.527	UNSAT TT=381.706	UNSAT TT=790.889	UNSAT TT=1113.44	UNSAT TT=31.28	UNSAT TT=18.175	UNSAT TT=96.252	UNSAT TT=78.707	? TT=113.993	? TT=274.682	? TT=218.278
csp/bddSmall/ normalized-bdd-21-133-18-78-29_ext.xml	UNSAT TT=15.081	UNSAT TT=59.482	UNSAT TT=71.196	UNSAT TT=56.838	UNSAT TT=349.641	UNSAT TT=403.689	UNSAT TT=772.285	UNSAT TT=1181.89	UNSAT TT=31.713	UNSAT TT=15.081	UNSAT TT=85.914	UNSAT TT=98.048	? TT=88.48	? TT=118.25	? TT=107.276
csp/bddSmall/ normalized-bdd-21-133-18-78-2_ext.xml	SAT TT=10.142	SAT TT=33.266	SAT TT=39.288	SAT TT=45.888	SAT TT=172.586	SAT TT=101.598	SAT TT=684.952	SAT TT=1042.53	SAT TT=11.904	SAT TT=10.142	SAT TT=48.065	SAT TT=22.882	? TT=85.965	? TT=223.897	? TT=209.646
csp/bddSmall/ normalized-bdd-21-133-18-78-30_ext.xml	UNSAT TT=16.008	UNSAT TT=62.602	UNSAT TT=68.133	UNSAT TT=94.77	UNSAT TT=380.985	UNSAT TT=393.582	UNSAT TT=684.226	UNSAT TT=1108.31	UNSAT TT=28.229	UNSAT TT=16.008	UNSAT TT=101.034	UNSAT TT=106.097	? TT=85.403	? TT=296.13	? TT=333.588
csp/bddSmall/ normalized-bdd-21-133-18-78-31_ext.xml	UNSAT TT=14.752	UNSAT TT=86.784	UNSAT TT=110.014	UNSAT TT=82.689	UNSAT TT=374.103	UNSAT TT=348.073	UNSAT TT=758.73	UNSAT TT=1088.87	UNSAT TT=30.516	UNSAT TT=14.752	UNSAT TT=112.716	UNSAT TT=108.761	? TT=111.338	? TT=293.322	? TT=283.009
csp/bddSmall/ normalized-bdd-21-133-18-78-32_ext.xml	SAT TT=0.8	SAT TT=9.623	SAT TT=4.206	SAT TT=11.717	SAT TT=6.821	SAT TT=7.741	SAT TT=16.267	SAT TT=337.562	SAT TT=3.086	SAT TT=4.8	SAT TT=0.8	SAT TT=0.825	? TT=87.872	? TT=271.514	? TT=282.007
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/bddSmall/ normalized-bdd-21-133-18-78-33_ext.xml	UNSAT TT=16.923	UNSAT TT=55.469	UNSAT TT=62.733	UNSAT TT=86.104	UNSAT TT=361.98	UNSAT TT=414.638	UNSAT TT=763.597	UNSAT TT=1154.03	UNSAT TT=32.803	UNSAT TT=16.923	UNSAT TT=134.295	UNSAT TT=170.232	? TT=108.224	? TT=105.3	? TT=105.86
csp/bddSmall/ normalized-bdd-21-133-18-78-34_ext.xml	SAT TT=9.06	SAT TT=28.644	SAT TT=34.421	SAT TT=38.038	SAT TT=45.636	SAT TT=109.623	SAT TT=657.593	SAT TT=1027.38	SAT TT=29.631	SAT TT=9.06	SAT TT=87.014	SAT TT=31.757	? TT=92.643	? TT=238.013	? TT=212.972
csp/bddSmall/ normalized-bdd-21-133-18-78-35_ext.xml	UNSAT TT=13.963	UNSAT TT=73.904	UNSAT TT=79.408	UNSAT TT=66.168	UNSAT TT=348.996	UNSAT TT=383.373	UNSAT TT=797.858	UNSAT TT=1151.81	UNSAT TT=28.942	UNSAT TT=13.963	UNSAT TT=100.35	UNSAT TT=93.378	? TT=95.967	? TT=234.34	? TT=245.678
csp/bddSmall/ normalized-bdd-21-133-18-78-3_ext.xml	SAT TT=0.78	SAT TT=26.305	SAT TT=4.355	SAT TT=13.197	SAT TT=85.264	SAT TT=100.125	SAT TT=14.679	SAT TT=334.487	SAT TT=3.009	SAT TT=4.872	SAT TT=0.78	SAT TT=0.787	? TT=115.164	? TT=265.954	? TT=285.389
csp/bddSmall/ normalized-bdd-21-133-18-78-4_ext.xml	SAT TT=0.866	SAT TT=27.751	SAT TT=34.78	SAT TT=89.052	SAT TT=276.328	SAT TT=96.662	SAT TT=615.869	SAT TT=960.867	SAT TT=3.09	SAT TT=9.716	SAT TT=0.926	SAT TT=0.866	? TT=96.819	? TT=264.704	? TT=269.561
csp/bddSmall/ normalized-bdd-21-133-18-78-5_ext.xml	SAT TT=0.775	SAT TT=31.931	SAT TT=4.413	SAT TT=11.966	SAT TT=236.645	SAT TT=17.468	SAT TT=15.33	SAT TT=337.34	SAT TT=3.015	SAT TT=6.497	SAT TT=0.775	SAT TT=0.777	? TT=94.091	? TT=250.151	? TT=269.311
csp/bddSmall/ normalized-bdd-21-133-18-78-6_ext.xml	UNSAT TT=14.308	UNSAT TT=55.46	UNSAT TT=71.04	UNSAT TT=52.536	UNSAT TT=332.529	UNSAT TT=395.65	UNSAT TT=773.38	UNSAT TT=1142.15	UNSAT TT=29.718	UNSAT TT=14.308	UNSAT TT=88.084	UNSAT TT=90.071	? TT=98.06	? TT=254.675	? TT=251.094
csp/bddSmall/ normalized-bdd-21-133-18-78-7_ext.xml	SAT TT=0.888	SAT TT=24.913	SAT TT=30.105	SAT TT=50.122	SAT TT=59.872	SAT TT=138.773	SAT TT=793.095	SAT TT=1189.72	SAT TT=3.119	SAT TT=9.872	SAT TT=0.912	SAT TT=0.888	? TT=101.975	? TT=211.927	? TT=232.994
csp/bddSmall/ normalized-bdd-21-133-18-78-8_ext.xml	SAT TT=0.793	SAT TT=17.372	SAT TT=4.21	SAT TT=12.902	SAT TT=7.581	SAT TT=7.438	SAT TT=14.769	SAT TT=336.24	SAT TT=2.978	SAT TT=6.268	SAT TT=0.794	SAT TT=0.793	? TT=83.979	? TT=269.738	? TT=261.286
csp/bddSmall/ normalized-bdd-21-133-18-78-9_ext.xml	SAT TT=0.944	SAT TT=18.952	SAT TT=4.093	SAT TT=37.066	SAT TT=17.413	SAT TT=8.512	SAT TT=113.124	SAT TT=465.426	SAT TT=3.023	SAT TT=9.144	SAT TT=0.952	SAT TT=0.944	? TT=100.206	? TT=289.303	? TT=288.281
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/dag-half/ normalized-half-n25-d5-e56-r7-10.xml	SAT TT=183.153	SAT TT=213.072	SAT TT=183.153	? (problem) TT=159.479	SAT TT=1174.13	? (TO) TT=1800.16	? (TO) TT=1800.04	? (TO) TT=1800.02	? (TO) TT=1800.02	SAT TT=386.99	? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.45	? (MO) TT=14.75	? (MO) TT=13.77
csp/dag-half/ normalized-half-n25-d5-e56-r7-12.xml	SAT TT=199.893	? (TO) TT=1800.17	? (TO) TT=1800.1	? (problem) TT=169.449	? (TO) TT=1800.12	? (TO) TT=1800.21	? (TO) TT=1800.06	? (TO) TT=1800.05	? (TO) TT=1800.09	SAT TT=199.893	? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.39	? (MO) TT=14.942	? (MO) TT=13.43
csp/dag-half/ normalized-half-n25-d5-e56-r7-15.xml	SAT TT=445.586	? (TO) TT=1800.1	? (TO) TT=1800.14	? (problem) TT=169.53	? (TO) TT=1800.21	? (TO) TT=1800.15	? (TO) TT=1800.05	? (TO) TT=1800.04	? (TO) TT=1800.02	SAT TT=445.586	? (TO) TT=1800.13	? (TO) TT=1800.1	SAT TT=1655.66	? (MO) TT=14.52	? (MO) TT=14.52
csp/dag-half/ normalized-half-n25-d5-e56-r7-16.xml	SAT TT=151.099	SAT TT=1055.73	SAT TT=1545.98	? (problem) TT=166.795	SAT TT=1087.69	? (TO) TT=1800.11	? (TO) TT=1800.05	? (TO) TT=1800	? (TO) TT=1800.09	SAT TT=151.099	? (TO) TT=1800.08	? (TO) TT=1800.06	? (TO) TT=1800.45	? (MO) TT=14.71	? (MO) TT=14.966
csp/dag-half/ normalized-half-n25-d5-e56-r7-18.xml	SAT TT=88.047	? (TO) TT=1800.15	? (TO) TT=1800.11	? (problem) TT=174.239	SAT TT=1330.72	SAT TT=547.641	SAT TT=590.251	SAT TT=558.884	SAT TT=1018.27	SAT TT=88.047	? (TO) TT=1800.09	? (TO) TT=1800.14	? (TO) TT=1800.41	? (MO) TT=14.84	? (MO) TT=14.948
csp/dag-half/ normalized-half-n25-d5-e56-r7-19.xml	SAT TT=133.727	? (TO) TT=1800.17	? (TO) TT=1800.15	? (problem) TT=167.896	? (TO) TT=1800.1	? (TO) TT=1800.18	SAT TT=845.452	SAT TT=903.554	SAT TT=195.127	SAT TT=133.727	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.3	? (MO) TT=13.859	? (MO) TT=13.36
csp/dag-half/ normalized-half-n25-d5-e56-r7-1.xml	SAT TT=29.765	? (TO) TT=1800.16	? (TO) TT=1800.08	? (problem) TT=161.65	SAT TT=1187.55	SAT TT=721.51	SAT TT=962.717	SAT TT=959.36	? (TO) TT=1800.09	SAT TT=29.765	? (TO) TT=1800.08	? (TO) TT=1800.07	? (TO) TT=1800.36	? (MO) TT=13.979	? (MO) TT=14.019
csp/dag-half/ normalized-half-n25-d5-e56-r7-20.xml	SAT TT=304.851	? (TO) TT=1800.09	? (TO) TT=1800.18	? (problem) TT=168.167	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.02	? (TO) TT=1800.04	? (TO) TT=1800.01	SAT TT=304.851	SAT TT=1025	? (TO) TT=1800.05	? (TO) TT=1800.45	? (MO) TT=14.981	? (MO) TT=13.904
csp/dag-half/ normalized-half-n25-d5-e56-r7-22.xml	SAT TT=156.785	? (TO) TT=1800.12	? (TO) TT=1800.07	? (problem) TT=159.946	SAT TT=800.824	? (TO) TT=1800.19	SAT TT=1005.18	SAT TT=935.915	? (TO) TT=1666.16	SAT TT=156.785	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800.49	? (MO) TT=13.99	? (MO) TT=13.901
csp/dag-half/ normalized-half-n25-d5-e56-r7-24.xml	SAT TT=325.562	SAT TT=902.385	SAT TT=847.902	? (problem) TT=164.573	SAT TT=325.562	SAT TT=920.308	SAT TT=924.48	SAT TT=919.027	? (TO) TT=1800.02	SAT TT=579.46	? (TO) TT=1800.12	? (TO) TT=1800.12	? (TO) TT=1800.43	? (MO) TT=15.058	? (MO) TT=15.043
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/dag-half/ normalized-half-n25-d5-e56-r7-2.xml	SAT TT=11.058	? (TO) TT=1800.13	? (TO) TT=1800.13	? (problem) TT=170.763	? (TO) TT=1800.19	SAT TT=1421.54	SAT TT=273.21	SAT TT=266.283	? (TO) TT=1800.03	SAT TT=11.058	SAT TT=1252.66	? (TO) TT=1800.11	? (TO) TT=1800.4	? (MO) TT=13.896	? (MO) TT=13.99
csp/dag-half/ normalized-half-n25-d5-e56-r7-3.xml	SAT TT=184.611	? (TO) TT=1800.19	? (TO) TT=1800.13	? (problem) TT=172.375	? (TO) TT=1800.17	? (TO) TT=1800.16	SAT TT=1521.25	SAT TT=1419.07	? (TO) TT=1800.02	SAT TT=184.611	? (TO) TT=1800.07	SAT TT=1334.4	? (TO) TT=1800.43	? (MO) TT=14.62	? (MO) TT=13.962
csp/dag-half/ normalized-half-n25-d5-e56-r7-5.xml	SAT TT=1486.86	? (TO) TT=1800.06	? (TO) TT=1800.16	? (problem) TT=169.605	? (TO) TT=1800.17	? (TO) TT=1800.09	? (TO) TT=1800.07	? (TO) TT=1800.1	? (TO) TT=1800.03	SAT TT=1486.86	? (TO) TT=1800.14	? (TO) TT=1800.07	? (TO) TT=1800.36	? (MO) TT=13.709	? (MO) TT=15.059
csp/dag-half/ normalized-half-n25-d5-e56-r7-7.xml	SAT TT=70.57	? (TO) TT=1800.17	? (TO) TT=1800.13	? (problem) TT=168.583	SAT TT=1159.82	? (TO) TT=1800.15	? (TO) TT=1800.07	? (TO) TT=1800.02	SAT TT=523.756	SAT TT=70.57	? (TO) TT=1800.11	? (TO) TT=1800.12	? (TO) TT=1800.4	? (MO) TT=13.906	? (MO) TT=13.886
csp/dag-half/ normalized-half-n25-d5-e56-r7-8.xml	SAT TT=158.178	? (TO) TT=1800.09	? (TO) TT=1800.17	? (problem) TT=171.341	SAT TT=629.481	? (TO) TT=1800.19	? (TO) TT=1800.08	? (TO) TT=1800.08	SAT TT=1207.63	SAT TT=158.178	? (TO) TT=1800.06	? (TO) TT=1800.07	? (TO) TT=1800.36	? (MO) TT=15.6	? (MO) TT=14.632
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-11.xml	UNSAT TT=81.116	UNSAT TT=81.116	UNSAT TT=88.741	? (problem) TT=297.718	? (TO) TT=1800.2	? (TO) TT=1800.21	UNSAT TT=888.107	UNSAT TT=1099.72	? (MO) TT=128.3	? (TO) TT=1800.09	UNSAT TT=642.999	UNSAT TT=678.05	? TT=139.826	? TT=156.597	? TT=156.878
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-12.xml	UNSAT TT=144.275	UNSAT TT=172.314	UNSAT TT=144.275	? (problem) TT=324.25	? (TO) TT=1800.18	? (TO) TT=1800.23	UNSAT TT=805.271	UNSAT TT=997.335	? (MO) TT=127.9	? (TO) TT=1800.15	UNSAT TT=1302.96	UNSAT TT=1404.1	? TT=135.833	? TT=180.677	? TT=163.571
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-16.xml	UNSAT TT=156.255	UNSAT TT=174.014	UNSAT TT=156.255	? (problem) TT=314.251	? (TO) TT=1800.21	? (TO) TT=1800.17	UNSAT TT=627.191	UNSAT TT=788.073	? (MO) TT=135.78	? (TO) TT=1800.11	UNSAT TT=1054.71	? (TO) TT=1800.15	? TT=113.603	? TT=159.013	? TT=167.999
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-17.xml	UNSAT TT=120.471	UNSAT TT=148.026	UNSAT TT=120.471	? (problem) TT=296.363	? (TO) TT=1800.2	? (TO) TT=1800.25	UNSAT TT=640.971	UNSAT TT=848.21	? (MO) TT=131.52	UNSAT TT=1601.13	UNSAT TT=803.139	UNSAT TT=788.535	? TT=144.651	? TT=165.723	? TT=170.635
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-18.xml	UNSAT TT=197.122	UNSAT TT=197.181	UNSAT TT=197.122	? (problem) TT=318.879	? (TO) TT=1800.25	? (TO) TT=1800.22	UNSAT TT=726.029	UNSAT TT=901.745	? (MO) TT=126.03	? (TO) TT=1800.06	? (TO) TT=1800.12	UNSAT TT=1757.03	? TT=118.359	? TT=165.369	? TT=159.92
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-19.xml	UNSAT TT=98.53	UNSAT TT=98.53	UNSAT TT=100.658	? (problem) TT=319.153	? (TO) TT=1800.21	? (TO) TT=1800.17	UNSAT TT=539.56	UNSAT TT=765.394	? (MO) TT=122.62	? (TO) TT=1800.15	UNSAT TT=743.594	UNSAT TT=1239.57	? TT=166.549	? TT=195.027	? TT=199.043
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-22.xml	UNSAT TT=98.539	UNSAT TT=99.446	UNSAT TT=98.539	? (problem) TT=322.231	? (TO) TT=1800.2	? (TO) TT=1800.21	UNSAT TT=419.382	UNSAT TT=596.379	? (MO) TT=126.21	? (TO) TT=1800.15	UNSAT TT=1060.53	UNSAT TT=1117.76	? TT=190.922	? TT=176.081	? TT=166.254
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-23.xml	UNSAT TT=110.566	UNSAT TT=137.465	UNSAT TT=110.566	? (problem) TT=298.39	? (TO) TT=1800.21	? (TO) TT=1800.23	UNSAT TT=817.123	UNSAT TT=983.591	? (MO) TT=129.61	? (TO) TT=1800.09	UNSAT TT=1471.4	UNSAT TT=1155.14	? TT=123.95	? TT=157.846	? TT=182.753
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-25.xml	UNSAT TT=152.004	UNSAT TT=168.203	UNSAT TT=152.004	? (problem) TT=315.01	? (TO) TT=1800.17	? (TO) TT=1800.2	UNSAT TT=912.889	UNSAT TT=1110.11	? (MO) TT=122.27	? (TO) TT=1800.15	UNSAT TT=902.285	UNSAT TT=875.148	? TT=118.798	? TT=176.707	? TT=183.129
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-2.xml	UNSAT TT=152.655	UNSAT TT=152.655	UNSAT TT=156.967	? (problem) TT=301.904	? (TO) TT=1800.22	? (TO) TT=1800.22	UNSAT TT=704.039	UNSAT TT=848.628	? (MO) TT=130.37	? (TO) TT=1800.11	UNSAT TT=836.783	UNSAT TT=881.09	? TT=152.879	? TT=155.58	? TT=161.059
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-3.xml	UNSAT TT=134.853	UNSAT TT=134.853	UNSAT TT=226.366	? (problem) TT=318.481	? (TO) TT=1800.26	? (TO) TT=1800.17	UNSAT TT=652.448	UNSAT TT=824.746	? (MO) TT=139.24	? (TO) TT=1800.07	UNSAT TT=1720.21	UNSAT TT=1419.77	? TT=139.464	? TT=176.26	? TT=171.09
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-4.xml	UNSAT TT=124.827	UNSAT TT=124.827	UNSAT TT=132.663	? (problem) TT=298.339	? (TO) TT=1800.17	? (TO) TT=1800.22	UNSAT TT=800.154	UNSAT TT=1020.21	? (MO) TT=121.18	? (TO) TT=1800.07	UNSAT TT=1701.09	UNSAT TT=1437.51	? TT=117.311	? TT=161.017	? TT=187.215
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-5.xml	UNSAT TT=122.486	UNSAT TT=127.88	UNSAT TT=122.486	? (problem) TT=294.904	? (TO) TT=1800.26	? (TO) TT=1800.19	UNSAT TT=656.182	UNSAT TT=821.403	? (MO) TT=119.36	? (TO) TT=1800.15	UNSAT TT=1019.26	UNSAT TT=950.932	? TT=140.975	? TT=174.992	? TT=172.648
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-6.xml	UNSAT TT=137.415	UNSAT TT=158.651	UNSAT TT=137.415	? (problem) TT=319.939	? (TO) TT=1800.19	? (TO) TT=1800.17	UNSAT TT=754.14	UNSAT TT=984.002	? (MO) TT=123.92	? (TO) TT=1800.12	UNSAT TT=951.987	UNSAT TT=906.163	? TT=159.546	? TT=166.489	? TT=177.796
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-9.xml	UNSAT TT=166.473	UNSAT TT=172.793	UNSAT TT=166.473	? (problem) TT=297.854	? (TO) TT=1800.18	? (TO) TT=1800.16	UNSAT TT=782.02	UNSAT TT=958.872	? (MO) TT=134.57	? (TO) TT=1800.09	UNSAT TT=1718.87	UNSAT TT=896.287	? TT=177.259	? TT=175.611	? TT=165.49

Statistics		Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Concrete 2009-07-14 (complete)	Concrete DC 2009-07-14 (complete)	Conquer 2009-07-10 (complete)	Mistral 1.545 (complete)	pcs 0.3.2 (complete)	pcs-restart 0.3.2 (complete)	SAT4J CSP 2.1.1 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
Number of times the solver is able to give the best known answer		88	88	70	78	74	92	92	74	86	86	85	1	0	0
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		5	11	0	1	0	0	0	0	54	17	13	0	0	0