CSP 2006 Competition (second round): results by benchmark

Results by benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory problems with patterns

This page displays the results of the different solvers for each benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory problems with patterns

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 109 ESAC
Abscon 109 AC
BPrologCSPSolver70a 2006-12-13
CSP4J - Combo 2006-12-19
CSP4J - MAC 2006-12-19
CSP4J - MAC 2007-01-16
CSPtoSAT+minisat 0.3
galac 1
galacJ beta 1
Mistral 2006-12-04
sat4jCSP 1.7 RC BF3
SAT4JCSP-CACHED 1.7 RC BF3
sugar 0.40
Tramontane 2006-12-04

Bench name	Best results on this instance	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	galac 1	galacJ beta 1	Mistral 2006-12-04	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04
cril/ cril-0_ext.xml	SAT TT=0.309	SAT TT=1.09 CHECKS=1076 NODES=0	SAT TT=0.988 CHECKS=1429 NODES=142	? (problem) TT=413.843	SAT TT=0.874 NODES=121	SAT TT=0.882 NODES=115	SAT TT=0.893 CHECKS=806 NODES=115	SAT TT=1.036	SAT TT=3.048	SAT TT=2.98	SAT TT=0.309	SAT TT=4.489	SAT TT=4.432	? TT=47.36	SAT TT=0.323
cril/ cril-6_ext.xml	UNSAT TT=0.961	UNSAT TT=27.747 CHECKS=26721736 NODES=379811	UNSAT TT=26.436 CHECKS=25274074 NODES=342436	? (TO) TT=1800.03	UNSAT TT=189.737 NODES=996967	UNSAT TT=68.906 NODES=381587	UNSAT TT=62.968 CHECKS=18381 NODES=373460	UNSAT TT=28.359	UNSAT TT=1.015	UNSAT TT=0.961	UNSAT TT=25.658	UNSAT TT=10.886	UNSAT TT=10.162	? (TO) TT=1800.04	UNSAT TT=2.051
cril/ cril-7.ext.xml	UNSAT TT=0.952	UNSAT TT=41.816 CHECKS=46151938 NODES=565822	UNSAT TT=55.81 CHECKS=58004485 NODES=702678	? (TO) TT=1800.1	UNSAT TT=228.851 NODES=1200998	UNSAT TT=114.176 NODES=769051	UNSAT TT=81.743 CHECKS=18506 NODES=555867	UNSAT TT=17.229	UNSAT TT=1.018	UNSAT TT=0.952	UNSAT TT=30.078	UNSAT TT=31.705	UNSAT TT=32.739	UNSAT TT=424.78	UNSAT TT=14.269
travellingSalesman/ travellingSalesman-20/ tsp-20-142_ext.xml	SAT TT=29.671	SAT TT=42.598 CHECKS=77296168 NODES=34647	SAT TT=29.671 CHECKS=52215791 NODES=25997	? (TO) TT=1800.07	? TT=35.301	? TT=34.73	? TT=40.19	? (MO) TT=593.134	? (TO) TT=1800.09	? (problem) TT=101.032	SAT TT=77.527	? (MO) TT=215.226	? (MO) TT=233.587	? TT=19.97	SAT TT=31.307
travellingSalesman/ travellingSalesman-20/ tsp-20-190_ext.xml	SAT TT=4.795	SAT TT=5.832 CHECKS=8032926 NODES=8411	SAT TT=4.795 CHECKS=5659192 NODES=6856	? (TO) TT=1800.07	SAT TT=143.512 NODES=3140	SAT TT=149.232 NODES=9439	? TT=44.57	? (TO) TT=1800.14	SAT TT=921.032	? (problem) TT=111.985	SAT TT=19.221	? (MO) TT=181.293	? (MO) TT=190.859	? TT=14.25	SAT TT=24.048
travellingSalesman/ travellingSalesman-20/ tsp-20-193_ext.xml	SAT TT=4.248	SAT TT=16.365 CHECKS=27383402 NODES=9809	SAT TT=4.248 CHECKS=5457990 NODES=1498	? (TO) TT=1800.03	? TT=33.307	? TT=33.542	? TT=33.453	? (MO) TT=13.736	? (TO) TT=1800.08	? (problem) TT=100.28	SAT TT=64.762	? (MO) TT=175.238	? (MO) TT=229.292	? TT=27.25	SAT TT=90.452
travellingSalesman/ travellingSalesman-20/ tsp-20-1_ext.xml	SAT TT=1.312	SAT TT=1.728 CHECKS=1511349 NODES=334	SAT TT=1.312 CHECKS=569013 NODES=358	? (TO) TT=1800.05	? TT=45.942	? TT=53.078	? TT=44.81	? (TO) TT=1800.16	SAT TT=665.841	? (problem) TT=106.247	SAT TT=13.821	? (MO) TT=182.602	? (MO) TT=246.618	? TT=16.44	SAT TT=15.632
travellingSalesman/ travellingSalesman-20/ tsp-20-29_ext.xml	SAT TT=0.946	SAT TT=1.585 CHECKS=1081125 NODES=77	SAT TT=0.946 CHECKS=118352 NODES=102	? (TO) TT=1800.05	SAT TT=91.032 NODES=81	SAT TT=86.622 NODES=67	SAT TT=86.418 CHECKS=48290 NODES=68	? (TO) TT=1800.13	SAT TT=405.161	? (problem) TT=111.584	SAT TT=12.658	? (MO) TT=221.083	? (MO) TT=181.395	? TT=10.1	SAT TT=12.585
travellingSalesman/ travellingSalesman-20/ tsp-20-2_ext.xml	SAT TT=3.566	SAT TT=3.566 CHECKS=3973614 NODES=2759	SAT TT=7.219 CHECKS=10544129 NODES=8292	? (TO) TT=1800.07	? TT=33.741	? TT=33.546	? TT=41.952	? (MO) TT=707.972	SAT TT=954.91	? (problem) TT=104.135	SAT TT=23.483	? (MO) TT=239.508	? (MO) TT=171.358	? TT=18.72	SAT TT=30.861
travellingSalesman/ travellingSalesman-20/ tsp-20-30_ext.xml	SAT TT=1.079	SAT TT=1.906 CHECKS=1797844 NODES=118	SAT TT=1.079 CHECKS=268735 NODES=150	? (TO) TT=1800.05	SAT TT=111.665 NODES=195	SAT TT=111.213 NODES=148	SAT TT=105.853 CHECKS=96323 NODES=134	? (TO) TT=1800.12	SAT TT=598.19	? (problem) TT=98.614	SAT TT=15.313	? (MO) TT=199.224	? (MO) TT=236.924	? TT=12.23	SAT TT=14.056
Bench name	Best results on this instance	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	galac 1	galacJ beta 1	Mistral 2006-12-04	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04
travellingSalesman/ travellingSalesman-20/ tsp-20-366_ext.xml	SAT TT=1.57	SAT TT=2.394 CHECKS=2734190 NODES=835	SAT TT=1.57 CHECKS=912402 NODES=612	? (TO) TT=1800.02	? TT=34.063	? TT=35.878	? TT=37.391	? (MO) TT=1076.57	? (TO) TT=1800.04	? (problem) TT=82.891	SAT TT=28.204	? (MO) TT=162.925	? (MO) TT=169.628	? TT=19.57	SAT TT=24.758
travellingSalesman/ travellingSalesman-20/ tsp-20-453_ext.xml	SAT TT=24.312	SAT TT=41.817 CHECKS=65348880 NODES=45778	SAT TT=24.312 CHECKS=40149486 NODES=29052	? (TO) TT=1800.11	? TT=31.939	? TT=39.679	? TT=39.131	? (MO) TT=584.555	? (TO) TT=1800.01	? (problem) TT=98.03	SAT TT=73.297	? (MO) TT=202.934	? (MO) TT=194.705	? TT=19.53	SAT TT=95.617
travellingSalesman/ travellingSalesman-20/ tsp-20-68_ext.xml	SAT TT=0.858	SAT TT=0.877 CHECKS=44035 NODES=0	SAT TT=0.858 CHECKS=47559 NODES=63	SAT TT=46.262	SAT TT=54.254 NODES=69	SAT TT=56.204 NODES=63	SAT TT=55.425 CHECKS=16839 NODES=63	SAT TT=305.877	SAT TT=249.581	SAT TT=135.092	SAT TT=6.674	? (MO) TT=450.347	? (MO) TT=448.284	? (TO) TT=1800.39	SAT TT=6.729
travellingSalesman/ travellingSalesman-20/ tsp-20-727_ext.xml	SAT TT=1.778	SAT TT=1.778 CHECKS=1413013 NODES=637	SAT TT=2.746 CHECKS=2999289 NODES=1760	? (TO) TT=1800.05	? TT=32.992	? TT=31.692	? TT=38.103	? (MO) TT=14.02	SAT TT=635.91	? (problem) TT=117.591	SAT TT=33.412	? (MO) TT=196.999	? (MO) TT=177.672	? TT=27.45	SAT TT=37.823
travellingSalesman/ travellingSalesman-20/ tsp-20-75_ext.xml	SAT TT=25.938	SAT TT=25.938 CHECKS=46112722 NODES=18661	SAT TT=31.267 CHECKS=53930185 NODES=23808	? (TO) TT=1800.03	? TT=32.905	? TT=35.888	? TT=40.322	? (MO) TT=685.298	? (TO) TT=1800.01	? (problem) TT=98.166	SAT TT=30.412	? (MO) TT=290.695	? (MO) TT=193.381	? TT=19.75	SAT TT=52.437
travellingSalesman/ travellingSalesman-20/ tsp-20-76_ext.xml	SAT TT=6.608	SAT TT=6.608 CHECKS=8944371 NODES=6783	SAT TT=9.712 CHECKS=13928994 NODES=12028	? (TO) TT=1800.12	? TT=33.48	? TT=39.158	? TT=36.647	? (MO) TT=12.599	? (TO) TT=1800.03	? (problem) TT=99.848	SAT TT=23.028	? (MO) TT=168.456	? (MO) TT=216.889	? TT=19.87	SAT TT=47.899
travellingSalesman/ travellingSalesman-20/ tsp-20-8_ext.xml	SAT TT=1.006	SAT TT=1.213 CHECKS=639941 NODES=190	SAT TT=1.006 CHECKS=149977 NODES=120	SAT TT=1643.8	SAT TT=77.064 NODES=80	SAT TT=81.972 NODES=112	SAT TT=76.831 CHECKS=64682 NODES=104	? (TO) TT=1800.1	SAT TT=324.125	? (problem) TT=105.345	SAT TT=13.284	? (MO) TT=205.706	? (MO) TT=193.122	? TT=8.81	SAT TT=13.675
travellingSalesman/ travellingSalesman-20/ tsp-20-901_ext.xml	SAT TT=16.903	SAT TT=16.903 CHECKS=26027182 NODES=15054	SAT TT=18.444 CHECKS=29544604 NODES=17602	? (TO) TT=1800.1	? TT=38.35	? TT=40.846	? TT=49.644	? (MO) TT=12.261	? (TO) TT=1800.02	? (problem) TT=104.27	SAT TT=50.75	? (MO) TT=276.917	? (MO) TT=226.22	? TT=24.31	SAT TT=36.565
travellingSalesman/ travellingSalesman-25/ tsp-25-13_ext.xml	SAT TT=2.513	SAT TT=2.878 CHECKS=3142874 NODES=1270	SAT TT=2.513 CHECKS=2752350 NODES=1615	? (TO) TT=1800.03	SAT TT=148.061 NODES=763	SAT TT=120.31 NODES=590	SAT TT=115.86 CHECKS=139265 NODES=753	? (TO) TT=1800.16	SAT TT=869.971	? (problem) TT=81.354	SAT TT=17.95	? (MO) TT=164.468	? (MO) TT=140.075	? TT=13.75	SAT TT=21.551
travellingSalesman/ travellingSalesman-25/ tsp-25-163_ext.xml	SAT TT=18.464	SAT TT=22.834 CHECKS=34990930 NODES=15511	SAT TT=18.464 CHECKS=28772400 NODES=14578	? (TO) TT=1800.09	? TT=30.598	? TT=32.96	? TT=38.933	? (MO) TT=14.279	? (TO) TT=1800.05	? (problem) TT=100.699	SAT TT=146.61	? (MO) TT=238.926	? (MO) TT=192.365	? TT=37.54	SAT TT=478.525
Bench name	Best results on this instance	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	galac 1	galacJ beta 1	Mistral 2006-12-04	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04
travellingSalesman/ travellingSalesman-25/ tsp-25-190_ext.xml	SAT TT=65.121	SAT TT=65.121 CHECKS=108054720 NODES=64740	SAT TT=147.691 CHECKS=244802389 NODES=152317	? (TO) TT=1800.1	? TT=36.793	? TT=38.598	? TT=38.302	? (MO) TT=14.252	? (TO) TT=1800.05	? (problem) TT=108.56	SAT TT=204.239	? (MO) TT=140.446	? (MO) TT=140.535	? TT=23.06	SAT TT=329.341
travellingSalesman/ travellingSalesman-25/ tsp-25-312_ext.xml	SAT TT=55.676	SAT TT=55.676 CHECKS=90564399 NODES=52043	SAT TT=198.213 CHECKS=349237626 NODES=214803	? (TO) TT=1800.04	? TT=33.142	? TT=34.274	? TT=46.757	? (MO) TT=12.89	? (TO) TT=1800	? (problem) TT=98.458	SAT TT=63.651	? (MO) TT=151.896	? (MO) TT=160.154	? TT=28.01	SAT TT=155.684
travellingSalesman/ travellingSalesman-25/ tsp-25-38_ext.xml	SAT TT=2.577	SAT TT=6.313 CHECKS=9054075 NODES=3219	SAT TT=2.577 CHECKS=2615130 NODES=1288	? (TO) TT=1800.04	? TT=33.596	? TT=37.915	? TT=46.81	? (TO) TT=1800.16	SAT TT=773.637	? (problem) TT=101.789	SAT TT=21.456	? (MO) TT=164.496	? (MO) TT=171.36	? TT=15.9	SAT TT=22.625
travellingSalesman/ travellingSalesman-25/ tsp-25-3_ext.xml	SAT TT=5.438	SAT TT=8.764 CHECKS=13628740 NODES=6579	SAT TT=5.438 CHECKS=7469759 NODES=3890	? (TO) TT=1800.04	? TT=37.964	? TT=37.577	? TT=46.557	? (TO) TT=1800.18	SAT TT=1072.38	? (problem) TT=106.206	SAT TT=22.495	? (MO) TT=148.571	? (MO) TT=135.929	? TT=16.59	SAT TT=19.907
travellingSalesman/ travellingSalesman-25/ tsp-25-456_ext.xml	SAT TT=172.385	SAT TT=410.67 CHECKS=655413427 NODES=410201	SAT TT=172.385 CHECKS=276606578 NODES=177045	? (TO) TT=1800.08	? TT=34.776	? TT=34.982	? TT=39.776	? (MO) TT=13.261	SAT TT=1369.92	? (problem) TT=127.218	SAT TT=274.097	? (MO) TT=225.21	? (MO) TT=168.707	? TT=28.49	SAT TT=748.222
travellingSalesman/ travellingSalesman-25/ tsp-25-48_ext.xml	SAT TT=65.6	SAT TT=234.533 CHECKS=407968421 NODES=247452	SAT TT=104.792 CHECKS=167984767 NODES=99788	? (TO) TT=1800.05	? TT=44.506	? TT=34.47	? TT=41.214	? (MO) TT=11.892	? (TO) TT=1800.07	? (problem) TT=88.502	SAT TT=264.952	? (MO) TT=162.23	? (MO) TT=169.491	? TT=27.3	SAT TT=65.6
travellingSalesman/ travellingSalesman-25/ tsp-25-523_ext.xml	SAT TT=18.232	SAT TT=62.003 CHECKS=98707542 NODES=64607	SAT TT=18.232 CHECKS=30861080 NODES=18878	? (TO) TT=1800.05	? TT=34.506	? TT=34.328	? TT=38.99	? (MO) TT=13.561	? (TO) TT=1800.01	? (problem) TT=101.4	SAT TT=200.606	? (MO) TT=157.039	? (MO) TT=192.835	? TT=24.32	SAT TT=309.398
travellingSalesman/ travellingSalesman-25/ tsp-25-54_ext.xml	SAT TT=1.42	SAT TT=1.529 CHECKS=998883 NODES=217	SAT TT=1.42 CHECKS=933033 NODES=433	? (TO) TT=1800.03	? TT=42.246	? TT=40.743	? TT=44.004	SAT TT=1661.47	SAT TT=752.099	? (problem) TT=98.709	SAT TT=15.626	? (MO) TT=156.548	? (MO) TT=157.888	? TT=14.12	SAT TT=15.203
travellingSalesman/ travellingSalesman-25/ tsp-25-561_ext.xml	SAT TT=19.54	SAT TT=47.855 CHECKS=78100244 NODES=42336	SAT TT=19.54 CHECKS=31686630 NODES=17327	? (TO) TT=1800.09	? TT=42.529	? TT=41.925	? TT=37.52	? (MO) TT=13.803	? (TO) TT=1800.07	? (problem) TT=105.378	SAT TT=80.036	? (MO) TT=143.032	? (MO) TT=137.738	? TT=24.4	SAT TT=38.682
travellingSalesman/ travellingSalesman-25/ tsp-25-66_ext.xml	SAT TT=2.033	SAT TT=2.033 CHECKS=1780095 NODES=784	SAT TT=3.274 CHECKS=3422235 NODES=2712	? (TO) TT=1800.03	SAT TT=92.377 NODES=1684	SAT TT=90.272 NODES=1990	SAT TT=84.479 CHECKS=182235 NODES=2210	? (TO) TT=1800.21	SAT TT=362.851	? (problem) TT=125.029	SAT TT=16.829	? (MO) TT=151.435	? (MO) TT=144.834	? (TO) TT=1800.24	SAT TT=14.123
Bench name	Best results on this instance	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	galac 1	galacJ beta 1	Mistral 2006-12-04	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04
travellingSalesman/ travellingSalesman-25/ tsp-25-681_ext.xml	SAT TT=111.18	SAT TT=128.074 CHECKS=208962146 NODES=139003	SAT TT=111.18 CHECKS=183507178 NODES=120943	? (TO) TT=1800.11	? TT=44.191	? TT=44.601	? TT=37.152	? (MO) TT=12.853	? (TO) TT=1800.07	? (problem) TT=103.962	SAT TT=220.967	? (MO) TT=161.765	? (MO) TT=172.571	? TT=27.99	SAT TT=426.419
travellingSalesman/ travellingSalesman-25/ tsp-25-715_ext.xml	SAT TT=205.171	SAT TT=491.037 CHECKS=859715351 NODES=425341	SAT TT=205.171 CHECKS=359984572 NODES=179989	? (TO) TT=1800.02	? TT=33.176	? TT=32.574	? TT=37.582	? (MO) TT=12.728	? (TO) TT=1800.1	? (problem) TT=106.592	SAT TT=1332.07	? (MO) TT=183.758	? (MO) TT=208.851	? TT=25.98	SAT TT=1492.45
travellingSalesman/ travellingSalesman-25/ tsp-25-843_ext.xml	SAT TT=37.801	SAT TT=60.854 CHECKS=105121773 NODES=64040	SAT TT=37.801 CHECKS=65756549 NODES=41883	? (TO) TT=1800.07	? TT=43.062	? TT=41.596	? TT=45.385	? (MO) TT=1299.98	? (TO) TT=1800.07	? (problem) TT=103.812	SAT TT=46.621	? (MO) TT=212.294	? (MO) TT=148.225	? TT=17.74	SAT TT=50.877

Some statistics...

	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	galac 1	galacJ beta 1	Mistral 2006-12-04	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04
Number of times the solver is able to give the best known answer	33	33	2	10	10	9	5	17	4	33	3	3	1	33
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	8	21	0	0	0	0	0	0	2	1	0	0	0	1