CSP 2006 Competition (second round): results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 109 ESAC
Abscon 109 AC
BPrologCSPSolver70a 2006-12-13
buggy_2_5 2007-01-08
buggy_2_5_s 2007-01-08
CSP4J - Combo 2006-12-19
CSP4J - MAC 2006-12-19
CSP4J - MAC 2007-01-16
CSPtoSAT+minisat 0.3
Diarmuid-rndi 2006-12-21
Diarmuid-rndi 2007-01-22
Diarmuid-wtdi 2006-12-21
galac 1
galacJ beta 1
Mistral 2006-12-04
rjw-solver 2006-12-09
rjw-solver 2007-01-21
sat4jCSP 1.7 RC BF3
SAT4JCSP-CACHED 1.7 RC BF3
sugar 0.40
Tramontane 2006-12-04
VALCSP 3.0
VALCSP 3.1

Bench name	Best results on this instance	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	buggy_2_5 2007-01-08	buggy_2_5_s 2007-01-08	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	Diarmuid-rndi 2006-12-21	Diarmuid-rndi 2007-01-22	Diarmuid-wtdi 2006-12-21	galac 1	galacJ beta 1	Mistral 2006-12-04	rjw-solver 2006-12-09	rjw-solver 2007-01-21	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04	VALCSP 3.0	VALCSP 3.1
hanoi/ hanoi-3_ext.xml	SAT TT=0.003	SAT TT=0.611 CHECKS=164 NODES=0	SAT TT=0.644 CHECKS=159 NODES=6	SAT TT=0.02	SAT TT=0.061	SAT TT=0.057	SAT TT=0.295 NODES=0	SAT TT=0.286 NODES=6	SAT TT=0.285 CHECKS=10 NODES=6	SAT TT=0.016	SAT TT=0.039 CHECKS=713 NODES=6	SAT TT=0.038 CHECKS=713 NODES=6	SAT TT=0.038 CHECKS=713 NODES=6	SAT TT=1.03	SAT TT=0.86	SAT TT=0.014	SAT TT=0.036 CHECKS=713 NODES=6	SAT TT=0.036 CHECKS=713 NODES=6	SAT TT=0.574	SAT TT=0.557	SAT TT=1.003	SAT TT=0.012	SAT TT=0.003 CHECKS=840 NODES=6	SAT TT=0.001 CHECKS=840 NODES=6
hanoi/ hanoi-4_ext.xml	SAT TT=0.004	SAT TT=0.687 CHECKS=2210 NODES=0	SAT TT=0.622 CHECKS=2192 NODES=14	SAT TT=0.026	SAT TT=0.071	SAT TT=0.071	SAT TT=0.411 NODES=0	SAT TT=0.407 NODES=14	SAT TT=0.399 CHECKS=26 NODES=14	SAT TT=0.07	SAT TT=0.06 CHECKS=22569 NODES=14	SAT TT=0.06 CHECKS=22569 NODES=14	SAT TT=0.056 CHECKS=22569 NODES=14	SAT TT=1.991	SAT TT=1.065	SAT TT=0.025	SAT TT=0.055 CHECKS=22569 NODES=14	SAT TT=0.059 CHECKS=22569 NODES=14	SAT TT=0.996	SAT TT=0.87	SAT TT=19.999	SAT TT=0.014	SAT TT=0.004 CHECKS=29413 NODES=14	SAT TT=0.004 CHECKS=29413 NODES=14
hanoi/ hanoi-5_ext.xml	SAT TT=0.033	SAT TT=0.688 CHECKS=30105 NODES=0	SAT TT=0.655 CHECKS=30036 NODES=30	SAT TT=0.055	SAT TT=0.147	SAT TT=0.151	SAT TT=0.639 NODES=0	SAT TT=0.678 NODES=30	SAT TT=0.594 CHECKS=58 NODES=30	SAT TT=0.992	SAT TT=0.556 CHECKS=524982 NODES=30	SAT TT=0.555 CHECKS=524982 NODES=30	SAT TT=0.557 CHECKS=524982 NODES=30	SAT TT=16.514	SAT TT=2.611	SAT TT=0.061	SAT TT=0.557 CHECKS=524982 NODES=30	SAT TT=0.524 CHECKS=524982 NODES=30	SAT TT=1.504	SAT TT=1.413	? (TO) TT=1800.29	SAT TT=0.058	SAT TT=0.033 CHECKS=730645 NODES=30	SAT TT=0.031 CHECKS=730645 NODES=30
hanoi/ hanoi-6_ext.xml	SAT TT=0.235	SAT TT=0.741 CHECKS=570583 NODES=0	SAT TT=0.735 CHECKS=570582 NODES=62	SAT TT=0.235	SAT TT=0.858	SAT TT=0.864	SAT TT=2.914 NODES=0	SAT TT=3.135 NODES=62	SAT TT=2.754 CHECKS=122 NODES=62	SAT TT=14.184	SAT TT=20.975 CHECKS=10703232 NODES=62	SAT TT=21.454 CHECKS=10703232 NODES=62	SAT TT=21.078 CHECKS=10703232 NODES=62	SAT TT=313.84	? (problem) TT=100.119	SAT TT=0.552	SAT TT=21.434 CHECKS=10703232 NODES=62	SAT TT=21.556 CHECKS=10703232 NODES=62	SAT TT=4.183	SAT TT=5.707	? (TO) TT=1800.1	SAT TT=0.573	SAT TT=0.528 CHECKS=15463800 NODES=62	SAT TT=0.523 CHECKS=15463800 NODES=62
hanoi/ hanoi-7_ext.xml	SAT TT=1.007	SAT TT=1.007 CHECKS=10427738 NODES=0	SAT TT=1.039 CHECKS=10427737 NODES=126	SAT TT=2.034	SAT TT=10.163	SAT TT=9.748	SAT TT=35.954 NODES=0	SAT TT=39.442 NODES=126	SAT TT=35.85 CHECKS=250 NODES=126	? (MO) TT=13.207	? (MO) TT=29.06	? (MO) TT=30.63	? (MO) TT=31.02	? (TO) TT=1800	? (problem) TT=76.781	SAT TT=19.78	? (MO) TT=30.62	? (MO) TT=29.09	SAT TT=69.672	SAT TT=28.607	? (TO) TT=1800.97	SAT TT=19.808	? (MO) TT=3.397	? (MO) TT=3.123
langford/ langford-2-4-ext.xml	SAT TT=0.002	SAT TT=0.621 CHECKS=250 NODES=0	SAT TT=0.628 CHECKS=224 NODES=8	SAT TT=0.019	SAT TT=0.064	SAT TT=0.057	Wrong Cert. TT=0.306 NODES=9	Wrong Cert. TT=0.317 NODES=8	SAT TT=0.301 CHECKS=872 NODES=8	SAT TT=0.018	Wrong Cert. TT=0.04 CHECKS=1369 NODES=8	SAT TT=0.037 CHECKS=1126 NODES=9	Wrong Cert. TT=0.039 CHECKS=1277 NODES=8	SAT TT=0.899	SAT TT=0.926	SAT TT=0.013	Wrong Cert. TT=0.037 CHECKS=1277 NODES=8	SAT TT=0.04 CHECKS=1188 NODES=8	SAT TT=0.585	SAT TT=0.586	SAT TT=0.668	SAT TT=0.013	SAT TT=0.002 CHECKS=1253 NODES=8	SAT TT=0.002 CHECKS=1253 NODES=8
langford/ langford-3-11-ext.xml	UNSAT TT=1.698	UNSAT TT=5.978 CHECKS=12575922 NODES=22744	UNSAT TT=5.516 CHECKS=12124108 NODES=21936	UNSAT TT=4.915	UNSAT TT=4.699	UNSAT TT=5.021	UNSAT TT=41.013 NODES=23176	UNSAT TT=23.515 NODES=18927	UNSAT TT=19.32 CHECKS=380136 NODES=18970	UNSAT TT=15.908	Wrong Cert. TT=0.625 CHECKS=312004 NODES=33	UNSAT TT=0.531 CHECKS=46484 CHECKS=92968 NODES=0 NODES=0	Wrong Cert. TT=0.59 CHECKS=287586 NODES=33	UNSAT TT=11.886	UNSAT TT=30.906	UNSAT TT=13.549	Wrong Cert. TT=0.655 CHECKS=287586 NODES=33	UNSAT TT=0.414 CHECKS=46484 NODES=0	UNSAT TT=54.293	UNSAT TT=54.338	UNSAT TT=34.85	UNSAT TT=14.11	UNSAT TT=1.698 CHECKS=52402100 NODES=29457	UNSAT TT=1.757 CHECKS=52402100 NODES=29457
langford/ langford-3-9-ext.xml	SAT TT=0.037	SAT TT=0.921 CHECKS=151110 NODES=491	SAT TT=0.838 CHECKS=129940 NODES=489	SAT TT=0.063	SAT TT=0.122	SAT TT=0.132	Wrong Cert. TT=0.925 NODES=330	Wrong Cert. TT=1.03 NODES=435	SAT TT=0.829 CHECKS=150272 NODES=347	SAT TT=0.322	Wrong Cert. TT=0.26 CHECKS=147414 NODES=27	Wrong UNSAT TT=0.235 CHECKS=21956 CHECKS=43912 NODES=0 NODES=0	Wrong Cert. TT=0.259 CHECKS=128995 NODES=27	SAT TT=1.945	SAT TT=1.583	SAT TT=0.038	Wrong Cert. TT=0.271 CHECKS=128995 NODES=27	Wrong UNSAT TT=0.207 CHECKS=21956 NODES=0	SAT TT=1.61	SAT TT=1.563	SAT TT=2.542	SAT TT=0.037	SAT TT=0.069 CHECKS=704453 NODES=369	SAT TT=0.07 CHECKS=704453 NODES=369
langford/ slangford-3-11-ext.xml	UNSAT TT=1.66	UNSAT TT=6.055 CHECKS=12948930 NODES=23317	UNSAT TT=6.03 CHECKS=12867892 NODES=23255	UNSAT TT=6.607	UNSAT TT=4.974	UNSAT TT=4.761	UNSAT TT=0.828 NODES=0	UNSAT TT=0.82 NODES=0	UNSAT TT=17.916 CHECKS=381932 NODES=18891	UNSAT TT=94.36	Wrong Cert. TT=0.601 CHECKS=305811 NODES=33	UNSAT TT=0.534 CHECKS=46484 CHECKS=92968 NODES=0 NODES=0	Wrong Cert. TT=0.653 CHECKS=312632 NODES=33	UNSAT TT=14.282	UNSAT TT=32.411	UNSAT TT=18.127	Wrong Cert. TT=0.616 CHECKS=312632 NODES=33	UNSAT TT=0.436 CHECKS=46484 NODES=0	UNSAT TT=65.886	UNSAT TT=75	UNSAT TT=536.191	UNSAT TT=17.454	UNSAT TT=1.66 CHECKS=53561900 NODES=30645	UNSAT TT=1.852 CHECKS=53561900 NODES=30645

Some statistics...

	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	buggy_2_5 2007-01-08	buggy_2_5_s 2007-01-08	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	Diarmuid-rndi 2006-12-21	Diarmuid-rndi 2007-01-22	Diarmuid-wtdi 2006-12-21	galac 1	galacJ beta 1	Mistral 2006-12-04	rjw-solver 2006-12-09	rjw-solver 2007-01-21	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04	VALCSP 3.0	VALCSP 3.1
Number of times the solver is able to give the best known answer	9	9	9	9	9	0	0	9	8	0	0	0	8	7	9	0	0	9	9	6	9	8	0
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	6	0