CSP 2006 Competition (second round): results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 109 ESAC
Abscon 109 AC
BPrologCSPSolver70a 2006-12-13
buggy_2_5 2007-01-08
buggy_2_5_s 2007-01-08
CSP4J - Combo 2006-12-19
CSP4J - MAC 2006-12-19
CSP4J - MAC 2007-01-16
CSPtoSAT+minisat 0.3
Diarmuid-rndi 2006-12-21
Diarmuid-rndi 2007-01-22
Diarmuid-wtdi 2006-12-21
galac 1
galacJ beta 1
Mistral 2006-12-04
rjw-solver 2006-12-09
rjw-solver 2007-01-21
sat4jCSP 1.7 RC BF3
SAT4JCSP-CACHED 1.7 RC BF3
sugar 0.40
Tramontane 2006-12-04
VALCSP 3.0
VALCSP 3.1

Bench name	Best results on this instance	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	buggy_2_5 2007-01-08	buggy_2_5_s 2007-01-08	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	Diarmuid-rndi 2006-12-21	Diarmuid-rndi 2007-01-22	Diarmuid-wtdi 2006-12-21	galac 1	galacJ beta 1	Mistral 2006-12-04	rjw-solver 2006-12-09	rjw-solver 2007-01-21	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04	VALCSP 3.0	VALCSP 3.1
driver/ driverlogw-01c-sat_ext.xml	SAT TT=0.006	SAT TT=0.773 CHECKS=1235 NODES=0	SAT TT=0.699 CHECKS=1022 NODES=71	SAT TT=0.024	SAT TT=0.088	SAT TT=0.089	SAT TT=0.424 NODES=78	SAT TT=0.407 NODES=73	SAT TT=0.443 CHECKS=2788 NODES=73	SAT TT=0.026	? (problem) TT=0.036	Wrong Cert. TT=0.071 CHECKS=2573 NODES=72	? (problem) TT=0.036	SAT TT=1.166	SAT TT=1.236	SAT TT=0.021	? (problem) TT=0.039	SAT TT=0.077 CHECKS=2800 NODES=73	SAT TT=0.844	SAT TT=0.789	SAT TT=0.058	SAT TT=0.021	SAT TT=0.006 CHECKS=2601 NODES=71	SAT TT=0.007 CHECKS=2601 NODES=70
driver/ driverlogw-02c-sat_ext.xml	SAT TT=0.112	SAT TT=1.762 CHECKS=691438 NODES=1088	SAT TT=1.562 CHECKS=397877 NODES=1640	SAT TT=1.257	SAT TT=0.939	SAT TT=0.959	SAT TT=3.12 NODES=465	SAT TT=2.638 NODES=421	SAT TT=2.678 CHECKS=117036 NODES=441	SAT TT=0.276	? (problem) TT=0.054	? (problem) TT=0.062	? (problem) TT=0.053	SAT TT=3.173	SAT TT=2.791	SAT TT=0.325	? (problem) TT=0.051	? (problem) TT=0.06	SAT TT=4.05	SAT TT=4.161	SAT TT=3.836	SAT TT=0.355	SAT TT=0.112 CHECKS=529019 NODES=1870	SAT TT=0.11 CHECKS=529019 NODES=1870
driver/ driverlogw-04c-sat_ext.xml	SAT TT=0.094	SAT TT=1.526 CHECKS=452318 NODES=590	SAT TT=1.323 CHECKS=156952 NODES=513	SAT TT=0.245	SAT TT=0.822	SAT TT=0.803	SAT TT=2.545 NODES=330	SAT TT=2.394 NODES=332	SAT TT=2.296 CHECKS=100674 NODES=345	SAT TT=0.303	? (problem) TT=0.053	? (problem) TT=0.06	? (problem) TT=0.049	SAT TT=3.154	SAT TT=2.652	SAT TT=0.279	? (problem) TT=0.048	? (problem) TT=0.058	SAT TT=4.062	SAT TT=4.03	SAT TT=3.729	SAT TT=0.275	SAT TT=0.094 CHECKS=264073 NODES=683	SAT TT=0.101 CHECKS=264073 NODES=682
driver/ driverlogw-05c-sat_ext.xml	SAT TT=0.145	SAT TT=2.111 CHECKS=664672 NODES=0	SAT TT=1.532 CHECKS=283432 NODES=964	SAT TT=1.832	SAT TT=1.233	SAT TT=1.265	SAT TT=5.841 NODES=430	SAT TT=5.163 NODES=374	SAT TT=5.179 CHECKS=188044 NODES=375	SAT TT=0.43	? (problem) TT=0.063	? (problem) TT=0.068	? (problem) TT=0.063	SAT TT=4.242	SAT TT=3.226	SAT TT=0.404	? (problem) TT=0.059	? (problem) TT=0.072	? (MO) TT=4.979	SAT TT=4.565	SAT TT=5.05	SAT TT=0.395	SAT TT=0.145 CHECKS=351402 NODES=783	Wrong Cert. TT=0.153 CHECKS=351383 NODES=779
driver/ driverlogw-08cc-sat_ext.xml	SAT TT=0.528	SAT TT=4.177 CHECKS=4157158 NODES=3333	SAT TT=2.778 CHECKS=2277039 NODES=4118	SAT TT=332.337	SAT TT=1.642	SAT TT=1.752	SAT TT=17.46 NODES=751	SAT TT=16.237 NODES=646	SAT TT=16.719 CHECKS=291332 NODES=630	SAT TT=0.772	? (problem) TT=0.076	? (problem) TT=0.082	? (problem) TT=0.069	SAT TT=5.574	SAT TT=4.262	SAT TT=0.86	? (problem) TT=0.072	? (problem) TT=0.077	SAT TT=5.509	SAT TT=5.283	SAT TT=7.908	SAT TT=0.877	SAT TT=0.528 CHECKS=4059300 NODES=9331	SAT TT=0.55 CHECKS=4059300 NODES=9331
driver/ driverlogw-08c-sat_ext.xml	SAT TT=0.534	SAT TT=4.728 CHECKS=4961728 NODES=4953	SAT TT=3.057 CHECKS=2614961 NODES=4453	SAT TT=200.224	SAT TT=1.654	SAT TT=1.737	SAT TT=17.376 NODES=751	SAT TT=16.201 NODES=646	SAT TT=17.116 CHECKS=291332 NODES=630	SAT TT=0.673	? (problem) TT=0.07	? (problem) TT=0.084	? (problem) TT=0.074	SAT TT=5.613	SAT TT=4.387	SAT TT=0.916	? (problem) TT=0.078	? (problem) TT=0.084	SAT TT=5.285	SAT TT=5.944	SAT TT=7.21	SAT TT=0.875	SAT TT=0.534 CHECKS=4090100 NODES=8760	Wrong Cert. TT=0.559 CHECKS=4090080 NODES=8756
driver/ driverlogw-09-sat_ext.xml	SAT TT=1.897	SAT TT=26.787 CHECKS=30048711 NODES=29052	SAT TT=14.105 CHECKS=18229345 NODES=16248	? (TO) TT=1800.1	SAT TT=38.475	SAT TT=40.419	SAT TT=90.334 NODES=2094	SAT TT=152.716 NODES=4293	SAT TT=87.562 CHECKS=739554 NODES=1563	SAT TT=1.897	? (problem) TT=0.119	? (problem) TT=0.12	? (problem) TT=0.113	SAT TT=11.485	SAT TT=8.113	SAT TT=20.845	? (problem) TT=0.115	? (problem) TT=0.12	SAT TT=6.328	SAT TT=6.239	SAT TT=21.636	SAT TT=46.593	SAT TT=2.803 CHECKS=25337500 NODES=28624	Wrong Cert. TT=2.782 CHECKS=25337500 NODES=28618

Some statistics...

	Abscon 109 ESAC	Abscon 109 AC	BPrologCSPSolver70a 2006-12-13	buggy_2_5 2007-01-08	buggy_2_5_s 2007-01-08	CSP4J - Combo 2006-12-19	CSP4J - MAC 2006-12-19	CSP4J - MAC 2007-01-16	CSPtoSAT+minisat 0.3	Diarmuid-rndi 2006-12-21	Diarmuid-rndi 2007-01-22	Diarmuid-wtdi 2006-12-21	galac 1	galacJ beta 1	Mistral 2006-12-04	rjw-solver 2006-12-09	rjw-solver 2007-01-21	sat4jCSP 1.7 RC BF3	SAT4JCSP-CACHED 1.7 RC BF3	sugar 0.40	Tramontane 2006-12-04	VALCSP 3.0	VALCSP 3.1
Number of times the solver is able to give the best known answer	7	7	6	7	7	0	0	7	7	0	0	0	7	7	7	0	0	6	7	7	7	7	0
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	6	0