CSP 2008 Competition: results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out
T1=...	Time to get the best model (T1): this is the wall clock time (in seconds) ellapsed when the solver found its best model. This time is obtained from the 'o lines'. It is useful to compare both complete and incomplete solvers. It doesn't take into account the time to prove optimality of the solution.

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC
Abscon 112v4 ESAC
bpsolver 2008-06-27
casper zao
casper zito
choco2_dwdeg 2008-06-26
choco2_impwdeg 2008-06-26
Concrete + CPS4J 2008-05-30
Concrete + CSP4J - Tabu Engine 2008-05-30
Concrete + CSP4J - WMC Engine 2008-05-30
cpHydra k_10
cpHydra k_40
galac E
galac M
MDG-noprobe 2008-06-27
MDG-probe 2008-06-27
Minion/Tailor 2008-07-04
Mistral-option 1.314
Mistral-prime 1.313
SAT4J CSP 2008-06-13
spider 2008-06-14
Sugar v1.13+minisat
Sugar v1.13+picosat

Bench name	Best results on this instance	Abscon 112v4 AC	Abscon 112v4 ESAC	bpsolver 2008-06-27	casper zao	casper zito	choco2_dwdeg 2008-06-26	choco2_impwdeg 2008-06-26	Concrete + CPS4J 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Concrete + CSP4J - WMC Engine 2008-05-30	cpHydra k_10	cpHydra k_40	galac E	galac M	MDG-noprobe 2008-06-27	MDG-probe 2008-06-27	Minion/Tailor 2008-07-04	Mistral-option 1.314	Mistral-prime 1.313	SAT4J CSP 2008-06-13	spider 2008-06-14	Sugar v1.13+minisat	Sugar v1.13+picosat
csp/hanoi/ normalized-hanoi-3_ext.xml	SAT TT=0.006	SAT TT=0.611 CHECKS=164 NODES=6	SAT TT=0.593 CHECKS=164 NODES=0	SAT TT=0.027	SAT TT=0.006 CHECKS=17	SAT TT=0.009 CHECKS=17	SAT TT=0.327 ASSIGNMENTS=1	SAT TT=0.338 ASSIGNMENTS=1	SAT TT=0.284 CHECKS=145 NODES=6	SAT TT=0.273 CHECKS=165 NODES=0	SAT TT=0.275 CHECKS=165 NODES=0	SAT TT=0.02 ASSIGNMENTS=0 CHECKS=16	SAT TT=0.017 ASSIGNMENTS=0 CHECKS=16	SAT TT=0.962	SAT TT=0.837	SAT TT=0.007 ASSIGNMENTS=0 CHECKS=16	SAT TT=0.007 ASSIGNMENTS=0 CHECKS=16	SAT TT=0.269	SAT TT=0.006 ASSIGNMENTS=0 CHECKS=16	SAT TT=0.006 ASSIGNMENTS=0 CHECKS=16	SAT TT=0.702	SAT TT=0.01 ASSIGNMENTS=7 CHECKS=835	SAT TT=0.462	SAT TT=0.482
csp/hanoi/ normalized-hanoi-4_ext.xml	SAT TT=0.007	SAT TT=0.619 CHECKS=2211 NODES=14	SAT TT=0.618 CHECKS=2210 NODES=0	SAT TT=0.032	SAT TT=0.018 CHECKS=49	SAT TT=0.019 CHECKS=49	SAT TT=0.37 ASSIGNMENTS=1	SAT TT=0.365 ASSIGNMENTS=1	SAT TT=0.329 CHECKS=2970 NODES=14	SAT TT=0.305 CHECKS=3157 NODES=0	SAT TT=0.305 CHECKS=3157 NODES=0	SAT TT=0.021 ASSIGNMENTS=0 CHECKS=48	SAT TT=0.018 ASSIGNMENTS=0 CHECKS=48	SAT TT=1.742	SAT TT=1.673	SAT TT=0.008 ASSIGNMENTS=0 CHECKS=48	SAT TT=0.007 ASSIGNMENTS=0 CHECKS=48	SAT TT=0.301	SAT TT=0.007 ASSIGNMENTS=0 CHECKS=48	SAT TT=0.008 ASSIGNMENTS=0 CHECKS=48	SAT TT=0.973	SAT TT=0.009 ASSIGNMENTS=14 CHECKS=13167	SAT TT=0.733	SAT TT=0.618
csp/hanoi/ normalized-hanoi-5_ext.xml	SAT TT=0.009	SAT TT=0.673 CHECKS=30105 NODES=30	SAT TT=1.231 CHECKS=30105 NODES=0	SAT TT=0.066	SAT TT=0.208 CHECKS=113	SAT TT=0.215 CHECKS=113	SAT TT=0.578 ASSIGNMENTS=1	SAT TT=0.571 ASSIGNMENTS=1	SAT TT=0.754 CHECKS=55621 NODES=30	SAT TT=0.372 CHECKS=58513 NODES=0	SAT TT=0.39 CHECKS=58513 NODES=0	SAT TT=0.021 ASSIGNMENTS=0 CHECKS=112	SAT TT=0.019 ASSIGNMENTS=0 CHECKS=112	SAT TT=19.202	SAT TT=14.622	SAT TT=0.01 ASSIGNMENTS=0 CHECKS=112	SAT TT=0.011 ASSIGNMENTS=0 CHECKS=112	? (problem) TT=0.443	SAT TT=0.009 ASSIGNMENTS=0 CHECKS=112	SAT TT=0.009 ASSIGNMENTS=0 CHECKS=112	SAT TT=1.543	SAT TT=0.101 ASSIGNMENTS=31 CHECKS=528638	SAT TT=1.397	SAT TT=1.684
csp/hanoi/ normalized-hanoi-6_ext.xml	SAT TT=0.041	SAT TT=0.868 CHECKS=570583 NODES=62	SAT TT=0.732 CHECKS=570583 NODES=0	SAT TT=0.356	SAT TT=5.463 CHECKS=241	SAT TT=5.519 CHECKS=241	SAT TT=3.422 ASSIGNMENTS=1	SAT TT=3.392 ASSIGNMENTS=1	SAT TT=5.704 CHECKS=1045260 NODES=62	SAT TT=0.533 CHECKS=1096760 NODES=0	SAT TT=0.55 CHECKS=1096760 NODES=0	SAT TT=0.062 ASSIGNMENTS=0 CHECKS=240	SAT TT=0.058 ASSIGNMENTS=0 CHECKS=240	SAT TT=274.011	SAT TT=271.254	SAT TT=0.046 ASSIGNMENTS=0 CHECKS=240	SAT TT=0.041 ASSIGNMENTS=0 CHECKS=240	? (problem) TT=1.33	SAT TT=0.048 ASSIGNMENTS=0 CHECKS=240	SAT TT=0.044 ASSIGNMENTS=0 CHECKS=240	SAT TT=4.14	SAT TT=0.925 ASSIGNMENTS=62 CHECKS=5075380	SAT TT=17.11	SAT TT=16.641
csp/hanoi/ normalized-hanoi-7_ext.xml	SAT TT=0.391	SAT TT=1.042 CHECKS=10427700 NODES=126	SAT TT=1.09 CHECKS=10427700 NODES=126	SAT TT=5.435	SAT TT=203.012 CHECKS=497	SAT TT=203.165 CHECKS=497	SAT TT=55.823 ASSIGNMENTS=1	SAT TT=55.218 ASSIGNMENTS=1	SAT TT=60.605 CHECKS=19969000 NODES=126	SAT TT=1.075 CHECKS=20567700 NODES=0	SAT TT=1.117 CHECKS=20567700 NODES=0	SAT TT=0.409 ASSIGNMENTS=0 CHECKS=496	SAT TT=0.442 ASSIGNMENTS=0 CHECKS=496	? (TO) TT=1800.03	? (TO) TT=1800.09	SAT TT=0.425 ASSIGNMENTS=0 CHECKS=496	SAT TT=0.391 ASSIGNMENTS=0 CHECKS=496	? (problem) TT=5.34	SAT TT=0.407 ASSIGNMENTS=0 CHECKS=496	SAT TT=0.425 ASSIGNMENTS=0 CHECKS=496	SAT TT=28.386	SAT TT=30.896 ASSIGNMENTS=127 CHECKS=185321000	SAT TT=475.201	SAT TT=368.853
csp/langford/ normalized-langford-2-4-ext.xml	SAT TT=0.005	SAT TT=0.669 CHECKS=242 NODES=8	SAT TT=0.704 CHECKS=242 NODES=0	SAT TT=0.029	SAT TT=0.011 CHECKS=3422	SAT TT=0.011 CHECKS=2609	SAT TT=0.395 ASSIGNMENTS=2	SAT TT=0.412 ASSIGNMENTS=2	SAT TT=0.315 CHECKS=270 NODES=10	SAT TT=0.324 CHECKS=2762 NODES=55	SAT TT=0.325 CHECKS=32159 NODES=349	SAT TT=0.017 ASSIGNMENTS=2 CHECKS=200	SAT TT=0.018 ASSIGNMENTS=2 CHECKS=200	SAT TT=0.809	SAT TT=0.812	SAT TT=0.006 ASSIGNMENTS=2 CHECKS=200	SAT TT=0.008 ASSIGNMENTS=2 CHECKS=192	SAT TT=0.274	SAT TT=0.005 ASSIGNMENTS=2 CHECKS=226	SAT TT=0.008 ASSIGNMENTS=2 CHECKS=200	SAT TT=0.715	SAT TT=0.009 ASSIGNMENTS=10 CHECKS=1024	SAT TT=0.503	SAT TT=0.499
csp/langford/ normalized-langford-3-11-ext.xml	UNSAT TT=1.162	UNSAT TT=7.291 CHECKS=10053800 NODES=34361	UNSAT TT=7.19 CHECKS=10075500 NODES=34337	UNSAT TT=2.278	UNSAT TT=77.72 CHECKS=38638200	UNSAT TT=83.566 CHECKS=60527400	UNSAT TT=2.831 ASSIGNMENTS=4009	UNSAT TT=5.009 ASSIGNMENTS=7542	UNSAT TT=9.355 CHECKS=3182420 NODES=19411	? (TO) TT=1800.09	? (TO) TT=1800.08	UNSAT TT=1.203 ASSIGNMENTS=8266 CHECKS=7322500	UNSAT TT=1.214 ASSIGNMENTS=8266 CHECKS=7322500	UNSAT TT=13.894	UNSAT TT=11.687	UNSAT TT=1.174 ASSIGNMENTS=8211 CHECKS=7306140	UNSAT TT=1.602 ASSIGNMENTS=11454 CHECKS=10053200	UNSAT TT=128.27	UNSAT TT=1.162 ASSIGNMENTS=8504 CHECKS=7670600	UNSAT TT=1.168 ASSIGNMENTS=8266 CHECKS=7322500	UNSAT TT=38.073	UNSAT TT=37.46 ASSIGNMENTS=38939 CHECKS=51069000	UNSAT TT=10.274	UNSAT TT=12.742
csp/langford/ normalized-langford-3-9-ext.xml	SAT TT=0.019	SAT TT=0.954 CHECKS=123859 NODES=438	SAT TT=0.977 CHECKS=140446 NODES=432	SAT TT=0.088	SAT TT=1.919 CHECKS=1011290	SAT TT=0.121 CHECKS=61258	SAT TT=0.808 ASSIGNMENTS=47	SAT TT=0.815 ASSIGNMENTS=93	SAT TT=0.555 CHECKS=5001 NODES=32	SAT TT=0.784 CHECKS=1932420 NODES=4662	? (TO) TT=1800.03	SAT TT=0.04 ASSIGNMENTS=110 CHECKS=81976	SAT TT=0.039 ASSIGNMENTS=110 CHECKS=81976	SAT TT=1.94	SAT TT=1.838	SAT TT=0.03 ASSIGNMENTS=110 CHECKS=81548	SAT TT=0.019 ASSIGNMENTS=21 CHECKS=14078	SAT TT=0.61	SAT TT=0.048 ASSIGNMENTS=249 CHECKS=236445	SAT TT=0.028 ASSIGNMENTS=110 CHECKS=81976	SAT TT=1.696	SAT TT=0.144 ASSIGNMENTS=47 CHECKS=59638	SAT TT=1.102	SAT TT=1.25
csp/langford/ normalized-slangford-3-11-ext.xml	UNSAT TT=1.246	UNSAT TT=7.164 CHECKS=10089200 NODES=34311	UNSAT TT=7.188 CHECKS=10107900 NODES=34344	UNSAT TT=2.366	UNSAT TT=291.228 CHECKS=132919000	UNSAT TT=89.689 CHECKS=48258100	UNSAT TT=3.31 ASSIGNMENTS=4019	UNSAT TT=6.048 ASSIGNMENTS=7542	UNSAT TT=9.424 CHECKS=3191300 NODES=19411	? (TO) TT=1800.01	? (TO) TT=1800.01	UNSAT TT=1.36 ASSIGNMENTS=8213 CHECKS=7771110	UNSAT TT=1.246 ASSIGNMENTS=8213 CHECKS=7771110	UNSAT TT=13.656	UNSAT TT=19.671	UNSAT TT=1.345 ASSIGNMENTS=8273 CHECKS=7820000	UNSAT TT=1.852 ASSIGNMENTS=11450 CHECKS=10681500	UNSAT TT=145.631	UNSAT TT=1.397 ASSIGNMENTS=8465 CHECKS=8144540	UNSAT TT=1.31 ASSIGNMENTS=8213 CHECKS=7771110	UNSAT TT=37.618	UNSAT TT=41.949 ASSIGNMENTS=50339 CHECKS=65965600	UNSAT TT=11.155	UNSAT TT=14.969

Some statistics...

	Abscon 112v4 AC	Abscon 112v4 ESAC	bpsolver 2008-06-27	casper zao	casper zito	choco2_dwdeg 2008-06-26	choco2_impwdeg 2008-06-26	Concrete + CPS4J 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	cpHydra k_10	cpHydra k_40	galac E	galac M	MDG-noprobe 2008-06-27	MDG-probe 2008-06-27	Minion/Tailor 2008-07-04	Mistral-option 1.314	Mistral-prime 1.313	SAT4J CSP 2008-06-13	spider 2008-06-14	Sugar v1.13+minisat	Sugar v1.13+picosat
Number of times the solver is able to give the best known answer	9	9	9	9	9	9	9	9	7	9	9	8	8	9	9	6	9	9	9	9	9	9
Number of times the solver is able to give the best known answer from an incomplete solver point of view (i.e. without considering optimality proof)	9	9	9	9	9	9	9	9	7	9	9	8	8	9	9	6	9	9	9	9	9	9
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	0	0	0	1	0	0	0	0	0	0	1	0	0	0	4	0	5	2	0	0	0	0
Number of times the solver is the best solver from an incomplete solver point of view (i.e. best known answer and best T1 time)	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0