MAX-CSP 2006 Competition: results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
AbsconMax 109 EPFC
AbsconMax 109 PFC
aolibdvo 2007-01-17
aolibpvo 2007-01-17
CSP4J - MaxCSP 2006-12-19
toolbar 2007-01-12
Toolbar_BTD 2007-01-12
Toolbar_MaxSat 2007-01-19
Toulbar2 2007-01-12

Bench name	Best results on this instance	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
MaxCSP/maxclique/ brock-200-1_ext.xml	MSAT TO f=5087 TT=2400.08	MSAT (TO) f=5085 TT=2400.07 T1=2206.16	MSAT (TO) f=5085 TT=2400.08 T1=35.16	MSAT (TO) f=5081 TT=2400.1 T1=76.57	MSAT (TO) f=5081 TT=2400.11 T1=70.43	? (problem) TT=3.415 NODES=0	? (TO) TT=2400.06	? (TO) TT=2400.09	? (TO) TT=2400.06	MSAT (TO) f=5087 TT=2400.08
MaxCSP/maxclique/ brock-200-2_ext.xml	MOPT f=10036 TT=282.955	MSAT (TO) f=10035 TT=2400.05 T1=813.22	MSAT (TO) f=10034 TT=2400.13 T1=39.34	? (TO) TT=2400.03	? (TO) TT=2400.02	? (problem) TT=17.566 NODES=0	OPTIMUM f=10036 TT=285.044	OPTIMUM f=10036 TT=282.955	? (TO) TT=2400.03	? (TO) TT=2400.12
MaxCSP/maxclique/ brock-400-1_ext.xml	MSAT TO f=20099 TT=2400.1	MSAT (TO) f=20097 TT=2400.13 T1=51.58	MSAT (TO) f=20099 TT=2400.1 T1=169.94	MSAT (TO) f=20091 TT=2400.07 T1=46.91	MSAT (TO) f=20091 TT=2400.11 T1=41.67	? (problem) TT=85.658 NODES=0	? (TO) TT=2400.05	? (TO) TT=2400.1	? (TO) TT=2400.05	? (TO) TT=2400.12
MaxCSP/maxclique/ brock-400-2_ext.xml	MSAT TO f=20036 TT=2400.14	MSAT (TO) f=20034 TT=2400.07 T1=811.96	MSAT (TO) f=20036 TT=2400.14 T1=188	MSAT (TO) f=20024 TT=2400.05 T1=29.58	MSAT (TO) f=20024 TT=2400.07 T1=26.64	? (problem) TT=84.738 NODES=0	? (TO) TT=2400.09	? (TO) TT=2400.09	? (TO) TT=2400.07	? (TO) TT=2400.08
MaxCSP/maxclique/ c-fat200-1_ext.xml	MOPT f=18378 TT=9.391	OPTIMUM f=18378 TT=91.78 T1=1.52	OPTIMUM f=18378 TT=87.103 T1=1.45	OPTIMUM f=18378 TT=9.481 T1=9.54	OPTIMUM f=18378 TT=9.391 T1=9.4	? (problem) TT=73.682 NODES=0	OPTIMUM f=18378 TT=21.794	OPTIMUM f=18378 TT=20.895	OPTIMUM f=18378 TT=6.355	OPTIMUM f=18378 TT=28.677
MaxCSP/maxclique/ hamming-6-2_ext.xml	MOPT f=224 TT=0.097	OPTIMUM f=224 TT=85.704 T1=0.59	OPTIMUM f=224 TT=129.39 T1=0.61	OPTIMUM f=224 TT=0.101 T1=0.1	OPTIMUM f=224 TT=0.097 T1=0.1	? (problem) TT=0.405 NODES=0	OPTIMUM f=224 TT=2.907	OPTIMUM f=224 TT=3.113	OPTIMUM f=224 TT=0.476	OPTIMUM f=224 TT=3.134
MaxCSP/maxclique/ hamming-6-4_ext.xml	MOPT f=1316 TT=0.927	OPTIMUM f=1316 TT=39.588 T1=0.83	OPTIMUM f=1316 TT=32.664 T1=0.78	OPTIMUM f=1316 TT=0.927 T1=0.92	OPTIMUM f=1316 TT=0.945 T1=0.94	? (problem) TT=0.735 NODES=0	OPTIMUM f=1316 TT=5.293	OPTIMUM f=1316 TT=5.42	OPTIMUM f=1316 TT=3.445	OPTIMUM f=1316 TT=6.259
MaxCSP/maxclique/ hamming-8-2_ext.xml	MOPT f=1152 TT=0.614	MSAT (TO) f=1152 TT=2400.07 T1=0.91	MSAT (TO) f=1152 TT=2400.09 T1=0.85	OPTIMUM f=1152 TT=0.671 T1=0.66	OPTIMUM f=1152 TT=0.614 T1=0.6	? (problem) TT=0.704 NODES=0	OPTIMUM f=1152 TT=9.846	OPTIMUM f=1152 TT=10.533	OPTIMUM f=1152 TT=2.844	OPTIMUM f=1152 TT=9.846
MaxCSP/maxclique/ hamming-8-4_ext.xml	MSAT TO f=11789 TT=2400.07	MSAT (TO) f=11789 TT=2400.07 T1=1.14	MSAT (TO) f=11789 TT=2400.08 T1=1.21	? (TO) TT=2400.09	? (TO) TT=2400.05	? (problem) TT=25.546 NODES=0	? (TO) TT=2400.19	? (TO) TT=2400.06	? (TO) TT=2400.1	? (TO) TT=2400.32
MaxCSP/maxclique/ johnson-8-2-4_ext.xml	MOPT f=172 TT=0.08	OPTIMUM f=172 TT=34.25 T1=3.92	OPTIMUM f=172 TT=34.43 T1=4.16	OPTIMUM f=172 TT=0.08 T1=0.09	OPTIMUM f=172 TT=0.081 T1=0.08	? (problem) TT=0.37 NODES=0	OPTIMUM f=172 TT=2.095	OPTIMUM f=172 TT=2.242	OPTIMUM f=172 TT=0.409	OPTIMUM f=172 TT=2.15
Bench name	Best results on this instance	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
MaxCSP/maxclique/ keller-4_ext.xml	MSAT TO f=5111 TT=2400.08	MSAT (TO) f=5109 TT=2400.05 T1=101.91	MSAT (TO) f=5111 TT=2400.08 T1=38.67	? (TO) TT=2400.03	? (TO) TT=2400.06	? (problem) TT=3.595 NODES=0	? (TO) TT=2400.1	? (TO) TT=2400.03	? (TO) TT=2400.04	MSAT (TO) f=5111 TT=2400.08
MaxCSP/maxclique/ mann-a27_ext.xml	MOPT f=828 TT=346.696	MSAT (TO) f=825 TT=2400.15 T1=340.65	MSAT (TO) f=825 TT=2400.16 T1=404.05	? (TO) TT=2400.1	? (TO) TT=2400.02	? (problem) TT=0.673 NODES=0	MSAT (TO) f=827 TT=2400.14	? (TO) TT=2400	? (TO) TT=2400.08	OPTIMUM f=828 TT=346.696
MaxCSP/ maxclique/ mann-a9_ext.xml	MOPT f=88 TT=0.189	OPTIMUM f=88 TT=797.586 T1=0.72	OPTIMUM f=88 TT=2183.24 T1=0.79	OPTIMUM f=88 TT=0.272 T1=0.27	OPTIMUM f=88 TT=0.189 T1=0.19	? (problem) TT=0.343 NODES=0	OPTIMUM f=88 TT=1.535	OPTIMUM f=88 TT=1.63	OPTIMUM f=88 TT=0.652	OPTIMUM f=88 TT=1.488
MaxCSP/maxclique/ p-hat300-1_ext.xml	MOPT f=33925 TT=291.942	MSAT (TO) f=33925 TT=2400.12 T1=268.65	MSAT (TO) f=33924 TT=2400.1 T1=2.12	MSAT (TO) f=33923 TT=2400.11 T1=73.12	OPTIMUM f=33925 TT=2239.19 T1=2239.69	? (problem) TT=258.078 NODES=0	OPTIMUM f=33925 TT=291.942	OPTIMUM f=33925 TT=308.519	? (TO) TT=2400.09	? (TO) TT=2400.12
MaxCSP/maxclique/ p-hat500-1_ext.xml	MSAT TO f=93189 TT=2400.1	MSAT (TO) f=93189 TT=2400.16 T1=818.15	MSAT (TO) f=93189 TT=2400.1 T1=2054.4	? (exit code) TT=0.001	? (exit code) TT=0	? (problem) TT=1951.43 NODES=0	? (TO) TT=2400.07	? (TO) TT=2400.01	? (TO) TT=2400.09	? (TO) TT=2400.02
MaxCSP/maxclique/ p-hat700-1_ext.xml	MSAT TO f=183660 TT=2400.16	MSAT (TO) f=183660 TT=2400.16 T1=1633.27	MSAT (TO) f=183659 TT=2400.1 T1=300.12	? (exit code) TT=0.001	? (exit code) TT=0.002	? (TO) TT=2400.05	? (TO) TT=2400.06	? (TO) TT=2400.01	? (MO) TT=37.37	? (TO) TT=2400.07
MaxCSP/maxclique/ san-200-0.7-1_ext.xml	MSAT TO f=6000 TT=2400.06	MSAT (TO) f=6000 TT=2400.1 T1=33.56	MSAT (TO) f=6000 TT=2400.06 T1=32.19	? (TO) TT=2400.11	? (TO) TT=2400.02	? (problem) TT=4.973 NODES=0	? (TO) TT=2400.14	? (TO) TT=2400.04	? (TO) TT=2400.07	? (TO) TT=2400.06
MaxCSP/maxclique/ san-200-0.9-1_ext.xml	MSAT TO f=2060 TT=2400.08	MSAT (TO) f=2060 TT=2400.12 T1=1.45	MSAT (TO) f=2060 TT=2400.08 T1=1.51	? (TO) TT=2400.11	? (TO) TT=2400.05	? (problem) TT=1.012 NODES=0	MSAT (TO) f=2036 TT=2400.08	? (TO) TT=2400.05	? (TO) TT=2400.12	? (TO) TT=2400.05
MaxCSP/maxclique/ san-400-0.5-1_ext.xml	MSAT TO f=39908 TT=2400.1	MSAT (TO) f=39908 TT=2400.13 T1=792.42	MSAT (TO) f=39908 TT=2400.1 T1=776.43	? (TO) TT=2400.04	? (TO) TT=2400.05	? (problem) TT=353.247 NODES=0	? (TO) TT=2400.08	? (TO) TT=2400.04	? (TO) TT=2400.04	? (TO) TT=2400.13
MaxCSP/maxclique/ sanr-200-0.7_ext.xml	MSAT TO f=6049 TT=2400.08	MSAT (TO) f=6048 TT=2400.13 T1=620.46	MSAT (TO) f=6049 TT=2400.08 T1=1733.25	MSAT (TO) f=6045 TT=2400.07 T1=9.05	MSAT (TO) f=6045 TT=2400.03 T1=8.43	? (problem) TT=4.771 NODES=0	? (TO) TT=2400.07	? (TO) TT=2400.05	? (TO) TT=2400.01	? (TO) TT=2400.06

Some statistics...

	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
Number of times the solver is able to give the best known answer	11	14	6	7	0	8	8	0	9
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	6	9	2	4	0	1	1	0	3