MAX-CSP 2006 Competition: results by benchmark

Results by benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory random problems

This page displays the results of the different solvers for each benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory random problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
AbsconMax 109 EPFC
AbsconMax 109 PFC
aolibdvo 2007-01-17
aolibpvo 2007-01-17
CSP4J - MaxCSP 2006-12-19
toolbar 2007-01-12
Toolbar_BTD 2007-01-12
Toolbar_MaxSat 2007-01-19
Toulbar2 2007-01-12

Bench name	Best results on this instance	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
random/rand-10-20-10/ rand-10-20-10-5-10000-0_ext.xml	MOPT f=3 TT=1160.59	OPTIMUM f=3 TT=1160.59 T1=166.48	MSAT (TO) f=2 TT=2400.07 T1=9.98	Signal TT=0.004	Signal TT=0.005	? TT=0.511	? (exit code) TT=0.048	? (exit code) TT=0.05	? (exit code) TT=0.042	? (TO) TT=2400.06
random/rand-10-20-10/ rand-10-20-10-5-10000-10_ext.xml	MSAT TO f=3 TT=2400.11	MSAT (TO) f=3 TT=2400.11 T1=1054.68	MSAT (TO) f=3 TT=2400.14 T1=1958.54	Signal TT=0.004	Signal TT=0.006	? TT=0.537	? (exit code) TT=0.044	? (exit code) TT=0.052	? (exit code) TT=0.045	? (TO) TT=2400.08
random/rand-10-20-10/ rand-10-20-10-5-10000-11_ext.xml	MSAT TO f=3 TT=2400.06	MSAT (TO) f=3 TT=2400.06 T1=1187.58	MSAT (TO) f=3 TT=2400.06 T1=695.55	Signal TT=0.004	Signal TT=0.004	? TT=0.508	? (exit code) TT=0.051	? (exit code) TT=0.048	? (exit code) TT=0.041	MSAT (TO) f=1 TT=2400.06
random/rand-10-20-10/ rand-10-20-10-5-10000-12_ext.xml	MOPT f=3 TT=750.774	MSAT (TO) f=3 TT=2400.13 T1=2029.9	OPTIMUM f=3 TT=750.774 T1=531.45	Signal TT=0.005	Signal TT=0.005	? TT=0.512	? (exit code) TT=0.048	? (exit code) TT=0.048	? (exit code) TT=0.044	? (TO) TT=2400.2
random/rand-10-20-10/ rand-10-20-10-5-10000-13_ext.xml	MSAT TO f=2 TT=2400.04	MSAT (TO) f=2 TT=2400.04 T1=23.9	MSAT (TO) f=2 TT=2400.12 T1=24.26	Signal TT=0.005	Signal TT=0.005	? TT=0.505	? (exit code) TT=0.05	? (exit code) TT=0.045	? (exit code) TT=0.041	? (TO) TT=2400.11
random/rand-10-20-10/ rand-10-20-10-5-10000-14_ext.xml	MSAT TO f=3 TT=2400.11	MSAT (TO) f=3 TT=2400.13 T1=112.19	MSAT (TO) f=3 TT=2400.11 T1=624.69	Signal TT=0.004	Signal TT=0.004	? TT=0.539	? (exit code) TT=0.049	? (exit code) TT=0.048	? (exit code) TT=0.04	MSAT (TO) f=2 TT=2400.07
random/rand-10-20-10/ rand-10-20-10-5-10000-15_ext.xml	MOPT f=3 TT=1153.56	OPTIMUM f=3 TT=2092.17 T1=53.67	OPTIMUM f=3 TT=1153.56 T1=152.34	Signal TT=0.004	Signal TT=0.005	? TT=0.537	? (exit code) TT=0.051	? (exit code) TT=0.048	? (exit code) TT=0.041	MSAT (TO) f=1 TT=2400.13
random/rand-10-20-10/ rand-10-20-10-5-10000-16_ext.xml	MSAT TO f=2 TT=2400.12	MSAT (TO) f=2 TT=2400.13 T1=11.45	MSAT (TO) f=2 TT=2400.12 T1=12.28	Signal TT=0.006	Signal TT=0.004	? TT=0.521	? (exit code) TT=0.049	? (exit code) TT=0.048	? (exit code) TT=0.042	MSAT (TO) f=2 TT=2400.14
random/rand-10-20-10/ rand-10-20-10-5-10000-17_ext.xml	MOPT f=3 TT=1022.9	OPTIMUM f=3 TT=1022.9 T1=851.51	OPTIMUM f=3 TT=1154.08 T1=984.42	Signal TT=0.005	Signal TT=0.004	? TT=0.541	? (exit code) TT=0.049	? (exit code) TT=0.049	? (exit code) TT=0.044	MSAT (TO) f=1 TT=2400.14
random/rand-10-20-10/ rand-10-20-10-5-10000-18_ext.xml	MSAT TO f=2 TT=2400.08	MSAT (TO) f=2 TT=2400.08 T1=17.14	MSAT (TO) f=2 TT=2400.16 T1=17.34	Signal TT=0.004	Signal TT=0.004	? TT=0.537	? (exit code) TT=0.048	? (exit code) TT=0.049	? (exit code) TT=0.04	? (TO) TT=2400.05
Bench name	Best results on this instance	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
random/rand-10-20-10/ rand-10-20-10-5-10000-19_ext.xml	MSAT TO f=3 TT=2400.12	MSAT (TO) f=2 TT=2400.05 T1=32.28	MSAT (TO) f=3 TT=2400.12 T1=1889.43	Signal TT=0.003	Signal TT=0.006	? TT=0.528	? (exit code) TT=0.045	? (exit code) TT=0.052	? (exit code) TT=0.044	? (TO) TT=2400.23
random/rand-10-20-10/ rand-10-20-10-5-10000-1_ext.xml	MOPT f=3 TT=670.579	OPTIMUM f=3 TT=1986.79 T1=1060.31	OPTIMUM f=3 TT=670.579 T1=284.33	Signal TT=0.005	Signal TT=0.004	? TT=0.504	? (exit code) TT=0.046	? (exit code) TT=0.05	? (exit code) TT=0.042	? (TO) TT=2400.05
random/rand-10-20-10/ rand-10-20-10-5-10000-2_ext.xml	MSAT TO f=2 TT=2400.04	MSAT (TO) f=2 TT=2400.08 T1=34.3	MSAT (TO) f=2 TT=2400.04 T1=33.17	Signal TT=0.004	Signal TT=0.005	? TT=0.537	? (exit code) TT=0.045	? (exit code) TT=0.05	? (exit code) TT=0.044	MSAT (TO) f=2 TT=2400.14
random/rand-10-20-10/ rand-10-20-10-5-10000-3_ext.xml	MSAT TO f=3 TT=2400.1	MSAT (TO) f=3 TT=2400.1 T1=808.85	MSAT (TO) f=3 TT=2400.14 T1=870.49	Signal TT=0.003	Signal TT=0.004	? TT=0.526	? (exit code) TT=0.052	? (exit code) TT=0.048	? (exit code) TT=0.041	? (TO) TT=2400.11
random/rand-10-20-10/ rand-10-20-10-5-10000-4_ext.xml	MOPT f=3 TT=406.94	OPTIMUM f=3 TT=681.578 T1=75.01	OPTIMUM f=3 TT=406.94 T1=86.55	Signal TT=0.005	Signal TT=0.006	? TT=0.537	? (exit code) TT=0.046	? (exit code) TT=0.051	? (exit code) TT=0.046	MSAT (TO) f=1 TT=2400.07
random/rand-10-20-10/ rand-10-20-10-5-10000-5_ext.xml	MOPT f=3 TT=1309.67	OPTIMUM f=3 TT=2190.48 T1=960.27	OPTIMUM f=3 TT=1309.67 T1=192.75	Signal TT=0.005	Signal TT=0.005	? TT=0.532	? (exit code) TT=0.049	? (exit code) TT=0.047	? (exit code) TT=0.045	? (TO) TT=2400.05
random/rand-10-20-10/ rand-10-20-10-5-10000-6_ext.xml	MSAT TO f=2 TT=2400.09	MSAT (TO) f=2 TT=2400.09 T1=39.04	MSAT (TO) f=2 TT=2400.12 T1=38.04	Signal TT=0.005	Signal TT=0.005	? TT=0.518	? (exit code) TT=0.047	? (exit code) TT=0.049	? (exit code) TT=0.042	? (TO) TT=2400.05
random/rand-10-20-10/ rand-10-20-10-5-10000-7_ext.xml	MOPT f=3 TT=242.092	OPTIMUM f=3 TT=2106.49 T1=1255.36	OPTIMUM f=3 TT=242.092 T1=48.85	Signal TT=0.005	Signal TT=0.004	? TT=0.539	? (exit code) TT=0.049	? (exit code) TT=0.05	? (exit code) TT=0.042	? (TO) TT=2400.07
random/rand-10-20-10/ rand-10-20-10-5-10000-8_ext.xml	MOPT f=3 TT=893.957	MSAT (TO) f=3 TT=2400.08 T1=1139.71	OPTIMUM f=3 TT=893.957 T1=182.29	Signal TT=0.005	Signal TT=0.005	? TT=0.532	? (exit code) TT=0.048	? (exit code) TT=0.047	? (exit code) TT=0.042	? (TO) TT=2400.07
random/rand-10-20-10/ rand-10-20-10-5-10000-9_ext.xml	MSAT TO f=3 TT=2400.15	MSAT (TO) f=2 TT=2400.1 T1=12.01	MSAT (TO) f=3 TT=2400.15 T1=136.18	Signal TT=0.005	Signal TT=0.005	? TT=0.533	? (exit code) TT=0.047	? (exit code) TT=0.052	? (exit code) TT=0.044	? (TO) TT=2400.05
Bench name	Best results on this instance	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
random/rand-3-20-20/ rand-3-20-20-60-632-11_ext.xml	MOPT f=59 TT=237.937	MSAT (TO) f=59 TT=2400.13 T1=302.92	OPTIMUM f=59 TT=1775.96 T1=567.7	OPTIMUM f=59 TT=681.298 T1=687.13	OPTIMUM f=59 TT=890.604 T1=894.02	MSAT (TO) f=54 TT=2400.05	OPTIMUM f=59 TT=237.937	OPTIMUM f=59 TT=251.245	? (exit code) TT=6.952	OPTIMUM f=59 TT=1895.53
random/rand-3-20-20/ rand-3-20-20-60-632-16_ext.xml	MOPT f=57 TT=220.504	MSAT (TO) f=57 TT=2400.1 T1=1357.2	OPTIMUM f=57 TT=2330.78 T1=525.03	OPTIMUM f=57 TT=1328.73 T1=1334.99	OPTIMUM f=57 TT=1258.73 T1=1268.06	MSAT (TO) f=54 TT=2400.08	OPTIMUM f=57 TT=220.504	OPTIMUM f=57 TT=254.93	? (exit code) TT=23.788	OPTIMUM f=57 TT=1394.27
random/rand-3-20-20/ rand-3-20-20-60-632-18_ext.xml	MOPT f=54 TT=30.804	OPTIMUM f=54 TT=427.062 T1=208.87	OPTIMUM f=54 TT=368.405 T1=230.7	OPTIMUM f=54 TT=60.126 T1=60.23	OPTIMUM f=54 TT=51.137 T1=51.15	MSAT (TO) f=48 TT=2400.01	OPTIMUM f=54 TT=30.804	OPTIMUM f=54 TT=32.051	? (exit code) TT=29.988	OPTIMUM f=54 TT=145.212
random/rand-3-20-20/ rand-3-20-20-60-632-19_ext.xml	MOPT f=57 TT=59.743	OPTIMUM f=57 TT=1771.72 T1=313.84	OPTIMUM f=57 TT=873.002 T1=173.89	? (TO) TT=2400.02	OPTIMUM f=57 TT=985.473 T1=989.78	MSAT (TO) f=51 TT=2400.01	OPTIMUM f=57 TT=59.743	OPTIMUM f=57 TT=63.856	? (exit code) TT=30.086	OPTIMUM f=57 TT=507.423
random/rand-3-20-20/ rand-3-20-20-60-632-21_ext.xml	MOPT f=57 TT=47.133	OPTIMUM f=57 TT=1210.88 T1=700.64	OPTIMUM f=57 TT=666.384 T1=431.41	OPTIMUM f=57 TT=47.133 T1=47.55	OPTIMUM f=57 TT=48.424 T1=48.67	MSAT (TO) f=54 TT=2400.02	OPTIMUM f=57 TT=377.661	OPTIMUM f=57 TT=413.7	? (exit code) TT=30.404	OPTIMUM f=57 TT=235.33
random/rand-3-20-20/ rand-3-20-20-60-632-22_ext.xml	MOPT f=58 TT=234.383	MSAT (TO) f=57 TT=2400.1 T1=260.69	MSAT (TO) f=58 TT=2400.12 T1=2020.79	OPTIMUM f=58 TT=1523.63 T1=1524.02	OPTIMUM f=58 TT=1182.77 T1=1186.77	MSAT (TO) f=52 TT=2400.05	OPTIMUM f=58 TT=234.383	OPTIMUM f=58 TT=254.134	? (exit code) TT=30.841	OPTIMUM f=58 TT=1626.72
random/rand-3-20-20/ rand-3-20-20-60-632-23_ext.xml	MOPT f=56 TT=36.664	OPTIMUM f=56 TT=1202.95 T1=346.05	OPTIMUM f=56 TT=426.199 T1=84.59	? (TO) TT=2400.04	OPTIMUM f=56 TT=260.789 T1=262.74	MSAT (TO) f=51 TT=2400.01 T1=1.43	OPTIMUM f=56 TT=36.664	OPTIMUM f=56 TT=41.066	? (exit code) TT=29.528	OPTIMUM f=56 TT=330.067
random/rand-3-20-20/ rand-3-20-20-60-632-2_ext.xml	MOPT f=59 TT=179.693	OPTIMUM f=59 TT=1916.42 T1=1395.81	OPTIMUM f=59 TT=1459.22 T1=934	? (TO) TT=2400.05	OPTIMUM f=59 TT=689.13 T1=689.53	MSAT (TO) f=53 TT=2400.1 T1=1.59	OPTIMUM f=59 TT=179.693	OPTIMUM f=59 TT=223.518	? (exit code) TT=32.093	OPTIMUM f=59 TT=207.483
random/rand-3-20-20/ rand-3-20-20-60-632-30_ext.xml	MOPT f=59 TT=202.418	OPTIMUM f=59 TT=660.637 T1=58.05	OPTIMUM f=59 TT=672.799 T1=362.14	OPTIMUM f=59 TT=687.558 T1=693.81	OPTIMUM f=59 TT=133.373 T1=134.37	MSAT (TO) f=49 TT=2400.05 T1=1.52	OPTIMUM f=59 TT=202.418	OPTIMUM f=59 TT=242.026	? (exit code) TT=19.919	OPTIMUM f=59 TT=439.5
random/rand-3-20-20/ rand-3-20-20-60-632-31_ext.xml	MOPT f=58 TT=255.417	MSAT (TO) f=58 TT=2400.12 T1=2292.63	OPTIMUM f=58 TT=1959.25 T1=1272.83	OPTIMUM f=58 TT=1534.2 T1=1536.2	OPTIMUM f=58 TT=234.158 T1=236	MSAT (TO) f=52 TT=2400.06	OPTIMUM f=58 TT=255.417	OPTIMUM f=58 TT=296.96	? (exit code) TT=1.11	OPTIMUM f=58 TT=667.865
Bench name	Best results on this instance	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
random/rand-3-20-20/ rand-3-20-20-60-632-33_ext.xml	MOPT f=58 TT=351.726	MSAT (TO) f=58 TT=2400.12 T1=1783.02	MSAT (TO) f=58 TT=2400.12 T1=1434.7	OPTIMUM f=58 TT=673.952 T1=679.9	OPTIMUM f=58 TT=370.698 T1=373.96	MSAT (TO) f=51 TT=2400.09	OPTIMUM f=58 TT=351.726	OPTIMUM f=58 TT=377.141	? (exit code) TT=30.108	OPTIMUM f=58 TT=1738.24
random/rand-3-20-20/ rand-3-20-20-60-632-34_ext.xml	MOPT f=58 TT=124.182	OPTIMUM f=58 TT=1099.01 T1=91.16	OPTIMUM f=58 TT=678.059 T1=71.5	OPTIMUM f=58 TT=1138.05 T1=1138.63	OPTIMUM f=58 TT=1119.15 T1=1124.04	MSAT (TO) f=50 TT=2400.05 T1=1.45	OPTIMUM f=58 TT=124.182	OPTIMUM f=58 TT=139.677	? (exit code) TT=29.649	OPTIMUM f=58 TT=1091.07
random/rand-3-20-20/ rand-3-20-20-60-632-35_ext.xml	MOPT f=56 TT=66.526	OPTIMUM f=56 TT=1524.62 T1=1159.54	OPTIMUM f=56 TT=642.466 T1=421.52	? (TO) TT=2400.08	OPTIMUM f=56 TT=254.003 T1=256.15	MSAT (TO) f=47 TT=2400.06 T1=1.43	OPTIMUM f=56 TT=66.526	OPTIMUM f=56 TT=72.969	? (exit code) TT=1.079	OPTIMUM f=56 TT=545.449
random/rand-3-20-20/ rand-3-20-20-60-632-38_ext.xml	MOPT f=55 TT=21.832	OPTIMUM f=55 TT=1068.5 T1=784.74	OPTIMUM f=55 TT=223.763 T1=115.32	OPTIMUM f=55 TT=323.739 T1=324.16	OPTIMUM f=55 TT=276.946 T1=279.35	MSAT (TO) f=49 TT=2400.08 T1=1.39	OPTIMUM f=55 TT=21.832	OPTIMUM f=55 TT=23.05	? (exit code) TT=23.954	OPTIMUM f=55 TT=145.917
random/rand-3-20-20/ rand-3-20-20-60-632-39_ext.xml	MOPT f=58 TT=85.381	OPTIMUM f=58 TT=1522.61 T1=337.06	OPTIMUM f=58 TT=673.634 T1=143.36	OPTIMUM f=58 TT=227.394 T1=229.48	OPTIMUM f=58 TT=909.556 T1=913.77	MSAT (TO) f=53 TT=2400.08	OPTIMUM f=58 TT=85.381	OPTIMUM f=58 TT=95.646	? (exit code) TT=30.501	OPTIMUM f=58 TT=568.788

Some statistics...

	AbsconMax 109 EPFC	AbsconMax 109 PFC	aolibdvo 2007-01-17	aolibpvo 2007-01-17	CSP4J - MaxCSP 2006-12-19	toolbar 2007-01-12	Toolbar_BTD 2007-01-12	Toolbar_MaxSat 2007-01-19	Toulbar2 2007-01-12
Number of times the solver is able to give the best known answer	26	32	11	0	0	15	15	0	17
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	11	18	1	0	0	14	0	0	2