MAX-CSP 2008 Competition: results by benchmark

Results by benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory quasi random

This page displays the results of the different solvers for each benchmark for category n-ary constraints in extension (N-ARY-EXT), subcategory quasi random

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out
T1=...	Time to get the best model (T1): this is the wall clock time (in seconds) ellapsed when the solver found its best model. This time is obtained from the 'o lines'. It is useful to compare both complete and incomplete solvers. It doesn't take into account the time to prove optimality of the solution.

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
AbsconMax 112 pc-d
AbsconMax 112 pc-w
Concrete + CSP4J - MCRW Engine 2008-05-30
Concrete + CSP4J - Tabu Engine 2008-05-30
Sugar v1.13+minisat
Sugar++ v1.13+minisat-inc
toulbar2 2008-06-27
toulbar2/BTD 2008-06-27

Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc	toulbar2 2008-06-27	toulbar2/BTD 2008-06-27
csp/bddLarge/ normalized-bdd-21-2713-15-79-10_ext.xml	MOPT f=88 TT=1162.19 T1=16.91	OPTIMUM f=88 TT=1285.93 T1=16.91 CHECKS=727170000 NODES=13755	OPTIMUM f=88 TT=1162.19 T1=17.01 CHECKS=675318000 NODES=12895	MSAT (TO) f=88 TT=3600.06	MSAT (TO) f=88 TT=3600.06	? TT=129.212	? TT=132.056	? (MO) TT=68.36	? (MO) TT=67.19
csp/bddLarge/ normalized-bdd-21-2713-15-79-12_ext.xml	MOPT f=125 TT=1405.93 T1=8.23	OPTIMUM f=125 TT=1591.69 T1=8.86 CHECKS=954422000 NODES=18942	OPTIMUM f=125 TT=1405.93 T1=8.23 CHECKS=889706000 NODES=17092	MSAT (TO) f=125 TT=3600.07	MSAT (TO) f=125 TT=3600.09	? TT=165.624	? TT=175.061	? (MO) TT=63.29	? (MO) TT=62.94
csp/bddLarge/ normalized-bdd-21-2713-15-79-15_ext.xml	MSAT TO f=701 TT=3600.03 T1=165.26	MSAT (TO) f=701 TT=3600.1 T1=2163.56	MSAT (TO) f=701 TT=3600.07 T1=165.26	MSAT (TO) f=701 TT=3600.08	MSAT (TO) f=701 TT=3600.03	? TT=137.243	? TT=136.944	? (MO) TT=66.85	? (MO) TT=66.75
csp/bddLarge/ normalized-bdd-21-2713-15-79-16_ext.xml	MOPT f=179 TT=1781.32 T1=3.69	OPTIMUM f=179 TT=1940.98 T1=4.22 CHECKS=1212100000 NODES=25250	OPTIMUM f=179 TT=1781.32 T1=3.69 CHECKS=1154530000 NODES=24156	MSAT (TO) f=179 TT=3600.09	MSAT (TO) f=179 TT=3600.09	? TT=123.074	? TT=125.15	? (MO) TT=66.86	? (MO) TT=62.57
csp/bddLarge/ normalized-bdd-21-2713-15-79-17_ext.xml	MOPT f=198 TT=1916.69 T1=7.25	OPTIMUM f=198 TT=2263.44 T1=8.1 CHECKS=1300620000 NODES=27073	OPTIMUM f=198 TT=1916.69 T1=7.25 CHECKS=1209160000 NODES=25489	MSAT (TO) f=198 TT=3600.02	MSAT (TO) f=198 TT=3600.05	? TT=122.619	? TT=129.684	? (MO) TT=63.08	? (MO) TT=62.68
csp/bddLarge/ normalized-bdd-21-2713-15-79-18_ext.xml	MSAT TO f=698 TT=3600.03	MSAT (TO) f=921 TT=3600.13 T1=43.09	MSAT (TO) f=921 TT=3600.15 T1=53.32	MSAT (TO) f=698 TT=3600.08	MSAT (TO) f=698 TT=3600.03	? TT=134.717	? TT=144.284	? (MO) TT=67.09	? (MO) TT=62.52
csp/bddLarge/ normalized-bdd-21-2713-15-79-1_ext.xml	MSAT TO f=758 TT=3600.04 T1=5.43	MSAT (TO) f=758 TT=3600.14 T1=5.43	MSAT (TO) f=758 TT=3600.11 T1=5.7	MSAT (TO) f=758 TT=3600.06	MSAT (TO) f=758 TT=3600.04	? TT=166.049	? TT=184.804	? (MO) TT=62.58	? (MO) TT=66.87
csp/bddLarge/ normalized-bdd-21-2713-15-79-20_ext.xml	MSAT TO f=420 TT=3600 T1=33.78	MSAT (TO) f=420 TT=3600.06 T1=33.78	MSAT (TO) f=420 TT=3600.13 T1=50.69	MSAT (TO) f=420 TT=3600	MSAT (TO) f=420 TT=3600.09	? TT=144.064	? TT=149.214	? (MO) TT=62.98	? (MO) TT=63.18
csp/bddLarge/ normalized-bdd-21-2713-15-79-21_ext.xml	MSAT TO f=37 TT=3600	MSAT (TO) f=780 TT=2218.84 T1=49.44	MSAT (TO) f=780 TT=3600.1 T1=1231.21	MSAT (TO) f=37 TT=3600.05	MSAT (TO) f=37 TT=3600	? TT=89.936	? TT=83.62	? (MO) TT=66.85	? (MO) TT=66.85
csp/bddLarge/ normalized-bdd-21-2713-15-79-23_ext.xml	MSAT TO f=557 TT=3600	MSAT (TO) f=758 TT=3600.15 T1=2310.64	MSAT (TO) f=758 TT=3600.15 T1=455.52	MSAT (TO) f=557 TT=3600	MSAT (TO) f=557 TT=3600.04	? TT=148.448	? TT=177.495	? (MO) TT=67.39	? (MO) TT=67.58
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc	toulbar2 2008-06-27	toulbar2/BTD 2008-06-27
csp/bddLarge/ normalized-bdd-21-2713-15-79-25_ext.xml	MSAT TO f=371 TT=3600.03	MSAT (TO) f=1231 TT=3600.14 T1=3197.77	MSAT (TO) f=1180 TT=3600.09 T1=1141.67	MSAT (TO) f=371 TT=3600.03	MSAT (TO) f=371 TT=3600.09	? TT=157.582	? TT=135.423	? (MO) TT=62.57	? (MO) TT=66.96
csp/bddLarge/ normalized-bdd-21-2713-15-79-26_ext.xml	MSAT TO f=354 TT=3600.06	MSAT (TO) f=799 TT=3600.09 T1=36.95	MSAT (TO) f=799 TT=3600.11 T1=450.2	MSAT (TO) f=354 TT=3600.09	MSAT (TO) f=354 TT=3600.06	? TT=103.603	? TT=106.23	? (MO) TT=63.03	? (MO) TT=67.59
csp/bddLarge/ normalized-bdd-21-2713-15-79-27_ext.xml	MSAT TO f=203 TT=3600.02	MSAT (TO) f=783 TT=3600.14 T1=3533.02	MSAT (TO) f=766 TT=3600.09 T1=3435.94	MSAT (TO) f=203 TT=3600.02	MSAT (TO) f=203 TT=3600.06	? TT=147.598	? TT=148.018	? (MO) TT=62.98	? (MO) TT=66.85
csp/bddLarge/ normalized-bdd-21-2713-15-79-29_ext.xml	MSAT TO f=770 TT=3600.06 T1=10.68	MSAT (TO) f=770 TT=3600.16 T1=10.98	MSAT (TO) f=770 TT=3600.15 T1=10.68	MSAT (TO) f=770 TT=3600.08	MSAT (TO) f=770 TT=3600.06	? TT=141.354	? TT=134.721	? (MO) TT=64.36	? (MO) TT=62.58
csp/bddLarge/ normalized-bdd-21-2713-15-79-2_ext.xml	MSAT TO f=500 TT=3600.03 T1=40.89	MSAT (TO) f=500 TT=3600.09 T1=1837.8	MSAT (TO) f=500 TT=3600.08 T1=40.89	MSAT (TO) f=500 TT=3600.03	MSAT (TO) f=500 TT=3600.06	? TT=155.656	? TT=133.12	? (MO) TT=63.3	? (MO) TT=62.46
csp/bddLarge/ normalized-bdd-21-2713-15-79-30_ext.xml	MOPT f=43 TT=805.326 T1=1.16	OPTIMUM f=43 TT=913.575 T1=1.16 CHECKS=530643000 NODES=10510	OPTIMUM f=43 TT=805.326 T1=1.25 CHECKS=486176000 NODES=9522	MSAT (TO) f=43 TT=3600.05	MSAT (TO) f=43 TT=3600.02	? TT=249.617	? TT=253.494	? (MO) TT=67.38	? (MO) TT=68.15
csp/bddLarge/ normalized-bdd-21-2713-15-79-33_ext.xml	MSAT TO f=651 TT=3600 T1=21.67	MSAT (TO) f=651 TT=3600.16 T1=23.89	MSAT (TO) f=651 TT=3600.05 T1=21.67	MSAT (TO) f=651 TT=3600	MSAT (TO) f=651 TT=3600.05	? TT=130.197	? TT=131.351	? (MO) TT=67.54	? (MO) TT=62.57
csp/bddLarge/ normalized-bdd-21-2713-15-79-6_ext.xml	MSAT TO f=263 TT=3600	MSAT (TO) f=1216 TT=3600.1 T1=17.39	MSAT (TO) f=1216 TT=3600.16 T1=17.15	MSAT (TO) f=263 TT=3600	MSAT (TO) f=263 TT=3600.05	? TT=133.478	? TT=140.256	? (MO) TT=64.88	? (MO) TT=64.96
csp/bddLarge/ normalized-bdd-21-2713-15-79-7_ext.xml	MSAT TO f=1263 TT=3600.01 T1=2.27	MSAT (TO) f=1263 TT=3600.11 T1=2.46	MSAT (TO) f=1263 TT=3600.09 T1=2.27	MSAT (TO) f=1263 TT=3600.01	MSAT (TO) f=1263 TT=3600.07	? TT=135.655	? TT=140.934	? (MO) TT=67.17	? (MO) TT=62.64
csp/bddLarge/ normalized-bdd-21-2713-15-79-9_ext.xml	MSAT TO f=1019 TT=3600.01 T1=39.04	MSAT (TO) f=1019 TT=3600.15 T1=39.04	MSAT (TO) f=1019 TT=3600.17 T1=3106.02	MSAT (TO) f=1019 TT=3600.06	MSAT (TO) f=1019 TT=3600.01	? TT=162.275	? TT=141.368	? (MO) TT=62.92	? (MO) TT=62.81
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc	toulbar2 2008-06-27	toulbar2/BTD 2008-06-27
csp/bddSmall/ normalized-bdd-21-133-18-78-12_ext.xml	MOPT f=13 TT=497.844 T1=2.19	OPTIMUM f=13 TT=497.844 T1=2.19 CHECKS=238263000 NODES=98287	OPTIMUM f=13 TT=556.707 T1=2.36 CHECKS=240477000 NODES=98142	MSAT (TO) f=13 TT=3600.04	MSAT (TO) f=13 TT=3600.05	? TT=239.56	? TT=257.314	OPTIMUM f=13 TT=361.271 T1=211.37	OPTIMUM f=13 TT=554.236 T1=483.28
csp/bddSmall/ normalized-bdd-21-133-18-78-13_ext.xml	MOPT f=10 TT=548.07 T1=301.06	OPTIMUM f=10 TT=707.319 T1=703.07 CHECKS=322155000 NODES=129852	OPTIMUM f=10 TT=629.806 T1=574.57 CHECKS=289637000 NODES=113619	MSAT (TO) f=10 TT=3600.1	MSAT (TO) f=10 TT=3600.05	? TT=131.884	? TT=133.433	OPTIMUM f=10 TT=367.155 T1=98.55	OPTIMUM f=10 TT=548.07 T1=301.06
csp/bddSmall/ normalized-bdd-21-133-18-78-14_ext.xml	MOPT f=9 TT=479.687 T1=15.77	OPTIMUM f=9 TT=498.517 T1=16.82 CHECKS=210396000 NODES=84024	OPTIMUM f=9 TT=479.687 T1=15.77 CHECKS=209514000 NODES=80607	MSAT (TO) f=9 TT=3600.07	MSAT (TO) f=9 TT=3600	? TT=256.504	? TT=255.697	OPTIMUM f=9 TT=426.437 T1=423.21	OPTIMUM f=9 TT=588.18 T1=522.57
csp/bddSmall/ normalized-bdd-21-133-18-78-15_ext.xml	MOPT f=8 TT=407.304 T1=1.19	OPTIMUM f=8 TT=431.53 T1=1.47 CHECKS=184139000 NODES=72284	OPTIMUM f=8 TT=407.304 T1=1.19 CHECKS=184780000 NODES=72034	MSAT (TO) f=8 TT=3600.04	MSAT (TO) f=8 TT=3600.09	? TT=111.841	? TT=112.011	OPTIMUM f=8 TT=321.393 T1=47.97	OPTIMUM f=8 TT=480.957 T1=212.16
csp/bddSmall/ normalized-bdd-21-133-18-78-16_ext.xml	MOPT f=10 TT=461.628 T1=2.86	OPTIMUM f=10 TT=461.628 T1=3.05 CHECKS=202086000 NODES=79141	OPTIMUM f=10 TT=497.861 T1=2.86 CHECKS=212152000 NODES=82864	MSAT (TO) f=10 TT=3600.06	MSAT (TO) f=10 TT=3600.04	? TT=174.806	? TT=181.498	OPTIMUM f=10 TT=376.045 T1=243.05	OPTIMUM f=10 TT=551.199 T1=531.21
csp/bddSmall/ normalized-bdd-21-133-18-78-19_ext.xml	MOPT f=3 TT=269.981 T1=2.88	OPTIMUM f=3 TT=269.981 T1=2.93 CHECKS=103238000 NODES=42114	OPTIMUM f=3 TT=294.533 T1=2.88 CHECKS=112806000 NODES=43098	MSAT (TO) f=3 TT=3600.02	MSAT (TO) f=3 TT=3600.05	? TT=279.408	? TT=230.52	OPTIMUM f=3 TT=221.202 T1=79.57	OPTIMUM f=3 TT=398.53 T1=257.26
csp/bddSmall/ normalized-bdd-21-133-18-78-21_ext.xml	MOPT f=6 TT=358.685 T1=4.68	OPTIMUM f=6 TT=358.685 T1=4.68 CHECKS=161659000 NODES=62198	OPTIMUM f=6 TT=388.499 T1=4.68 CHECKS=161773000 NODES=60829	MSAT (TO) f=6 TT=3600.04	MSAT (TO) f=6 TT=3600.1	? TT=233.696	? TT=248.351	OPTIMUM f=6 TT=305.591 T1=69.08	OPTIMUM f=6 TT=482.844 T1=255.66
csp/bddSmall/ normalized-bdd-21-133-18-78-22_ext.xml	MOPT f=1 TT=211.044 T1=1.86	OPTIMUM f=1 TT=212.307 T1=1.86 CHECKS=82731500 NODES=30586	OPTIMUM f=1 TT=211.044 T1=2.16 CHECKS=74332000 NODES=26370	MSAT (TO) f=1 TT=3600.03	MSAT (TO) f=1 TT=3600.07	? TT=100.627	? TT=91.62	OPTIMUM f=1 TT=349.929 T1=339.09	OPTIMUM f=1 TT=470.267 T1=421.63
csp/bddSmall/ normalized-bdd-21-133-18-78-24_ext.xml	MOPT f=10 TT=407.156 T1=1.09	OPTIMUM f=10 TT=407.156 T1=1.09 CHECKS=191749000 NODES=78260	OPTIMUM f=10 TT=460.48 T1=1.38 CHECKS=196362000 NODES=75220	MSAT (TO) f=10 TT=3600.08	MSAT (TO) f=10 TT=3600.04	? TT=232.586	? TT=237.861	OPTIMUM f=10 TT=388.008 T1=261.19	OPTIMUM f=10 TT=523.401 T1=500.28
csp/bddSmall/ normalized-bdd-21-133-18-78-29_ext.xml	MOPT f=3 TT=274.533 T1=1.58	OPTIMUM f=3 TT=274.533 T1=1.58 CHECKS=117432000 NODES=45972	OPTIMUM f=3 TT=283.137 T1=1.59 CHECKS=118416000 NODES=44829	MSAT (TO) f=3 TT=3600.05	MSAT (TO) f=3 TT=3600.06	? TT=132.004	? TT=152.276	OPTIMUM f=3 TT=282.289 T1=41.98	OPTIMUM f=3 TT=458.367 T1=216.96
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc	toulbar2 2008-06-27	toulbar2/BTD 2008-06-27
csp/bddSmall/ normalized-bdd-21-133-18-78-30_ext.xml	MOPT f=11 TT=541.898 T1=9.13	OPTIMUM f=11 TT=541.898 T1=9.34 CHECKS=240737000 NODES=94515	OPTIMUM f=11 TT=548.564 T1=9.13 CHECKS=231973000 NODES=90408	MSAT (TO) f=11 TT=3600.01	MSAT (TO) f=11 TT=3600.03	? TT=289.66	? TT=272.233	OPTIMUM f=11 TT=406.002 T1=357.79	OPTIMUM f=11 TT=556.368 T1=461.5
csp/bddSmall/ normalized-bdd-21-133-18-78-31_ext.xml	MOPT f=14 TT=530.231 T1=2.58	OPTIMUM f=14 TT=544.027 T1=2.58 CHECKS=262885000 NODES=104407	OPTIMUM f=14 TT=594.074 T1=2.95 CHECKS=260033000 NODES=104747	MSAT (TO) f=14 TT=3600.09	MSAT (TO) f=14 TT=3600.07	? TT=238.481	? TT=252.823	OPTIMUM f=14 TT=346.646 T1=46.73	OPTIMUM f=14 TT=530.231 T1=218.86
csp/bddSmall/ normalized-bdd-21-133-18-78-33_ext.xml	MOPT f=1 TT=223.986 T1=25.92	OPTIMUM f=1 TT=257.369 T1=28.21 CHECKS=86542900 NODES=31273	OPTIMUM f=1 TT=223.986 T1=25.92 CHECKS=77980600 NODES=27218	MSAT (TO) f=1 TT=3600.04	MSAT (TO) f=1 TT=3600.01	? TT=255.879	? TT=218.741	OPTIMUM f=1 TT=172.91 T1=42.57	OPTIMUM f=1 TT=341.502 T1=214.95
csp/bddSmall/ normalized-bdd-21-133-18-78-35_ext.xml	MOPT f=8 TT=414.723 T1=5.27	OPTIMUM f=8 TT=414.723 T1=5.82 CHECKS=180432000 NODES=70396	OPTIMUM f=8 TT=416.442 T1=5.27 CHECKS=186397000 NODES=71636	MSAT (TO) f=8 TT=3600.01	MSAT (TO) f=8 TT=3600.01	? TT=246.79	? TT=253.853	OPTIMUM f=8 TT=400.092 T1=170.25	OPTIMUM f=8 TT=587.823 T1=348.5
csp/bddSmall/ normalized-bdd-21-133-18-78-6_ext.xml	MOPT f=2 TT=253.215 T1=1.16	OPTIMUM f=2 TT=253.215 T1=1.37 CHECKS=103468000 NODES=39827	OPTIMUM f=2 TT=255.833 T1=1.16 CHECKS=97450400 NODES=36234	MSAT (TO) f=2 TT=3600.04	MSAT (TO) f=2 TT=3600.06	? TT=248.369	? TT=255.002	OPTIMUM f=2 TT=318.32 T1=313.31	OPTIMUM f=2 TT=485.986 T1=473.33
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-11.xml	MSAT TO f=9 TT=3600.04 T1=2679.66	MSAT (TO) f=10 TT=3600.16 T1=24.92	MSAT (TO) f=10 TT=3600.21 T1=25.13	MSAT (TO) f=9 TT=3600.08	MSAT (TO) f=9 TT=3600.04	? (MO) TT=23.32	? (MO) TT=22.47	? (TO) TT=3600.04	? (TO) f=9 TT=3600.01 T1=2679.66
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-12.xml	MSAT TO f=9 TT=3600.01	MSAT (TO) f=10 TT=3600.2 T1=31.39	MSAT (TO) f=10 TT=3600.21 T1=30.16	MSAT (TO) f=9 TT=3600.05	MSAT (TO) f=9 TT=3600.01	? (MO) TT=21.9	? (MO) TT=22.604	? (TO) TT=3600.01	? (TO) f=10 TT=3600 T1=374.51
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-13.xml	MSAT TO f=9 TT=3600.01 T1=187.97	MSAT (TO) f=10 TT=3600.19 T1=24.73	MSAT (TO) f=9 TT=3600.14 T1=529.59	MSAT (TO) f=9 TT=3600.05	MSAT (TO) f=9 TT=3600.01	? (MO) TT=22.686	? (MO) TT=22.39	? (TO) TT=3600.08	? (TO) f=9 TT=3600 T1=187.97
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-14.xml	MSAT TO f=9 TT=3600 T1=1457.71	MSAT (TO) f=10 TT=3600.22 T1=32.11	MSAT (TO) f=9 TT=3600.2 T1=2686.4	MSAT (TO) f=9 TT=3600.03	MSAT (TO) f=9 TT=3600	? (MO) TT=23.42	? (MO) TT=23.16	? (TO) TT=3600	? (TO) f=9 TT=3600.01 T1=1457.71
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-15.xml	MSAT TO f=9 TT=3600.04	MSAT (TO) f=10 TT=3600.24 T1=25.54	MSAT (TO) f=10 TT=3600.15 T1=24.18	MSAT (TO) f=9 TT=3600.08	MSAT (TO) f=9 TT=3600.04	? (MO) TT=22.41	? (MO) TT=21.56	? (TO) TT=3600.04	? (TO) f=10 TT=3600.03 T1=155.84
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc	toulbar2 2008-06-27	toulbar2/BTD 2008-06-27
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-17.xml	MSAT TO f=9 TT=3600.03	MSAT (TO) f=10 TT=3600.14 T1=25.63	MSAT (TO) f=10 TT=3600.13 T1=29.88	MSAT (TO) f=9 TT=3600.04	MSAT (TO) f=9 TT=3600.03	? (MO) TT=23.668	? (MO) TT=22.37	? (TO) TT=3600.04	? (TO) f=10 TT=3600.01 T1=96.54
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-19.xml	MSAT TO f=9 TT=3600 T1=3331.58	MSAT (TO) f=10 TT=3600.15 T1=26.25	MSAT (TO) f=10 TT=3600.25 T1=26.32	MSAT (TO) f=9 TT=3600.09	MSAT (TO) f=9 TT=3600	? (MO) TT=23.58	? TT=107.974	? (TO) TT=3600.02	? (TO) f=9 TT=3600.08 T1=3331.58
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-21.xml	MSAT TO f=8 TT=3600.01	MSAT (TO) f=10 TT=3600.17 T1=28.79	MSAT (TO) f=10 TT=3600.2 T1=27.41	MSAT (TO) f=9 TT=3600.09	MSAT (TO) f=8 TT=3600.01	? (MO) TT=22.697	? (MO) TT=23.15	? (TO) TT=3600	? (TO) f=10 TT=3600.09 T1=131.72
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-22.xml	MSAT TO f=9 TT=3600.03	MSAT (TO) f=10 TT=3600.15 T1=34.16	MSAT (TO) f=10 TT=3600.2 T1=34.25	MSAT (TO) f=9 TT=3600.03	MSAT (TO) f=9 TT=3600.04	? TT=113.325	? (MO) TT=22.608	? (TO) TT=3600.08	? (TO) f=10 TT=3600.09 T1=500.81
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-25.xml	MSAT TO f=9 TT=3600.03 T1=2257.39	MSAT (TO) f=10 TT=3600.17 T1=30.54	MSAT (TO) f=10 TT=3600.15 T1=29.42	MSAT (TO) f=9 TT=3600.03	MSAT (TO) f=9 TT=3600.07	? (MO) TT=23.43	? (MO) TT=23.381	? (TO) TT=3600.1	? (TO) f=9 TT=3600.07 T1=2257.39
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-3.xml	MSAT TO f=9 TT=3600.08 T1=625.62	MSAT (TO) f=9 TT=3600.2 T1=625.62	MSAT (TO) f=10 TT=3600.23 T1=38.13	MSAT (TO) f=9 TT=3600.08	MSAT (TO) f=9 TT=3600.09	? (MO) TT=24.74	? (MO) TT=25.653	? (TO) TT=3600	? (TO) f=10 TT=3600 T1=111.97
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-5.xml	MSAT TO f=9 TT=3600.04 T1=2128.56	MSAT (TO) f=10 TT=3600.22 T1=25.92	MSAT (TO) f=9 TT=3600.24 T1=3041.06	MSAT (TO) f=9 TT=3600.04	MSAT (TO) f=9 TT=3600.08	? (MO) TT=23.46	? (MO) TT=22.3	? (TO) TT=3600.02	? (TO) f=9 TT=3600.01 T1=2128.56
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-6.xml	MSAT TO f=9 TT=3600.03	MSAT (TO) f=10 TT=3600.22 T1=28.81	MSAT (TO) f=10 TT=3600.2 T1=29	MSAT (TO) f=9 TT=3600.03	MSAT (TO) f=9 TT=3600.08	? (MO) TT=22.156	? (MO) TT=23.56	? (TO) TT=3600.07	? (TO) f=10 TT=3600.08 T1=819.06
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-7.xml	MSAT TO f=9 TT=3600.02	MSAT (TO) f=10 TT=3600.24 T1=30.49	MSAT (TO) f=10 TT=3600.21 T1=27.79	MSAT (TO) f=9 TT=3600.02	MSAT (TO) f=9 TT=3600.03	? (MO) TT=23.47	? (MO) TT=22.06	? (TO) TT=3600.08	? (TO) f=10 TT=3600.1 T1=112.95
csp/dag-rand/ normalized-rand-n23-d3-e16-r15-t150000-9.xml	MSAT TO f=9 TT=3600.03 T1=1765.17	MSAT (TO) f=10 TT=3600.33 T1=24.11	MSAT (TO) f=9 TT=3600.25 T1=2830.26	MSAT (TO) f=9 TT=3600.05	MSAT (TO) f=9 TT=3600.03	? (MO) TT=25.83	? (MO) TT=26.004	? (TO) TT=3600.01	? (TO) f=9 TT=3600.01 T1=1765.17

Some statistics...

	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	toulbar2/BTD 2008-06-27
Number of times the solver is able to give the best known answer	29	32	29	30	15
Number of times the solver is able to give the best known answer from an incomplete solver point of view (i.e. without considering optimality proof)	29	32	49	50	15
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	18	21	29	30	2
Number of times the solver is the best solver from an incomplete solver point of view (i.e. best known answer and best T1 time)	12	17	0	0	1