MAX-CSP 2008 Competition: results by benchmark

Results by benchmark for category binary constraints in intension (2-ARY-INT), subcategory real world problems

This page displays the results of the different solvers for each benchmark for category binary constraints in intension (2-ARY-INT), subcategory real world problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out
T1=...	Time to get the best model (T1): this is the wall clock time (in seconds) ellapsed when the solver found its best model. This time is obtained from the 'o lines'. It is useful to compare both complete and incomplete solvers. It doesn't take into account the time to prove optimality of the solution.

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
AbsconMax 112 pc-d
AbsconMax 112 pc-w
Concrete + CSP4J - MCRW Engine 2008-05-30
Concrete + CSP4J - Tabu Engine 2008-05-30
Sugar v1.13+minisat
Sugar++ v1.13+minisat-inc

Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/fapp/fapp01/ normalized-fapp01-0200-2.xml	MSAT TO f=4 TT=3600.01	MSAT (TO) f=9 TT=1641.28 T1=24.63	MSAT (TO) f=9 TT=1622.43 T1=23.3	MSAT (TO) f=4 TT=3600.01	MSAT (TO) f=12 TT=3600.01	MSAT (TO) f=186 TT=3600.15 T1=1001.64	MSAT (TO) f=196 TT=3600.23 T1=1974.35
csp/fapp/fapp02/ normalized-fapp02-0250-0.xml	MSAT TO f=16 TT=3600.02	MSAT (TO) f=26 TT=1727.15 T1=66.13	MSAT (TO) f=26 TT=1734.37 T1=64.35	MSAT (TO) f=16 TT=3600.02	MSAT (TO) f=48 TT=3600.06	MSAT (TO) f=474 TT=3600.34 T1=615.96	No Cert. TT=181.946
csp/fapp/fapp03/ normalized-fapp03-0300-0.xml	MSAT TO f=23 TT=3600.02	? TT=259.121	? TT=278.74	MSAT (TO) f=23 TT=3600.02	MSAT (TO) f=81 TT=3600.05	? (MO) TT=99.191	? (MO) TT=104.847
csp/fapp/fapp03/ normalized-fapp03-0300-4.xml	MSAT TO f=18 TT=3600.02	? TT=150.758	? TT=158.058	MSAT (TO) f=18 TT=3600.02	MSAT (TO) f=37 TT=3600.04	? (MO) TT=98.4	? (MO) TT=106.811
csp/fapp/fapp04/ normalized-fapp04-0300-0.xml	MSAT TO f=16 TT=3600.04	? TT=165.45	? TT=167.517	MSAT (TO) f=16 TT=3600.04	MSAT (TO) f=74 TT=3600.03	? TT=82.841	? (MO) TT=80.246
csp/fapp/fapp05/ normalized-fapp05-0350-1.xml	MSAT TO f=3 TT=3600.09	? TT=99.71	? TT=99.956	MSAT (TO) f=3 TT=3600.09	MSAT (TO) f=43 TT=3600.01	? (MO) TT=131.238	? (MO) TT=132.153
csp/fapp/fapp05/ normalized-fapp05-0350-6.xml	MSAT TO f=2 TT=3600.08	? TT=207.95	? TT=206.704	MSAT (TO) f=2 TT=3600.08	MSAT (TO) f=36 TT=3600.09	? (MO) TT=127.376	? (MO) TT=122.074
csp/fapp/fapp06/ normalized-fapp06-0500-0.xml	MSAT TO f=28 TT=3600.09	? TT=197.517	? TT=196.916	MSAT (TO) f=28 TT=3600.09	MSAT (TO) f=127 TT=3600.05	? (MO) TT=275.704	? (MO) TT=210.818
csp/fapp/fapp06/ normalized-fapp06-0500-4.xml	MSAT TO f=17 TT=3600.08	? TT=223.496	? TT=209.181	MSAT (TO) f=17 TT=3600.08	MSAT (TO) f=46 TT=3600.06	? (MO) TT=196.555	? (MO) TT=204.595
csp/fapp/fapp07/ normalized-fapp07-0600-6.xml	MSAT TO f=28 TT=3600.04	? TT=198.7	? TT=197.288	MSAT (TO) f=28 TT=3600.04	MSAT (TO) f=57 TT=3600.02	? TT=220.065	? TT=236.266
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/fapp/fapp07/ normalized-fapp07-0600-7.xml	MSAT TO f=25 TT=3600	? TT=201.794	? TT=165.632	MSAT (TO) f=25 TT=3600	MSAT (TO) f=47 TT=3600.05	? TT=234.254	? TT=238.989
csp/fapp/fapp08/ normalized-fapp08-0700-0.xml	MSAT TO f=73 TT=3600.1	? TT=189.193	? TT=202.501	MSAT (TO) f=73 TT=3600.1	MSAT (TO) f=136 TT=3600.01	? TT=224.438	? TT=223.73
csp/fapp/fapp08/ normalized-fapp08-0700-4.xml	MSAT TO f=17 TT=3600	? TT=213.467	? TT=211.484	MSAT (TO) f=17 TT=3600	MSAT (TO) f=58 TT=3600.01	? TT=201.564	? TT=203.859
csp/fapp/fapp09/ normalized-fapp09-0800-0.xml	MSAT TO f=78 TT=3600.02	? TT=384.055	? TT=368.735	MSAT (TO) f=78 TT=3600.02	MSAT (TO) f=114 TT=3600.04	? TT=257.548	? TT=239.961
csp/fapp/fapp10/ normalized-fapp10-0900-0.xml	MSAT TO f=95 TT=3600.01	? TT=399.332	? TT=424.144	MSAT (TO) f=95 TT=3600.01	MSAT (TO) f=213 TT=3600.07	? TT=276.72	? TT=272.374
csp/fapp/fapp11/ normalized-fapp11-1000-1.xml	MSAT TO f=37 TT=3600.06	? TT=573.146	? TT=542.742	MSAT (TO) f=37 TT=3600.06	MSAT (TO) f=210 TT=3600.07	? TT=312.471	? TT=298.186
csp/fapp/fapp11/ normalized-fapp11-1000-6.xml	MSAT TO f=29 TT=3600.05	? TT=485.176	? TT=510.293	MSAT (TO) f=29 TT=3600.05	MSAT (TO) f=151 TT=3600.01	? TT=325.403	? TT=294.067
csp/fapp/fapp12/ normalized-fapp12-1500-0.xml	MSAT TO f=92 TT=3600.03	? TT=1153.59	? TT=1098.92	MSAT (TO) f=92 TT=3600.03	MSAT (TO) f=750 TT=3600.02	? TT=427.552	? TT=403.068
csp/fapp/fapp14/ normalized-fapp14-2500-0.xml	MSAT TO f=1158 TT=3600	? TT=2078.47	? TT=2230.85	MSAT (TO) f=1158 TT=3600	MSAT (TO) f=1267 TT=3600.05	? TT=508.169	? TT=514.694
csp/fapp/fapp14/ normalized-fapp14-2500-2.xml	MSAT TO f=427 TT=3600.02	? TT=1516.6	? TT=1469.26	MSAT (TO) f=427 TT=3600.02	MSAT (TO) f=652 TT=3600.08	? TT=539.893	? TT=579.797
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/fapp/fapp15/ normalized-fapp15-3000-0.xml	MSAT TO f=1050 TT=3600.08	? TT=588.141	? TT=617.48	MSAT (TO) f=1050 TT=3600.08	MSAT (TO) f=1550 TT=3600.05	? TT=337.108	? TT=341.485
csp/fapp/fapp15/ normalized-fapp15-3000-2.xml	MSAT TO f=365 TT=3600.03	? TT=837.628	? TT=655.398	MSAT (TO) f=365 TT=3600.03	MSAT (TO) f=848 TT=3600.04	? TT=349.705	? TT=330.15
csp/fapp/fapp16/ normalized-fapp16-0260-10.xml	MSAT TO f=5 TT=1799.87 T1=53.19	MSAT (TO) f=5 TT=1806.33 T1=53.19	MSAT (TO) f=5 TT=1799.87 T1=53.25	MSAT (TO) f=5 TT=3600.05	MSAT (TO) f=5 TT=3600.01	? (MO) TT=83.42	? (MO) TT=81.395
csp/fapp/fapp16/ normalized-fapp16-0260-8.xml	MSAT TO f=5 TT=1770.96 T1=38.23	MSAT (TO) f=5 TT=1781.69 T1=38.23	MSAT (TO) f=5 TT=1770.96 T1=40.04	MSAT (TO) f=5 TT=3600.05	MSAT (TO) f=5 TT=3600.09	? (MO) TT=85.164	? (MO) TT=85.095
csp/fapp/fapp17/ normalized-fapp17-0300-0.xml	MSAT TO f=12 TT=3600.04	? TT=96.711	? TT=95.682	MSAT (TO) f=12 TT=3600.1	MSAT (TO) f=12 TT=3600.04	? (MO) TT=98.127	? (MO) TT=94.698
csp/fapp/fapp17/ normalized-fapp17-0300-1.xml	MSAT TO f=11 TT=3600.01	? TT=98.718	? TT=92.56	MSAT (TO) f=11 TT=3600.01	MSAT (TO) f=11 TT=3600.08	? (MO) TT=98.306	? (MO) TT=98.156
csp/fapp/fapp18/ normalized-fapp18-0350-4.xml	MSAT TO f=7 TT=3600.08	? (problem) TT=138.182 CHECKS=4597100 NODES=0	? (problem) TT=137.834 CHECKS=4597100 NODES=0	MSAT (TO) f=7 TT=3600.08	MSAT (TO) f=17 TT=3600.05	? (MO) TT=98.06	? (MO) TT=90.933
csp/fapp/fapp18/ normalized-fapp18-0350-6.xml	MSAT TO f=15 TT=3600.06	? (problem) TT=103.779 CHECKS=4783750 NODES=0	? (problem) TT=104.492 CHECKS=4792430 NODES=0	MSAT (TO) f=15 TT=3600.09	MSAT (TO) f=15 TT=3600.06	? (MO) TT=91.585	? (MO) TT=97.847
csp/fapp/fapp19/ normalized-fapp19-0350-2.xml	MSAT TO f=14 TT=3600.06	? TT=399.739	? TT=429.188	MSAT (TO) f=14 TT=3600.06	MSAT (TO) f=40 TT=3600.04	? (MO) TT=301.224	? (MO) TT=285.65
csp/fapp/fapp19/ normalized-fapp19-0350-5.xml	MSAT TO f=4 TT=3600.04	? TT=515.052	? TT=479.804	MSAT (TO) f=4 TT=3600.04	MSAT (TO) f=66 TT=3600.08	? (MO) TT=284.115	? (MO) TT=303.19
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/fapp/fapp20/ normalized-fapp20-0420-2.xml	MSAT TO f=14 TT=3600.02	? TT=177.112	? TT=178.005	MSAT (TO) f=14 TT=3600.02	MSAT (TO) f=23 TT=3600.03	? (MO) TT=131.261	? (MO) TT=130.97
csp/fapp/fapp20/ normalized-fapp20-0420-5.xml	MSAT TO f=8 TT=3600.06	? TT=210.547	? TT=193.025	MSAT (TO) f=8 TT=3600.06	MSAT (TO) f=15 TT=3600.07	? (MO) TT=129.36	? (MO) TT=131.054
csp/fapp/fapp22/ normalized-fapp22-1750-0.xml	MSAT TO f=172 TT=3600.06	? TT=1825.25	? TT=1818.59	MSAT (TO) f=330 TT=3600.01	MSAT (TO) f=172 TT=3600.06	? TT=706.325	? TT=713.638
csp/fapp/fapp22/ normalized-fapp22-1750-6.xml	MSAT TO f=309 TT=3600.03	? TT=1995.8	? TT=1986.04	MSAT (TO) f=309 TT=3600.03	? (TO) TT=3600.01	? TT=777.599	? TT=848.893
csp/fapp/fapp23/ normalized-fapp23-1800-4.xml		? TT=1720.88	? TT=1789.78	? (TO) TT=892.56	? (TO) TT=938.59	? TT=479.713	? TT=509.303
csp/fapp/fapp24/ normalized-fapp24-2000-1.xml	MSAT TO f=23 TT=3600.02	? TT=2132.59	? TT=2174.8	MSAT (TO) f=23 TT=3600.02	MSAT (TO) f=257 TT=3600.05	? TT=440.678	? TT=455.087
csp/fapp/fapp25/ normalized-fapp25-2230-2.xml	MSAT TO f=621 TT=3600.04	? TT=1159.69	? TT=1068.63	MSAT (TO) f=621 TT=3600.04	MSAT (TO) f=862 TT=3600.08	? TT=443.79	? TT=425.457
csp/fapp/fapp26/ normalized-fapp26-2300-4.xml	MSAT TO f=61 TT=3600.01	? TT=958.124	? TT=913.63	MSAT (TO) f=61 TT=3600.01	MSAT (TO) f=700 TT=3600	? TT=476.669	? TT=459.035
csp/fapp/fapp27/ normalized-fapp27-2550-3.xml	MSAT TO f=262 TT=3600	? TT=768.635	? TT=787.213	MSAT (TO) f=262 TT=3600	MSAT (TO) f=363 TT=3600.07	? TT=382.718	? TT=389.025
csp/fapp/fapp29/ normalized-fapp29-2900-3.xml		No Cert. TT=3600.12	No Cert. TT=3600.16	? (TO) TT=963.04	? (TO) TT=945.94	? TT=980.256	? TT=846.067
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/fapp/fapp30/ normalized-fapp30-3000-1.xml		No Cert. TT=3600.14	No Cert. TT=3600.14	? (TO) TT=1096.55	? (TO) TT=1093.14	? TT=869.3	? TT=806.452
csp/fapp/fapp32/ normalized-fapp32-0550-1.xml	MSAT TO f=52 TT=3600.07	? TT=588.596	? TT=560.922	MSAT (TO) f=52 TT=3600.07	MSAT (TO) f=127 TT=3600.1	? TT=607.339	? TT=635.112
csp/fapp/fapp33/ normalized-fapp33-0650-3.xml	MSAT TO f=39 TT=3600.06	? TT=475.987	? TT=457.915	MSAT (TO) f=39 TT=3600.06	MSAT (TO) f=87 TT=3600.09	? TT=310.04	? TT=320.098
csp/fapp/fapp34/ normalized-fapp34-0750-0.xml	MSAT TO f=36 TT=3600.03	? TT=503.924	? TT=499.027	MSAT (TO) f=36 TT=3600.03	MSAT (TO) f=77 TT=3600.06	? TT=336.024	? TT=357.176
csp/fapp/fapp35/ normalized-fapp35-1500-3.xml	MSAT TO f=439 TT=3600.01	? TT=2028.6	? TT=2100.06	MSAT (TO) f=441 TT=3600.01	MSAT (TO) f=439 TT=3600.01	? TT=721.902	? TT=726.581
csp/fapp/fapp36/ normalized-fapp36-2000-3.xml	MSAT TO f=232 TT=3600.01	? TT=729.694	? TT=721.017	MSAT (TO) f=232 TT=3600.01	MSAT (TO) f=458 TT=3600.01	? TT=552.1	? TT=571.128
csp/fapp/fapp37/ normalized-fapp37-2250-2.xml		? TT=1763.39	? TT=1704.03	? (TO) TT=633.91	? (TO) TT=641.7	? TT=1358.69	? TT=1062.11
csp/fapp/fapp40/ normalized-fapp40-3000-2.xml		? TT=2670.89	? TT=2618.3	? (TO) TT=865.67	? (TO) TT=822.63	? TT=642.451	? TT=632.329
csp/rlfapGraphsMod/ normalized-graph12-w1.xml	MOPT f=8 TT=58.589 T1=14.15	MSAT (TO) f=8 TT=1640.78 T1=32.75	MSAT (TO) f=8 TT=1647.07 T1=32.71	MSAT (TO) f=8 TT=3600.06	MSAT (TO) f=16 TT=3600.02	OPTIMUM f=8 TT=63.299 T1=63.11	OPTIMUM f=8 TT=58.589 T1=14.15
csp/rlfapGraphsMod/ normalized-graph13-w1.xml	MOPT f=9 TT=277.94 T1=277.41	MSAT (TO) f=12 TT=1657.29 T1=32.14	MSAT (TO) f=12 TT=1646.26 T1=35.5	MSAT (TO) f=9 TT=3600.07	MSAT (TO) f=19 TT=3600.07	OPTIMUM f=9 TT=277.94 T1=277.41	OPTIMUM f=9 TT=293.437 T1=292.86
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/rlfapGraphsMod/ normalized-graph14-f28.xml	MSAT TO f=16 TT=3600.08	MSAT (TO) f=70 TT=1681.32 T1=43.63	MSAT (TO) f=70 TT=1668.78 T1=40.9	MSAT (TO) f=16 TT=3600.08	MSAT (TO) f=69 TT=3600.04	? (MO) TT=69.19	? (MO) TT=69.349
csp/rlfapGraphsMod/ normalized-graph8-f11.xml	MSAT TO f=5 TT=3600.3 T1=869.79	MSAT (TO) f=30 TT=1661.12 T1=27.65	MSAT (TO) f=30 TT=1653.5 T1=28.32	MSAT (TO) f=14 TT=3600	MSAT (TO) f=34 TT=3600.03	MSAT (TO) f=5 TT=3600.3 T1=869.79	No Cert. TT=57.312
csp/rlfapGraphsMod/ normalized-graph9-f10.xml	MSAT TO f=19 TT=3600.01	MSAT (TO) f=52 TT=1684.36 T1=49.54	MSAT (TO) f=52 TT=1676.65 T1=53.43	MSAT (TO) f=19 TT=3600.01	MSAT (TO) f=39 TT=3600.08	? (MO) TT=92.055	? (MO) TT=90.453
csp/rlfapGraphs/ normalized-graph11.xml	MSAT TO f=26 TT=3600.07	MSAT (TO) f=44 TT=1676.43 T1=45.44	MSAT (TO) f=44 TT=1655.52 T1=44.93	MSAT (TO) f=26 TT=3600.07	MSAT (TO) f=194 TT=3600.1	MSAT (TO) f=310 TT=3600.21 T1=3147.31	No Cert. TT=56.47
csp/rlfapGraphs/ normalized-graph6.xml	MOPT f=10 TT=586.984 T1=586.7	MSAT (TO) f=20 TT=1620.31 T1=15.25	MSAT (TO) f=20 TT=1628.72 T1=15.19	MSAT (TO) f=11 TT=3600.01	MSAT (TO) f=55 TT=3600.07	OPTIMUM f=10 TT=836.867 T1=708.53	OPTIMUM f=10 TT=586.984 T1=586.7
csp/rlfapGraphs/ normalized-graph7.xml	MOPT f=10 TT=245.576 T1=225.29	MSAT (TO) f=19 TT=1646.4 T1=19.29	MSAT (TO) f=19 TT=1631.99 T1=19.28	MSAT (TO) f=10 TT=3600.07	MSAT (TO) f=60 TT=3600.01	OPTIMUM f=10 TT=386.808 T1=350.83	OPTIMUM f=10 TT=245.576 T1=225.29
csp/rlfapScens11/ normalized-scen11-f10.xml	MSAT TO f=17 TT=3600.39 T1=786.4	MSAT (TO) f=26 TT=1623.02 T1=23.68	MSAT (TO) f=26 TT=1632.63 T1=25.3	MSAT (TO) f=27 TT=3600.07	MSAT (TO) f=20 TT=3600.05	MSAT (TO) f=17 TT=3600.39 T1=786.4	? (MO) TT=64.096
csp/rlfapScens11/ normalized-scen11-f12.xml	MSAT TO f=20 TT=3600.32 T1=3193.59	MSAT (TO) f=28 TT=1672.53 T1=41.21	MSAT (TO) f=28 TT=1680.91 T1=44.13	MSAT (TO) f=35 TT=3600	MSAT (TO) f=29 TT=3600.09	MSAT (TO) f=20 TT=3600.32 T1=3193.59	? (MO) TT=75.089
csp/rlfapScens11/ normalized-scen11-f1.xml	MOPT f=2 TT=1525.22 T1=872.2	MSAT (TO) f=4 TT=1635.78 T1=7.1	MSAT (TO) f=4 TT=1630.98 T1=6.59	MSAT (TO) f=2 TT=3600.01	MSAT (TO) f=2 TT=3600.07	OPTIMUM f=2 TT=1525.22 T1=872.2	No Cert. TT=62.188
csp/rlfapScens11/ normalized-scen11-f2.xml	MOPT f=2 TT=1488.18 T1=23.96	MSAT (TO) f=2 TT=1636.64 T1=23.96	MSAT (TO) f=2 TT=1656.18 T1=25.81	MSAT (TO) f=2 TT=3600.09	MSAT (TO) f=2 TT=3600.07	OPTIMUM f=2 TT=1488.18 T1=833.77	? (MO) TT=67.261
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/rlfapScens11/ normalized-scen11-f3.xml	MSAT TO f=3 TT=1635.15 T1=10.69	MSAT (TO) f=3 TT=2135.45 T1=10.81	MSAT (TO) f=3 TT=1635.15 T1=10.69	MSAT (TO) f=3 TT=3600.06	MSAT (TO) f=3 TT=3600.07	MSAT (TO) f=3 TT=3600.42 T1=786.16	No Cert. TT=64.006
csp/rlfapScens11/ normalized-scen11-f4.xml	MSAT TO f=3 TT=1652.73 T1=38.17	MSAT (TO) f=3 TT=1652.73 T1=38.17	MSAT (TO) f=3 TT=1658.99 T1=40.56	MSAT (TO) f=3 TT=3600.08	MSAT (TO) f=3 TT=3600.04	MSAT (TO) f=4 TT=3600.34 T1=710.37	MSAT (MO) f=438 TT=69.168 T1=61.52
csp/rlfapScens11/ normalized-scen11-f5.xml	MSAT TO f=3 TT=1636.35 T1=23.62	MSAT (TO) f=3 TT=1636.35 T1=23.62	MSAT (TO) f=3 TT=1651.09 T1=25.91	MSAT (TO) f=4 TT=3600.07	MSAT (TO) f=3 TT=3600.01	MSAT (TO) f=3 TT=3600.29 T1=849.69	No Cert. TT=67.422
csp/rlfapScens11/ normalized-scen11-f6.xml	MSAT TO f=6 TT=3600.06 T1=766.98	MSAT (TO) f=10 TT=1634.42 T1=20.79	MSAT (TO) f=10 TT=1647.21 T1=19.76	MSAT (TO) f=7 TT=3600.05	MSAT (TO) f=6 TT=3600.06	MSAT (TO) f=6 TT=3600.35 T1=766.98	? (MO) TT=111.966
csp/rlfapScens11/ normalized-scen11-f7.xml	MSAT TO f=5 TT=3600.33 T1=793.21	MSAT (TO) f=8 TT=1694.79 T1=39.93	MSAT (TO) f=8 TT=1668.84 T1=39.96	MSAT (TO) f=9 TT=3600.09	MSAT (TO) f=9 TT=3600.01	MSAT (TO) f=5 TT=3600.33 T1=793.21	? (MO) TT=78.999
csp/rlfapScens11/ normalized-scen11-f8.xml	MSAT TO f=7 TT=3600.32 T1=907.34	MSAT (TO) f=11 TT=1636.61 T1=26.8	MSAT (TO) f=11 TT=1717.64 T1=25.05	MSAT (TO) f=16 TT=3600.06	MSAT (TO) f=12 TT=3600.01	MSAT (TO) f=7 TT=3600.32 T1=907.34	? (MO) TT=74.464
csp/rlfapScens11/ normalized-scen11-f9.xml	MSAT TO f=8 TT=3600.37 T1=944.99	MSAT (TO) f=10 TT=1694.1 T1=84.47	MSAT (TO) f=10 TT=1690.29 T1=92.21	MSAT (TO) f=17 TT=3600	MSAT (TO) f=16 TT=3600.09	MSAT (TO) f=8 TT=3600.37 T1=944.99	No Cert. TT=62.663
csp/rlfapScensMod/ normalized-scen10-w1-f3.xml	MOPT f=1 TT=13.866 T1=13.72	MSAT (TO) f=2 TT=1641.15 T1=26.02	MSAT (TO) f=2 TT=1629.05 T1=23.69	MSAT (TO) f=1 TT=3600.05	MSAT (TO) f=16 TT=3600.05	OPTIMUM f=1 TT=34.622 T1=32.09	OPTIMUM f=1 TT=13.866 T1=13.72
csp/rlfapScensMod/ normalized-scen1-f9.xml	MSAT TO f=2 TT=3600.07	MSAT (TO) f=12 TT=1665.15 T1=23.71	MSAT (TO) f=12 TT=1655.76 T1=23.82	MSAT (TO) f=2 TT=3600.07	MSAT (TO) f=11 TT=3600.04	? (MO) TT=106.284	? (MO) TT=105.846
csp/rlfapScensMod/ normalized-scen2-f25.xml	MOPT f=2 TT=12.633 T1=1.45	MSAT (TO) f=2 TT=1619.84 T1=1.45	MSAT (TO) f=2 TT=1611.21 T1=1.45	MSAT (TO) f=2 TT=3600.01	MSAT (TO) f=2 TT=3600.08	OPTIMUM f=2 TT=36.193 T1=35.86	OPTIMUM f=2 TT=12.633 T1=11.96
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/rlfapScensMod/ normalized-scen3-f11.xml	MOPT f=1 TT=43.828 T1=43.21	MSAT (TO) f=12 TT=1626.71 T1=26.99	MSAT (TO) f=12 TT=1638.22 T1=29.2	MSAT (TO) f=3 TT=3600.04	MSAT (TO) f=11 TT=3600.03	OPTIMUM f=1 TT=352.936 T1=326.68	OPTIMUM f=1 TT=43.828 T1=43.21
csp/rlfapScensMod/ normalized-scen6-w1-f2.xml	MOPT f=1 TT=4.304 T1=4.03	MSAT (TO) f=1 TT=1617.13 T1=18.12	MSAT (TO) f=1 TT=1603.89 T1=16.67	MSAT (TO) f=1 TT=3600.03	MSAT (TO) f=1 TT=3600.01	OPTIMUM f=1 TT=5.336 T1=5.07	OPTIMUM f=1 TT=4.304 T1=4.03
csp/rlfapScensMod/ normalized-scen9-w1-f3.xml	MOPT f=1 TT=13.489 T1=13	MSAT (TO) f=2 TT=1649.06 T1=25.53	MSAT (TO) f=2 TT=1647.91 T1=23.75	MSAT (TO) f=1 TT=3600.05	MSAT (TO) f=15 TT=3600.06	OPTIMUM f=1 TT=34.263 T1=31.46	OPTIMUM f=1 TT=13.489 T1=13
csp/rlfapScens/ normalized-scen10.xml	MSAT TO f=67 TT=3600.03	MSAT (TO) f=115 TT=1656.96 T1=37.71	MSAT (TO) f=115 TT=1672.94 T1=37.32	MSAT (TO) f=67 TT=3600.03	MSAT (TO) f=145 TT=3600.01	MSAT (TO) f=196 TT=3600.34 T1=2403.55	No Cert. TT=110.572
csp/rlfapScens/ normalized-scen6.xml	MSAT TO f=29 TT=3600.06	MSAT (TO) f=32 TT=1637.34 T1=62	MSAT (TO) f=32 TT=1666.97 T1=67.46	MSAT (TO) f=29 TT=3600.06	MSAT (TO) f=37 TT=3600.05	MSAT (TO) f=56 TT=3600.03 T1=56.55	MSAT (TO) f=95 TT=3600.13 T1=376.79
csp/rlfapScens/ normalized-scen7.xml	MSAT TO f=56 TT=3600.08	MSAT (TO) f=111 TT=1653.32 T1=34.43	MSAT (TO) f=111 TT=1632.26 T1=34.04	MSAT (TO) f=56 TT=3600.08	MSAT (TO) f=79 TT=3600.01	MSAT (TO) f=452 TT=3600.18 T1=170.78	MSAT (TO) f=268 TT=3600.31 T1=3377.31
csp/rlfapScens/ normalized-scen8.xml	MSAT TO f=139 TT=3600.07	MSAT (TO) f=255 TT=1706.5 T1=110.9	MSAT (TO) f=255 TT=1726.94 T1=116.27	MSAT (TO) f=139 TT=3600.07	MSAT (TO) f=222 TT=3600.04	? (MO) TT=108.56	? (MO) TT=104.53
csp/rlfapScens/ normalized-scen9.xml	MSAT TO f=68 TT=3600.06	MSAT (TO) f=115 TT=1658.44 T1=39.75	MSAT (TO) f=115 TT=1697.32 T1=37.1	MSAT (TO) f=68 TT=3600.06	MSAT (TO) f=141 TT=3600.01	MSAT (TO) f=196 TT=3600.34 T1=2401.11	MSAT (MO) f=842 TT=116.646 T1=104.71
csp/subs/ normalized-scen06-sub0.xml	MOPT f=6 TT=4.252 T1=1.26	MSAT (TO) f=6 TT=1589.1 T1=1.26	MSAT (TO) f=6 TT=1608.99 T1=1.38	MSAT (TO) f=6 TT=3600.03	MSAT (TO) f=6 TT=3600.1	OPTIMUM f=6 TT=5.627 T1=5.42	OPTIMUM f=6 TT=4.252 T1=4.15
csp/subs/ normalized-scen06-sub1.xml	MSAT TO f=21 TT=1662.53 T1=60.25	MSAT (TO) f=21 TT=1662.53 T1=60.25	MSAT (TO) f=22 TT=1622.44 T1=32.68	MSAT (TO) f=21 TT=3600.01	MSAT (TO) f=21 TT=3600.06	MSAT (TO) f=38 TT=3600.03 T1=3.9	MSAT (TO) f=34 TT=3600.09 T1=742.91
Bench name	Best results on this instance	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
csp/subs/ normalized-scen06-sub2.xml	MSAT TO f=21 TT=1612.19 T1=31.77	MSAT (TO) f=22 TT=1569.84 T1=1.95	MSAT (TO) f=21 TT=1612.19 T1=31.77	MSAT (TO) f=21 TT=3600.04	MSAT (TO) f=21 TT=3600.01	MSAT (TO) f=41 TT=3600.04 T1=14.61	MSAT (TO) f=49 TT=3600.12 T1=1344.53
csp/subs/ normalized-scen06-sub3.xml	MSAT TO f=23 TT=3600.07	MSAT (TO) f=24 TT=1606.38 T1=1.75	MSAT (TO) f=24 TT=1595.32 T1=1.67	MSAT (TO) f=23 TT=3600.07	MSAT (TO) f=23 TT=3600.1	MSAT (TO) f=35 TT=3600.05 T1=1543.04	MSAT (TO) f=27 TT=3600.12 T1=685.62
csp/subs/ normalized-scen06-sub4.xml	MSAT TO f=23 TT=3600.04	MSAT (TO) f=27 TT=1615.65 T1=2.07	MSAT (TO) f=27 TT=1594.05 T1=2.06	MSAT (TO) f=23 TT=3600.04	MSAT (TO) f=23 TT=3600.04	MSAT (TO) f=26 TT=3600.03 T1=240.46	MSAT (TO) f=52 TT=3600.1 T1=366.18
csp/subs/ normalized-scen07-sub1.xml	MSAT TO f=17 TT=1595.52 T1=4.44	MSAT (TO) f=17 TT=1595.99 T1=4.44	MSAT (TO) f=17 TT=1595.52 T1=4.54	MSAT (TO) f=17 TT=3600.04	MSAT (TO) f=17 TT=3600.06	MSAT (TO) f=17 TT=3600.09 T1=3575.85	MSAT (TO) f=17 TT=3600.11 T1=2183.28
csp/subs/ normalized-scen07-sub2.xml	MOPT f=17 TT=3368.59 T1=9.16	MSAT (TO) f=17 TT=1598.76 T1=9.2	MSAT (TO) f=17 TT=1618.12 T1=9.16	MSAT (TO) f=17 TT=3600.02	MSAT (TO) f=17 TT=3600.02	OPTIMUM f=17 TT=3368.59 T1=783.79	MSAT (TO) f=17 TT=3600.05 T1=1508.13
csp/subs/ normalized-scen07-sub3.xml	MSAT TO f=19 TT=1609.48 T1=46.24	MSAT (TO) f=19 TT=1609.48 T1=46.73	MSAT (TO) f=19 TT=1625.17 T1=46.24	MSAT (TO) f=19 TT=3600.03	MSAT (TO) f=19 TT=3600.09	MSAT (TO) f=22 TT=3600.09 T1=2844.3	MSAT (TO) f=28 TT=3600.12 T1=4.64
csp/subs/ normalized-scen07-sub4.xml	MSAT TO f=19 TT=1607.5 T1=10.22	MSAT (TO) f=19 TT=1607.5 T1=10.22	MSAT (TO) f=19 TT=1618.06 T1=10.98	MSAT (TO) f=19 TT=3600.03	MSAT (TO) f=19 TT=3600.03	MSAT (TO) f=36 TT=3600.08 T1=5.85	MSAT (TO) f=29 TT=3600.08 T1=678.58

Some statistics...

	AbsconMax 112 pc-d	AbsconMax 112 pc-w	Concrete + CSP4J - MCRW Engine 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Sugar v1.13+minisat	Sugar++ v1.13+minisat-inc
Number of times the solver is able to give the best known answer	9	9	59	18	23	11
Number of times the solver is able to give the best known answer from an incomplete solver point of view (i.e. without considering optimality proof)	15	15	70	24	23	12
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	5	5	50	8	11	9
Number of times the solver is the best solver from an incomplete solver point of view (i.e. best known answer and best T1 time)	10	5	0	0	9	7