CSP 2008 Competition: results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out
T1=...	Time to get the best model (T1): this is the wall clock time (in seconds) ellapsed when the solver found its best model. This time is obtained from the 'o lines'. It is useful to compare both complete and incomplete solvers. It doesn't take into account the time to prove optimality of the solution.

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC
Abscon 112v4 ESAC
bpsolver 2008-06-27
casper zao
casper zito
choco2_dwdeg 2008-06-26
choco2_impwdeg 2008-06-26
Concrete + CPS4J 2008-05-30
Concrete + CSP4J - Tabu Engine 2008-05-30
Concrete + CSP4J - WMC Engine 2008-05-30
cpHydra k_10
cpHydra k_40
galac E
galac M
MDG-noprobe 2008-06-27
MDG-probe 2008-06-27
Minion/Tailor 2008-07-04
Mistral-option 1.314
Mistral-prime 1.313
SAT4J CSP 2008-06-13
spider 2008-06-14
Sugar v1.13+minisat
Sugar v1.13+picosat

Bench name	Best results on this instance	Abscon 112v4 AC	Abscon 112v4 ESAC	bpsolver 2008-06-27	casper zao	casper zito	choco2_dwdeg 2008-06-26	choco2_impwdeg 2008-06-26	Concrete + CPS4J 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	Concrete + CSP4J - WMC Engine 2008-05-30	cpHydra k_10	cpHydra k_40	galac E	galac M	MDG-noprobe 2008-06-27	MDG-probe 2008-06-27	Minion/Tailor 2008-07-04	Mistral-option 1.314	Mistral-prime 1.313	SAT4J CSP 2008-06-13	spider 2008-06-14	Sugar v1.13+minisat	Sugar v1.13+picosat
csp/driver/ normalized-driverlogw-01c-sat_ext.xml	SAT TT=0.01	SAT TT=0.771 CHECKS=1271 NODES=71	SAT TT=0.736 CHECKS=1242 NODES=0	SAT TT=0.047	SAT TT=0.018 CHECKS=4665	SAT TT=0.016 CHECKS=3190	SAT TT=0.509 ASSIGNMENTS=24	SAT TT=0.518 ASSIGNMENTS=24	SAT TT=0.378 CHECKS=733 NODES=71	SAT TT=0.412 CHECKS=3422 NODES=31	SAT TT=0.389 CHECKS=6207 NODES=19	SAT TT=0.023 ASSIGNMENTS=24 CHECKS=535	SAT TT=0.023 ASSIGNMENTS=24 CHECKS=535	SAT TT=0.998	SAT TT=1.01	SAT TT=0.01 ASSIGNMENTS=24 CHECKS=535	SAT TT=0.011 ASSIGNMENTS=20 CHECKS=512	SAT TT=0.386	SAT TT=0.01 ASSIGNMENTS=23 CHECKS=460	SAT TT=0.01 ASSIGNMENTS=24 CHECKS=535	SAT TT=0.844	SAT TT=0.019 ASSIGNMENTS=71 CHECKS=1442	SAT TT=0.699	SAT TT=0.688
csp/driver/ normalized-driverlogw-02c-sat_ext.xml	SAT TT=0.136	SAT TT=2.307 CHECKS=1742190 NODES=4000	SAT TT=3.196 CHECKS=2504230 NODES=4111	SAT TT=0.477	SAT TT=4.127 CHECKS=3171710	SAT TT=1.333 CHECKS=1153110	SAT TT=1.573 ASSIGNMENTS=65	SAT TT=1.774 ASSIGNMENTS=65	SAT TT=3.114 CHECKS=35400 NODES=423	SAT TT=5.566 CHECKS=9618040 NODES=59971	SAT TT=4.35 CHECKS=7845390 NODES=3763	SAT TT=0.139 ASSIGNMENTS=180 CHECKS=117043	SAT TT=0.142 ASSIGNMENTS=180 CHECKS=117043	SAT TT=2.919	SAT TT=3.028	SAT TT=0.143 ASSIGNMENTS=201 CHECKS=150798	SAT TT=0.781 ASSIGNMENTS=9610 CHECKS=2701420	SAT TT=1.352	SAT TT=0.447 ASSIGNMENTS=1308 CHECKS=1602120	SAT TT=0.136 ASSIGNMENTS=180 CHECKS=117043	SAT TT=3.679	SAT TT=0.897 ASSIGNMENTS=1248 CHECKS=609322	SAT TT=1.567	SAT TT=1.728
csp/driver/ normalized-driverlogw-04c-sat_ext.xml	SAT TT=0.118	SAT TT=1.635 CHECKS=420610 NODES=884	SAT TT=1.865 CHECKS=563609 NODES=627	SAT TT=0.52	? (TO) TT=1800.05	SAT TT=1.208 CHECKS=1017290	SAT TT=1.173 ASSIGNMENTS=31	SAT TT=1.18 ASSIGNMENTS=31	SAT TT=2.69 CHECKS=24260 NODES=292	SAT TT=2.716 CHECKS=479876 NODES=2435	SAT TT=2.669 CHECKS=851220 NODES=355	SAT TT=0.129 ASSIGNMENTS=95 CHECKS=64460	SAT TT=0.131 ASSIGNMENTS=95 CHECKS=64460	SAT TT=2.928	SAT TT=2.808	SAT TT=0.124 ASSIGNMENTS=97 CHECKS=66005	SAT TT=0.145 ASSIGNMENTS=458 CHECKS=150402	SAT TT=1.136	SAT TT=0.285 ASSIGNMENTS=591 CHECKS=754943	SAT TT=0.118 ASSIGNMENTS=95 CHECKS=64460	SAT TT=4.484	SAT TT=0.415 ASSIGNMENTS=539 CHECKS=390557	SAT TT=1.495	SAT TT=1.625
csp/driver/ normalized-driverlogw-05c-sat_ext.xml	SAT TT=0.174	SAT TT=1.939 CHECKS=517412 NODES=1333	SAT TT=1.71 CHECKS=216933 NODES=0	SAT TT=0.733	SAT TT=18.462 CHECKS=10742900	SAT TT=3.754 CHECKS=2207510	SAT TT=3.474 ASSIGNMENTS=95	SAT TT=3.633 ASSIGNMENTS=95	SAT TT=5.845 CHECKS=38628 NODES=378	SAT TT=8.709 CHECKS=10323200 NODES=41956	SAT TT=6 CHECKS=1051790 NODES=289	SAT TT=0.183 ASSIGNMENTS=74 CHECKS=59750	SAT TT=0.179 ASSIGNMENTS=74 CHECKS=59750	SAT TT=3.677	SAT TT=3.603	SAT TT=0.196 ASSIGNMENTS=117 CHECKS=133099	SAT TT=0.736 ASSIGNMENTS=5825 CHECKS=2512870	SAT TT=1.874	SAT TT=0.603 ASSIGNMENTS=873 CHECKS=1787770	SAT TT=0.174 ASSIGNMENTS=74 CHECKS=59750	SAT TT=4.455	SAT TT=0.927 ASSIGNMENTS=1279 CHECKS=418105	SAT TT=2.115	SAT TT=1.861
csp/driver/ normalized-driverlogw-08cc-sat_ext.xml	SAT TT=0.359	SAT TT=2.762 CHECKS=2027910 NODES=2613	SAT TT=3.832 CHECKS=4430730 NODES=2706	SAT TT=1.308	SAT TT=30.815 CHECKS=15004800	SAT TT=6.917 CHECKS=2820000	SAT TT=6.203 ASSIGNMENTS=76	SAT TT=6.139 ASSIGNMENTS=76	SAT TT=13.984 CHECKS=76029 NODES=489	SAT TT=17.597 CHECKS=13907500 NODES=47630	SAT TT=20.227 CHECKS=20735900 NODES=5463	SAT TT=0.365 ASSIGNMENTS=174 CHECKS=476259	SAT TT=0.367 ASSIGNMENTS=174 CHECKS=476259	SAT TT=5.484	SAT TT=4.615	SAT TT=0.53 ASSIGNMENTS=470 CHECKS=1122820	SAT TT=2.093 ASSIGNMENTS=9657 CHECKS=6612560	SAT TT=3.392	SAT TT=1.383 ASSIGNMENTS=1191 CHECKS=4009560	SAT TT=0.359 ASSIGNMENTS=174 CHECKS=476259	SAT TT=5.288	SAT TT=2.164 ASSIGNMENTS=1141 CHECKS=648302	SAT TT=2.208	SAT TT=2.282
csp/driver/ normalized-driverlogw-08c-sat_ext.xml	SAT TT=0.351	SAT TT=3.349 CHECKS=3976790 NODES=3591	SAT TT=4.645 CHECKS=5487000 NODES=3194	SAT TT=1.269	SAT TT=31.053 CHECKS=17547300	SAT TT=5.419 CHECKS=2209600	SAT TT=6.107 ASSIGNMENTS=76	SAT TT=6.154 ASSIGNMENTS=76	SAT TT=14.138 CHECKS=76056 NODES=489	SAT TT=57.988 CHECKS=134653000 NODES=490165	SAT TT=15.427 CHECKS=6441340 NODES=2234	SAT TT=0.366 ASSIGNMENTS=174 CHECKS=476259	SAT TT=0.363 ASSIGNMENTS=174 CHECKS=476259	SAT TT=5.097	SAT TT=4.959	SAT TT=0.516 ASSIGNMENTS=470 CHECKS=1122820	SAT TT=2.111 ASSIGNMENTS=9682 CHECKS=6424930	SAT TT=3.491	SAT TT=1.407 ASSIGNMENTS=1191 CHECKS=4009560	SAT TT=0.351 ASSIGNMENTS=174 CHECKS=476259	SAT TT=5.217	SAT TT=2.209 ASSIGNMENTS=1141 CHECKS=648251	SAT TT=2.264	SAT TT=2.23
csp/driver/ normalized-driverlogw-09-sat_ext.xml	SAT TT=2.291	SAT TT=152.666 CHECKS=35571000 NODES=27204	SAT TT=143.575 CHECKS=33533600 NODES=23806	SAT TT=42.351	SAT TT=80.622 CHECKS=34515400	SAT TT=93.872 CHECKS=41022300	SAT TT=15.952 ASSIGNMENTS=345	SAT TT=17.887 ASSIGNMENTS=345	SAT TT=141.088 CHECKS=1964450 NODES=6371	? (TO) TT=1800.06	? (TO) TT=1800.06	SAT TT=34.14 ASSIGNMENTS=28499 CHECKS=36491500	SAT TT=37.224 ASSIGNMENTS=28499 CHECKS=36491500	SAT TT=10.527	SAT TT=9.623	SAT TT=2.291 ASSIGNMENTS=1581 CHECKS=5209070	SAT TT=7.406 ASSIGNMENTS=14916 CHECKS=17398700	SAT TT=8.216	SAT TT=6.433 ASSIGNMENTS=2328 CHECKS=14564800	SAT TT=2.975 ASSIGNMENTS=2280 CHECKS=6973980	SAT TT=6.641	SAT TT=39.181 ASSIGNMENTS=22420 CHECKS=20207800	SAT TT=3.539	SAT TT=3.934

Some statistics...

	Abscon 112v4 AC	Abscon 112v4 ESAC	bpsolver 2008-06-27	casper zao	casper zito	choco2_dwdeg 2008-06-26	choco2_impwdeg 2008-06-26	Concrete + CPS4J 2008-05-30	Concrete + CSP4J - Tabu Engine 2008-05-30	cpHydra k_10	cpHydra k_40	galac E	galac M	MDG-noprobe 2008-06-27	MDG-probe 2008-06-27	Minion/Tailor 2008-07-04	Mistral-option 1.314	Mistral-prime 1.313	SAT4J CSP 2008-06-13	spider 2008-06-14	Sugar v1.13+minisat	Sugar v1.13+picosat
Number of times the solver is able to give the best known answer	7	7	7	6	7	7	7	7	6	7	7	7	7	7	7	7	7	7	7	7	7	7
Number of times the solver is able to give the best known answer from an incomplete solver point of view (i.e. without considering optimality proof)	7	7	7	6	7	7	7	7	6	7	7	7	7	7	7	7	7	7	7	7	7	7
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0	0	0	6	0	0	0	0
Number of times the solver is the best solver from an incomplete solver point of view (i.e. best known answer and best T1 time)	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0