CSP 2009 Competition: results by benchmark

Results by benchmark for category alldiff,element,weightedsum (Alldiff+Elt+WSum), subcategory real world problems

This page displays the results of the different solvers for each benchmark for category alldiff,element,weightedsum (Alldiff+Elt+WSum), subcategory real world problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC (complete)
Abscon 112v4 ESAC (complete)
bpsolver 09 (complete)
Choco2.1.1 2009-06-10 (complete)
Choco2.1.1b 2009-07-16 (complete)
Mistral 1.545 (complete)
Sugar v1.14.6+minisat (complete)
Sugar v1.14.6+picosat (complete)

Bench name	Best results on this instance	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/CabinetStart1/ normalized-cabinet-5561.xml	UNSAT TT=0.37	? (TO) TT=1800.09	? (TO) TT=1800.11	UNSAT TT=0.37	UNSAT TT=81.684	UNSAT TT=69.946	UNSAT TT=1.602	UNSAT TT=97.149	UNSAT TT=66.739
csp/CabinetStart1/ normalized-cabinet-5562.xml	UNSAT TT=0.371	? (TO) TT=1800.11	? (TO) TT=1800.16	UNSAT TT=0.371	UNSAT TT=75.452	UNSAT TT=65.773	UNSAT TT=1.667	UNSAT TT=99.912	UNSAT TT=66.279
csp/CabinetStart1/ normalized-cabinet-5563.xml	UNSAT TT=0.368	? (TO) TT=1800.15	? (TO) TT=1800.09	UNSAT TT=0.368	UNSAT TT=63.505	UNSAT TT=74.047	UNSAT TT=1.624	UNSAT TT=96.742	UNSAT TT=68.842
csp/CabinetStart1/ normalized-cabinet-5564.xml	UNSAT TT=0.366	? (TO) TT=1800.15	? (TO) TT=1800.08	UNSAT TT=0.366	UNSAT TT=62.66	UNSAT TT=72.614	UNSAT TT=1.616	UNSAT TT=102.013	UNSAT TT=65.619
csp/CabinetStart1/ normalized-cabinet-5565.xml	UNSAT TT=0.369	? (TO) TT=1800.1	? (TO) TT=1800.05	UNSAT TT=0.369	UNSAT TT=58.208	UNSAT TT=72.721	UNSAT TT=1.607	UNSAT TT=111.101	UNSAT TT=70.072
csp/CabinetStart1/ normalized-cabinet-5566.xml	UNSAT TT=0.367	? (TO) TT=1800.1	? (TO) TT=1800.18	UNSAT TT=0.367	UNSAT TT=63.645	UNSAT TT=58.988	UNSAT TT=1.739	UNSAT TT=111.206	UNSAT TT=75.508
csp/CabinetStart1/ normalized-cabinet-5567.xml	UNSAT TT=0.366	? (TO) TT=1800.07	? (TO) TT=1800.11	UNSAT TT=0.366	UNSAT TT=62.702	UNSAT TT=71.98	UNSAT TT=1.741	UNSAT TT=104.758	UNSAT TT=68.924
csp/CabinetStart1/ normalized-cabinet-5568.xml	UNSAT TT=0.372	? (TO) TT=1800.15	? (TO) TT=1800.1	UNSAT TT=0.372	UNSAT TT=56.568	UNSAT TT=60.079	UNSAT TT=1.633	UNSAT TT=108.929	UNSAT TT=70.611
csp/CabinetStart1/ normalized-cabinet-5569.xml	UNSAT TT=0.368	? (TO) TT=1800.12	? (TO) TT=1800.17	UNSAT TT=0.368	UNSAT TT=50.275	UNSAT TT=56.732	UNSAT TT=1.592	UNSAT TT=108.959	UNSAT TT=62.309
csp/CabinetStart1/ normalized-cabinet-5570.xml	UNSAT TT=0.366	? (TO) TT=1800.16	? (TO) TT=1800.12	UNSAT TT=0.366	UNSAT TT=59.146	UNSAT TT=72.632	UNSAT TT=1.594	UNSAT TT=126.218	UNSAT TT=69.645
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/CabinetStart1/ normalized-cabinet-5571.xml	UNSAT TT=0.374	? (TO) TT=1800.15	? (TO) TT=1800.12	UNSAT TT=0.374	UNSAT TT=71.134	UNSAT TT=59.244	UNSAT TT=2.129	UNSAT TT=105.455	UNSAT TT=73.915
csp/CabinetStart1/ normalized-cabinet-5572.xml	UNSAT TT=0.367	? (TO) TT=1800.18	? (TO) TT=1800.14	UNSAT TT=0.367	UNSAT TT=53.379	UNSAT TT=61.312	UNSAT TT=1.157	UNSAT TT=82.427	UNSAT TT=78.546
csp/CabinetStart1/ normalized-cabinet-5573.xml	UNSAT TT=0.371	? (TO) TT=1800.11	? (TO) TT=1800.1	UNSAT TT=0.371	UNSAT TT=66.533	UNSAT TT=65.825	UNSAT TT=1.17	UNSAT TT=91.555	UNSAT TT=76.685
csp/CabinetStart1/ normalized-cabinet-5574.xml	UNSAT TT=0.351	? (TO) TT=1800.1	? (TO) TT=1800.13	UNSAT TT=0.351	UNSAT TT=54.013	UNSAT TT=70.198	UNSAT TT=1.087	UNSAT TT=76.473	UNSAT TT=74.978
csp/CabinetStart1/ normalized-cabinet-5575.xml	UNSAT TT=0.371	? (TO) TT=1800.14	? (TO) TT=1800.13	UNSAT TT=0.371	UNSAT TT=59.115	UNSAT TT=51.782	UNSAT TT=1.096	UNSAT TT=92.156	UNSAT TT=72.018
csp/CabinetStart1/ normalized-cabinet-5576.xml	UNSAT TT=0.343	? (TO) TT=1800.11	? (TO) TT=1800.17	UNSAT TT=0.343	UNSAT TT=56.808	UNSAT TT=56.244	UNSAT TT=1.413	UNSAT TT=93.812	UNSAT TT=77.788
csp/CabinetStart1/ normalized-cabinet-5577.xml	UNSAT TT=0.348	? (TO) TT=1800.13	? (TO) TT=1800.1	UNSAT TT=0.348	UNSAT TT=64.833	UNSAT TT=75.066	UNSAT TT=1.423	UNSAT TT=94.865	UNSAT TT=82.712
csp/CabinetStart1/ normalized-cabinet-5578.xml	UNSAT TT=0.366	? (TO) TT=1800.14	? (TO) TT=1800.07	UNSAT TT=0.366	UNSAT TT=64.547	UNSAT TT=55.301	UNSAT TT=1.512	UNSAT TT=110.982	UNSAT TT=68.28
csp/CabinetStart1/ normalized-cabinet-5579.xml	UNSAT TT=0.368	? (TO) TT=1800.06	? (TO) TT=1800.12	UNSAT TT=0.368	UNSAT TT=68.587	UNSAT TT=66.078	UNSAT TT=1.43	UNSAT TT=98.497	UNSAT TT=72.519
csp/CabinetStart1/ normalized-cabinet-5580.xml	UNSAT TT=0.367	? (TO) TT=1800.1	? (TO) TT=1800.1	UNSAT TT=0.367	UNSAT TT=60.531	UNSAT TT=55.823	UNSAT TT=1.406	UNSAT TT=109.408	UNSAT TT=75.189
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/CabinetStart1/ normalized-cabinet-5581.xml	SAT TT=0.024	? (TO) TT=1800.17	? (TO) TT=1800.14	SAT TT=0.06	SAT TT=33.609	SAT TT=36.714	SAT TT=0.024	SAT TT=10.778	SAT TT=22.464
csp/CabinetStart1/ normalized-cabinet-5582.xml	SAT TT=0.022	? (TO) TT=1800.13	? (TO) TT=1800.12	SAT TT=0.059	SAT TT=34.24	SAT TT=34.057	SAT TT=0.022	SAT TT=10.416	SAT TT=20.874
csp/CabinetStart1/ normalized-cabinet-5583.xml	SAT TT=0.022	? (TO) TT=1800.11	? (TO) TT=1800.15	SAT TT=0.06	SAT TT=33.739	SAT TT=33.996	SAT TT=0.022	SAT TT=10.407	SAT TT=22.465
csp/CabinetStart1/ normalized-cabinet-5584.xml	SAT TT=0.024	? (TO) TT=1800.12	? (TO) TT=1800.06	SAT TT=0.062	SAT TT=33.545	SAT TT=36.872	SAT TT=0.024	SAT TT=13.694	SAT TT=22.255
csp/CabinetStart1/ normalized-cabinet-5585.xml	SAT TT=0.021	? (TO) TT=1800.12	? (TO) TT=1800.1	SAT TT=0.061	SAT TT=33.723	SAT TT=36.906	SAT TT=0.021	SAT TT=13.727	SAT TT=22.662
csp/CabinetStart1/ normalized-cabinet-5586.xml	SAT TT=0.025	? (TO) TT=1800.15	? (TO) TT=1800.09	SAT TT=0.068	SAT TT=33.558	SAT TT=36.86	SAT TT=0.025	SAT TT=13.989	SAT TT=25.361
csp/CabinetStart1/ normalized-cabinet-5587.xml	SAT TT=0.022	? (TO) TT=1800.12	? (TO) TT=1800.14	SAT TT=0.066	SAT TT=34.258	SAT TT=34.408	SAT TT=0.022	SAT TT=13.9	SAT TT=22.041
csp/CabinetStart1/ normalized-cabinet-5588.xml	SAT TT=0.024	? (TO) TT=1800.11	? (TO) TT=1800.09	SAT TT=0.067	SAT TT=37.325	SAT TT=36.372	SAT TT=0.024	SAT TT=15.218	SAT TT=22.381
csp/CabinetStart1/ normalized-cabinet-5589.xml	SAT TT=0.021	? (TO) TT=1800.11	? (TO) TT=1800.11	SAT TT=0.061	SAT TT=34.191	SAT TT=35.121	SAT TT=0.021	SAT TT=13.782	SAT TT=20.464
csp/CabinetStart1/ normalized-cabinet-5590.xml	SAT TT=0.025	? (TO) TT=1800.11	? (TO) TT=1800.12	SAT TT=0.066	SAT TT=33.784	SAT TT=37.106	SAT TT=0.025	SAT TT=13.759	SAT TT=20.078
Bench name	Best results	Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/CabinetStart1/ normalized-cabinet-5591.xml	SAT TT=0.024	? (TO) TT=1800.15	? (TO) TT=1800.15	SAT TT=0.063	SAT TT=37.14	SAT TT=34.184	SAT TT=0.024	SAT TT=14.07	SAT TT=20.891
csp/CabinetStart1/ normalized-cabinet-5592.xml	SAT TT=0.025	? (TO) TT=1800.17	? (TO) TT=1800.13	SAT TT=0.066	SAT TT=36.653	SAT TT=36.909	SAT TT=0.025	SAT TT=13.439	SAT TT=22.488
csp/CabinetStart1/ normalized-cabinet-5593.xml	SAT TT=0.025	? (TO) TT=1800.09	? (TO) TT=1800.09	SAT TT=0.06	SAT TT=34.39	SAT TT=36.916	SAT TT=0.025	SAT TT=14.328	SAT TT=21.671
csp/CabinetStart1/ normalized-cabinet-5594.xml	SAT TT=0.023	? (TO) TT=1800.07	? (TO) TT=1800.16	SAT TT=0.064	SAT TT=34.16	SAT TT=37.002	SAT TT=0.023	SAT TT=13.15	SAT TT=20.03
csp/CabinetStart1/ normalized-cabinet-5595.xml	SAT TT=0.025	? (TO) TT=1800.16	? (TO) TT=1800.17	SAT TT=0.063	SAT TT=34.272	SAT TT=37.089	SAT TT=0.025	SAT TT=13.688	SAT TT=20.092
csp/CabinetStart1/ normalized-cabinet-5596.xml	SAT TT=0.025	? (TO) TT=1800.14	? (TO) TT=1800.17	SAT TT=0.063	SAT TT=37.182	SAT TT=34.313	SAT TT=0.025	SAT TT=13.993	SAT TT=19.997
csp/CabinetStart1/ normalized-cabinet-5597.xml	SAT TT=0.026	? (TO) TT=1800.17	? (TO) TT=1800.14	SAT TT=0.065	SAT TT=36.5	SAT TT=34.531	SAT TT=0.026	SAT TT=15.994	SAT TT=21.639
csp/CabinetStart1/ normalized-cabinet-5598.xml	SAT TT=0.024	? (TO) TT=1800.19	? (TO) TT=1800.15	SAT TT=0.065	SAT TT=36.747	SAT TT=37.054	SAT TT=0.024	SAT TT=14.828	SAT TT=20.109
csp/CabinetStart1/ normalized-cabinet-5599.xml	SAT TT=0.023	? (TO) TT=1800.12	? (TO) TT=1800.16	SAT TT=0.061	SAT TT=36.607	SAT TT=35.296	SAT TT=0.023	SAT TT=10.234	SAT TT=21.77
csp/CabinetStart1/ normalized-cabinet-5600.xml	SAT TT=0.023	? (TO) TT=1800.15	? (TO) TT=1800.16	SAT TT=0.064	SAT TT=34.425	SAT TT=34.165	SAT TT=0.023	SAT TT=12.158	SAT TT=20.36

Statistics		Abscon 112v4 AC (complete)	Abscon 112v4 ESAC (complete)	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
Number of times the solver is able to give the best known answer		0	0	40	40	40	40	40	40
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		0	0	20	0	0	20	0	0