CSP 2009 Competition: results by benchmark

Results by benchmark for category alldiff,cumulative,element,weightedsum (Alldiff+Cumul+Elt+WSum), subcategory academic problems

This page displays the results of the different solvers for each benchmark for category alldiff,cumulative,element,weightedsum (Alldiff+Cumul+Elt+WSum), subcategory academic problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
bpsolver 09 (complete)
Choco2.1.1 2009-06-10 (complete)
Choco2.1.1b 2009-07-16 (complete)
Mistral 1.545 (complete)
Sugar v1.14.6+minisat (complete)
Sugar v1.14.6+picosat (complete)

Bench name	Best results on this instance	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-10-10.xml	SAT TT=0.039	SAT TT=0.397	SAT TT=1.917	SAT TT=1.796	SAT TT=0.039	SAT TT=1.209	SAT TT=1.172
csp/allsquares/ normalized-squares-11-11.xml	SAT TT=0.038	SAT TT=591.919	SAT TT=2.338	SAT TT=2.507	SAT TT=0.038	SAT TT=1.464	SAT TT=1.518
csp/allsquares/ normalized-squares-12-12.xml	SAT TT=0.039	SAT TT=1345.16	SAT TT=2.731	SAT TT=2.672	SAT TT=0.039	SAT TT=1.38	SAT TT=1.706
csp/allsquares/ normalized-squares-13-13.xml	SAT TT=0.053	SAT TT=1339.98	SAT TT=4.723	SAT TT=3.561	SAT TT=0.053	SAT TT=1.488	SAT TT=1.657
csp/allsquares/ normalized-squares-14-14.xml	SAT TT=0.059	? (TO) TT=1800.09	SAT TT=4.294	SAT TT=7.824	SAT TT=0.059	SAT TT=1.594	SAT TT=1.925
csp/allsquares/ normalized-squares-15-15.xml	SAT TT=0.073	SAT TT=1344.66	SAT TT=6.097	SAT TT=6.204	SAT TT=0.073	SAT TT=1.759	SAT TT=2.188
csp/allsquares/ normalized-squares-16-16.xml	SAT TT=2.345	SAT TT=1350	SAT TT=70.017	SAT TT=66.098	SAT TT=4.819	SAT TT=2.345	SAT TT=2.582
csp/allsquares/ normalized-squares-17-17.xml	SAT TT=0.967	SAT TT=1340.49	SAT TT=36.922	SAT TT=28.523	SAT TT=0.967	SAT TT=2.342	SAT TT=2.907
csp/allsquares/ normalized-squares-18-18.xml	SAT TT=2.373	SAT TT=1340.65	SAT TT=771.659	SAT TT=11.398	SAT TT=2.376	SAT TT=2.373	SAT TT=3.658
csp/allsquares/ normalized-squares-19-19.xml	SAT TT=3.83	SAT TT=1524.67	SAT TT=1047.6	? (TO) TT=1800.14	SAT TT=63.694	SAT TT=20.152	SAT TT=3.83
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-20-20.xml	SAT TT=3.478	? (TO) TT=1800.01	? (TO) TT=1800.16	? (TO) TT=1800.11	SAT TT=17.174	SAT TT=3.478	SAT TT=5.132
csp/allsquares/ normalized-squares-21-21.xml	SAT TT=4.428	SAT TT=1347.75	? (TO) TT=1800.18	? (TO) TT=1800.14	SAT TT=124.504	SAT TT=4.428	SAT TT=6.041
csp/allsquares/ normalized-squares-22-22.xml	SAT TT=159.864	? (TO) TT=1800.02	? (TO) TT=1800.06	? (TO) TT=1800.15	? (TO) TT=1800.1	SAT TT=242.447	SAT TT=159.864
csp/allsquares/ normalized-squares-2-2.xml	SAT TT=0.008	SAT TT=0.027	SAT TT=0.575	SAT TT=0.635	SAT TT=0.008	SAT TT=0.554	SAT TT=0.541
csp/allsquares/ normalized-squares-23-23.xml	SAT TT=320.803	? (TO) TT=1800.05	? (TO) TT=1800.11	? (TO) TT=1800.07	? (TO) TT=1800.1	SAT TT=978.401	SAT TT=320.803
csp/allsquares/ normalized-squares-24-24.xml	SAT TT=8.316	? (TO) TT=1800.07	? (TO) TT=1800.16	? (TO) TT=1800.16	? (TO) TT=1800.02	SAT TT=8.316	SAT TT=10.208
csp/allsquares/ normalized-squares-25-25.xml	SAT TT=50.821	? (TO) TT=1800.03	? (TO) TT=1800.12	? (TO) TT=1800.12	? (TO) TT=1800.09	SAT TT=50.821	SAT TT=84.517
csp/allsquares/ normalized-squares-26-26.xml	SAT TT=11.586	? (TO) TT=1800.09	? (TO) TT=1800.12	? (TO) TT=1800.29	SAT TT=699.949	SAT TT=11.586	SAT TT=16.758
csp/allsquares/ normalized-squares-27-27.xml	SAT TT=10.988	? (TO) TT=1800.1	? (TO) TT=1800.13	? (TO) TT=1800.11	? (TO) TT=1800.07	SAT TT=10.988	SAT TT=29.622
csp/allsquares/ normalized-squares-28-28.xml	SAT TT=15.827	? (TO) TT=1800.1	? (TO) TT=1800.15	? (TO) TT=1800.09	? (TO) TT=1800.07	SAT TT=15.827	SAT TT=34.434
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-29-29.xml		? (TO) TT=1800.07	? (TO) TT=1800.11	? (TO) TT=1800.15	? (TO) TT=1800.05	? (TO) TT=1800.01	? (TO) TT=1800.09
csp/allsquares/ normalized-squares-30-30.xml	SAT TT=367.186	? (TO) TT=1800.1	? (TO) TT=1800.13	? (TO) TT=1800.1	? (TO) TT=1800.09	SAT TT=557.803	SAT TT=367.186
csp/allsquares/ normalized-squares-31-31.xml	SAT TT=388.083	? (TO) TT=1800.08	? (TO) TT=1800.11	? (TO) TT=1800.06	? (TO) TT=1800.06	SAT TT=388.083	SAT TT=445.238
csp/allsquares/ normalized-squares-32-32.xml	SAT TT=190.2	? (TO) TT=1800.06	? (TO) TT=1800.13	? (TO) TT=1800.07	? (TO) TT=1800.04	SAT TT=530.447	SAT TT=190.2
csp/allsquares/ normalized-squares-33-33.xml	SAT TT=567.061	? (TO) TT=1800.06	? (TO) TT=1800.12	? (TO) TT=1800.17	? (TO) TT=1800.1	SAT TT=567.061	SAT TT=814.452
csp/allsquares/ normalized-squares-3-3.xml	SAT TT=0.008	SAT TT=0.029	SAT TT=0.67	SAT TT=0.659	SAT TT=0.008	SAT TT=0.598	SAT TT=0.598
csp/allsquares/ normalized-squares-34-34.xml	SAT TT=160.441	? (TO) TT=1800.09	? (TO) TT=1800.06	? (TO) TT=1800.16	? (TO) TT=1800.04	SAT TT=160.441	? (TO) TT=1800.07
csp/allsquares/ normalized-squares-35-35.xml		? (TO) TT=1800.05	? (TO) TT=1800.11	? (TO) TT=1800.15	? (TO) TT=1800.04	? (TO) TT=1800.05	? (TO) TT=1800.01
csp/allsquares/ normalized-squares-36-36.xml		? (TO) TT=1800.05	? (TO) TT=1800.14	? (TO) TT=1800.06	? (TO) TT=1800.11	? (TO) TT=1800.03	? (TO) TT=1800.03
csp/allsquares/ normalized-squares-37-37.xml		? (TO) TT=1800.1	? (TO) TT=1800.06	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.06	? (TO) TT=1800.09
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-38-38.xml		? (TO) TT=1800.06	? (TO) TT=1800.11	? (TO) TT=1800.12	? (TO) TT=1800.11	? (TO) TT=1800.08	? (TO) TT=1800.13
csp/allsquares/ normalized-squares-4-4.xml	SAT TT=0.008	SAT TT=0.035	SAT TT=0.764	SAT TT=0.78	SAT TT=0.008	SAT TT=0.734	SAT TT=0.722
csp/allsquares/ normalized-squares-5-5.xml	SAT TT=0.027	SAT TT=0.039	SAT TT=0.859	SAT TT=0.961	SAT TT=0.027	SAT TT=0.804	SAT TT=0.924
csp/allsquares/ normalized-squares-6-6.xml	SAT TT=0.014	SAT TT=0.05	SAT TT=1.005	SAT TT=1.039	SAT TT=0.014	SAT TT=0.898	SAT TT=0.967
csp/allsquares/ normalized-squares-7-7.xml	SAT TT=0.015	SAT TT=0.054	SAT TT=1.149	SAT TT=1.149	SAT TT=0.015	SAT TT=0.964	SAT TT=0.961
csp/allsquares/ normalized-squares-8-8.xml	SAT TT=0.018	SAT TT=0.064	SAT TT=1.363	SAT TT=1.32	SAT TT=0.018	SAT TT=1.035	SAT TT=1.071
csp/allsquares/ normalized-squares-9-9.xml	SAT TT=0.023	SAT TT=0.088	SAT TT=1.594	SAT TT=1.685	SAT TT=0.023	SAT TT=1.086	SAT TT=1.131
csp/allsquaresUnsat/ normalized-squaresUnsat-10-10.xml	UNSAT TT=0.025	UNSAT TT=108.482	UNSAT TT=1.197	UNSAT TT=1.184	UNSAT TT=0.025	UNSAT TT=1.091	UNSAT TT=1.198
csp/allsquaresUnsat/ normalized-squaresUnsat-11-11.xml	UNSAT TT=0.036	UNSAT TT=1349.74	UNSAT TT=1.284	UNSAT TT=1.469	UNSAT TT=0.036	UNSAT TT=1.239	UNSAT TT=1.32
csp/allsquaresUnsat/ normalized-squaresUnsat-12-12.xml	UNSAT TT=0.051	UNSAT TT=1340.11	UNSAT TT=1.509	UNSAT TT=1.499	UNSAT TT=0.051	UNSAT TT=1.523	UNSAT TT=1.67
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-13-13.xml	UNSAT TT=0.046	UNSAT TT=1344.77	UNSAT TT=1.779	UNSAT TT=1.627	UNSAT TT=0.046	UNSAT TT=1.598	UNSAT TT=1.757
csp/allsquaresUnsat/ normalized-squaresUnsat-14-14.xml	UNSAT TT=0.052	UNSAT TT=1345.11	UNSAT TT=1.693	UNSAT TT=1.849	UNSAT TT=0.052	UNSAT TT=1.539	UNSAT TT=1.999
csp/allsquaresUnsat/ normalized-squaresUnsat-15-15.xml	UNSAT TT=0.059	UNSAT TT=1349.71	UNSAT TT=1.139	UNSAT TT=1.126	UNSAT TT=0.059	UNSAT TT=1.655	UNSAT TT=2.075
csp/allsquaresUnsat/ normalized-squaresUnsat-16-16.xml	UNSAT TT=0.095	UNSAT TT=1339.37	UNSAT TT=1.235	UNSAT TT=1.252	UNSAT TT=0.095	UNSAT TT=1.755	UNSAT TT=2.404
csp/allsquaresUnsat/ normalized-squaresUnsat-17-17.xml	UNSAT TT=1.256	UNSAT TT=1344.39	UNSAT TT=1.256	UNSAT TT=1.324	UNSAT TT=11.014	UNSAT TT=2.871	UNSAT TT=3.208
csp/allsquaresUnsat/ normalized-squaresUnsat-18-18.xml	UNSAT TT=10.247	UNSAT TT=1612.84	? (TO) TT=1800.09	? (TO) TT=1800.09	UNSAT TT=544.005	UNSAT TT=10.247	UNSAT TT=10.817
csp/allsquaresUnsat/ normalized-squaresUnsat-19-19.xml	UNSAT TT=1.344	UNSAT TT=1531.02	UNSAT TT=1.344	UNSAT TT=1.542	UNSAT TT=232.555	UNSAT TT=10.121	UNSAT TT=8.34
csp/allsquaresUnsat/ normalized-squaresUnsat-20-20.xml	UNSAT TT=1.456	? (TO) TT=1800.07	UNSAT TT=1.456	UNSAT TT=1.497	UNSAT TT=427.064	UNSAT TT=26.507	UNSAT TT=24.364
csp/allsquaresUnsat/ normalized-squaresUnsat-21-21.xml	UNSAT TT=1.433	? (TO) TT=1800.11	UNSAT TT=1.433	UNSAT TT=1.486	? (TO) TT=1800.04	UNSAT TT=59.366	UNSAT TT=61.101
csp/allsquaresUnsat/ normalized-squaresUnsat-22-22.xml	UNSAT TT=1.527	? (TO) TT=1800.02	UNSAT TT=1.527	UNSAT TT=1.726	? (TO) TT=1800.05	UNSAT TT=97.286	UNSAT TT=79.838
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-2-2.xml	UNSAT TT=0.007	UNSAT TT=0.028	UNSAT TT=0.586	UNSAT TT=0.487	UNSAT TT=0.007	UNSAT TT=0.38	UNSAT TT=0.374
csp/allsquaresUnsat/ normalized-squaresUnsat-23-23.xml	UNSAT TT=1.592	? (TO) TT=1800.04	UNSAT TT=1.592	UNSAT TT=1.729	? (TO) TT=1800.09	UNSAT TT=96.291	UNSAT TT=130.302
csp/allsquaresUnsat/ normalized-squaresUnsat-24-24.xml		? (TO) TT=1800.03	? (TO) TT=1800.16	? (TO) TT=1800.1	? (TO) TT=1800.09	? (TO) TT=1800.01	? (TO) TT=1800.04
csp/allsquaresUnsat/ normalized-squaresUnsat-25-25.xml	UNSAT TT=1.736	? (TO) TT=1800.08	UNSAT TT=1.736	UNSAT TT=1.805	? (TO) TT=1800.01	? (TO) TT=1800.06	? (TO) TT=1800.03
csp/allsquaresUnsat/ normalized-squaresUnsat-26-26.xml		? (TO) TT=1800.07	? (TO) TT=1800.06	? (TO) TT=1800.1	? (TO) TT=1800.11	? (TO) TT=1800.05	? (TO) TT=1800.08
csp/allsquaresUnsat/ normalized-squaresUnsat-27-27.xml	UNSAT TT=1.969	? (TO) TT=1800.05	UNSAT TT=1.969	UNSAT TT=2.044	? (TO) TT=1800.07	? (TO) TT=1800.07	? (TO) TT=1800.07
csp/allsquaresUnsat/ normalized-squaresUnsat-28-28.xml		? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.16	? (TO) TT=1800.06	? (TO) TT=1800.01	? (TO) TT=1800.01
csp/allsquaresUnsat/ normalized-squaresUnsat-29-29.xml	UNSAT TT=2.054	? (TO) TT=1800.07	UNSAT TT=2.054	UNSAT TT=2.627	? (TO) TT=1800.05	? (TO) TT=1800.04	? (TO) TT=1800.02
csp/allsquaresUnsat/ normalized-squaresUnsat-30-30.xml	UNSAT TT=2.145	? (TO) TT=1800.09	UNSAT TT=2.145	UNSAT TT=2.324	? (TO) TT=1800.11	? (TO) TT=1800.04	? (TO) TT=1800.01
csp/allsquaresUnsat/ normalized-squaresUnsat-31-31.xml	UNSAT TT=2.002	? (TO) TT=1800.07	UNSAT TT=2.155	UNSAT TT=2.002	? (TO) TT=1800.04	? (TO) TT=1800.07	? (TO) TT=1800.05
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-32-32.xml		? (TO) TT=1800.07	? (TO) TT=1800.17	? (TO) TT=1800.09	? (TO) TT=1800.02	? (TO) TT=1800.02	? (TO) TT=1800
csp/allsquaresUnsat/ normalized-squaresUnsat-33-33.xml		? (TO) TT=1800.06	? (TO) TT=1800.16	? (TO) TT=1800.12	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800.09
csp/allsquaresUnsat/ normalized-squaresUnsat-3-3.xml	UNSAT TT=0.007	UNSAT TT=0.03	UNSAT TT=0.505	UNSAT TT=0.552	UNSAT TT=0.007	UNSAT TT=0.419	UNSAT TT=0.428
csp/allsquaresUnsat/ normalized-squaresUnsat-34-34.xml		? (TO) TT=1800.09	? (TO) TT=1800.13	? (TO) TT=1800.1	? (TO) TT=1800.05	? (TO) TT=1800.09	? (TO) TT=1800.02
csp/allsquaresUnsat/ normalized-squaresUnsat-35-35.xml	UNSAT TT=2.864	? (TO) TT=1800.03	UNSAT TT=2.864	UNSAT TT=3.468	? (TO) TT=1800.09	? (TO) TT=1800.11	? (TO) TT=1800.08
csp/allsquaresUnsat/ normalized-squaresUnsat-36-36.xml	UNSAT TT=2.408	? (TO) TT=1800.12	UNSAT TT=2.408	UNSAT TT=2.486	? (TO) TT=1800.05	? (TO) TT=1800.01	? (TO) TT=1800.06
csp/allsquaresUnsat/ normalized-squaresUnsat-37-37.xml	UNSAT TT=2.933	? (TO) TT=1800.04	UNSAT TT=3.943	UNSAT TT=2.933	? (TO) TT=1800.11	? (TO) TT=1800.1	? (TO) TT=1800.06
csp/allsquaresUnsat/ normalized-squaresUnsat-38-38.xml		? (TO) TT=1800.07	? (TO) TT=1800.11	? (TO) TT=1800.04	? (TO) TT=1800.03	? (TO) TT=1800.08	? (TO) TT=1800.05
csp/allsquaresUnsat/ normalized-squaresUnsat-4-4.xml	UNSAT TT=0.02	UNSAT TT=0.034	UNSAT TT=0.599	UNSAT TT=0.548	UNSAT TT=0.02	UNSAT TT=0.563	UNSAT TT=0.565
csp/allsquaresUnsat/ normalized-squaresUnsat-5-5.xml	UNSAT TT=0.01	UNSAT TT=0.038	UNSAT TT=0.585	UNSAT TT=0.603	UNSAT TT=0.01	UNSAT TT=0.626	UNSAT TT=0.6
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-6-6.xml	UNSAT TT=0.012	UNSAT TT=0.044	UNSAT TT=0.646	UNSAT TT=0.635	UNSAT TT=0.012	UNSAT TT=0.654	UNSAT TT=0.801
csp/allsquaresUnsat/ normalized-squaresUnsat-7-7.xml	UNSAT TT=0.013	UNSAT TT=0.054	UNSAT TT=0.673	UNSAT TT=0.749	UNSAT TT=0.013	UNSAT TT=0.742	UNSAT TT=0.685
csp/allsquaresUnsat/ normalized-squaresUnsat-8-8.xml	UNSAT TT=0.016	UNSAT TT=0.058	UNSAT TT=0.799	UNSAT TT=0.779	UNSAT TT=0.016	UNSAT TT=0.758	UNSAT TT=0.759
csp/allsquaresUnsat/ normalized-squaresUnsat-9-9.xml	UNSAT TT=0.023	UNSAT TT=1.406	UNSAT TT=1.049	UNSAT TT=1.087	UNSAT TT=0.023	UNSAT TT=1.064	UNSAT TT=1.094

Statistics		bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
Number of times the solver is able to give the best known answer		36	47	46	40	54	53
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		0	12	2	30	13	5