CSP 2009 Competition: results by benchmark

Results by benchmark for category alldiff,cumulative,element,weightedsum (Alldiff+Cumul+Elt+WSum)

This page displays the results of the different solvers for each benchmark for category alldiff,cumulative,element,weightedsum (Alldiff+Cumul+Elt+WSum)

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver gave a definitive answer (SAT or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
bpsolver 09 (complete)
Choco2.1.1 2009-06-10 (complete)
Choco2.1.1b 2009-07-16 (complete)
Mistral 1.545 (complete)
Sugar v1.14.6+minisat (complete)
Sugar v1.14.6+picosat (complete)

Bench name	Best results on this instance	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-10-10.xml	SAT TT=0.039	SAT TT=0.397	SAT TT=1.917	SAT TT=1.796	SAT TT=0.039	SAT TT=1.209	SAT TT=1.172
csp/allsquares/ normalized-squares-11-11.xml	SAT TT=0.038	SAT TT=591.919	SAT TT=2.338	SAT TT=2.507	SAT TT=0.038	SAT TT=1.464	SAT TT=1.518
csp/allsquares/ normalized-squares-12-12.xml	SAT TT=0.039	SAT TT=1345.16	SAT TT=2.731	SAT TT=2.672	SAT TT=0.039	SAT TT=1.38	SAT TT=1.706
csp/allsquares/ normalized-squares-13-13.xml	SAT TT=0.053	SAT TT=1339.98	SAT TT=4.723	SAT TT=3.561	SAT TT=0.053	SAT TT=1.488	SAT TT=1.657
csp/allsquares/ normalized-squares-14-14.xml	SAT TT=0.059	? (TO) TT=1800.09	SAT TT=4.294	SAT TT=7.824	SAT TT=0.059	SAT TT=1.594	SAT TT=1.925
csp/allsquares/ normalized-squares-15-15.xml	SAT TT=0.073	SAT TT=1344.66	SAT TT=6.097	SAT TT=6.204	SAT TT=0.073	SAT TT=1.759	SAT TT=2.188
csp/allsquares/ normalized-squares-16-16.xml	SAT TT=2.345	SAT TT=1350	SAT TT=70.017	SAT TT=66.098	SAT TT=4.819	SAT TT=2.345	SAT TT=2.582
csp/allsquares/ normalized-squares-17-17.xml	SAT TT=0.967	SAT TT=1340.49	SAT TT=36.922	SAT TT=28.523	SAT TT=0.967	SAT TT=2.342	SAT TT=2.907
csp/allsquares/ normalized-squares-18-18.xml	SAT TT=2.373	SAT TT=1340.65	SAT TT=771.659	SAT TT=11.398	SAT TT=2.376	SAT TT=2.373	SAT TT=3.658
csp/allsquares/ normalized-squares-19-19.xml	SAT TT=3.83	SAT TT=1524.67	SAT TT=1047.6	? (TO) TT=1800.14	SAT TT=63.694	SAT TT=20.152	SAT TT=3.83
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-20-20.xml	SAT TT=3.478	? (TO) TT=1800.01	? (TO) TT=1800.16	? (TO) TT=1800.11	SAT TT=17.174	SAT TT=3.478	SAT TT=5.132
csp/allsquares/ normalized-squares-21-21.xml	SAT TT=4.428	SAT TT=1347.75	? (TO) TT=1800.18	? (TO) TT=1800.14	SAT TT=124.504	SAT TT=4.428	SAT TT=6.041
csp/allsquares/ normalized-squares-22-22.xml	SAT TT=159.864	? (TO) TT=1800.02	? (TO) TT=1800.06	? (TO) TT=1800.15	? (TO) TT=1800.1	SAT TT=242.447	SAT TT=159.864
csp/allsquares/ normalized-squares-2-2.xml	SAT TT=0.008	SAT TT=0.027	SAT TT=0.575	SAT TT=0.635	SAT TT=0.008	SAT TT=0.554	SAT TT=0.541
csp/allsquares/ normalized-squares-23-23.xml	SAT TT=320.803	? (TO) TT=1800.05	? (TO) TT=1800.11	? (TO) TT=1800.07	? (TO) TT=1800.1	SAT TT=978.401	SAT TT=320.803
csp/allsquares/ normalized-squares-24-24.xml	SAT TT=8.316	? (TO) TT=1800.07	? (TO) TT=1800.16	? (TO) TT=1800.16	? (TO) TT=1800.02	SAT TT=8.316	SAT TT=10.208
csp/allsquares/ normalized-squares-25-25.xml	SAT TT=50.821	? (TO) TT=1800.03	? (TO) TT=1800.12	? (TO) TT=1800.12	? (TO) TT=1800.09	SAT TT=50.821	SAT TT=84.517
csp/allsquares/ normalized-squares-26-26.xml	SAT TT=11.586	? (TO) TT=1800.09	? (TO) TT=1800.12	? (TO) TT=1800.29	SAT TT=699.949	SAT TT=11.586	SAT TT=16.758
csp/allsquares/ normalized-squares-27-27.xml	SAT TT=10.988	? (TO) TT=1800.1	? (TO) TT=1800.13	? (TO) TT=1800.11	? (TO) TT=1800.07	SAT TT=10.988	SAT TT=29.622
csp/allsquares/ normalized-squares-28-28.xml	SAT TT=15.827	? (TO) TT=1800.1	? (TO) TT=1800.15	? (TO) TT=1800.09	? (TO) TT=1800.07	SAT TT=15.827	SAT TT=34.434
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-29-29.xml		? (TO) TT=1800.07	? (TO) TT=1800.11	? (TO) TT=1800.15	? (TO) TT=1800.05	? (TO) TT=1800.01	? (TO) TT=1800.09
csp/allsquares/ normalized-squares-30-30.xml	SAT TT=367.186	? (TO) TT=1800.1	? (TO) TT=1800.13	? (TO) TT=1800.1	? (TO) TT=1800.09	SAT TT=557.803	SAT TT=367.186
csp/allsquares/ normalized-squares-31-31.xml	SAT TT=388.083	? (TO) TT=1800.08	? (TO) TT=1800.11	? (TO) TT=1800.06	? (TO) TT=1800.06	SAT TT=388.083	SAT TT=445.238
csp/allsquares/ normalized-squares-32-32.xml	SAT TT=190.2	? (TO) TT=1800.06	? (TO) TT=1800.13	? (TO) TT=1800.07	? (TO) TT=1800.04	SAT TT=530.447	SAT TT=190.2
csp/allsquares/ normalized-squares-33-33.xml	SAT TT=567.061	? (TO) TT=1800.06	? (TO) TT=1800.12	? (TO) TT=1800.17	? (TO) TT=1800.1	SAT TT=567.061	SAT TT=814.452
csp/allsquares/ normalized-squares-3-3.xml	SAT TT=0.008	SAT TT=0.029	SAT TT=0.67	SAT TT=0.659	SAT TT=0.008	SAT TT=0.598	SAT TT=0.598
csp/allsquares/ normalized-squares-34-34.xml	SAT TT=160.441	? (TO) TT=1800.09	? (TO) TT=1800.06	? (TO) TT=1800.16	? (TO) TT=1800.04	SAT TT=160.441	? (TO) TT=1800.07
csp/allsquares/ normalized-squares-35-35.xml		? (TO) TT=1800.05	? (TO) TT=1800.11	? (TO) TT=1800.15	? (TO) TT=1800.04	? (TO) TT=1800.05	? (TO) TT=1800.01
csp/allsquares/ normalized-squares-36-36.xml		? (TO) TT=1800.05	? (TO) TT=1800.14	? (TO) TT=1800.06	? (TO) TT=1800.11	? (TO) TT=1800.03	? (TO) TT=1800.03
csp/allsquares/ normalized-squares-37-37.xml		? (TO) TT=1800.1	? (TO) TT=1800.06	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.06	? (TO) TT=1800.09
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquares/ normalized-squares-38-38.xml		? (TO) TT=1800.06	? (TO) TT=1800.11	? (TO) TT=1800.12	? (TO) TT=1800.11	? (TO) TT=1800.08	? (TO) TT=1800.13
csp/allsquares/ normalized-squares-4-4.xml	SAT TT=0.008	SAT TT=0.035	SAT TT=0.764	SAT TT=0.78	SAT TT=0.008	SAT TT=0.734	SAT TT=0.722
csp/allsquares/ normalized-squares-5-5.xml	SAT TT=0.027	SAT TT=0.039	SAT TT=0.859	SAT TT=0.961	SAT TT=0.027	SAT TT=0.804	SAT TT=0.924
csp/allsquares/ normalized-squares-6-6.xml	SAT TT=0.014	SAT TT=0.05	SAT TT=1.005	SAT TT=1.039	SAT TT=0.014	SAT TT=0.898	SAT TT=0.967
csp/allsquares/ normalized-squares-7-7.xml	SAT TT=0.015	SAT TT=0.054	SAT TT=1.149	SAT TT=1.149	SAT TT=0.015	SAT TT=0.964	SAT TT=0.961
csp/allsquares/ normalized-squares-8-8.xml	SAT TT=0.018	SAT TT=0.064	SAT TT=1.363	SAT TT=1.32	SAT TT=0.018	SAT TT=1.035	SAT TT=1.071
csp/allsquares/ normalized-squares-9-9.xml	SAT TT=0.023	SAT TT=0.088	SAT TT=1.594	SAT TT=1.685	SAT TT=0.023	SAT TT=1.086	SAT TT=1.131
csp/allsquaresUnsat/ normalized-squaresUnsat-10-10.xml	UNSAT TT=0.025	UNSAT TT=108.482	UNSAT TT=1.197	UNSAT TT=1.184	UNSAT TT=0.025	UNSAT TT=1.091	UNSAT TT=1.198
csp/allsquaresUnsat/ normalized-squaresUnsat-11-11.xml	UNSAT TT=0.036	UNSAT TT=1349.74	UNSAT TT=1.284	UNSAT TT=1.469	UNSAT TT=0.036	UNSAT TT=1.239	UNSAT TT=1.32
csp/allsquaresUnsat/ normalized-squaresUnsat-12-12.xml	UNSAT TT=0.051	UNSAT TT=1340.11	UNSAT TT=1.509	UNSAT TT=1.499	UNSAT TT=0.051	UNSAT TT=1.523	UNSAT TT=1.67
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-13-13.xml	UNSAT TT=0.046	UNSAT TT=1344.77	UNSAT TT=1.779	UNSAT TT=1.627	UNSAT TT=0.046	UNSAT TT=1.598	UNSAT TT=1.757
csp/allsquaresUnsat/ normalized-squaresUnsat-14-14.xml	UNSAT TT=0.052	UNSAT TT=1345.11	UNSAT TT=1.693	UNSAT TT=1.849	UNSAT TT=0.052	UNSAT TT=1.539	UNSAT TT=1.999
csp/allsquaresUnsat/ normalized-squaresUnsat-15-15.xml	UNSAT TT=0.059	UNSAT TT=1349.71	UNSAT TT=1.139	UNSAT TT=1.126	UNSAT TT=0.059	UNSAT TT=1.655	UNSAT TT=2.075
csp/allsquaresUnsat/ normalized-squaresUnsat-16-16.xml	UNSAT TT=0.095	UNSAT TT=1339.37	UNSAT TT=1.235	UNSAT TT=1.252	UNSAT TT=0.095	UNSAT TT=1.755	UNSAT TT=2.404
csp/allsquaresUnsat/ normalized-squaresUnsat-17-17.xml	UNSAT TT=1.256	UNSAT TT=1344.39	UNSAT TT=1.256	UNSAT TT=1.324	UNSAT TT=11.014	UNSAT TT=2.871	UNSAT TT=3.208
csp/allsquaresUnsat/ normalized-squaresUnsat-18-18.xml	UNSAT TT=10.247	UNSAT TT=1612.84	? (TO) TT=1800.09	? (TO) TT=1800.09	UNSAT TT=544.005	UNSAT TT=10.247	UNSAT TT=10.817
csp/allsquaresUnsat/ normalized-squaresUnsat-19-19.xml	UNSAT TT=1.344	UNSAT TT=1531.02	UNSAT TT=1.344	UNSAT TT=1.542	UNSAT TT=232.555	UNSAT TT=10.121	UNSAT TT=8.34
csp/allsquaresUnsat/ normalized-squaresUnsat-20-20.xml	UNSAT TT=1.456	? (TO) TT=1800.07	UNSAT TT=1.456	UNSAT TT=1.497	UNSAT TT=427.064	UNSAT TT=26.507	UNSAT TT=24.364
csp/allsquaresUnsat/ normalized-squaresUnsat-21-21.xml	UNSAT TT=1.433	? (TO) TT=1800.11	UNSAT TT=1.433	UNSAT TT=1.486	? (TO) TT=1800.04	UNSAT TT=59.366	UNSAT TT=61.101
csp/allsquaresUnsat/ normalized-squaresUnsat-22-22.xml	UNSAT TT=1.527	? (TO) TT=1800.02	UNSAT TT=1.527	UNSAT TT=1.726	? (TO) TT=1800.05	UNSAT TT=97.286	UNSAT TT=79.838
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-2-2.xml	UNSAT TT=0.007	UNSAT TT=0.028	UNSAT TT=0.586	UNSAT TT=0.487	UNSAT TT=0.007	UNSAT TT=0.38	UNSAT TT=0.374
csp/allsquaresUnsat/ normalized-squaresUnsat-23-23.xml	UNSAT TT=1.592	? (TO) TT=1800.04	UNSAT TT=1.592	UNSAT TT=1.729	? (TO) TT=1800.09	UNSAT TT=96.291	UNSAT TT=130.302
csp/allsquaresUnsat/ normalized-squaresUnsat-24-24.xml		? (TO) TT=1800.03	? (TO) TT=1800.16	? (TO) TT=1800.1	? (TO) TT=1800.09	? (TO) TT=1800.01	? (TO) TT=1800.04
csp/allsquaresUnsat/ normalized-squaresUnsat-25-25.xml	UNSAT TT=1.736	? (TO) TT=1800.08	UNSAT TT=1.736	UNSAT TT=1.805	? (TO) TT=1800.01	? (TO) TT=1800.06	? (TO) TT=1800.03
csp/allsquaresUnsat/ normalized-squaresUnsat-26-26.xml		? (TO) TT=1800.07	? (TO) TT=1800.06	? (TO) TT=1800.1	? (TO) TT=1800.11	? (TO) TT=1800.05	? (TO) TT=1800.08
csp/allsquaresUnsat/ normalized-squaresUnsat-27-27.xml	UNSAT TT=1.969	? (TO) TT=1800.05	UNSAT TT=1.969	UNSAT TT=2.044	? (TO) TT=1800.07	? (TO) TT=1800.07	? (TO) TT=1800.07
csp/allsquaresUnsat/ normalized-squaresUnsat-28-28.xml		? (TO) TT=1800.07	? (TO) TT=1800.08	? (TO) TT=1800.16	? (TO) TT=1800.06	? (TO) TT=1800.01	? (TO) TT=1800.01
csp/allsquaresUnsat/ normalized-squaresUnsat-29-29.xml	UNSAT TT=2.054	? (TO) TT=1800.07	UNSAT TT=2.054	UNSAT TT=2.627	? (TO) TT=1800.05	? (TO) TT=1800.04	? (TO) TT=1800.02
csp/allsquaresUnsat/ normalized-squaresUnsat-30-30.xml	UNSAT TT=2.145	? (TO) TT=1800.09	UNSAT TT=2.145	UNSAT TT=2.324	? (TO) TT=1800.11	? (TO) TT=1800.04	? (TO) TT=1800.01
csp/allsquaresUnsat/ normalized-squaresUnsat-31-31.xml	UNSAT TT=2.002	? (TO) TT=1800.07	UNSAT TT=2.155	UNSAT TT=2.002	? (TO) TT=1800.04	? (TO) TT=1800.07	? (TO) TT=1800.05
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-32-32.xml		? (TO) TT=1800.07	? (TO) TT=1800.17	? (TO) TT=1800.09	? (TO) TT=1800.02	? (TO) TT=1800.02	? (TO) TT=1800
csp/allsquaresUnsat/ normalized-squaresUnsat-33-33.xml		? (TO) TT=1800.06	? (TO) TT=1800.16	? (TO) TT=1800.12	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800.09
csp/allsquaresUnsat/ normalized-squaresUnsat-3-3.xml	UNSAT TT=0.007	UNSAT TT=0.03	UNSAT TT=0.505	UNSAT TT=0.552	UNSAT TT=0.007	UNSAT TT=0.419	UNSAT TT=0.428
csp/allsquaresUnsat/ normalized-squaresUnsat-34-34.xml		? (TO) TT=1800.09	? (TO) TT=1800.13	? (TO) TT=1800.1	? (TO) TT=1800.05	? (TO) TT=1800.09	? (TO) TT=1800.02
csp/allsquaresUnsat/ normalized-squaresUnsat-35-35.xml	UNSAT TT=2.864	? (TO) TT=1800.03	UNSAT TT=2.864	UNSAT TT=3.468	? (TO) TT=1800.09	? (TO) TT=1800.11	? (TO) TT=1800.08
csp/allsquaresUnsat/ normalized-squaresUnsat-36-36.xml	UNSAT TT=2.408	? (TO) TT=1800.12	UNSAT TT=2.408	UNSAT TT=2.486	? (TO) TT=1800.05	? (TO) TT=1800.01	? (TO) TT=1800.06
csp/allsquaresUnsat/ normalized-squaresUnsat-37-37.xml	UNSAT TT=2.933	? (TO) TT=1800.04	UNSAT TT=3.943	UNSAT TT=2.933	? (TO) TT=1800.11	? (TO) TT=1800.1	? (TO) TT=1800.06
csp/allsquaresUnsat/ normalized-squaresUnsat-38-38.xml		? (TO) TT=1800.07	? (TO) TT=1800.11	? (TO) TT=1800.04	? (TO) TT=1800.03	? (TO) TT=1800.08	? (TO) TT=1800.05
csp/allsquaresUnsat/ normalized-squaresUnsat-4-4.xml	UNSAT TT=0.02	UNSAT TT=0.034	UNSAT TT=0.599	UNSAT TT=0.548	UNSAT TT=0.02	UNSAT TT=0.563	UNSAT TT=0.565
csp/allsquaresUnsat/ normalized-squaresUnsat-5-5.xml	UNSAT TT=0.01	UNSAT TT=0.038	UNSAT TT=0.585	UNSAT TT=0.603	UNSAT TT=0.01	UNSAT TT=0.626	UNSAT TT=0.6
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/allsquaresUnsat/ normalized-squaresUnsat-6-6.xml	UNSAT TT=0.012	UNSAT TT=0.044	UNSAT TT=0.646	UNSAT TT=0.635	UNSAT TT=0.012	UNSAT TT=0.654	UNSAT TT=0.801
csp/allsquaresUnsat/ normalized-squaresUnsat-7-7.xml	UNSAT TT=0.013	UNSAT TT=0.054	UNSAT TT=0.673	UNSAT TT=0.749	UNSAT TT=0.013	UNSAT TT=0.742	UNSAT TT=0.685
csp/allsquaresUnsat/ normalized-squaresUnsat-8-8.xml	UNSAT TT=0.016	UNSAT TT=0.058	UNSAT TT=0.799	UNSAT TT=0.779	UNSAT TT=0.016	UNSAT TT=0.758	UNSAT TT=0.759
csp/allsquaresUnsat/ normalized-squaresUnsat-9-9.xml	UNSAT TT=0.023	UNSAT TT=1.406	UNSAT TT=1.049	UNSAT TT=1.087	UNSAT TT=0.023	UNSAT TT=1.064	UNSAT TT=1.094
csp/cjss/ normalized-ft06x2-54.xml	UNSAT TT=2.839	? (TO) TT=1800.08	? (TO) TT=1800.08	? (TO) TT=1800.07	UNSAT TT=216.779	UNSAT TT=2.839	UNSAT TT=3.127
csp/cjss/ normalized-ft06x2-55.xml	SAT TT=1.656	? (TO) TT=1800.09	SAT TT=31.377	SAT TT=17.183	SAT TT=1.656	SAT TT=2.096	SAT TT=2.325
csp/cjss/ normalized-ft06x3-54.xml	UNSAT TT=74.417	? (TO) TT=1800.08	? (TO) TT=1800.09	? (TO) TT=1800.16	? (TO) TT=1800.02	UNSAT TT=200.583	UNSAT TT=74.417
csp/cjss/ normalized-ft06x3-55.xml	SAT TT=3.564	? (TO) TT=1800.08	? (TO) TT=1800.05	? (TO) TT=1800.15	? (TO) TT=1800.1	SAT TT=3.564	SAT TT=3.837
csp/cjss/ normalized-ft10x2-837.xml		? (TO) TT=1800.08	? (TO) TT=1800.1	? (TO) TT=1800.12	? (TO) TT=1800.1	? (TO) TT=1800.1	? (TO) TT=1800.02
csp/cjss/ normalized-ft10x3-838.xml		? (TO) TT=1800.02	? (TO) TT=1800.16	? (TO) TT=1800.08	? (TO) TT=1800.09	? (TO) TT=1800.07	? (TO) TT=1800.09
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/cjss/ normalized-la03x2-593.xml		? (TO) TT=1800.08	? (TO) TT=1800.11	? (TO) TT=1800.14	? (TO) TT=1800.06	? (TO) TT=1800.06	? (TO) TT=1800.07
csp/cjss/ normalized-la03x3-590.xml		? (TO) TT=1800.09	? (TO) TT=1800.14	? (TO) TT=1800.11	? (TO) TT=1800.04	? (TO) TT=1800.01	? (TO) TT=1800.11
csp/cjss/ normalized-la04x2-572.xml		? (TO) TT=1800.08	? (TO) TT=1800.14	? (TO) TT=1800.12	? (TO) TT=1800.09	? (TO) TT=1800.03	? (TO) TT=1800.02
csp/cjss/ normalized-la04x3-570.xml		? (TO) TT=1800.09	? (TO) TT=1800.12	? (TO) TT=1800.11	? (TO) TT=1800.06	? (TO) TT=1800.02	? (TO) TT=1800.07
csp/rcpsp/ normalized-rcpsp20-02.xml	SAT TT=0.031	SAT TT=0.085	SAT TT=1.269	SAT TT=1.42	SAT TT=0.031	SAT TT=1.064	SAT TT=1.063
csp/rcpsp/ normalized-rcpsp20-03.xml	SAT TT=0.063	SAT TT=0.094	SAT TT=1.772	SAT TT=1.88	SAT TT=0.063	SAT TT=1.015	SAT TT=1.063
csp/rcpsp/ normalized-rcpsp20-04.xml	SAT TT=0.028	SAT TT=0.035	SAT TT=1.194	SAT TT=1.064	SAT TT=0.028	SAT TT=1.041	SAT TT=1.304
csp/rcpsp/ normalized-rcpsp20-05.xml	UNSAT TT=1.392	UNSAT TT=34.019	UNSAT TT=145.365	UNSAT TT=175.51	UNSAT TT=28.088	UNSAT TT=1.392	UNSAT TT=1.694
csp/rcpsp/ normalized-rcpsp20-06.xml	SAT TT=0.02	SAT TT=0.033	SAT TT=1.023	SAT TT=1.05	SAT TT=0.02	SAT TT=0.965	SAT TT=0.99
csp/rcpsp/ normalized-rcpsp20-07.xml	SAT TT=1.159	SAT TT=41.692	SAT TT=15.543	SAT TT=128.081	SAT TT=1.705	SAT TT=1.159	SAT TT=1.189
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpsp/ normalized-rcpsp20-08.xml	SAT TT=0.026	SAT TT=0.139	SAT TT=1.097	SAT TT=1.077	SAT TT=0.026	SAT TT=1.045	SAT TT=1.124
csp/rcpsp/ normalized-rcpsp20-09.xml	SAT TT=0.029	SAT TT=0.034	SAT TT=0.997	SAT TT=1.157	SAT TT=0.029	SAT TT=0.961	SAT TT=1.013
csp/rcpsp/ normalized-rcpsp20-10.xml	SAT TT=0.02	SAT TT=0.041	SAT TT=1.008	SAT TT=1.076	SAT TT=0.02	SAT TT=0.95	SAT TT=1.047
csp/rcpsp/ normalized-rcpsp20-11.xml	SAT TT=0.017	SAT TT=0.032	SAT TT=0.931	SAT TT=0.95	SAT TT=0.017	SAT TT=0.917	SAT TT=0.944
csp/rcpsp/ normalized-rcpsp20-12.xml	SAT TT=0.023	SAT TT=0.223	SAT TT=7.357	SAT TT=5.572	SAT TT=0.023	SAT TT=1.304	SAT TT=1.129
csp/rcpsp/ normalized-rcpsp20-13.xml	SAT TT=0.027	SAT TT=0.107	SAT TT=1.078	SAT TT=1.135	SAT TT=0.027	SAT TT=1.006	SAT TT=1.124
csp/rcpsp/ normalized-rcpsp20-14.xml	SAT TT=0.02	SAT TT=0.064	SAT TT=1.54	SAT TT=1.514	SAT TT=0.02	SAT TT=1.095	SAT TT=1.175
csp/rcpsp/ normalized-rcpsp20-15.xml	SAT TT=0.036	SAT TT=0.052	SAT TT=1.132	SAT TT=1.154	SAT TT=0.036	SAT TT=0.934	SAT TT=0.945
csp/rcpsp/ normalized-rcpsp20-16.xml	SAT TT=0.024	SAT TT=9.973	SAT TT=212.19	SAT TT=198.747	SAT TT=0.024	SAT TT=1.046	SAT TT=1.086
csp/rcpsp/ normalized-rcpsp20-17.xml	SAT TT=0.037	SAT TT=0.088	SAT TT=1.012	SAT TT=1.006	SAT TT=0.037	SAT TT=0.966	SAT TT=0.982
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpsp/ normalized-rcpsp20-18.xml	SAT TT=0.04	SAT TT=3.745	SAT TT=1.09	SAT TT=1.176	SAT TT=0.04	SAT TT=1.031	SAT TT=1.104
csp/rcpsp/ normalized-rcpsp20-19.xml	SAT TT=0.017	SAT TT=0.032	SAT TT=1.057	SAT TT=1.004	SAT TT=0.017	SAT TT=0.902	SAT TT=0.915
csp/rcpsp/ normalized-rcpsp20-20.xml	SAT TT=0.03	SAT TT=0.049	SAT TT=1.399	SAT TT=1.652	SAT TT=0.03	SAT TT=0.983	SAT TT=1.037
csp/rcpsp/ normalized-rcpsp25-21.xml	SAT TT=0.361	SAT TT=0.361	SAT TT=1772.74	SAT TT=33.378	SAT TT=0.83	SAT TT=1.87	SAT TT=2.165
csp/rcpsp/ normalized-rcpsp25-22.xml	SAT TT=0.04	SAT TT=33.401	SAT TT=15.346	SAT TT=120.977	SAT TT=0.04	SAT TT=1.455	SAT TT=1.647
csp/rcpsp/ normalized-rcpsp25-23.xml	SAT TT=0.671	SAT TT=280.086	SAT TT=46.701	SAT TT=433.152	SAT TT=0.671	SAT TT=1.292	SAT TT=1.441
csp/rcpsp/ normalized-rcpsp25-24.xml	SAT TT=0.04	SAT TT=0.04	SAT TT=1.238	SAT TT=1.382	SAT TT=0.041	SAT TT=1.243	SAT TT=1.291
csp/rcpsp/ normalized-rcpsp25-25.xml	SAT TT=0.047	SAT TT=8.886	SAT TT=2.049	SAT TT=3.22	SAT TT=0.047	SAT TT=1.28	SAT TT=1.367
csp/rcpsp/ normalized-rcpsp25-26.xml	SAT TT=0.037	SAT TT=0.037	SAT TT=19.3	SAT TT=23.725	SAT TT=0.062	SAT TT=1.508	SAT TT=1.64
csp/rcpsp/ normalized-rcpsp25-27.xml	SAT TT=0.04	SAT TT=0.042	SAT TT=1.53	SAT TT=1.608	SAT TT=0.04	SAT TT=1.496	SAT TT=1.64
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpsp/ normalized-rcpsp25-28.xml	SAT TT=0.044	SAT TT=1.104	SAT TT=2.859	SAT TT=3.437	SAT TT=0.044	SAT TT=1.19	SAT TT=1.368
csp/rcpsp/ normalized-rcpsp25-29.xml	SAT TT=0.073	SAT TT=0.073	SAT TT=1.957	SAT TT=2.109	SAT TT=0.111	SAT TT=1.39	SAT TT=1.4
csp/rcpsp/ normalized-rcpsp25-30.xml	SAT TT=0.035	SAT TT=0.039	SAT TT=1.503	SAT TT=1.495	SAT TT=0.035	SAT TT=1.306	SAT TT=1.246
csp/rcpsp/ normalized-rcpsp25-31.xml	SAT TT=0.109	SAT TT=0.109	SAT TT=468.083	SAT TT=1565.59	SAT TT=0.136	SAT TT=1.427	SAT TT=1.274
csp/rcpsp/ normalized-rcpsp25-32.xml	SAT TT=1.726	SAT TT=112.82	SAT TT=1067.64	? (TO) TT=1800.11	SAT TT=1.819	SAT TT=1.726	SAT TT=1.735
csp/rcpsp/ normalized-rcpsp25-33.xml	SAT TT=0.041	SAT TT=30.838	SAT TT=41.528	SAT TT=41.645	SAT TT=0.041	SAT TT=1.141	SAT TT=1.207
csp/rcpsp/ normalized-rcpsp25-34.xml	SAT TT=0.047	SAT TT=2.727	SAT TT=1.582	SAT TT=1.699	SAT TT=0.047	SAT TT=1.54	SAT TT=1.98
csp/rcpsp/ normalized-rcpsp25-35.xml	SAT TT=0.033	SAT TT=0.051	SAT TT=1.405	SAT TT=1.484	SAT TT=0.033	SAT TT=1.152	SAT TT=1.266
csp/rcpsp/ normalized-rcpsp25-36.xml	SAT TT=0.028	SAT TT=0.046	SAT TT=1.693	SAT TT=27.047	SAT TT=0.028	SAT TT=1.187	SAT TT=1.256
csp/rcpsp/ normalized-rcpsp25-37.xml	SAT TT=0.033	SAT TT=29.073	SAT TT=1.611	SAT TT=1.696	SAT TT=0.033	SAT TT=1.533	SAT TT=1.53
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpsp/ normalized-rcpsp25-38.xml	SAT TT=0.049	SAT TT=0.383	SAT TT=1.647	SAT TT=1.789	SAT TT=0.049	SAT TT=1.348	SAT TT=1.606
csp/rcpsp/ normalized-rcpsp25-39.xml	SAT TT=0.156	SAT TT=169.293	SAT TT=16.577	SAT TT=6.695	SAT TT=0.156	SAT TT=1.422	SAT TT=1.58
csp/rcpsp/ normalized-rcpsp25-40.xml	SAT TT=0.086	SAT TT=0.086	SAT TT=4.124	SAT TT=4.689	SAT TT=0.163	SAT TT=1.443	SAT TT=1.569
csp/rcpspTighter/ normalized-rcpsptight20-02.xml	UNSAT TT=0.546	UNSAT TT=3.558	UNSAT TT=0.635	UNSAT TT=0.546	UNSAT TT=1.311	UNSAT TT=1.023	UNSAT TT=1.303
csp/rcpspTighter/ normalized-rcpsptight20-03.xml	UNSAT TT=0.81	UNSAT TT=1.319	UNSAT TT=12.017	UNSAT TT=14.637	UNSAT TT=0.81	UNSAT TT=1.027	UNSAT TT=1.092
csp/rcpspTighter/ normalized-rcpsptight20-04.xml	SAT TT=0.033	SAT TT=0.033	SAT TT=1.187	SAT TT=1.139	SAT TT=0.045	SAT TT=1.008	SAT TT=1.07
csp/rcpspTighter/ normalized-rcpsptight20-05.xml	UNSAT TT=0.588	UNSAT TT=3.767	UNSAT TT=0.602	UNSAT TT=0.588	UNSAT TT=1.121	UNSAT TT=1.086	UNSAT TT=1.14
csp/rcpspTighter/ normalized-rcpsptight20-06.xml	UNSAT TT=0.094	UNSAT TT=0.142	UNSAT TT=0.595	UNSAT TT=0.605	UNSAT TT=0.094	UNSAT TT=0.93	UNSAT TT=0.966
csp/rcpspTighter/ normalized-rcpsptight20-07.xml	UNSAT TT=0.197	UNSAT TT=3.974	UNSAT TT=0.587	UNSAT TT=0.596	UNSAT TT=0.197	UNSAT TT=1.215	UNSAT TT=1.089
csp/rcpspTighter/ normalized-rcpsptight20-08.xml	UNSAT TT=0.579	UNSAT TT=46.87	UNSAT TT=0.579	UNSAT TT=0.622	UNSAT TT=3.583	UNSAT TT=1.074	UNSAT TT=1.144
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpspTighter/ normalized-rcpsptight20-09.xml	UNSAT TT=0.583	UNSAT TT=3.086	UNSAT TT=0.583	UNSAT TT=0.601	UNSAT TT=0.62	UNSAT TT=0.964	UNSAT TT=1.003
csp/rcpspTighter/ normalized-rcpsptight20-10.xml	UNSAT TT=0.18	UNSAT TT=5.66	UNSAT TT=0.596	UNSAT TT=0.556	UNSAT TT=0.18	UNSAT TT=0.951	UNSAT TT=1.193
csp/rcpspTighter/ normalized-rcpsptight20-11.xml	UNSAT TT=0.017	UNSAT TT=0.032	UNSAT TT=0.547	UNSAT TT=0.601	UNSAT TT=0.017	UNSAT TT=0.687	UNSAT TT=0.694
csp/rcpspTighter/ normalized-rcpsptight20-12.xml	UNSAT TT=0.904	UNSAT TT=269.922	UNSAT TT=1065.6	UNSAT TT=1072.33	UNSAT TT=0.904	UNSAT TT=1.042	UNSAT TT=1.122
csp/rcpspTighter/ normalized-rcpsptight20-13.xml	UNSAT TT=0.57	UNSAT TT=1.27	UNSAT TT=0.576	UNSAT TT=0.57	UNSAT TT=0.592	UNSAT TT=1.031	UNSAT TT=1.266
csp/rcpspTighter/ normalized-rcpsptight20-14.xml	UNSAT TT=0.606	UNSAT TT=45.037	UNSAT TT=0.606	UNSAT TT=0.63	UNSAT TT=12.524	UNSAT TT=1.468	UNSAT TT=1.544
csp/rcpspTighter/ normalized-rcpsptight20-15.xml	UNSAT TT=0.019	UNSAT TT=0.035	UNSAT TT=0.588	UNSAT TT=0.579	UNSAT TT=0.019	UNSAT TT=1.076	UNSAT TT=1.107
csp/rcpspTighter/ normalized-rcpsptight20-16.xml	UNSAT TT=0.606	? (TO) TT=1800.03	UNSAT TT=0.606	UNSAT TT=0.613	UNSAT TT=1.707	UNSAT TT=1.002	UNSAT TT=0.984
csp/rcpspTighter/ normalized-rcpsptight20-17.xml	UNSAT TT=0.569	UNSAT TT=1114.89	UNSAT TT=0.569	UNSAT TT=0.569	UNSAT TT=4.219	UNSAT TT=1.124	UNSAT TT=1.368
csp/rcpspTighter/ normalized-rcpsptight20-18.xml	UNSAT TT=0.311	UNSAT TT=1.207	UNSAT TT=0.572	UNSAT TT=0.619	UNSAT TT=0.311	UNSAT TT=0.954	UNSAT TT=1.073
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpspTighter/ normalized-rcpsptight20-19.xml	UNSAT TT=0.121	UNSAT TT=0.391	UNSAT TT=0.624	UNSAT TT=0.614	UNSAT TT=0.121	UNSAT TT=0.91	UNSAT TT=0.941
csp/rcpspTighter/ normalized-rcpsptight20-20.xml	UNSAT TT=0.225	UNSAT TT=0.573	UNSAT TT=0.564	UNSAT TT=0.591	UNSAT TT=0.225	UNSAT TT=0.973	UNSAT TT=1.009
csp/rcpspTighter/ normalized-rcpsptight25-21.xml	UNSAT TT=1.913	UNSAT TT=141.72	UNSAT TT=549.467	UNSAT TT=814.747	UNSAT TT=8.289	UNSAT TT=1.913	UNSAT TT=2.072
csp/rcpspTighter/ normalized-rcpsptight25-22.xml	UNSAT TT=1.449	UNSAT TT=123.516	UNSAT TT=1626.56	? (TO) TT=1800.09	UNSAT TT=1.963	UNSAT TT=1.449	UNSAT TT=1.522
csp/rcpspTighter/ normalized-rcpsptight25-23.xml	UNSAT TT=0.58	UNSAT TT=11.46	UNSAT TT=0.654	UNSAT TT=0.58	UNSAT TT=3.358	UNSAT TT=1.458	UNSAT TT=1.512
csp/rcpspTighter/ normalized-rcpsptight25-24.xml	UNSAT TT=0.042	UNSAT TT=0.133	UNSAT TT=1.095	UNSAT TT=1.043	UNSAT TT=0.042	UNSAT TT=1.326	UNSAT TT=1.254
csp/rcpspTighter/ normalized-rcpsptight25-25.xml	UNSAT TT=0.163	UNSAT TT=5.432	UNSAT TT=28.603	UNSAT TT=32.434	UNSAT TT=0.163	UNSAT TT=1.231	UNSAT TT=1.324
csp/rcpspTighter/ normalized-rcpsptight25-26.xml	UNSAT TT=0.291	UNSAT TT=1.675	UNSAT TT=6.102	UNSAT TT=6.488	UNSAT TT=0.291	UNSAT TT=1.486	UNSAT TT=1.594
csp/rcpspTighter/ normalized-rcpsptight25-27.xml	SAT TT=0.038	SAT TT=0.045	SAT TT=1.475	SAT TT=1.586	SAT TT=0.038	SAT TT=1.451	SAT TT=1.561
csp/rcpspTighter/ normalized-rcpsptight25-28.xml	UNSAT TT=0.262	UNSAT TT=4.602	UNSAT TT=9.034	UNSAT TT=11.502	UNSAT TT=0.262	UNSAT TT=1.324	UNSAT TT=1.211
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpspTighter/ normalized-rcpsptight25-29.xml	UNSAT TT=0.229	UNSAT TT=7.425	UNSAT TT=1.227	UNSAT TT=1.144	UNSAT TT=0.229	UNSAT TT=1.36	UNSAT TT=1.431
csp/rcpspTighter/ normalized-rcpsptight25-30.xml	UNSAT TT=0.049	UNSAT TT=10.296	UNSAT TT=0.682	UNSAT TT=0.736	UNSAT TT=0.049	UNSAT TT=1.044	UNSAT TT=1.221
csp/rcpspTighter/ normalized-rcpsptight25-31.xml	UNSAT TT=0.162	UNSAT TT=110.483	UNSAT TT=2.593	UNSAT TT=2.714	UNSAT TT=0.162	UNSAT TT=1.365	UNSAT TT=1.263
csp/rcpspTighter/ normalized-rcpsptight25-32.xml	UNSAT TT=2.789	? (TO) TT=1800.07	? (TO) TT=1800.06	? (TO) TT=1800.04	UNSAT TT=112.921	UNSAT TT=2.789	UNSAT TT=2.791
csp/rcpspTighter/ normalized-rcpsptight25-33.xml	UNSAT TT=0.059	UNSAT TT=648.057	UNSAT TT=293.974	UNSAT TT=270.247	UNSAT TT=0.059	UNSAT TT=1.053	UNSAT TT=1.185
csp/rcpspTighter/ normalized-rcpsptight25-34.xml	UNSAT TT=0.098	UNSAT TT=0.725	UNSAT TT=1.135	UNSAT TT=1.139	UNSAT TT=0.098	UNSAT TT=1.545	UNSAT TT=1.446
csp/rcpspTighter/ normalized-rcpsptight25-35.xml	UNSAT TT=0.084	UNSAT TT=0.148	UNSAT TT=1.364	UNSAT TT=1.428	UNSAT TT=0.084	UNSAT TT=1.169	UNSAT TT=1.218
csp/rcpspTighter/ normalized-rcpsptight25-36.xml	UNSAT TT=0.042	UNSAT TT=0.366	UNSAT TT=1.275	UNSAT TT=1.368	UNSAT TT=0.042	UNSAT TT=1.129	UNSAT TT=1.106
csp/rcpspTighter/ normalized-rcpsptight25-37.xml	UNSAT TT=0.054	UNSAT TT=0.619	UNSAT TT=0.798	UNSAT TT=0.833	UNSAT TT=0.054	UNSAT TT=1.379	UNSAT TT=1.518
csp/rcpspTighter/ normalized-rcpsptight25-38.xml	UNSAT TT=1.361	UNSAT TT=218.451	UNSAT TT=654.673	UNSAT TT=1673.9	UNSAT TT=1.611	UNSAT TT=1.361	UNSAT TT=1.485
Bench name	Best results	bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
csp/rcpspTighter/ normalized-rcpsptight25-39.xml	UNSAT TT=0.583	? (TO) TT=1800.04	UNSAT TT=0.668	UNSAT TT=0.583	UNSAT TT=2.174	UNSAT TT=1.358	UNSAT TT=1.359
csp/rcpspTighter/ normalized-rcpsptight25-40.xml	UNSAT TT=1.51	UNSAT TT=148.117	UNSAT TT=769.368	UNSAT TT=724.165	UNSAT TT=6.399	UNSAT TT=1.51	UNSAT TT=1.597

Statistics		bpsolver 09 (complete)	Choco2.1.1 2009-06-10 (complete)	Choco2.1.1b 2009-07-16 (complete)	Mistral 1.545 (complete)	Sugar v1.14.6+minisat (complete)	Sugar v1.14.6+picosat (complete)
Number of times the solver is able to give the best known answer		111	125	122	120	136	135
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		7	17	8	84	23	6