2019 XCSP3 competition: mini-solver track (sequential and parallel solvers): results by benchmark

Results by benchmark for category decision problem (CSP)

This page displays the results of the different solvers for each benchmark for category decision problem (CSP)

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance (complete solvers point of view)
text	the solver performed better than the other ones on that instance (incomplete solvers point of view)
text	the solver was ended by a signal or other problem
text	the solver gave an incomplete answer
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
cosoco 2 (complete)
cosoco 2.0 (complete)
cosoco 2.0 (complete)
miniBTD 19.06.16 (complete)
NACRE 1.0.5 (complete)
NACRE 1.0.5-Hybrid (complete)
(reference) PicatSAT 2019-09-12 (complete)

Bench name	Best results on this instance	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
AllInterval/ AllInterval-m1-s1/ AllInterval-040.xml	SAT TT=0.048	SAT TT=0.348	SAT TT=0.347	SAT TT=0.349	SAT TT=0.048	SAT TT=0.073	SAT TT=0.07	SAT TT=291.992
AllInterval/ AllInterval-m1-s1/ AllInterval-075.xml	SAT TT=0.179	? (TO) TT=2400.01	? (TO) TT=2400.04	? (TO) TT=2400.01	SAT TT=0.501	SAT TT=0.181	SAT TT=0.179	? (TO) TT=2400.08
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07-2_X2.xml	UNSAT TT=524.722	? (TO) TT=2400.01	? (TO) TT=2400.01	? (TO) TT=2400.06	? TT=2400.03	? (TO) TT=2400.06	? (TO) TT=2400.01	UNSAT TT=524.722
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07-3_X2.xml	UNSAT TT=523.635	? (TO) TT=2400.04	? (TO) TT=2400.01	? (TO) TT=2400.01	? TT=2400.03	? (TO) TT=2400.06	? (TO) TT=2400.06	UNSAT TT=523.635
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07-4_X2.xml	UNSAT TT=565.312	? (TO) TT=2400.08	? (TO) TT=2400.03	? (TO) TT=2400.03	? TT=2400.03	? (TO) TT=2400.08	? (TO) TT=2400.09	UNSAT TT=565.312
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07-7_X2.xml	UNSAT TT=515.662	? (TO) TT=2400.09	? (TO) TT=2400.02	? (TO) TT=2400.01	? TT=2400.03	? (TO) TT=2400.08	? (TO) TT=2400.03	UNSAT TT=515.662
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07-8_X2.xml	UNSAT TT=654.324	? (TO) TT=2400.06	? (TO) TT=2400.02	? (TO) TT=2400.05	? TT=2400.03	? (TO) TT=2400.07	? (TO) TT=2400.01	UNSAT TT=654.324
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07-9_X2.xml	UNSAT TT=573.798	? (TO) TT=2400.02	? (TO) TT=2400.01	? (TO) TT=2400.01	? (TO) TT=2400.01	? (TO) TT=2400.08	? (TO) TT=2400.07	UNSAT TT=573.798
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07h-0_X2.xml	UNSAT TT=704.404	? (TO) TT=2400.08	? (TO) TT=2400.06	? (TO) TT=2400.1	? TT=2400.03	? (TO) TT=2400.03	? (TO) TT=2400.03	UNSAT TT=704.404
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07h-3_X2.xml	UNSAT TT=404.557	? (TO) TT=2400.08	? (TO) TT=2400.04	? (TO) TT=2400.04	? (TO) TT=2400.04	? (TO) TT=2400.05	? (TO) TT=2400.04	UNSAT TT=404.557
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07h-6_X2.xml	UNSAT TT=538.997	? (TO) TT=2400.11	? (TO) TT=2400.04	? (TO) TT=2400.09	? (TO) TT=2400.03	? (TO) TT=2400.02	? (TO) TT=2400.03	UNSAT TT=538.997
Blackhole/ Blackhole-xcsp2-s07/ Blackhole-4-07h-8_X2.xml	UNSAT TT=537.808	? (TO) TT=2400.03	? (TO) TT=2400.08	? (TO) TT=2400.06	? TT=2400.03	? (TO) TT=2400.04	? (TO) TT=2400.02	UNSAT TT=537.808
ColouredQueens/ ColouredQueens-m1-s1/ ColouredQueens-08.xml	UNSAT TT=8.526	? (TO) TT=2400.07	? (TO) TT=2400.08	? (TO) TT=2400.05	? TT=2400	? (TO) TT=2400.02	? (TO) TT=2400.1	UNSAT TT=8.526
ColouredQueens/ ColouredQueens-m1-s1/ ColouredQueens-13.xml		? (TO) TT=2400.02	? (TO) TT=2400.04	? (TO) TT=2400.02	? TT=2400	? (TO) TT=2400.05	? (TO) TT=2400.02	? (TO) TT=2400.01
ColouredQueens/ ColouredQueens-m1-s1/ ColouredQueens-15.xml		? (TO) TT=2400.07	? (TO) TT=2400.09	? (TO) TT=2400.01	? TT=2400.01	? (TO) TT=2400.05	? (TO) TT=2400.09	? (TO) TT=2400.05
ColouredQueens/ ColouredQueens-m1-s1/ ColouredQueens-16.xml		? (TO) TT=2400.01	? (TO) TT=2400.08	? (TO) TT=2400.07	? TT=2400	? (TO) TT=2400.05	? (TO) TT=2400.11	? (TO) TT=2400.03
CoveringArray/ CoveringArray-elt-s1/ CoveringArray-elt-3-08-2-11.xml		? (TO) TT=2400.03	? (TO) TT=2400.02	? (TO) TT=2400.05	? (TO) TT=2400.05	? (TO) TT=2400.1	? (TO) TT=2400.1	? (TO) TT=2400.02
CoveringArray/ CoveringArray-elt-s1/ CoveringArray-elt-3-09-2-12.xml	SAT TT=805.886	SAT TT=999.958	SAT TT=1004.68	SAT TT=1008.33	? (TO) TT=2400.06	? (TO) TT=2400.07	? (TO) TT=2400.08	SAT TT=805.886
CoveringArray/ CoveringArray-elt-s1/ CoveringArray-elt-3-10-2-11.xml		? (TO) TT=2400.05	? (TO) TT=2400.02	? (TO) TT=2400.05	? (TO) TT=2400.1	? (TO) TT=2400.06	? (TO) TT=2400.01	? (TO) TT=2400.07
CoveringArray/ CoveringArray-elt-s1/ CoveringArray-elt-3-11-2-11.xml		? (TO) TT=2400.06	? (TO) TT=2400.1	? (TO) TT=2400.05	? (TO) TT=2400.02	? (TO) TT=2400.06	? (TO) TT=2400.09	? (TO) TT=2400.09
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Crossword/ Crossword-m1c-lex-herald/ Crossword-m1c-lex-h1502.xml	SAT TT=0.318	SAT TT=0.653	SAT TT=0.653	SAT TT=0.74	SAT TT=0.318	SAT TT=1.115	SAT TT=1.807	SAT TT=16.942
Crossword/ Crossword-m1c-lex-herald/ Crossword-m1c-lex-h1503.xml	SAT TT=0.099	SAT TT=0.129	SAT TT=0.128	SAT TT=0.148	SAT TT=0.099	SAT TT=0.285	SAT TT=0.316	SAT TT=24.376
Crossword/ Crossword-m1c-lex-herald/ Crossword-m1c-lex-h1901.xml	SAT TT=0.134	SAT TT=0.208	SAT TT=0.209	SAT TT=0.237	SAT TT=0.134	SAT TT=0.318	SAT TT=0.324	SAT TT=28.364
Crossword/ Crossword-m1c-lex-herald/ Crossword-m1c-lex-h1906.xml	SAT TT=0.107	SAT TT=0.205	SAT TT=0.205	SAT TT=0.235	SAT TT=0.107	SAT TT=0.27	SAT TT=0.262	SAT TT=15.467
Crossword/ Crossword-m1c-lex-herald/ Crossword-m1c-lex-h2102.xml	SAT TT=0.243	SAT TT=0.249	SAT TT=0.251	SAT TT=0.285	SAT TT=0.243	SAT TT=1.275	SAT TT=0.413	SAT TT=93.951
Crossword/ Crossword-m1c-lex-herald/ Crossword-m1c-lex-h2304.xml	SAT TT=5.616	SAT TT=33.277	SAT TT=33.566	SAT TT=37.782	? (TO) TT=2400.04	SAT TT=108.105	SAT TT=5.616	? (TO) TT=2400.04
Crossword/ Crossword-m1c-lex-herald/ Crossword-m1c-lex-h2307.xml	SAT TT=0.984	SAT TT=0.984	SAT TT=0.996	SAT TT=1.125	SAT TT=2.812	SAT TT=1.878	SAT TT=1.002	SAT TT=809.533
Crossword/ Crossword-m1c-lex-vg/ Crossword-m1c-lex-vg-4-8.xml	SAT TT=0.314	SAT TT=8.612	SAT TT=8.735	SAT TT=9.884	SAT TT=0.314	SAT TT=27.921	SAT TT=25.918	SAT TT=66.27
Crossword/ Crossword-m1c-lex-vg/ Crossword-m1c-lex-vg-5-6.xml	SAT TT=0.477	SAT TT=0.477	SAT TT=0.477	SAT TT=0.545	SAT TT=3.475	SAT TT=11.849	SAT TT=1.135	SAT TT=16.094
Crossword/Crossword-m1c-lex-vg/ Crossword-m1c-lex-vg-7-10.xml	UNSAT TT=3.397	UNSAT TT=9.331	UNSAT TT=9.355	UNSAT TT=10.691	UNSAT TT=3.397	UNSAT TT=19.408	UNSAT TT=4.422	UNSAT TT=43.044
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Crossword/ Crossword-m1c-ogd2008-herald/ Crossword-m1c-ogd2008-h1905.xml	SAT TT=0.567	SAT TT=1.396	SAT TT=1.601	SAT TT=1.401	SAT TT=0.567	SAT TT=1.688	SAT TT=1.689	SAT TT=161.526
Crossword/ Crossword-m1c-ogd2008-herald/ Crossword-m1c-ogd2008-h1907.xml	SAT TT=0.904	SAT TT=3.112	SAT TT=3.547	SAT TT=3.125	SAT TT=0.904	SAT TT=2.193	SAT TT=2.044	SAT TT=530.579
Crossword/ Crossword-m1c-ogd2008-puzzle/ Crossword-m1c-ogd2008-p16.xml	SAT TT=0.263	SAT TT=0.849	SAT TT=0.97	SAT TT=0.851	SAT TT=0.263	SAT TT=0.681	SAT TT=0.672	SAT TT=18.2
Crossword/ Crossword-m1c-ogd2008-puzzle/ Crossword-m1c-ogd2008-p19.xml	SAT TT=0.732	SAT TT=2.58	SAT TT=2.977	SAT TT=2.595	SAT TT=0.732	SAT TT=1.704	SAT TT=1.685	SAT TT=267.234
Crossword/Crossword-m1c-ogd2008-vg/ Crossword-m1c-ogd2008-vg-7-7.xml	SAT TT=3.18	SAT TT=5.404	SAT TT=6.14	SAT TT=5.512	SAT TT=3.18	SAT TT=12.974	SAT TT=66.101	SAT TT=144.224
Crossword/ Crossword-m1c-ogd-puzzle/ Crossword-m1c-ogd-p14.xml	SAT TT=0.097	SAT TT=0.315	SAT TT=0.317	SAT TT=0.316	SAT TT=0.097	SAT TT=0.242	SAT TT=0.241	SAT TT=6.334
Crossword/Crossword-m1c-ogd-vg/ Crossword-m1c-ogd-vg-14-15.xml	UNSAT TT=4.037	UNSAT TT=18.138	UNSAT TT=17.899	UNSAT TT=18.245	UNSAT TT=4.037	UNSAT TT=167.224	UNSAT TT=44.451	UNSAT TT=1251.18
Crossword/Crossword-m1c-ogd-vg/ Crossword-m1c-ogd-vg-14-18.xml	UNSAT TT=0.382	UNSAT TT=0.816	UNSAT TT=0.814	UNSAT TT=0.922	UNSAT TT=0.382	UNSAT TT=1.442	UNSAT TT=1.45	UNSAT TT=363.298
Crossword/Crossword-m1c-ogd-vg/ Crossword-m1c-ogd-vg-15-15.xml	UNSAT TT=1.609	UNSAT TT=5.992	UNSAT TT=6.012	UNSAT TT=6.787	UNSAT TT=1.609	UNSAT TT=17.947	UNSAT TT=12.71	UNSAT TT=644.141
Crossword/Crossword-m1c-ogd-vg/ Crossword-m1c-ogd-vg-9-12.xml		? (TO) TT=2400.04	? (TO) TT=2400.08	? (TO) TT=2400.01	? (TO) TT=2400.01	? (TO) TT=2400.04	? (TO) TT=2400.04	? (TO) TT=2400.1
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Crossword/ Crossword-m1c-uk-herald/ Crossword-m1c-uk-h1506.xml	SAT TT=0.467	SAT TT=1.09	SAT TT=1.255	SAT TT=1.096	SAT TT=0.467	SAT TT=1.269	SAT TT=1.212	SAT TT=226.984
Crossword/ Crossword-m1c-uk-herald/ Crossword-m1c-uk-h1903.xml	SAT TT=0.529	SAT TT=1.157	SAT TT=1.334	SAT TT=1.164	SAT TT=0.529	SAT TT=1.637	SAT TT=1.661	SAT TT=214.864
Crossword/ Crossword-m1c-uk-herald/ Crossword-m1c-uk-h1908.xml	SAT TT=0.428	SAT TT=1.267	SAT TT=1.456	SAT TT=1.276	SAT TT=0.428	SAT TT=1.063	SAT TT=1.067	SAT TT=97.286
Crossword/ Crossword-m1c-uk-herald/ Crossword-m1c-uk-h2302.xml	SAT TT=0.68	SAT TT=1.923	SAT TT=2.219	SAT TT=1.932	SAT TT=0.68	SAT TT=1.787	SAT TT=1.799	SAT TT=190.26
Crossword/ Crossword-m1c-uk-herald/ Crossword-m1c-uk-h2304.xml	SAT TT=0.813	SAT TT=1.307	SAT TT=1.507	SAT TT=1.314	SAT TT=0.813	SAT TT=2.485	SAT TT=2.437	SAT TT=480.048
Crossword/ Crossword-m1c-uk-herald/ Crossword-m1c-uk-h2308.xml	SAT TT=1.061	SAT TT=1.287	SAT TT=1.478	SAT TT=1.294	SAT TT=1.061	SAT TT=2.389	SAT TT=2.381	SAT TT=260.024
Crossword/Crossword-m1c-uk-vg/ Crossword-m1c-uk-vg-14-14.xml	UNSAT TT=52.534	UNSAT TT=108.061	UNSAT TT=122.351	UNSAT TT=106.873	UNSAT TT=52.534	UNSAT TT=251.856	UNSAT TT=83.411	UNSAT TT=426.994
Crossword/Crossword-m1c-uk-vg/ Crossword-m1c-uk-vg-16-20.xml	UNSAT TT=0.071	UNSAT TT=0.093	UNSAT TT=0.093	UNSAT TT=0.093	UNSAT TT=0.071	UNSAT TT=0.132	UNSAT TT=0.138	UNSAT TT=16.344
Crossword/Crossword-m1c-uk-vg/ Crossword-m1c-uk-vg-5-7.xml	SAT TT=0.098	SAT TT=0.353	SAT TT=0.355	SAT TT=0.355	SAT TT=0.098	SAT TT=0.777	SAT TT=0.845	SAT TT=21.535
Crossword/Crossword-m1c-uk-vg/ Crossword-m1c-uk-vg-8-9.xml		? (TO) TT=2400.02	? (TO) TT=2400.04	? (TO) TT=2400.03	? (TO) TT=2400.03	? (TO) TT=2400.03	? (TO) TT=2400.02	? (TO) TT=2400.09
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h0509.xml	SAT TT=0.013	SAT TT=0.04	SAT TT=0.046	SAT TT=0.04	SAT TT=0.013	SAT TT=0.023	SAT TT=0.022	SAT TT=0.145
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h1502.xml	SAT TT=0.365	SAT TT=0.412	SAT TT=0.472	SAT TT=0.418	SAT TT=0.365	SAT TT=1.203	SAT TT=1.538	SAT TT=18.14
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h1510.xml	SAT TT=3.756	SAT TT=9.36	SAT TT=10.687	SAT TT=9.404	SAT TT=3.756	SAT TT=24.707	SAT TT=9.573	SAT TT=1022.31
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h1901.xml	SAT TT=0.158	SAT TT=0.319	SAT TT=0.365	SAT TT=0.324	SAT TT=0.158	SAT TT=0.418	SAT TT=0.423	SAT TT=32.266
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h1903.xml	SAT TT=0.688	SAT TT=3.762	SAT TT=4.301	SAT TT=3.778	SAT TT=0.688	SAT TT=9.716	SAT TT=5.523	SAT TT=307.325
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h1908.xml	SAT TT=0.157	SAT TT=0.316	SAT TT=0.363	SAT TT=0.317	SAT TT=0.157	SAT TT=0.347	SAT TT=0.362	SAT TT=29.316
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h2104.xml	SAT TT=1.421	SAT TT=1.438	SAT TT=1.615	SAT TT=1.421	SAT TT=14.639	SAT TT=2.114	SAT TT=2.643	SAT TT=615.879
Crossword/ Crossword-m1c-words-herald/ Crossword-m1c-words-h2307.xml	SAT TT=0.571	SAT TT=0.632	SAT TT=0.725	SAT TT=0.64	SAT TT=0.571	SAT TT=1.905	SAT TT=1.184	SAT TT=311.945
Crossword/Crossword-m1c-words-vg/ Crossword-m1c-words-vg-10-10.xml	UNSAT TT=7.471	UNSAT TT=11.4	UNSAT TT=12.881	UNSAT TT=11.326	UNSAT TT=7.471	UNSAT TT=10.994	UNSAT TT=9.119	UNSAT TT=100.7
Crossword/Crossword-m1c-words-vg/ Crossword-m1c-words-vg-6-10.xml	UNSAT TT=24.632	UNSAT TT=126.471	UNSAT TT=143.554	UNSAT TT=127.073	UNSAT TT=24.632	UNSAT TT=251.911	UNSAT TT=45.166	UNSAT TT=264.028
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Dubois/ Dubois-m1-s1/ Dubois-028.xml	UNSAT TT=0.007	? (TO) TT=2400.01	? (TO) TT=2400.08	? (TO) TT=2400.03	UNSAT TT=0.007	? (TO) TT=2400.02	UNSAT TT=0.009	UNSAT TT=0.007
GracefulGraph/ GracefulGraph-K04-P04_c18.xml	SAT TT=2.511	SAT TT=2.511	SAT TT=2.515	SAT TT=2.527	SAT TT=175.743	? (TO) TT=2400.06	? (TO) TT=2400.07	SAT TT=82.523
GracefulGraph/ GracefulGraph-K05-P02_c18.xml	SAT TT=15.723	SAT TT=18.504	SAT TT=18.537	SAT TT=18.615	SAT TT=32.889	? (TO) TT=2400.05	? (TO) TT=2400.05	SAT TT=15.723
GracefulGraph/ GracefulGraph-K07-P03_c18.xml		? (TO) TT=2400.09	? (TO) TT=2400.05	? (TO) TT=2400.04	? TT=2400.02	? (TO) TT=2400.05	? (TO) TT=2400.06	? (TO) TT=2400.11
GracefulGraph/ GracefulGraph-m1-s1/ GracefulGraph-K05-P02.xml	SAT TT=15.5	SAT TT=18.688	SAT TT=18.555	SAT TT=18.647	SAT TT=33.213	? (TO) TT=2400.06	? (TO) TT=2400.08	SAT TT=15.5
GracefulGraph/ GracefulGraph-m1-s1/ GracefulGraph-K08-P03.xml		? (TO) TT=2400.07	? (TO) TT=2400.02	? (TO) TT=2400.03	? TT=2400.03	? (TO) TT=2400.05	? (TO) TT=2400.05	? (TO) TT=2400.1
GracefulGraph/ GracefulGraph-m1-s1/ GracefulGraph-K12-P08.xml		? (TO) TT=2400.08	? (TO) TT=2400.02	? (TO) TT=2400.04	? (TO) TT=2400.01	? (TO) TT=2400.02	? (TO) TT=2400.05	? (TO) TT=2400.1
GracefulGraph/ GracefulGraph-m1-s1/ GracefulGraph-K13-P04.xml		? (TO) TT=2400.1	? (TO) TT=2400.06	? (TO) TT=2400.1	? (TO) TT=2400.04	? (TO) TT=2400	? (TO) TT=2400.07	? (TO) TT=2400.08
GracefulGraph/ GracefulGraph-m1-s1/ GracefulGraph-K14-P05.xml		? (TO) TT=2400.02	? (TO) TT=2400.04	? (TO) TT=2400.01	? (TO) TT=2400.05	? (TO) TT=2400.01	? (TO) TT=2400.07	? (TO) TT=2400.05
Langford/ Langford-m1-k3/ Langford-3-45.xml	SAT TT=65.653	? (TO) TT=2400.03	? (TO) TT=2400.03	? (TO) TT=2400.11	? (TO) TT=2400.01	SAT TT=440.419	SAT TT=65.653	? (TO) TT=2400.09
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Langford/ Langford-m1-k4/ Langford-4-16.xml	UNSAT TT=84.703	UNSAT TT=1528.47	UNSAT TT=1714.87	UNSAT TT=1492.98	? TT=2400	UNSAT TT=1188.5	? (TO) TT=2400.03	UNSAT TT=84.703
Langford/ Langford-m2-s1/ LangfordBin-20.xml	SAT TT=0.008	SAT TT=0.03	SAT TT=0.029	SAT TT=0.029	? TT=2400.01	SAT TT=0.008	SAT TT=0.009	SAT TT=4.725
Langford/ Langford-m2-s1/ LangfordBin-24.xml	SAT TT=5.862	? (TO) TT=2400.07	? (TO) TT=2400.01	? (TO) TT=2400.01	? TT=2400.01	? (TO) TT=2400.06	? (TO) TT=2400.07	SAT TT=5.862
Langford/ Langford-m2-s1/ LangfordBin-70.xml		? (TO) TT=2400.03	? (TO) TT=2400.04	? (TO) TT=2400.02	? (TO) TT=2400.06	? (TO) TT=2400.05	? (TO) TT=2400.14	? (TO) TT=2400.06
LatinSquare/ LatinSquare-xcsp2-qcp20/ qcp-20-187-03_X2.xml	SAT TT=4.701	SAT TT=311.564	SAT TT=322.85	SAT TT=315.684	SAT TT=429.87	? (TO) TT=2400.03	SAT TT=4.701	SAT TT=5.626
LatinSquare/ LatinSquare-xcsp2-qcp25/ qcp-25-264-09_X2.xml	SAT TT=14.06	? (TO) TT=2400.03	? (TO) TT=2400.02	? (TO) TT=2400.01	? (TO) TT=2400.08	? (TO) TT=2400.04	SAT TT=191.236	SAT TT=14.06
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-04-0003.xml	SAT TT=0.556	SAT TT=1.546	SAT TT=1.546	SAT TT=1.544	SAT TT=3.167	SAT TT=2.039	SAT TT=0.556	SAT TT=119.281
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-05-0033.xml		? (TO) TT=2400.08	? (TO) TT=2400.04	? (TO) TT=2400.1	? TT=2400.01	? (TO) TT=2400.06	? (TO) TT=2400.1	? (TO) TT=2400.01
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-05-m012.xml	SAT TT=626.863	SAT TT=636.031	SAT TT=627.174	SAT TT=626.863	? TT=2400	? (TO) TT=2400.06	? (TO) TT=2400.1	? (TO) TT=2400.03
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-06-m111.xml		? (TO) TT=2400.11	? (TO) TT=2400.03	? (TO) TT=2400.04	? (TO) TT=2400.01	? (TO) TT=2400.04	? (TO) TT=2400.09	? (TO) TT=2400.08
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-07-0080.xml	UNSAT TT=1154.64	? (TO) TT=2400.01	? (TO) TT=2400.07	? (TO) TT=2400.07	? TT=2400.01	? (TO) TT=2400.02	? (TO) TT=2400.04	UNSAT TT=1154.64
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-07-m102.xml		? (TO) TT=2400.03	? (TO) TT=2400.06	? (TO) TT=2400.09	? TT=2400.01	? (TO) TT=2400.04	? (TO) TT=2400.06	? (TO) TT=2400.03
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-07-m167.xml		? (TO) TT=2400.05	? (TO) TT=2400.1	? (TO) TT=2400.05	? TT=2400.01	? (TO) TT=2400.05	? (TO) TT=2400.08	? (TO) TT=2400.03
MagicHexagon/ MagicHexagon-m1-s1/ MagicHexagon-11-m018.xml		? (TO) TT=2400.07	? (TO) TT=2400.01	? (TO) TT=2400.07	? TT=2400.03	? (TO) TT=2400.07	? (TO) TT=2400.03	? (TO) TT=2400.07
MagicSquare/ MagicSquare-sum-s1/ MagicSquare-27-sum.xml		? (TO) TT=2400.09	? (TO) TT=2400.07	? (TO) TT=2400.01	? (TO) TT=2400.06	? (TO) TT=2400.03	? (TO) TT=2400.07	? (TO) TT=2400.03
MarketSplit/ MarketSplit-m1-gp/ MarketSplit-08.xml	SAT TT=7.19	SAT TT=68.844	SAT TT=80.33	SAT TT=69.228	SAT TT=7.19	SAT TT=466.269	SAT TT=226.792	SAT TT=52.747
MaxCSP/MaxCSP-maxclique-s1/ MaxCSP-p-hat1000-2.xml	UNSAT TT=1.198	UNSAT TT=4.506	UNSAT TT=4.399	UNSAT TT=4.471	UNSAT TT=901.776	UNSAT TT=1.223	UNSAT TT=1.198	UNSAT TT=2.833
MaxCSP/ MaxCSP-maxclique-s1/MaxCSP-san-1000.xml	UNSAT TT=1.18	UNSAT TT=4.33	UNSAT TT=4.225	UNSAT TT=4.313	UNSAT TT=865.364	UNSAT TT=1.18	UNSAT TT=1.182	UNSAT TT=2.746
MaxCSP/ MaxCSP-random-denseLoose/rand-30-10-25-48-32.xml	UNSAT TT=0.011	UNSAT TT=0.02	UNSAT TT=0.023	UNSAT TT=0.022	UNSAT TT=0.022	UNSAT TT=0.012	UNSAT TT=0.011	UNSAT TT=0.189
Mixed/ Mixed-xcsp2-cril/ cril-2_X2.xml	UNSAT TT=0.037	UNSAT TT=0.104	UNSAT TT=0.104	UNSAT TT=0.104	UNSAT TT=198.029	UNSAT TT=0.038	UNSAT TT=0.037	UNSAT TT=0.142
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
MultiKnapsack/ MultiKnapsack-m1-gp/ MultiKnapsack-1-06.xml	SAT TT=18.947	SAT TT=30.038	SAT TT=29.928	SAT TT=34.39	SAT TT=18.947	SAT TT=1567.38	SAT TT=221.406	SAT TT=123.101
Nonogram/ Nonogram-table-s1/ Nonogram-019-table.xml	SAT TT=18.078	SAT TT=110.277	SAT TT=110.059	SAT TT=109.539	SAT TT=18.078	SAT TT=30.598	SAT TT=32.18	SAT TT=78.424
Nonogram/ Nonogram-table-s1/ Nonogram-122-table.xml	SAT TT=28.527	SAT TT=173.555	SAT TT=173.028	SAT TT=173.128	SAT TT=28.527	SAT TT=48.48	SAT TT=47.186	SAT TT=163.414
Nonogram/ Nonogram-table-s1/ Nonogram-151-table.xml	SAT TT=3.436	SAT TT=20.891	SAT TT=20.978	SAT TT=20.852	SAT TT=3.436	SAT TT=5.829	SAT TT=5.917	SAT TT=16.571
PigeonsPlus/ PigeonsPlus-m1-s1/ pigeonsPlus-06-10.xml	UNSAT TT=25.708	UNSAT TT=209.597	UNSAT TT=209.061	UNSAT TT=209.079	UNSAT TT=25.708	UNSAT TT=235.2	UNSAT TT=218.817	Signal TT=631.463
PigeonsPlus/ PigeonsPlus-m1-s1/ pigeonsPlus-10-07.xml	UNSAT TT=703.608	? (TO) TT=2400.03	? (TO) TT=2400.02	? (TO) TT=2400.11	UNSAT TT=703.608	? (TO) TT=2400.07	? (TO) TT=2400.12	? (TO) TT=2400.1
PigeonsPlus/ PigeonsPlus-m1-s1/ pigeonsPlus-11-07.xml		? (TO) TT=2400.07	? (TO) TT=2400.04	? (TO) TT=2400.04	? (TO) TT=2400.05	? (TO) TT=2400.22	? (TO) TT=2400.3	? (MO) TT=876.365
Primes/Primes-m1-p10/ Primes-10-40-3-5.xml	SAT TT=261.548	? (TO) TT=2400.02	? (TO) TT=2400.05	? (TO) TT=2400.09	SAT TT=261.548	? (TO) TT=2400.07	? (TO) TT=2400.06	SAT TT=2266.26
Primes/Primes-m1-p10/ Primes-10-60-3-5.xml	SAT TT=0.312	SAT TT=0.331	SAT TT=0.335	SAT TT=0.331	SAT TT=0.312	SAT TT=2.206	SAT TT=16.821	SAT TT=38.764
Primes/Primes-m1-p20/ Primes-20-20-3-3.xml	SAT TT=0.007	SAT TT=0.008	SAT TT=0.007	SAT TT=0.008	SAT TT=0.008	SAT TT=0.007	SAT TT=0.013	SAT TT=5.436
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Primes/Primes-m1-p20/ Primes-20-40-2-7.xml	SAT TT=1.826	? (TO) TT=2400.08	? (TO) TT=2400.02	? (TO) TT=2400.03	SAT TT=1.826	SAT TT=15.722	SAT TT=81.722	SAT TT=1520.8
Primes/Primes-m1-p20/ Primes-20-60-3-5.xml	SAT TT=517.414	? (TO) TT=2400.1	? (TO) TT=2400.1	? (TO) TT=2400.1	SAT TT=517.414	? (TO) TT=2400.08	SAT TT=1703.2	? (TO) TT=2400.1
Primes/Primes-m1-p25/ Primes-25-60-3-3.xml	SAT TT=0.046	SAT TT=9.436	SAT TT=9.321	SAT TT=9.285	SAT TT=0.345	SAT TT=0.046	SAT TT=0.088	SAT TT=28.875
Primes/Primes-m1-p25/ Primes-25-60-3-7.xml		? (TO) TT=2400.03	? (TO) TT=2400.04	? (TO) TT=2400.06	? TT=2400	? (TO) TT=2400.04	? (TO) TT=2400.03	? (TO) TT=2400.07
Primes/Primes-m1-p25/ Primes-25-80-3-5.xml	SAT TT=0.42	SAT TT=101.484	SAT TT=101.021	SAT TT=100.777	SAT TT=0.42	SAT TT=25.401	SAT TT=0.681	SAT TT=896.754
Primes/Primes-m1-p30/ Primes-30-80-3-7.xml	SAT TT=0.237	SAT TT=1.25	SAT TT=1.239	SAT TT=1.24	SAT TT=0.353	SAT TT=0.298	SAT TT=0.237	SAT TT=114.646
PropStress/ PropStress-m1-s1/ PropStress-0200.xml	UNSAT TT=72.383	? (TO) TT=2400.06	? (TO) TT=2400.02	? (TO) TT=2400.04	UNSAT TT=588.109	? (TO) TT=2400.04	? (TO) TT=2400.02	UNSAT TT=72.383
PropStress/ PropStress-m1-s1/ PropStress-0450.xml	UNSAT TT=503.862	? (TO) TT=2400.03	? (TO) TT=2400.04	? (TO) TT=2400.08	? (MO) TT=342.388	? (exit code) TT=16.113	? (exit code) TT=15.8	UNSAT TT=503.862
PropStress/ PropStress-m1-s1/ PropStress-0650.xml	UNSAT TT=1671.21	? (TO) TT=2400.36	? (TO) TT=2400.28	? (TO) TT=2400.36	? (MO) TT=66.566	? (exit code) TT=15.725	? (exit code) TT=15.575	UNSAT TT=1671.21
PseudoBoolean/ PseudoBoolean-dec-fpga/ Pb-chnl10-11.xml	UNSAT TT=56.641	UNSAT TT=2113.98	UNSAT TT=2128.69	UNSAT TT=2136.77	? TT=2400.01	? (TO) TT=2400.01	UNSAT TT=213.336	UNSAT TT=56.641
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
PseudoBoolean/ PseudoBoolean-dec-fpga/ Pb-chnl40-50.xml		? (TO) TT=2400.01	? (TO) TT=2400.01	? (TO) TT=2400.09	? (TO) TT=2400.07	? (TO) TT=2400.08	? (TO) TT=2400.05	? (TO) TT=2400.1
PseudoBoolean/ PseudoBoolean-dec-fpga/ Pb-fpga45-44-sat.xml	SAT TT=16.378	? (TO) TT=2400.06	? (TO) TT=2400.09	? (TO) TT=2400.02	? (TO) TT=2400.08	? (TO) TT=2400.08	SAT TT=16.378	SAT TT=135.487
PseudoBoolean/ PseudoBoolean-dec-pigeon/ Pb-pigeon-card-011-010.xml	UNSAT TT=29.588	UNSAT TT=1690.99	UNSAT TT=1706.72	UNSAT TT=1706.11	? TT=2400.01	? (TO) TT=2400.06	UNSAT TT=223.709	UNSAT TT=29.588
PseudoBoolean/ PseudoBoolean-dec-pigeon/ Pb-pigeon-card-015-010.xml	UNSAT TT=106.65	? (TO) TT=2400.06	? (TO) TT=2400.08	? (TO) TT=2400.1	? TT=2400	? (TO) TT=2400.03	UNSAT TT=121.72	UNSAT TT=106.65
PseudoBoolean/ PseudoBoolean-dec-pigeon/ Pb-pigeon-clau-011-010.xml	UNSAT TT=6.61	? (TO) TT=2400.05	? (TO) TT=2400.04	? (TO) TT=2400.05	UNSAT TT=1511.17	? (TO) TT=2400.06	UNSAT TT=6.61	UNSAT TT=63.564
PseudoBoolean/ PseudoBoolean-dec-spba/ Pb-BeauxArts-K67.xml	SAT TT=165.471	? (TO) TT=2400.07	? (TO) TT=2400.08	? (TO) TT=2400.03	? (TO) TT=2400.04	? (TO) TT=2400.06	? (TO) TT=2400.15	SAT TT=165.471
PseudoBoolean/ PseudoBoolean-dec-spba/ Pb-BeauxArts-K75.xml	SAT TT=16.783	? (TO) TT=2400.06	? (TO) TT=2400.11	? (TO) TT=2400.07	? (TO) TT=2400.1	? (TO) TT=2400.04	SAT TT=99.403	SAT TT=16.783
PseudoBoolean/ PseudoBoolean-dec-spba/ Pb-BeauxArts-K80.xml	SAT TT=15.801	? (TO) TT=2400.03	? (TO) TT=2400.06	? (TO) TT=2400.02	? (TO) TT=2400.05	SAT TT=20.912	SAT TT=26.556	SAT TT=15.801
PseudoBoolean/ PseudoBoolean-dec-spng/ Pb-NG-K138.xml		? (TO) TT=2400.03	? (TO) TT=2400.09	? (TO) TT=2400.09	? (TO) TT=2400.03	? (TO) TT=2400.06	? (TO) TT=2400.22	? (TO) TT=2400.01
PseudoBoolean/ PseudoBoolean-dec-spng/ Pb-NG-K151.xml		? (TO) TT=2400.08	? (TO) TT=2400.08	? (TO) TT=2400.09	? (TO) TT=2400.08	? (TO) TT=2400.06	? (TO) TT=2400.12	? (TO) TT=2400.05
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
PseudoBoolean/ PseudoBoolean-dec-spng/ Pb-NG-K152.xml		? (TO) TT=2400.07	? (TO) TT=2400.07	? (TO) TT=2400.12	? (TO) TT=2400.01	? (TO) TT=2400.1	? (TO) TT=2400.11	? (TO) TT=2400.06
PseudoBoolean/ PseudoBoolean-dec-spng/ Pb-NG-K171.xml		? (TO) TT=2400.13	? (TO) TT=2400.06	? (TO) TT=2400.03	? (TO) TT=2400.12	? (TO) TT=2400.02	? (TO) TT=2400.15	? (TO) TT=2400.03
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K090.xml	SAT TT=606.023	? (TO) TT=2400.09	? (TO) TT=2400.09	? (TO) TT=2400.09	? (TO) TT=2400.03	? (TO) TT=2400.02	? (TO) TT=2400.15	SAT TT=606.023
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K097.xml	SAT TT=120.497	? (TO) TT=2400.04	? (TO) TT=2400.07	? (TO) TT=2400.01	? (TO) TT=2400.04	? (TO) TT=2400.06	? (TO) TT=2400.18	SAT TT=120.497
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K100.xml	SAT TT=119.847	? (TO) TT=2400.08	? (TO) TT=2400.05	? (TO) TT=2400.06	? (TO) TT=2400.09	? (TO) TT=2400.07	? (TO) TT=2400.11	SAT TT=119.847
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K104.xml	SAT TT=110.658	? (TO) TT=2400.04	? (TO) TT=2400.01	? (TO) TT=2400.05	? (TO) TT=2400.02	? (TO) TT=2400.07	SAT TT=485.692	SAT TT=110.658
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K105.xml	SAT TT=107.681	? (TO) TT=2400.04	? (TO) TT=2400.04	? (TO) TT=2400.01	? (TO) TT=2400.03	? (TO) TT=2400.05	? (TO) TT=2400.31	SAT TT=107.681
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K106.xml	SAT TT=102.276	? (TO) TT=2400.01	? (TO) TT=2400.04	? (TO) TT=2400.03	? (TO) TT=2400.1	? (TO) TT=2400.06	? (TO) TT=2400.18	SAT TT=102.276
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K108.xml	SAT TT=78.684	? (TO) TT=2400.09	? (TO) TT=2400.07	? (TO) TT=2400.09	? (TO) TT=2400.05	? (TO) TT=2400.04	? (TO) TT=2400.15	SAT TT=78.684
PseudoBoolean/ PseudoBoolean-dec-sptate/ Pb-Tate-K109.xml	SAT TT=80.14	? (TO) TT=2400.1	? (TO) TT=2400.07	? (TO) TT=2400.07	? (TO) TT=2400.06	? (TO) TT=2400.04	? (TO) TT=2400.21	SAT TT=80.14
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
QRandom/QRandom-bdd-15-21-2/ bdd-15-21-2-2713-79-01.xml	UNSAT TT=19.555	UNSAT TT=183.041	UNSAT TT=182.357	UNSAT TT=184.388	UNSAT TT=19.555	UNSAT TT=671.514	UNSAT TT=201.124	? (TO) TT=2400.11
QRandom/QRandom-bdd-15-21-2/ bdd-15-21-2-2713-79-19.xml	SAT TT=21.154	SAT TT=21.154	SAT TT=21.409	SAT TT=21.755	SAT TT=22.42	SAT TT=493.326	SAT TT=162.56	? (TO) TT=2400.04
QRandom/QRandom-bdd-15-21-2/ bdd-15-21-2-2713-79-21.xml	UNSAT TT=20.978	UNSAT TT=191.603	UNSAT TT=191.901	UNSAT TT=195.946	UNSAT TT=20.978	UNSAT TT=487.414	UNSAT TT=177.9	? (TO) TT=2400.03
QRandom/QRandom-bdd-15-21-2/ bdd-15-21-2-2713-79-30.xml	UNSAT TT=19.106	UNSAT TT=163.898	UNSAT TT=166.326	UNSAT TT=169.772	UNSAT TT=19.106	UNSAT TT=531.404	UNSAT TT=191.743	? (TO) TT=2400.09
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-04.xml	SAT TT=7.753	SAT TT=66.693	SAT TT=71.024	SAT TT=67.689	SAT TT=7.753	SAT TT=600.365	SAT TT=129.862	? (TO) TT=2400.06
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-06.xml	UNSAT TT=8.68	UNSAT TT=129.97	UNSAT TT=128.771	UNSAT TT=131.044	UNSAT TT=8.68	UNSAT TT=497.131	UNSAT TT=110.398	? (TO) TT=2400.09
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-11.xml	SAT TT=7.29	SAT TT=43.933	SAT TT=41.289	SAT TT=45.715	SAT TT=7.29	SAT TT=670.531	SAT TT=127.583	? (TO) TT=2400.09
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-15.xml	UNSAT TT=8.394	UNSAT TT=132.877	UNSAT TT=132.053	UNSAT TT=128.152	UNSAT TT=8.394	UNSAT TT=532.053	UNSAT TT=107.705	? (TO) TT=2400.07
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-16.xml	UNSAT TT=8.691	UNSAT TT=122.879	UNSAT TT=124.662	UNSAT TT=125.327	UNSAT TT=8.691	UNSAT TT=542.236	UNSAT TT=106.5	? (TO) TT=2400.11
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-18.xml	SAT TT=1.664	SAT TT=1.666	SAT TT=1.664	SAT TT=1.669	SAT TT=1.958	SAT TT=13.584	SAT TT=13.847	? (TO) TT=2400.08
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-23.xml	SAT TT=1.668	SAT TT=1.668	SAT TT=1.903	SAT TT=1.684	SAT TT=1.959	SAT TT=14.1	SAT TT=13.684	? (TO) TT=2400.09
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-24.xml	UNSAT TT=7.972	UNSAT TT=121.572	UNSAT TT=120.053	UNSAT TT=123.522	UNSAT TT=7.972	UNSAT TT=508.636	UNSAT TT=97.315	? (TO) TT=2400.02
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-28.xml	UNSAT TT=8.565	UNSAT TT=135.607	UNSAT TT=137.575	UNSAT TT=133.704	UNSAT TT=8.565	UNSAT TT=756.382	UNSAT TT=123.504	? (TO) TT=2400.01
QRandom/QRandom-bdd-18-21-2/ bdd-18-21-2-133-78-30.xml	UNSAT TT=8.845	UNSAT TT=137.302	UNSAT TT=136.081	UNSAT TT=135.809	UNSAT TT=8.845	UNSAT TT=607.642	UNSAT TT=128.242	? (TO) TT=2400.09
QRandom/QRandom-geometric-rw/ geometric-50-20-d4-75-31.xml	SAT TT=0.035	SAT TT=0.108	SAT TT=0.108	SAT TT=0.107	SAT TT=0.045	SAT TT=0.035	SAT TT=0.035	SAT TT=0.309
QRandom/QRandom-geometric-rw/ geometric-50-20-d4-75-70.xml	UNSAT TT=29.397	UNSAT TT=29.397	UNSAT TT=29.771	UNSAT TT=29.491	UNSAT TT=690.9	UNSAT TT=37.999	UNSAT TT=69.722	UNSAT TT=246.683
QRandom/ QRandom-mdd-7-25-5/ mdd-7-25-5-56-01.xml	SAT TT=95.049	SAT TT=1913.61	SAT TT=2057.74	SAT TT=1930.23	SAT TT=95.049	? (TO) TT=2400.11	SAT TT=1845.57	SAT TT=864.608
QRandom/ QRandom-mdd-7-25-5/ mdd-7-25-5-56-02.xml	SAT TT=92.114	? (TO) TT=2400.07	? (TO) TT=2400.1	? (TO) TT=2400.06	SAT TT=92.114	? (TO) TT=2400.07	SAT TT=1286.69	SAT TT=390.716
QRandom/ QRandom-mdd-7-25-5/ mdd-7-25-5-56-06.xml	SAT TT=22.492	? (TO) TT=2400.03	? (TO) TT=2400.09	? (TO) TT=2400.02	SAT TT=22.492	? (TO) TT=2400.01	? (TO) TT=2400.09	SAT TT=1021.45
QRandom/ QRandom-mdd-7-25-5/ mdd-7-25-5-56-17.xml	SAT TT=253.884	? (TO) TT=2400.06	? (TO) TT=2400.05	? (TO) TT=2400.08	SAT TT=253.884	? (TO) TT=2400.04	SAT TT=1919.53	SAT TT=384.883
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
QRandom/QRandom-mdd-7-25-5-p7/ mdd-7-25-5-ps05-psh07-2.xml	UNSAT TT=20.864	UNSAT TT=1494.06	UNSAT TT=1486.06	UNSAT TT=1562.06	UNSAT TT=20.864	? (TO) TT=2400.03	UNSAT TT=487.812	UNSAT TT=1035.87
QRandom/QRandom-mdd-7-25-5-p7/ mdd-7-25-5-ps05-psh07-3.xml	UNSAT TT=56.078	? (TO) TT=2400.06	? (TO) TT=2400.07	? (TO) TT=2400.09	UNSAT TT=56.078	? (TO) TT=2400.1	UNSAT TT=1754.39	UNSAT TT=1066.75
QRandom/QRandom-mdd-7-25-5-p7/ mdd-7-25-5-ps05-psh07-9.xml	UNSAT TT=42.968	? (TO) TT=2400.1	? (TO) TT=2400.09	? (TO) TT=2400.08	UNSAT TT=42.968	? (TO) TT=2400.1	UNSAT TT=916.896	UNSAT TT=1121.99
Random/Random-D-t050/ rand-2-40-25-180-050-28.xml	SAT TT=1.87	SAT TT=1.88	SAT TT=1.882	SAT TT=1.87	SAT TT=36.301	SAT TT=2.623	SAT TT=12.837	SAT TT=11.594
Random/Random-D-t065/ rand-2-40-40-135-065-20.xml	SAT TT=0.293	SAT TT=0.293	SAT TT=0.296	SAT TT=0.294	SAT TT=0.392	SAT TT=0.453	SAT TT=0.706	SAT TT=28.783
Random/Random-m1-10-60-20/ rand-10-60-20-30-51200-43.xml	UNSAT TT=13.594	UNSAT TT=520.06	UNSAT TT=520.343	UNSAT TT=554.703	UNSAT TT=13.594	UNSAT TT=586.304	UNSAT TT=314.037	? (TO) TT=2400.1
Random/Random-m1-5-12-12/ rand-5-12-12-200-12442-18.xml	UNSAT TT=3.354	UNSAT TT=59.498	UNSAT TT=59.376	UNSAT TT=59.658	UNSAT TT=3.354	UNSAT TT=153.877	UNSAT TT=85.434	UNSAT TT=1337.44
Random/Random-m1-7-40-8t/ rand-7-40-8-40-20000-2.xml	UNSAT TT=32.706	UNSAT TT=1527.79	UNSAT TT=1521.6	UNSAT TT=1546.96	UNSAT TT=32.706	? (TO) TT=2400.03	UNSAT TT=1599.55	UNSAT TT=1275.55
Random/Random-RB-2-40-19f/ rand-2-40-19-443-230f-30.xml	SAT TT=44.079	SAT TT=99.845	SAT TT=102.84	SAT TT=114.1	SAT TT=1733.97	SAT TT=44.079	SAT TT=975.988	SAT TT=60.222
Random/Random-RB-2-50-23/ rand-2-50-23-587-230-39.xml	SAT TT=2044.77	? (TO) TT=2400.04	? (TO) TT=2400.06	? (TO) TT=2400.01	SAT TT=2253.38	? (TO) TT=2400.05	? (TO) TT=2400.02	SAT TT=2044.77
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Random/Random-RB-3-20-20f/ rand-3-20-20-60-632f-11.xml	SAT TT=33.067	SAT TT=613.434	SAT TT=615.08	SAT TT=610.67	SAT TT=33.067	SAT TT=1414.87	SAT TT=398.745	SAT TT=104.938
Random/Random-RB-3-24-24f/ rand-3-24-24-76-632f-04.xml	SAT TT=74.75	SAT TT=1063.69	SAT TT=1069.73	SAT TT=1072.32	SAT TT=112.408	? (TO) TT=2400.07	SAT TT=1685.52	SAT TT=74.75
Random/Random-RB-3-28-28f/ rand-3-28-28-93-632f-36.xml		? (TO) TT=2400.08	? (TO) TT=2400.1	? (TO) TT=2400.1	? (TO) TT=2400.04	? (TO) TT=2400.05	? (TO) TT=2400.07	? (TO) TT=2400.05
Rlfap_mini/ Rlfap-dec-scens11_mini/ Rlfap-ext-scen-11-f02_c18.xml	UNSAT TT=95.944	UNSAT TT=300.456	UNSAT TT=298.414	UNSAT TT=299.338	? (TO) TT=2400.07	UNSAT TT=1877.41	UNSAT TT=95.944	UNSAT TT=181.22
Rlfap_mini/ Rlfap-dec-scens11_mini/ Rlfap-ext-scen-11-f04_c18.xml	UNSAT TT=4.986	UNSAT TT=25.654	UNSAT TT=27.325	UNSAT TT=25.636	UNSAT TT=289.421	UNSAT TT=110.35	UNSAT TT=4.986	UNSAT TT=37.918
Sat/ Sat-m1-various/ hole-10.xml	UNSAT TT=20.505	UNSAT TT=1385.04	UNSAT TT=1439.43	UNSAT TT=1389.57	UNSAT TT=1842.91	? (TO) TT=2400.06	UNSAT TT=20.505	UNSAT TT=63.383
SchurrLemma/ SchurrLemma-mod-s1/ SchurrLemma-mod-015-9.xml	UNSAT TT=7.195	UNSAT TT=22.425	UNSAT TT=22.523	UNSAT TT=22.777	UNSAT TT=7.195	UNSAT TT=14.232	UNSAT TT=113.973	UNSAT TT=25.574
SchurrLemma/ SchurrLemma-mod-s1/ SchurrLemma-mod-100-9.xml	UNSAT TT=35.313	UNSAT TT=530.765	UNSAT TT=530.667	UNSAT TT=528.464	UNSAT TT=288.742	UNSAT TT=323.946	UNSAT TT=141.179	UNSAT TT=35.313
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g04-g43.xml	SAT TT=0.029	SAT TT=0.723	SAT TT=0.831	SAT TT=0.723	SAT TT=0.029	SAT TT=0.701	SAT TT=12.85	SAT TT=0.102
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g07-g50.xml	SAT TT=0.024	SAT TT=0.024	SAT TT=0.027	SAT TT=0.024	SAT TT=0.126	SAT TT=0.232	SAT TT=0.229	SAT TT=1.086
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g10-g15.xml	UNSAT TT=0.01	UNSAT TT=1406.37	UNSAT TT=1405.14	UNSAT TT=1409.54	UNSAT TT=0.01	? (TO) TT=2400.11	? (TO) TT=2400.09	UNSAT TT=699.086
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g10-g22.xml	UNSAT TT=0.03	UNSAT TT=210.658	UNSAT TT=210.696	UNSAT TT=212.231	UNSAT TT=0.03	UNSAT TT=1075.87	UNSAT TT=855.805	UNSAT TT=269
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g10-g47.xml	SAT TT=0.096	SAT TT=0.22	SAT TT=0.252	SAT TT=0.222	SAT TT=2.212	SAT TT=0.096	SAT TT=0.109	SAT TT=16.792
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g13-g27.xml	UNSAT TT=2.654	UNSAT TT=388.116	UNSAT TT=384.926	UNSAT TT=384.612	UNSAT TT=2.654	UNSAT TT=1270.82	UNSAT TT=483.059	UNSAT TT=72.33
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g19-g25.xml	UNSAT TT=0.015	UNSAT TT=1961.87	UNSAT TT=2237.25	UNSAT TT=2238.06	UNSAT TT=0.015	? (TO) TT=2400.05	? (TO) TT=2400.09	UNSAT TT=110.884
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g29-g30.xml	UNSAT TT=0.029	UNSAT TT=0.03	UNSAT TT=0.03	UNSAT TT=0.029	UNSAT TT=0.313	UNSAT TT=0.633	UNSAT TT=0.626	UNSAT TT=1.22
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g34-g41.xml	UNSAT TT=2.905	UNSAT TT=3.623	UNSAT TT=3.643	UNSAT TT=3.611	UNSAT TT=11.394	UNSAT TT=2.905	UNSAT TT=3.074	UNSAT TT=41.601
Subisomorphism/ Subisomorphism-m1-LV/ Subisomorphism-g41-g47.xml	UNSAT TT=0.475	UNSAT TT=0.476	UNSAT TT=0.542	UNSAT TT=0.475	UNSAT TT=10.928	UNSAT TT=1.244	UNSAT TT=1.358	UNSAT TT=22.344
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-B-07.xml	SAT TT=0.719	SAT TT=0.722	SAT TT=0.72	SAT TT=0.719	SAT TT=10.564	SAT TT=2.652	SAT TT=2.661	SAT TT=24.062
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-04.xml	SAT TT=1.8	SAT TT=1.807	SAT TT=1.8	SAT TT=1.819	SAT TT=34.227	SAT TT=9.185	SAT TT=9.233	SAT TT=160.832
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-05.xml	SAT TT=3.102	SAT TT=3.109	SAT TT=3.108	SAT TT=3.102	SAT TT=48.611	SAT TT=8.364	SAT TT=8.383	SAT TT=275.677
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-07.xml	SAT TT=1.623	SAT TT=1.628	SAT TT=1.623	SAT TT=1.629	SAT TT=117.279	SAT TT=8.459	SAT TT=8.521	SAT TT=264.112
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-08.xml	SAT TT=2.656	SAT TT=2.656	SAT TT=2.656	SAT TT=2.663	SAT TT=60.674	SAT TT=9.614	SAT TT=9.599	SAT TT=159.531
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-09.xml	SAT TT=1.583	SAT TT=1.586	SAT TT=1.583	SAT TT=1.588	SAT TT=36.136	SAT TT=8.797	SAT TT=8.801	SAT TT=227.58
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-11.xml	SAT TT=1.579	SAT TT=1.582	SAT TT=1.579	SAT TT=1.583	SAT TT=43.342	SAT TT=8.611	SAT TT=8.561	SAT TT=278.211
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-15.xml	SAT TT=2.015	SAT TT=2.019	SAT TT=2.015	SAT TT=2.021	SAT TT=34.369	SAT TT=9.724	SAT TT=9.871	SAT TT=134.611
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-C-17.xml	SAT TT=2.1	SAT TT=2.103	SAT TT=2.1	SAT TT=2.103	SAT TT=52.388	SAT TT=8.4	SAT TT=8.458	SAT TT=228.914
Subisomorphism/ Subisomorphism-m1-SF/ Subisomorphism-F-20.xml	SAT TT=233.201	? (TO) TT=2400.02	? (TO) TT=2400.02	? (TO) TT=2400.01	SAT TT=233.201	? (TO) TT=2400.09	? (TO) TT=2400.13	? (TO) TT=2400.12
Subisomorphism/ Subisomorphism-m1-si2-bvg/ Subisomorphism-si2-b03m-m800-09.xml	SAT TT=0.042	SAT TT=0.042	SAT TT=0.047	SAT TT=0.042	SAT TT=3.61	SAT TT=0.227	SAT TT=0.232	SAT TT=3.628
Subisomorphism/ Subisomorphism-m1-si2-bvg/ Subisomorphism-si2-b06m-m400-03.xml	SAT TT=0.019	SAT TT=0.019	SAT TT=0.021	SAT TT=0.019	SAT TT=0.297	SAT TT=0.131	SAT TT=0.132	SAT TT=1.957
Bench name	Best results	cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Subisomorphism/ Subisomorphism-m1-si2-m4D/ Subisomorphism-si2-m4Dr6-m1296-03.xml	SAT TT=0.176	SAT TT=0.176	SAT TT=0.176	SAT TT=0.178	SAT TT=13.669	SAT TT=2.015	SAT TT=1.973	SAT TT=12.673
Subisomorphism/ Subisomorphism-m1-si4-rand/ Subisomorphism-si4-r005-m600-07.xml	SAT TT=1144.16	SAT TT=1144.97	SAT TT=1144.45	SAT TT=1144.16	? (MO) TT=30.265	SAT TT=1662.46	SAT TT=1381.41	? (TO) TT=2400.12
Subisomorphism/ Subisomorphism-m1-si6-bvg/ Subisomorphism-si6-b09m-m200-07.xml	SAT TT=0.022	SAT TT=0.022	SAT TT=0.022	SAT TT=0.022	SAT TT=0.286	SAT TT=0.286	SAT TT=0.283	SAT TT=1.234
Subisomorphism/ Subisomorphism-m1-si6-m4D/ Subisomorphism-si6-m4D-m1296-05.xml	SAT TT=2.21	SAT TT=2.21	SAT TT=2.233	SAT TT=2.224	SAT TT=1270.65	SAT TT=10.879	SAT TT=10.856	SAT TT=311.003
Subisomorphism/ Subisomorphism-m1-si6-m4D/ Subisomorphism-si6-m4D-m256-08.xml	SAT TT=0.031	SAT TT=0.031	SAT TT=0.031	SAT TT=0.031	SAT TT=0.489	SAT TT=0.236	SAT TT=0.233	SAT TT=6.71
Subisomorphism/ Subisomorphism-m1-si6-m4D/ Subisomorphism-si6-m4Dr2-m1296-00.xml	SAT TT=0.896	SAT TT=0.899	SAT TT=0.899	SAT TT=0.896	SAT TT=153.597	SAT TT=7.211	SAT TT=7.204	SAT TT=48.077
Subisomorphism/ Subisomorphism-m1-si6-m4D/ Subisomorphism-si6-m4Dr4-m1296-00.xml	SAT TT=0.404	SAT TT=0.404	SAT TT=0.404	SAT TT=0.404	SAT TT=94.255	SAT TT=7.882	SAT TT=7.863	SAT TT=32.337
Subisomorphism/ Subisomorphism-m1-si6-m4D/ Subisomorphism-si6-m4Dr4-m1296-05.xml	SAT TT=1.112	SAT TT=1.112	SAT TT=1.113	SAT TT=1.114	SAT TT=100.664	SAT TT=7.888	SAT TT=8.038	SAT TT=218.584
TravellingSalesman/ TravellingSalesman-xcsp2-s20a4/ TravellingSalesman-4-20-075-a4_X2.xml	SAT TT=9.148	SAT TT=175.219	SAT TT=174.218	SAT TT=176.071	SAT TT=9.148	SAT TT=158.664	SAT TT=39.359	SAT TT=24.598
TravellingSalesman/ TravellingSalesman-xcsp2-s20/ TravellingSalesman-20-453_X2.xml	SAT TT=1.132	SAT TT=1.132	SAT TT=1.134	SAT TT=1.147	SAT TT=20.448	SAT TT=23.544	SAT TT=25.033	SAT TT=28.818

Statistics		cosoco 2 (complete)	cosoco 2.0 (complete)	cosoco 2.0 (complete)	miniBTD 19.06.16 (complete)	NACRE 1.0.5 (complete)	NACRE 1.0.5-Hybrid (complete)	(reference) PicatSAT 2019-09-12 (complete)
Number of times the solver is able to give the best known answer		127	127	127	133	116	135	148
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		13	10	11	78	7	15	38