PB'16 competition: satisfaction and optimization track: results by benchmark

Results by benchmark for category no optimisation, small integers, linear constraints (DEC-SMALLINT-LIN), subcategory Industrial, subsubcategory UCLID Benchmarks

This page displays the results of the different solvers for each benchmark for category no optimisation, small integers, linear constraints (DEC-SMALLINT-LIN), subcategory Industrial, subsubcategory UCLID Benchmarks

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance (complete solvers point of view)
text	the solver performed better than the other ones on that instance (incomplete solvers point of view)
text	the solver was ended by a signal or other problem
text	the solver gave an incomplete answer
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
cdcl-cuttingplanes DEC 2016-05-01 (complete)
minisatp 2012-10-02 git-d91742b (complete)
NaPS 1.02 (complete)
Open-WBO PB16 (complete)
Open-WBO-LSU PB16 (complete)
Sat4j PB 2.3.6 Res+CP PB16 (complete)
Sat4j PB 2.3.6 Resolution PB16 (complete)
toysat 2016-05-02 (complete)

Bench name	Best results on this instance	cdcl-cuttingplanes DEC 2016-05-01 (complete)	minisatp 2012-10-02 git-d91742b (complete)	NaPS 1.02 (complete)	Open-WBO PB16 (complete)	Open-WBO-LSU PB16 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
normalized-PB06/SATUNSAT-SMALLINT/ web/ uclid_pb_benchmarks/normalized-22s.smv.opb	UNSAT TT=0.088	UNSAT TT=0.088	UNSAT TT=0.529	UNSAT TT=7.772	UNSAT TT=2.306	UNSAT TT=2.311	UNSAT TT=2.084	UNSAT TT=1.203	UNSAT TT=0.986
normalized-PB06/SATUNSAT-SMALLINT/ web/ uclid_pb_benchmarks/normalized-25s.smv.opb	UNSAT TT=0.069	UNSAT TT=0.069	UNSAT TT=0.426	UNSAT TT=5.827	UNSAT TT=2.249	UNSAT TT=2.248	UNSAT TT=1.994	UNSAT TT=1.044	UNSAT TT=0.481
normalized-PB06/SATUNSAT-SMALLINT/ web/ uclid_pb_benchmarks/normalized-37s.smv.opb	UNSAT TT=0.029	UNSAT TT=0.029	UNSAT TT=0.067	UNSAT TT=0.242	UNSAT TT=0.041	UNSAT TT=0.041	UNSAT TT=1.023	UNSAT TT=0.764	UNSAT TT=0.199
normalized-PB06/SATUNSAT-SMALLINT/ web/ uclid_pb_benchmarks/normalized-43s.smv.opb	UNSAT TT=0.129	UNSAT TT=0.129	UNSAT TT=0.291	UNSAT TT=3.93	UNSAT TT=1.716	UNSAT TT=1.706	UNSAT TT=3.203	UNSAT TT=1.296	UNSAT TT=1.115
normalized-PB06/SATUNSAT-SMALLINT/ web/ uclid_pb_benchmarks/normalized-44s.smv.opb	UNSAT TT=0.016	UNSAT TT=0.016	UNSAT TT=0.037	UNSAT TT=0.401	UNSAT TT=0.057	UNSAT TT=0.059	UNSAT TT=1.044	UNSAT TT=0.564	UNSAT TT=0.09
normalized-PB06/SATUNSAT-SMALLINT/ web/ uclid_pb_benchmarks/normalized-46s.smv.opb	UNSAT TT=0.04	UNSAT TT=0.04	UNSAT TT=0.122	UNSAT TT=1.179	UNSAT TT=0.191	UNSAT TT=0.191	UNSAT TT=1.744	UNSAT TT=0.935	UNSAT TT=0.318
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-floppy1-2.ucl.opb	UNSAT TT=0.009	UNSAT TT=0.01	UNSAT TT=0.009	UNSAT TT=0.158	UNSAT TT=0.083	UNSAT TT=0.084	UNSAT TT=1.114	UNSAT TT=0.443	UNSAT TT=0.058
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-floppy1-3.ucl.opb	UNSAT TT=0.003	UNSAT TT=0.01	UNSAT TT=0.009	UNSAT TT=0.003	UNSAT TT=0.012	UNSAT TT=0.014	UNSAT TT=0.275	UNSAT TT=0.253	UNSAT TT=0.023
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-floppy1-4.ucl.opb	UNSAT TT=0.003	UNSAT TT=0.011	UNSAT TT=0.007	UNSAT TT=0.003	UNSAT TT=0.033	UNSAT TT=0.035	UNSAT TT=0.264	UNSAT TT=0.235	UNSAT TT=0.02
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-floppy1-6.ucl.opb	UNSAT TT=0.015	UNSAT TT=0.015	UNSAT TT=0.043	UNSAT TT=1.173	UNSAT TT=0.387	UNSAT TT=0.389	UNSAT TT=0.875	UNSAT TT=0.685	UNSAT TT=0.11
Bench name	Best results	cdcl-cuttingplanes DEC 2016-05-01 (complete)	minisatp 2012-10-02 git-d91742b (complete)	NaPS 1.02 (complete)	Open-WBO PB16 (complete)	Open-WBO-LSU PB16 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-floppy1-7.ucl.opb	UNSAT TT=0.035	UNSAT TT=0.035	UNSAT TT=0.302	UNSAT TT=48.71	UNSAT TT=9.768	UNSAT TT=9.83	UNSAT TT=1.164	UNSAT TT=0.794	UNSAT TT=0.18
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-floppy1-8.ucl.opb	UNSAT TT=0.04	UNSAT TT=0.04	UNSAT TT=0.179	UNSAT TT=0.208	UNSAT TT=12.153	UNSAT TT=12.302	UNSAT TT=1.374	UNSAT TT=0.923	UNSAT TT=0.217
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-tlan2.ucl.opb	UNSAT TT=0.031	UNSAT TT=0.031	UNSAT TT=0.179	UNSAT TT=8.422	UNSAT TT=1.157	UNSAT TT=1.16	UNSAT TT=1.295	UNSAT TT=1.224	UNSAT TT=0.216
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-blast-tlan3.ucl.opb	UNSAT TT=0.164	UNSAT TT=0.164	UNSAT TT=1.219	UNSAT TT=0.515	UNSAT TT=6.67	UNSAT TT=6.722	UNSAT TT=2.954	UNSAT TT=1.803	UNSAT TT=1.152
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-cache-ibm-q-full.all.ucl.opb	UNSAT TT=53.252	? (TO) TT=1800.03	UNSAT TT=53.252	UNSAT TT=225.322	UNSAT TT=222.064	UNSAT TT=207.56	? (TO) TT=1800.01	UNSAT TT=812.361	? (TO) TT=1800.07
normalized-PB06/SATUNSAT-SMALLINT/web/ uclid_pb_benchmarks/ normalized-cache-ibm-q-unbounded.Ic22arity.ucl.opb	UNSAT TT=5.698	UNSAT TT=6.882	UNSAT TT=5.698	UNSAT TT=50.062	UNSAT TT=15.846	UNSAT TT=15.848	UNSAT TT=55.506	UNSAT TT=38.149	UNSAT TT=132.829
normalized-PB06/SATUNSAT-SMALLINT/web/ uclid_pb_benchmarks/ normalized-cache-ibm-q-unbounded.Icl2arity.ucl.opb	UNSAT TT=0.835	UNSAT TT=0.835	UNSAT TT=1.574	UNSAT TT=11.461	UNSAT TT=2.307	UNSAT TT=2.291	UNSAT TT=7.553	UNSAT TT=2.794	UNSAT TT=14.455
normalized-PB06/SATUNSAT-SMALLINT/web/ uclid_pb_benchmarks/ normalized-cache-ibm-q-unbounded.Ih1arity.ucl.opb	UNSAT TT=55.624	? (TO) TT=1800.01	UNSAT TT=55.624	UNSAT TT=376.761	UNSAT TT=403.441	UNSAT TT=392.94	? (TO) TT=1800.11	UNSAT TT=923.7	? (TO) TT=1800.06
normalized-PB06/SATUNSAT-SMALLINT/web/ uclid_pb_benchmarks/ normalized-cache-ibm-q-unbounded.Ih2arity.ucl.opb	SAT TT=0.923 T1=0.9	SAT TT=0.923 T1=0.9	SAT TT=4.799 T1=4.76	SAT TT=95.685 T1=95.6	SAT TT=5.735 T1=5.66	SAT TT=5.732 T1=5.65	SAT TT=7.083 T1=3.4	SAT TT=3.832 T1=2.59	SAT TT=8.571 T1=8.44
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-cache.inv10.ucl.opb	UNSAT TT=0.104	UNSAT TT=0.104	UNSAT TT=0.418	UNSAT TT=0.425	UNSAT TT=0.11	UNSAT TT=0.11	UNSAT TT=2.713	UNSAT TT=1.133	UNSAT TT=0.663
Bench name	Best results	cdcl-cuttingplanes DEC 2016-05-01 (complete)	minisatp 2012-10-02 git-d91742b (complete)	NaPS 1.02 (complete)	Open-WBO PB16 (complete)	Open-WBO-LSU PB16 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-cache.inv12.ucl.opb	UNSAT TT=1.61	UNSAT TT=25.798	UNSAT TT=1.719	UNSAT TT=1.61	UNSAT TT=2.206	UNSAT TT=2.191	UNSAT TT=26.101	UNSAT TT=10.591	UNSAT TT=61.824
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-cache.inv14.ucl.opb	UNSAT TT=99.91	? (TO) TT=1800.02	UNSAT TT=122.439	UNSAT TT=99.91	UNSAT TT=143.793	UNSAT TT=141.303	? (TO) TT=1800.84	UNSAT TT=1604.29	? (TO) TT=1800.05
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-cache.inv8.ucl.opb	UNSAT TT=0.029	UNSAT TT=0.033	UNSAT TT=0.084	UNSAT TT=0.077	UNSAT TT=0.029	UNSAT TT=0.029	UNSAT TT=1.555	UNSAT TT=0.645	UNSAT TT=0.133
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-dlx1c.rwmem1.ucl.opb	UNSAT TT=0.29	UNSAT TT=0.291	UNSAT TT=0.461	UNSAT TT=1.531	UNSAT TT=0.29	UNSAT TT=0.291	UNSAT TT=4.312	UNSAT TT=1.937	UNSAT TT=2.251
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-dlx1c.rwmem.ucl.opb	UNSAT TT=0.276	UNSAT TT=0.276	UNSAT TT=0.543	UNSAT TT=1.566	UNSAT TT=0.444	UNSAT TT=0.441	UNSAT TT=3.462	UNSAT TT=1.684	UNSAT TT=2.678
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-dlx1c.ucl.opb	UNSAT TT=0.14	UNSAT TT=0.14	UNSAT TT=0.373	UNSAT TT=1.363	UNSAT TT=0.262	UNSAT TT=0.263	UNSAT TT=2.714	UNSAT TT=1.494	UNSAT TT=0.949
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-elf.rf10.ucl.opb	UNSAT TT=3.067	UNSAT TT=3.067	UNSAT TT=3.413	UNSAT TT=14.798	UNSAT TT=3.292	UNSAT TT=3.305	UNSAT TT=11.702	UNSAT TT=4.693	UNSAT TT=23.679
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-elf.rf6.ucl.opb	UNSAT TT=0.001	UNSAT TT=0.01	UNSAT TT=0.003	UNSAT TT=0.004	UNSAT TT=0.001	UNSAT TT=0.001	UNSAT TT=0.185	UNSAT TT=0.172	UNSAT TT=0.017
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-elf.rf7.ucl.opb	UNSAT TT=0.024	UNSAT TT=0.024	UNSAT TT=0.072	UNSAT TT=0.206	UNSAT TT=0.029	UNSAT TT=0.034	UNSAT TT=0.994	UNSAT TT=0.613	UNSAT TT=0.119
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-elf.rf8.ucl.opb	UNSAT TT=0.088	UNSAT TT=0.088	UNSAT TT=0.344	UNSAT TT=1.457	UNSAT TT=0.18	UNSAT TT=0.182	UNSAT TT=2.234	UNSAT TT=1.175	UNSAT TT=0.538
Bench name	Best results	cdcl-cuttingplanes DEC 2016-05-01 (complete)	minisatp 2012-10-02 git-d91742b (complete)	NaPS 1.02 (complete)	Open-WBO PB16 (complete)	Open-WBO-LSU PB16 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-elf.rf9.ucl.opb	UNSAT TT=0.447	UNSAT TT=0.447	UNSAT TT=1.198	UNSAT TT=5.6	UNSAT TT=0.64	UNSAT TT=0.646	UNSAT TT=3.632	UNSAT TT=2.055	UNSAT TT=3.278
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.burch_dill.2.accl.ucl.opb	UNSAT TT=0.291	UNSAT TT=0.77	UNSAT TT=0.291	UNSAT TT=1.27	UNSAT TT=1.189	UNSAT TT=1.186	UNSAT TT=7.29	UNSAT TT=2.454	UNSAT TT=6.455
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.burch_dill.3.accl.ucl.opb	UNSAT TT=0.865	UNSAT TT=3.52	UNSAT TT=0.865	UNSAT TT=3.383	UNSAT TT=3.921	UNSAT TT=3.988	UNSAT TT=11.761	UNSAT TT=4.402	UNSAT TT=20.333
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.burch_dill.4.accl.ucl.opb	UNSAT TT=0.103	UNSAT TT=0.103	UNSAT TT=0.268	UNSAT TT=2.285	UNSAT TT=0.359	UNSAT TT=0.36	UNSAT TT=3.504	UNSAT TT=1.634	UNSAT TT=0.565
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.burch_dill.6.accl.ucl.opb	UNSAT TT=477.303	? (TO) TT=1800.1	UNSAT TT=489.47	UNSAT TT=477.303	UNSAT TT=923.75	UNSAT TT=959.686	? (TO) TT=1800.05	UNSAT TT=583.905	? (TO) TT=1800.07
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.burch_dill.8.accl.ucl.opb	UNSAT TT=61.157	? (TO) TT=1800.02	UNSAT TT=61.157	UNSAT TT=122.528	UNSAT TT=216.423	UNSAT TT=190.479	UNSAT TT=463.753	UNSAT TT=193.13	? (TO) TT=1800.08
normalized-PB06/SATUNSAT-SMALLINT/web/ uclid_pb_benchmarks/ normalized-ooo.ex.br.mem.RobRegValid_bar.ucl.opb	UNSAT TT=6.301	UNSAT TT=179.854	UNSAT TT=6.301	UNSAT TT=169.183	UNSAT TT=45.136	UNSAT TT=45.39	UNSAT TT=77.371	UNSAT TT=22.162	UNSAT TT=131.046
normalized-PB06/SATUNSAT-SMALLINT/web/uclid_pb_benchmarks/ normalized-ooo.ex.br.mem.Src1Valid_Src1ValidBar.ucl.opb	UNSAT TT=40.521	? (TO) TT=1800.02	UNSAT TT=40.521	UNSAT TT=1449.56	UNSAT TT=472.589	UNSAT TT=495.587	UNSAT TT=786.752	UNSAT TT=354.14	? (TO) TT=1800.1
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.ex.mem.LdValue.ucl.opb	UNSAT TT=34.559	? (TO) TT=1800.02	UNSAT TT=34.559	? (TO) TT=1800.05	UNSAT TT=305.186	UNSAT TT=344.729	UNSAT TT=607.091	UNSAT TT=294.009	? (TO) TT=1800.05
normalized-PB06/SATUNSAT-SMALLINT/web/uclid_pb_benchmarks/ normalized-ooo.ex.mem.LsqHdStrong.ucl.opb	UNSAT TT=62.995	? (TO) TT=1800.11	UNSAT TT=62.995	? (TO) TT=1800.06	UNSAT TT=1321.3	UNSAT TT=1298.24	? (TO) TT=1800.18	? (TO) TT=1800.77	? (TO) TT=1800.04
Bench name	Best results	cdcl-cuttingplanes DEC 2016-05-01 (complete)	minisatp 2012-10-02 git-d91742b (complete)	NaPS 1.02 (complete)	Open-WBO PB16 (complete)	Open-WBO-LSU PB16 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.rf10.ucl.opb	UNSAT TT=2.392	UNSAT TT=10.574	UNSAT TT=2.392	UNSAT TT=4.463	UNSAT TT=5.828	UNSAT TT=5.917	UNSAT TT=31.559	UNSAT TT=16.241	UNSAT TT=348.415
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.rf6.ucl.opb	UNSAT TT=0.034	UNSAT TT=0.034	UNSAT TT=0.096	UNSAT TT=0.107	UNSAT TT=0.035	UNSAT TT=0.035	UNSAT TT=1.483	UNSAT TT=0.834	UNSAT TT=0.245
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.rf7.ucl.opb	UNSAT TT=0.096	UNSAT TT=0.096	UNSAT TT=0.203	UNSAT TT=0.661	UNSAT TT=0.103	UNSAT TT=0.106	UNSAT TT=3.943	UNSAT TT=1.164	UNSAT TT=0.954
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.rf8.ucl.opb	UNSAT TT=0.232	UNSAT TT=0.334	UNSAT TT=0.39	UNSAT TT=1.019	UNSAT TT=0.233	UNSAT TT=0.232	UNSAT TT=4.004	UNSAT TT=1.704	UNSAT TT=3.458
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.rf9.ucl.opb	UNSAT TT=0.762	UNSAT TT=1.42	UNSAT TT=0.762	UNSAT TT=1.687	UNSAT TT=0.951	UNSAT TT=0.962	UNSAT TT=13.552	UNSAT TT=4.892	UNSAT TT=23.041
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.tag10.ucl.opb	UNSAT TT=0.504	UNSAT TT=0.663	UNSAT TT=0.894	UNSAT TT=1.874	UNSAT TT=0.504	UNSAT TT=0.504	UNSAT TT=21.661	UNSAT TT=6.303	UNSAT TT=45.578
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.tag12.ucl.opb	UNSAT TT=1.582	UNSAT TT=4.577	UNSAT TT=3.571	UNSAT TT=3.159	UNSAT TT=1.643	UNSAT TT=1.582	UNSAT TT=103.977	UNSAT TT=62.41	UNSAT TT=183.174
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.tag14.ucl.opb	UNSAT TT=4.165	UNSAT TT=12.741	UNSAT TT=22.036	UNSAT TT=8.87	UNSAT TT=4.165	UNSAT TT=4.215	UNSAT TT=425.25	UNSAT TT=110.41	UNSAT TT=489.829
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.tag8.ucl.opb	UNSAT TT=0.205	UNSAT TT=0.567	UNSAT TT=0.205	UNSAT TT=0.678	UNSAT TT=0.346	UNSAT TT=0.372	UNSAT TT=6.705	UNSAT TT=2.423	UNSAT TT=4.307
normalized-PB06/SATUNSAT-SMALLINT/ web/uclid_pb_benchmarks/ normalized-ooo.unbounded.all.ucl.opb	UNSAT TT=31.401	? (TO) TT=1800.02	UNSAT TT=31.401	UNSAT TT=845.324	UNSAT TT=160.95	UNSAT TT=160.755	UNSAT TT=622.92	UNSAT TT=214.432	UNSAT TT=1675.13

Statistics		cdcl-cuttingplanes DEC 2016-05-01 (complete)	minisatp 2012-10-02 git-d91742b (complete)	NaPS 1.02 (complete)	Open-WBO PB16 (complete)	Open-WBO-LSU PB16 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
Number of times the solver is able to give the best known answer		41	50	48	50	50	45	49	42
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		23	15	5	4	4	0	0	0