PB'16 competition: WBO track: results by benchmark

Results by benchmark for category only soft constraints, big integers, linear constraints (SOFT-BIGINT-LIN)

This page displays the results of the different solvers for each benchmark for category only soft constraints, big integers, linear constraints (SOFT-BIGINT-LIN)

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance
text	the solver was ended by a signal
text	the solver gave an incomplete answer
text	the solver was disqualified in the category
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
NaPS 1.02 (complete)
Sat4j PB 2.3.6 Res+CP PB16 (complete)
Sat4j PB 2.3.6 Resolution PB16 (complete)
toysat 2016-05-02 (complete)

Bench name	Best results on this instance	NaPS 1.02 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-blast-tlan3.ucl--soft-0-100-0.wbo	MOPT f=1 TT=9.181 T1=6.94	MSAT (MO) f=83112 TT=196.927 T1=196.4	OPTIMUM f=1 TT=49.127 T1=40.88	OPTIMUM f=1 TT=9.181 T1=6.94	OPTIMUM f=1 TT=11.029 T1=10.9
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-cache-ibm-q-full.all.ucl--soft-0-100-0.wbo	MOPT f=1 TT=1395.83 T1=461.94	MSAT (MO) f=43204 TT=308.692 T1=308.31	MSAT (TO) f=134 TT=1800.88 T1=1774.22	OPTIMUM f=1 TT=1395.83 T1=461.94	? (TO) TT=1800.07
/SOFT-BIGINT-LIN/PB06/web/uclid_pb_benchmarks/ normalized-cache-ibm-q-unbounded.Ic22arity.ucl--soft-0-100-0.wbo	MOPT f=1 TT=200.72 T1=196.42	MSAT f=119524 TT=287.575 T1=287.3	MSAT (TO) f=397 TT=1800.98 T1=1778.82	OPTIMUM f=1 TT=223.3 T1=196.42	OPTIMUM f=1 TT=200.72 T1=200.43
/SOFT-BIGINT-LIN/PB06/web/uclid_pb_benchmarks/ normalized-cache-ibm-q-unbounded.Icl2arity.ucl--soft-0-100-0.wbo	MOPT f=1 TT=6.653 T1=3.94	MSAT (MO) f=43222 TT=87.446 T1=86.91	OPTIMUM f=1 TT=24.531 T1=16.94	OPTIMUM f=1 TT=6.653 T1=3.94	OPTIMUM f=1 TT=20.578 T1=20.44
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-cache.inv10.ucl--soft-0-100-0.wbo	MOPT f=2 TT=4.742 T1=1.34	OPTIMUM f=2 TT=191.438 T1=189.89	OPTIMUM f=2 TT=17.821 T1=1.86	OPTIMUM f=2 TT=4.742 T1=1.34	OPTIMUM f=2 TT=14.984 T1=14.9
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-cache.inv12.ucl--soft-0-100-0.wbo	MOPT f=1 TT=55.155 T1=52.09	MSAT (MO) f=10578 TT=81.71 T1=81.2	MSAT (TO) f=5 TT=1800.47 T1=1725.04	OPTIMUM f=1 TT=55.155 T1=52.09	OPTIMUM f=1 TT=147.588 T1=147.47
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-cache.inv14.ucl--soft-0-100-0.wbo	MSAT f=1 TT=1800.06 T1=236.28	MSAT f=17977 TT=131.627 T1=131.37	MSAT (TO) f=9 TT=1800.79 T1=1619.82	MSAT (TO) f=1 TT=1800.06 T1=236.28	? (TO) TT=1800.05
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-dlx1c.rwmem1.ucl--soft-0-100-0.wbo	MOPT f=1 TT=4.523 T1=2.28	OPTIMUM f=1 TT=191.296 T1=189.91	OPTIMUM f=1 TT=6.794 T1=4.52	OPTIMUM f=1 TT=4.523 T1=2.28	OPTIMUM f=1 TT=7.326 T1=7.28
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-dlx1c.rwmem.ucl--soft-0-100-0.wbo	MOPT f=1 TT=4.003 T1=2.23	MSAT f=55 TT=438.123 T1=437.63	OPTIMUM f=1 TT=12.342 T1=18.39	OPTIMUM f=1 TT=4.003 T1=2.23	OPTIMUM f=1 TT=11.024 T1=10.97
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-elf.rf10.ucl--soft-0-100-0.wbo	MOPT f=1 TT=10.072 T1=4.11	? (MO) TT=156.984	OPTIMUM f=1 TT=71.403 T1=43.28	OPTIMUM f=1 TT=10.072 T1=4.11	OPTIMUM f=1 TT=95.818 T1=95.61
Bench name	Best results	NaPS 1.02 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-elf.rf9.ucl--soft-0-100-0.wbo	MOPT f=1 TT=3.832 T1=1.63	MSAT f=45717 TT=79.283 T1=78.72	OPTIMUM f=1 TT=7.384 T1=3.46	OPTIMUM f=1 TT=3.832 T1=1.63	OPTIMUM f=1 TT=12.126 T1=12.01
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.burch_dill.8.accl.ucl--soft-0-100-0.wbo	MOPT f=1 TT=267.41 T1=5.24	? (MO) TT=86.642	OPTIMUM f=1 TT=392.765 T1=7.08	OPTIMUM f=1 TT=267.41 T1=5.24	? (TO) TT=1800.24
/SOFT-BIGINT-LIN/PB06/web/uclid_pb_benchmarks/ normalized-ooo.ex.br.mem.RobRegValid_bar.ucl--soft-0-100-0.wbo	MOPT f=1 TT=56.675 T1=34.92	MSAT f=119899 TT=170.171 T1=169.72	OPTIMUM f=1 TT=865.401 T1=766.07	OPTIMUM f=1 TT=56.675 T1=34.92	OPTIMUM f=1 TT=406.857 T1=406.68
/SOFT-BIGINT-LIN/PB06/web/uclid_pb_benchmarks/ normalized-ooo.ex.br.mem.Src1Valid_Src1ValidBar.ucl--soft-0-100-0.wbo	MOPT f=1 TT=612.98 T1=163.23	MSAT f=212983 TT=245.369 T1=245.12	MSAT (TO) f=112 TT=1800.94 T1=63.94	OPTIMUM f=1 TT=612.98 T1=163.23	? (TO) TT=1800.21
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.ex.mem.LdValue.ucl--soft-0-100-0.wbo	MOPT f=1 TT=666.8 T1=135.57	MSAT f=147423 TT=259.273 T1=258.99	MSAT (TO) f=162 TT=1801.58 T1=6.87	OPTIMUM f=1 TT=666.8 T1=135.57	? (TO) TT=1800.17
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.rf10.ucl--soft-0-100-0.wbo	MOPT f=1 TT=20.59 T1=5.71	MSAT (MO) f=36208 TT=85.168 T1=84.6	OPTIMUM f=1 TT=75.953 T1=48.17	OPTIMUM f=1 TT=20.59 T1=5.71	? (TO) TT=1800.11
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.rf9.ucl--soft-0-100-0.wbo	MOPT f=1 TT=6.513 T1=1.76	MSAT f=273 TT=257.704 T1=257.22	OPTIMUM f=1 TT=11.212 T1=3.7	OPTIMUM f=1 TT=6.513 T1=1.76	OPTIMUM f=1 TT=72.88 T1=72.81
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.tag10.ucl--soft-0-100-0.wbo	MOPT f=1 TT=5.063 T1=3.08	MSAT f=293 TT=243.267 T1=242.71	OPTIMUM f=1 TT=10.753 T1=16.3	OPTIMUM f=1 TT=5.063 T1=3.08	OPTIMUM f=1 TT=95.278 T1=95.23
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.tag12.ucl--soft-0-100-0.wbo	MOPT f=1 TT=16.814 T1=4.73	MSAT f=22583 TT=76.946 T1=76.38	OPTIMUM f=1 TT=119.195 T1=17.14	OPTIMUM f=1 TT=16.814 T1=4.73	OPTIMUM f=1 TT=258.571 T1=258.48
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.tag14.ucl--soft-0-100-0.wbo	MOPT f=1 TT=46.428 T1=22.84	MSAT (MO) f=32164 TT=113.499 T1=113.01	OPTIMUM f=1 TT=266.276 T1=23.93	OPTIMUM f=1 TT=46.428 T1=22.84	OPTIMUM f=1 TT=1526.29 T1=1526.28
Bench name	Best results	NaPS 1.02 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
/SOFT-BIGINT-LIN/PB06/web/ uclid_pb_benchmarks/ normalized-ooo.unbounded.all.ucl--soft-0-100-0.wbo	MOPT f=1 TT=292.738 T1=103.66	MSAT f=144358 TT=297.532 T1=297.1	MSAT (TO) f=129 TT=1800.3 T1=1750.34	OPTIMUM f=1 TT=292.738 T1=103.66	? (TO) TT=1800.2
/SOFT-BIGINT-LIN/PB10/oliveras/ j120/ normalized-j12013_1-unsat--soft-0-100-0.wbo	MSAT f=3541 TT=1800.03 T1=1795.24	MSAT f=21722 TT=98.276 T1=97.75	MSAT (TO) f=4943 TT=1800.92 T1=899.73	MSAT (TO) f=3541 TT=1800.03 T1=1795.24	? (TO) TT=1800.22
/SOFT-BIGINT-LIN/PB10/oliveras/ j120/ normalized-j12015_3-unsat--soft-0-100-0.wbo	MOPT f=3 TT=40.542 T1=40.44	MSAT (MO) f=23029 TT=93.31 T1=92.8	MSAT (TO) f=730 TT=1800.58 T1=940.33	OPTIMUM f=3 TT=379.8 T1=375.84	OPTIMUM f=3 TT=40.542 T1=40.44
/SOFT-BIGINT-LIN/PB10/oliveras/ j120/ normalized-j12023_3-unsat--soft-0-100-0.wbo	MOPT f=7 TT=2.863 T1=1.29	MSAT (MO) f=28617 TT=86.015 T1=85.5	OPTIMUM f=7 TT=5.423 T1=1.78	OPTIMUM f=7 TT=2.863 T1=1.29	OPTIMUM f=7 TT=10.33 T1=10.13
/SOFT-BIGINT-LIN/PB10/oliveras/ j120/ normalized-j12026_5-unsat--soft-0-100-0.wbo	MSAT f=4417 TT=1800.15 T1=899.74	MSAT f=28950 TT=112.416 T1=111.94	MSAT (TO) f=4417 TT=1800.15 T1=899.74	MSAT (TO) f=5991 TT=1800.04 T1=1793.05	? (TO) TT=1800.15
/SOFT-BIGINT-LIN/PB10/oliveras/ j120/ normalized-j12027_7-unsat--soft-0-100-0.wbo	MSAT f=3968 TT=1800.06 T1=899.13	MSAT f=21065 TT=89.362 T1=88.83	MSAT (TO) f=3968 TT=1800.82 T1=899.13	MSAT (TO) f=3968 TT=1800.06 T1=1794.34	? (TO) TT=1800.1
/SOFT-BIGINT-LIN/PB10/oliveras/ j120/ normalized-j12041_4-unsat--soft-0-100-0.wbo	MOPT f=1 TT=627.581 T1=621.87	MSAT f=25015 TT=93.875 T1=93.34	MSAT (TO) f=14 TT=1800.55 T1=1651.52	OPTIMUM f=1 TT=627.581 T1=621.87	? (TO) TT=1800.06
/SOFT-BIGINT-LIN/PB10/oliveras/ j120/ normalized-j1205_1-unsat--soft-0-100-0.wbo	MOPT f=3 TT=2.725 T1=1.19	MSAT (MO) f=30020 TT=81.418 T1=80.9	OPTIMUM f=3 TT=5.115 T1=1.71	OPTIMUM f=3 TT=2.725 T1=1.19	OPTIMUM f=3 TT=9.97 T1=9.77
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j6010_2-unsat--soft-0-100-0.wbo	MOPT f=9 TT=2.024 T1=0.78	OPTIMUM f=9 TT=179.664 T1=178.27	OPTIMUM f=9 TT=3.972 T1=1.4	OPTIMUM f=9 TT=2.024 T1=0.78	OPTIMUM f=9 TT=8.826 T1=8.71
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j6027_4-unsat--soft-0-100-0.wbo	MOPT f=6 TT=1.983 T1=0.78	OPTIMUM f=6 TT=156.272 T1=155.04	OPTIMUM f=6 TT=4.054 T1=1.42	OPTIMUM f=6 TT=1.983 T1=0.78	OPTIMUM f=6 TT=8.942 T1=8.82
Bench name	Best results	NaPS 1.02 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j6030_9-unsat--soft-0-100-0.wbo	MOPT f=1 TT=2.484 T1=0.94	MSAT f=11117 TT=117.14 T1=116.51	OPTIMUM f=1 TT=5.012 T1=1.62	OPTIMUM f=1 TT=2.484 T1=0.94	OPTIMUM f=1 TT=16.247 T1=16.15
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j603_2-unsat--soft-0-100-0.wbo	MOPT f=4 TT=1.796 T1=0.76	MSAT f=20 TT=251.872 T1=251.38	OPTIMUM f=4 TT=3.533 T1=1.33	OPTIMUM f=4 TT=1.796 T1=0.76	OPTIMUM f=4 TT=2.717 T1=2.64
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j6033_2-unsat--soft-0-100-0.wbo	MOPT f=20 TT=2.193 T1=0.86	MSAT f=12883 TT=110.143 T1=109.59	OPTIMUM f=20 TT=4.413 T1=1.48	OPTIMUM f=20 TT=2.193 T1=0.86	OPTIMUM f=20 TT=3.87 T1=3.76
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j605_10-unsat--soft-0-100-0.wbo	MSAT f=200 TT=1800.16 T1=911.43	MSAT f=5057 TT=238.663 T1=238.17	MSAT (TO) f=200 TT=1800.16 T1=911.43	MSAT (TO) f=7470 TT=1800.07 T1=1788.83	? (TO) TT=1800.02
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j605_1-unsat--soft-0-100-0.wbo	MOPT f=1 TT=97.83 T1=94.09	MSAT f=5435 TT=228.873 T1=228.36	OPTIMUM f=1 TT=1026.16 T1=164.33	OPTIMUM f=1 TT=97.83 T1=94.09	OPTIMUM f=1 TT=561.63 T1=561.78
/SOFT-BIGINT-LIN/PB10/ oliveras/j60/ normalized-j606_1-unsat--soft-0-100-0.wbo	MOPT f=13 TT=2.094 T1=0.82	OPTIMUM f=13 TT=178.96 T1=177.68	OPTIMUM f=13 TT=4.393 T1=1.37	OPTIMUM f=13 TT=2.094 T1=0.82	OPTIMUM f=13 TT=4.32 T1=4.23
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j9010_6-unsat--soft-0-100-0.wbo	MOPT f=5 TT=2.543 T1=1.05	MSAT (MO) f=15540 TT=89.551 T1=89	OPTIMUM f=5 TT=5.163 T1=1.62	OPTIMUM f=5 TT=2.543 T1=1.05	OPTIMUM f=5 TT=25.446 T1=25.29
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j9012_2-unsat--soft-0-100-0.wbo	MOPT f=9 TT=2.283 T1=0.92	MSAT (MO) f=16579 TT=118.718 T1=118.2	OPTIMUM f=9 TT=4.253 T1=1.62	OPTIMUM f=9 TT=2.283 T1=0.92	OPTIMUM f=9 TT=38.503 T1=38.37
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j9014_1-unsat--soft-0-100-0.wbo	MOPT f=2 TT=2.942 T1=1.29	MSAT (MO) f=14780 TT=96.537 T1=96	OPTIMUM f=2 TT=7.122 T1=3.5	OPTIMUM f=2 TT=2.942 T1=1.29	OPTIMUM f=2 TT=41.704 T1=41.6
/SOFT-BIGINT-LIN/PB10/oliveras/ j90/ normalized-j9015_10-unsat--soft-0-100-0.wbo	MOPT f=11 TT=2.523 T1=1.02	MSAT (MO) f=15901 TT=102.536 T1=102	OPTIMUM f=11 TT=4.803 T1=1.66	OPTIMUM f=11 TT=2.523 T1=1.02	OPTIMUM f=11 TT=45.439 T1=45.31
Bench name	Best results	NaPS 1.02 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j9028_2-unsat--soft-0-100-0.wbo	MOPT f=8 TT=2.384 T1=0.97	MSAT (MO) TT=100.209	OPTIMUM f=8 TT=4.395 T1=1.66	OPTIMUM f=8 TT=2.384 T1=0.97	OPTIMUM f=8 TT=35.444 T1=35.35
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j903_10-unsat--soft-0-100-0.wbo	MOPT f=8 TT=2.094 T1=0.83	MSAT f=43 TT=231.283 T1=230.78	OPTIMUM f=8 TT=4.183 T1=1.43	OPTIMUM f=8 TT=2.094 T1=0.83	OPTIMUM f=8 TT=4.779 T1=4.69
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j9035_1-unsat--soft-0-100-0.wbo	MOPT f=8 TT=2.713 T1=1.04	? (MO) TT=80.223	OPTIMUM f=8 TT=4.993 T1=1.61	OPTIMUM f=8 TT=2.713 T1=1.04	OPTIMUM f=8 TT=7.425 T1=7.28
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j9039_3-unsat--soft-0-100-0.wbo	MOPT f=23 TT=2.574 T1=1.07	MSAT (MO) f=18546 TT=87.061 T1=86.5	OPTIMUM f=23 TT=4.882 T1=1.61	OPTIMUM f=23 TT=2.574 T1=1.07	OPTIMUM f=23 TT=12.927 T1=12.8
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j9048_1-unsat--soft-0-100-0.wbo	MOPT f=18 TT=2.494 T1=1.06	MSAT (MO) f=17735 TT=92.65 T1=92.1	OPTIMUM f=18 TT=4.903 T1=1.73	OPTIMUM f=18 TT=2.494 T1=1.06	OPTIMUM f=18 TT=75.425 T1=75.28
/SOFT-BIGINT-LIN/PB10/ oliveras/j90/ normalized-j907_8-unsat--soft-0-100-0.wbo	MOPT f=11 TT=2.124 T1=0.88	OPTIMUM f=11 TT=295.009 T1=292.57	OPTIMUM f=11 TT=4.215 T1=1.41	OPTIMUM f=11 TT=2.124 T1=0.88	OPTIMUM f=11 TT=11.775 T1=11.69

Statistics		NaPS 1.02 (complete)	Sat4j PB 2.3.6 Res+CP PB16 (complete)	Sat4j PB 2.3.6 Resolution PB16 (complete)	toysat 2016-05-02 (complete)
Number of times the solver is able to give the best known answer		6	36	44	34
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		0	3	42	2