PB'09 competition: results by benchmark

Results by benchmark for category optimisation, small integers, linear constraints (OPT-SMALLINT-LIN), subcategory Handmade, subsubcategory Queens Problems

This page displays the results of the different solvers for each benchmark for category optimisation, small integers, linear constraints (OPT-SMALLINT-LIN), subcategory Handmade, subsubcategory Queens Problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell example	Meaning
Answer	Solver result
f=...	value of the objective function for the model reported by the solver
TT=...	Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out

Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

Abbreviation	Meaning
f=...	Value of the objective function
TO	Time Out
MO	Mem. Out (out of memory)

Meaning of the different colors:

Color	Meaning
text	the solver cannot handle this instance
text	the solver gave no answer
text	the solver could give an answer (SAT)
text	the solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
text	the solver performed better than the other ones on that instance (complete solvers point of view)
text	the solver performed better than the other ones on that instance (incomplete solvers point of view)
text	the solver was ended by a signal or other problem
text	the solver gave an incomplete answer
text	the solver gave a wrong answer

For better readability, you may choose to hide some solvers:
bsolo 3.1 (complete)
bsolo 3.1 cl (complete)
bsolo 3.1 pb (complete)
pbclasp 2009-04-24 (complete)
SAT4J Pseudo CP 2.1.1 (complete)
SAT4J Pseudo Resolution 2.1.1 (complete)
SCIPclp SCIP 1.1.0.7 with CLP 1.8.2 (complete)
SCIPspx SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009) (complete)
wbo 1.0 (complete)

Bench name	Best results on this instance	bsolo 3.1 (complete)	bsolo 3.1 cl (complete)	bsolo 3.1 pb (complete)	pbclasp 2009-04-24 (complete)	SAT4J Pseudo CP 2.1.1 (complete)	SAT4J Pseudo Resolution 2.1.1 (complete)	SCIPclp SCIP 1.1.0.7 with CLP 1.8.2 (complete)	SCIPspx SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009) (complete)	wbo 1.0 (complete)
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_0.opb.PB06.opb	SAT f=5483 TT=1364.87 T1=234.14	SAT f=5860 TT=1798.24 T1=1523.8	SAT (MO) f=5483 TT=1364.87 T1=234.14	SAT (MO) f=5624 TT=1757.37 T1=29.72	SAT (TO) f=6226 TT=1800 T1=1708.01	? (TO) TT=1800.41	? (TO) TT=1800.39	? (TO) TT=1802.3	? (TO) TT=1802.33	? (MO) TT=1724.24
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_10.opb.PB06.opb	SAT f=5589 TT=1603.78 T1=571.48	SAT f=5737 TT=1798.15 T1=1460.43	SAT (MO) f=5589 TT=1603.78 T1=571.48	SAT (MO) f=5670 TT=1432.81 T1=29.71	SAT (TO) f=5973 TT=1800.09 T1=1158.87	? (TO) TT=1800.42	? (TO) TT=1800.36	? (TO) TT=1802.31	? (TO) TT=1802.3	? (MO) TT=1346.94
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_11.opb.PB06.opb	SAT f=5461 TT=1178.47 T1=42.79	SAT f=5905 TT=1798.54 T1=208.2	SAT (MO) f=5461 TT=1178.47 T1=42.79	SAT (MO) f=6252 TT=1110.33 T1=30.14	SAT (TO) f=6131 TT=1800.01 T1=1342.62	? (TO) TT=1800.4	? (TO) TT=1800.46	? (TO) TT=1802.28	? (TO) TT=1802.28	? (MO) TT=1329.55
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_12.opb.PB06.opb	SAT f=5676 TT=1470.62 T1=662.2	SAT f=5746 TT=1798.43 T1=332	SAT (MO) f=5676 TT=1470.62 T1=662.2	SAT (MO) f=6046 TT=1414.63 T1=29.69	SAT (TO) f=6232 TT=1800.1 T1=1788.69	? (TO) TT=1800.37	? (TO) TT=1800.47	? (TO) TT=1802.3	? (TO) TT=1802.26	? (MO) TT=1372.82
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_13.opb.PB06.opb	SAT f=5596 TT=1240.97 T1=366.88	SAT f=5473 TT=1798.19 T1=222.32	SAT (MO) f=5596 TT=1240.97 T1=366.88	SAT (MO) f=6378 TT=1392.13 T1=34.74	SAT (TO) f=6262 TT=1800.05 T1=1160.91	? (TO) TT=1800.43	? (TO) TT=1800.43	? (TO) TT=1802.24	? (TO) TT=1802.31	? (TO) TT=1800.33
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_14.opb.PB06.opb	SAT f=5623 TT=1190.7 T1=1138.79	SAT f=5972 TT=1798.27 T1=267.56	SAT (MO) f=5623 TT=1190.7 T1=1138.79	SAT (MO) f=6726 TT=1555.25 T1=194.81	SAT (TO) f=6228 TT=1800.01 T1=285.73	? (TO) TT=1800.48	? (TO) TT=1800.36	? (TO) TT=1802.29	? (TO) TT=1802.27	? (MO) TT=1437.15
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_1.opb.PB06.opb	SAT f=5450 TT=1635.75 T1=616.78	SAT f=5772 TT=1798.29 T1=1477.48	SAT (MO) f=5450 TT=1635.75 T1=616.78	SAT (MO) f=6144 TT=1343.57 T1=29.23	SAT (TO) f=5908 TT=1800.07 T1=1556.8	? (TO) TT=1800.44	? (TO) TT=1800.47	? (TO) TT=1802.29	? (TO) TT=1802.31	? (MO) TT=1354.2
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_2.opb.PB06.opb	SAT f=5657 TT=1383.63 T1=899.33	SAT f=5773 TT=1798.31 T1=514.99	SAT (MO) f=5657 TT=1383.63 T1=899.33	SAT (MO) f=5798 TT=1312.53 T1=29.71	SAT (TO) f=6009 TT=1800.02 T1=969.14	? (TO) TT=1800.47	? (TO) TT=1800.42	? (TO) TT=1802.28	? (TO) TT=1802.31	? (MO) TT=1644.15
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_3.opb.PB06.opb	SAT f=5789 TT=1798.32 T1=1561.36	SAT f=5867 TT=1798.17 T1=1676.55	SAT f=5789 TT=1798.32 T1=1561.36	SAT (MO) f=6278 TT=1350.21 T1=32.5	SAT (TO) f=6464 TT=1800 T1=1424.46	? (TO) TT=1800.4	? (TO) TT=1800.58	? (TO) TT=1802.27	? (TO) TT=1802.28	? (MO) TT=1363.59
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_4.opb.PB06.opb	SAT f=5248 TT=1292.98 T1=166.35	SAT f=5688 TT=1798.32 T1=553.89	SAT (MO) f=5248 TT=1292.98 T1=166.35	SAT (MO) f=6227 TT=1269.38 T1=30.66	SAT (TO) f=5951 TT=1800.01 T1=1721.37	? (TO) TT=1800.44	? (TO) TT=1800.47	? (TO) TT=1802.28	? (TO) TT=1802.29	? (TO) TT=1800.28
Bench name	Best results	bsolo 3.1 (complete)	bsolo 3.1 cl (complete)	bsolo 3.1 pb (complete)	pbclasp 2009-04-24 (complete)	SAT4J Pseudo CP 2.1.1 (complete)	SAT4J Pseudo Resolution 2.1.1 (complete)	SCIPclp SCIP 1.1.0.7 with CLP 1.8.2 (complete)	SCIPspx SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009) (complete)	wbo 1.0 (complete)
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_5.opb.PB06.opb	SAT f=5675 TT=1449.29 T1=1290.34	SAT f=5607 TT=1798.3 T1=97.28	SAT (MO) f=5675 TT=1449.29 T1=1290.34	SAT (MO) f=5956 TT=1231.07 T1=29.71	SAT (TO) f=6052 TT=1800.04 T1=1667.51	? (TO) TT=1800.44	? (TO) TT=1800.45	? (TO) TT=1802.24	? (TO) TT=1802.27	? (TO) TT=1800.35
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_6.opb.PB06.opb	SAT f=5701 TT=1531.68 T1=1136.76	SAT f=5613 TT=1798.15 T1=1462.28	SAT (MO) f=5701 TT=1531.68 T1=1136.76	SAT (MO) f=5821 TT=1473.46 T1=29.73	SAT (TO) f=6339 TT=1800.03 T1=1591.34	? (TO) TT=1800.42	? (TO) TT=1800.41	? (TO) TT=1802.26	? (TO) TT=1802.27	? (TO) TT=1800.34
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_7.opb.PB06.opb	SAT f=5502 TT=1336.92 T1=195.7	SAT f=5676 TT=1798.2 T1=1352.51	SAT (MO) f=5502 TT=1336.92 T1=195.7	SAT (MO) f=5632 TT=1346.01 T1=30.07	SAT (TO) f=6380 TT=1800.01 T1=1398.23	? (TO) TT=1800.48	? (TO) TT=1800.49	? (TO) TT=1802.32	? (TO) TT=1802.26	? (TO) TT=1800.38
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_8.opb.PB06.opb	SAT f=5597 TT=1449.52 T1=1034.53	SAT f=5586 TT=1798.38 T1=258.37	SAT (MO) f=5597 TT=1449.52 T1=1034.53	SAT (MO) f=6228 TT=1394.85 T1=30.12	SAT (TO) f=6258 TT=1800.1 T1=1193.99	? (TO) TT=1800.39	? (TO) TT=1800.45	? (TO) TT=1802.25	? (TO) TT=1802.31	? (MO) TT=1559.35
normalized-PB06/OPT-SMALLINT/ submitted-PB06/liu/wnq/ normalized-wnq_n100_mw99_9.opb.PB06.opb	SAT f=5514 TT=1600.95 T1=972.1	SAT f=5778 TT=1798.19 T1=1264.72	SAT (MO) f=5514 TT=1600.95 T1=972.1	SAT (MO) f=5784 TT=1660.7 T1=30.57	SAT (TO) f=5986 TT=1800.08 T1=1623.01	? (TO) TT=1800.42	? (TO) TT=1800.45	? (TO) TT=1802.32	? (TO) TT=1802.28	? (MO) TT=1293.51

Statistics		bsolo 3.1 (complete)	bsolo 3.1 cl (complete)	bsolo 3.1 pb (complete)	pbclasp 2009-04-24 (complete)	SAT4J Pseudo CP 2.1.1 (complete)	SAT4J Pseudo Resolution 2.1.1 (complete)	SCIPclp SCIP 1.1.0.7 with CLP 1.8.2 (complete)	SCIPspx SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009) (complete)	wbo 1.0 (complete)
Number of times the solver is able to give the best known answer		0	15	0	0	0	0	0	0	0
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)		0	15	0	0	0	0	0	0	0