PB'24 competition: ranking of solvers

Ranking on final answers (complete solver point of view)

This section presents the ranking of the different solvers. Solvers are ranked on the number of solved instances, i.e. where a final answer was given. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "decision problem, linear constraints, no UNSAT certificate" (DEC-LIN): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 397
Virtual Best Solver (VBS)			362	147 SAT, 215 UNSAT	91%	100%	44497.46	0.28	122.92	-	-	-	-
1	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	312	126 SAT, 186 UNSAT	79%	86%	100811.90	7.67	323.12	101115.31	7.70	324.09	1.00
2	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	309	123 SAT, 186 UNSAT	78%	85%	103032.17	9.07	333.44	103311.95	9.09	334.34	1.00
3	Exact_veripb2	2024-07-05	291	108 SAT, 8 UNSAT, 175 UNSATC	73%	80%	50490.50	1.07	173.51	51748.63	1.46	177.83	0.98
4	RoundingSat	log (3)	288	99 SAT, 189 UNSATC	73%	80%	65214.06	1.11	226.44	65491.20	1.65	227.40	1.00
5	Exact	2024-07-05	286	102 SAT, 184 UNSAT	72%	79%	46106.69	0.73	161.21	46238.86	0.73	161.67	1.00
6	Exact_veripb2_no_SoPlex	2024-07-05	284	112 SAT, 7 UNSAT, 165 UNSATC	72%	78%	47724.15	1.24	168.04	48787.69	1.26	171.79	0.98
7	RoundingSat	nolog (2)	282	98 SAT, 184 UNSAT	71%	78%	55497.78	0.75	196.80	55647.12	0.76	197.33	1.00
8	Exact_no_SoPlex	2024-07-05	275	104 SAT, 171 UNSAT	69%	76%	41618.15	1.05	151.34	41734.45	1.06	151.76	1.00
9	CASHWMaxSAT-DisjCad-S	2024-07-10	273	124 SAT, 149 UNSAT	69%	75%	36208.04	3.07	132.63	36313.36	3.09	133.02	1.00
10	NaPS	1.03	273	122 SAT, 151 UNSAT	69%	75%	38754.87	4.34	141.96	38872.29	4.36	142.39	1.00
11	Picat	2024-06-18	273	123 SAT, 150 UNSAT	69%	75%	39385.08	7.30	144.27	39494.89	7.32	144.67	1.00
12	FiberSCIP	20 cores, 2024-08-12	270	101 SAT, 169 UNSAT	68%	75%	757612.17	7.57	2805.97	52620.84	0.72	194.89	14.40
13	CASHWMaxSAT-DisjCad-SP	2024-07-10	268	119 SAT, 149 UNSAT	68%	74%	31007.85	2.97	115.70	31101.81	2.99	116.05	1.00
14	CASHWMaxSAT-DisjCom-S	2024-07-10	267	113 SAT, 154 UNSAT	67%	74%	37537.14	3.08	140.59	37646.36	3.09	141.00	1.00
15	CASHWMaxSAT-DisjCom-SP	2024-07-10	267	113 SAT, 154 UNSAT	67%	74%	38682.14	3.19	144.88	38790.42	3.19	145.28	1.00
16	Sat4j Resolution Default	2024-07-07	246	104 SAT, 142 UNSAT	62%	68%	55274.18	4.48	224.69	54797.59	2.90	222.75	1.01
17	Sat4j Resolution 2024	2024-07-07	244	102 SAT, 1 UNSAT, 141 UNSATC	61%	67%	51902.91	5.01	212.72	51688.89	3.17	211.84	1.00
18	NaPS-GM	1.03	241	91 SAT, 150 UNSAT	61%	67%	28719.32	3.24	119.17	28807.21	3.25	119.53	1.00
19	SCIP	2024-07-07	241	80 SAT, 161 UNSAT	61%	67%	34871.39	0.73	144.69	34978.04	0.74	145.14	1.00
20	NaPS-PB16	1.02b5	236	86 SAT, 150 UNSAT	59%	65%	30594.34	3.04	129.64	30685.23	3.05	130.02	1.00
21	FiberSCIP	2024-07-30	235	76 SAT, 159 UNSAT	59%	65%	35677.52	0.70	151.82	35787.12	0.72	152.29	1.00
22	Sat4j Partial RoundingSAT 2024	2024-07-12	217	68 SAT, 149 UNSAT	55%	60%	39937.87	4.95	184.05	38499.50	2.66	177.42	1.04
23	Sat4j Cutting Planes 2024	2024-07-12	199	60 SAT, 139 UNSAT	50%	55%	35796.77	4.71	179.88	35078.60	2.56	176.27	1.02
24	ParLS-PBO-1.1	20 cores, 2024-07-09	134	124 SAT, 10 UNSAT	34%	37%	210615.13	1.68	1571.75	10680.52	0.14	79.71	19.72
25	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	130	130 SAT	33%	36%	40563.01	22.35	312.02	40680.99	22.44	312.93	1.00
26	ParLS-PBO-1.1	2024-07-09	122	112 SAT, 10 UNSAT	31%	34%	3503.07	0.42	28.71	3486.46	0.43	28.58	1.00
27	DLS-PBO	2024-07-09	121	111 SAT, 10 UNSAT	30%	33%	8091.23	1.13	66.87	8109.21	1.13	67.02	1.00
28	Hybrid-LSMabP+Exact	2024-07-11	117	117 SAT	29%	32%	30630.47	7.33	261.80	30698.24	7.35	262.38	1.00
29	Hybrid-LSMabSP+Exact	2024-07-11	115	115 SAT	29%	32%	40228.90	28.02	349.82	40338.32	28.10	350.77	1.00
30	LS-MabSP	2024-07-10	112	112 SAT	28%	31%	56548.45	26.43	504.90	56704.74	26.51	506.29	1.00
31	LS-MabS	2024-07-10	110	110 SAT	28%	30%	44996.95	7.99	409.06	45123.86	8.02	410.22	1.00
32	LS-MabP	2024-07-10	108	108 SAT	27%	30%	10033.58	1.79	92.90	10056.03	1.79	93.11	1.00
33	LS-Mab	2024-07-10	105	105 SAT	26%	29%	3595.46	0.61	34.24	3602.51	0.61	34.31	1.00

Category "decision problem, linear constraints, no UNSAT certificate" (DEC-LIN): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 397
Virtual Best Solver (VBS)			147	37%	100%	13971.37	0.60	95.04	-	-	-	-
1	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	130	33%	88%	40563.01	22.35	312.02	40680.99	22.44	312.93	1.00
2	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	126	32%	86%	19043.31	12.36	151.14	19107.82	12.39	151.65	1.00
3	CASHWMaxSAT-DisjCad-S	2024-07-10	124	31%	84%	22346.46	6.96	180.21	22408.94	6.98	180.72	1.00
4	ParLS-PBO-1.1	20 cores, 2024-07-09	124	31%	84%	210614.00	2.81	1698.50	10679.93	0.20	86.13	19.72
5	Picat	2024-06-18	123	31%	84%	19100.34	9.25	155.29	19154.73	9.27	155.73	1.00
6	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	123	31%	84%	30687.61	20.03	249.49	30776.44	20.09	250.21	1.00
7	NaPS	1.03	122	31%	83%	18885.04	8.21	154.80	18941.68	8.23	155.26	1.00
8	CASHWMaxSAT-DisjCad-SP	2024-07-10	119	30%	81%	15832.52	11.26	133.05	15881.78	11.29	133.46	1.00
9	Hybrid-LSMabP+Exact	2024-07-11	117	29%	80%	30630.47	7.33	261.80	30698.24	7.35	262.38	1.00
10	Hybrid-LSMabSP+Exact	2024-07-11	115	29%	78%	40228.90	28.02	349.82	40338.32	28.10	350.77	1.00
11	CASHWMaxSAT-DisjCom-S	2024-07-10	113	28%	77%	20618.22	12.16	182.46	20682.77	12.20	183.03	1.00
12	CASHWMaxSAT-DisjCom-SP	2024-07-10	113	28%	77%	22726.76	22.59	201.12	22792.17	22.66	201.70	1.00
13	ParLS-PBO-1.1	2024-07-09	112	28%	76%	3502.53	0.64	31.27	3485.89	0.65	31.12	1.00
14	Exact_veripb2_no_SoPlex	2024-07-05	112	28%	76%	24957.79	2.45	222.84	25563.11	2.46	228.24	0.98
15	LS-MabSP	2024-07-10	112	28%	76%	56548.45	26.43	504.90	56704.74	26.51	506.29	1.00
16	DLS-PBO	2024-07-09	111	28%	76%	8089.80	1.86	72.88	8107.76	1.88	73.04	1.00
17	LS-MabS	2024-07-10	110	28%	75%	44996.95	7.99	409.06	45123.86	8.02	410.22	1.00
18	LS-MabP	2024-07-10	108	27%	73%	10033.58	1.79	92.90	10056.03	1.79	93.11	1.00
19	Exact_veripb2	2024-07-05	108	27%	73%	24806.98	7.16	229.69	25315.62	7.19	234.40	0.98
20	LS-Mab	2024-07-10	105	26%	71%	3595.46	0.61	34.24	3602.51	0.61	34.31	1.00
21	Exact_no_SoPlex	2024-07-05	104	26%	71%	18423.25	1.95	177.15	18476.94	1.95	177.66	1.00
22	Sat4j Resolution Default	2024-07-07	104	26%	71%	22507.50	4.52	216.42	22270.25	2.90	214.14	1.01
23	Exact	2024-07-05	102	26%	69%	19694.36	5.68	193.08	19750.90	5.70	193.64	1.00
24	Sat4j Resolution 2024	2024-07-07	102	26%	69%	20811.63	4.98	204.04	20727.88	3.29	203.21	1.00
25	FiberSCIP	20 cores, 2024-08-12	101	25%	69%	383234.50	25.41	3794.40	28684.60	3.21	284.01	13.36
26	RoundingSat	log (3)	99	25%	67%	26929.53	11.30	272.02	27047.66	11.83	273.21	1.00
27	RoundingSat	nolog (2)	98	25%	67%	23927.53	13.85	244.16	23992.06	13.93	244.82	1.00
28	NaPS-GM	1.03	91	23%	62%	12635.67	7.83	138.85	12673.88	7.86	139.27	1.00
29	NaPS-PB16	1.02b5	86	22%	59%	14683.92	8.17	170.74	14727.44	8.19	171.25	1.00
30	SCIP	2024-07-07	80	20%	54%	14472.32	6.00	180.90	14516.54	6.02	181.46	1.00
31	FiberSCIP	2024-07-30	76	19%	52%	9677.90	3.09	127.34	9708.29	3.12	127.74	1.00
32	Sat4j Partial RoundingSAT 2024	2024-07-12	68	17%	46%	9336.70	12.64	137.30	8874.77	9.14	130.51	1.05
33	Sat4j Cutting Planes 2024	2024-07-12	60	15%	41%	11130.54	12.37	185.51	10895.77	8.30	181.60	1.02

Category "decision problem, linear constraints, no UNSAT certificate" (DEC-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 397
Virtual Best Solver (VBS)			215	215 UNSAT	54%	100%	30526.09	0.14	141.98	-	-	-	-
1	RoundingSat	log (3)	189	189 UNSATC	48%	88%	38284.53	0.52	202.56	38443.54	0.84	203.40	1.00
2	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	186	186 UNSAT	47%	87%	72344.56	4.64	388.95	72535.51	4.66	389.98	1.00
3	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	186	186 UNSAT	47%	87%	81768.58	4.37	439.62	82007.49	4.39	440.90	1.00
4	Exact	2024-07-05	184	184 UNSAT	46%	86%	26412.33	0.42	143.55	26487.96	0.43	143.96	1.00
5	RoundingSat	nolog (2)	184	184 UNSAT	46%	86%	31570.25	0.36	171.58	31655.06	0.37	172.04	1.00
6	Exact_veripb2	2024-07-05	183	8 UNSAT, 175 UNSATC	46%	85%	25683.52	0.48	140.35	26433.01	0.64	144.44	0.97
7	Exact_veripb2_no_SoPlex	2024-07-05	172	7 UNSAT, 165 UNSATC	43%	80%	22766.36	0.61	132.36	23224.58	0.76	135.03	0.98
8	Exact_no_SoPlex	2024-07-05	171	171 UNSAT	43%	80%	23194.90	0.81	135.64	23257.51	0.82	136.01	1.00
9	FiberSCIP	20 cores, 2024-08-12	169	169 UNSAT	43%	79%	374377.67	5.31	2215.25	23936.25	0.42	141.63	15.64
10	SCIP	2024-07-07	161	161 UNSAT	41%	75%	20399.06	0.50	126.70	20461.50	0.51	127.09	1.00
11	FiberSCIP	2024-07-30	159	159 UNSAT	40%	74%	25999.61	0.56	163.52	26078.82	0.57	164.02	1.00
12	CASHWMaxSAT-DisjCom-SP	2024-07-10	154	154 UNSAT	39%	72%	15955.38	1.64	103.61	15998.25	1.65	103.88	1.00
13	CASHWMaxSAT-DisjCom-S	2024-07-10	154	154 UNSAT	39%	72%	16918.92	1.53	109.86	16963.59	1.53	110.15	1.00
14	NaPS	1.03	151	151 UNSAT	38%	70%	19869.83	2.24	131.59	19930.62	2.25	131.99	1.00
15	NaPS-PB16	1.02b5	150	150 UNSAT	38%	70%	15910.42	2.52	106.07	15957.79	2.53	106.39	1.00
16	NaPS-GM	1.03	150	150 UNSAT	38%	70%	16083.65	2.53	107.22	16133.33	2.54	107.56	1.00
17	Picat	2024-06-18	150	150 UNSAT	38%	70%	20284.73	5.73	135.23	20340.16	5.75	135.60	1.00
18	CASHWMaxSAT-DisjCad-S	2024-07-10	149	149 UNSAT	38%	69%	13861.59	1.23	93.03	13904.43	1.24	93.32	1.00
19	CASHWMaxSAT-DisjCad-SP	2024-07-10	149	149 UNSAT	38%	69%	15175.33	1.44	101.85	15220.03	1.46	102.15	1.00
20	Sat4j Partial RoundingSAT 2024	2024-07-12	149	149 UNSAT	38%	69%	30601.17	3.09	205.38	29624.73	1.66	198.82	1.03
21	Sat4j Resolution 2024	2024-07-07	142	1 UNSAT, 141 UNSATC	36%	66%	31091.28	5.04	218.95	30961.01	3.06	218.04	1.00
22	Sat4j Resolution Default	2024-07-07	142	142 UNSAT	36%	66%	32766.68	4.43	230.75	32527.34	2.93	229.07	1.01
23	Sat4j Cutting Planes 2024	2024-07-12	139	139 UNSAT	35%	65%	24666.23	3.29	177.45	24182.82	1.82	173.98	1.02
24	ParLS-PBO-1.1	2024-07-09	10	10 UNSAT	3%	5%	0.55	0.06	0.05	0.56	0.06	0.06	0.98
25	ParLS-PBO-1.1	20 cores, 2024-07-09	10	10 UNSAT	3%	5%	1.13	0.12	0.11	0.59	0.07	0.06	1.92
26	DLS-PBO	2024-07-09	10	10 UNSAT	3%	5%	1.43	0.15	0.14	1.46	0.15	0.15	0.98
27	LS-MabSP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
28	LS-MabS	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
29	LS-MabP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
30	LS-Mab	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
31	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
32	Hybrid-LSMabSP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
33	Hybrid-LSMabP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "decision problem, non-linear constraints, no UNSAT certificate" (DEC-NLC): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 10
Virtual Best Solver (VBS)			10	8 SAT, 2 UNSAT	100%	100%	7.99	0.10	0.80	-	-	-	-
1	FiberSCIP	20 cores, 2024-08-12	10	8 SAT, 2 UNSAT	100%	100%	25.60	1.52	2.56	3.51	0.25	0.35	7.30
2	FiberSCIP	2024-07-30	9	7 SAT, 2 UNSAT	90%	90%	2.12	0.17	0.24	2.18	0.17	0.24	0.98
3	SCIP	2024-07-07	9	7 SAT, 2 UNSAT	90%	90%	39.41	0.15	4.38	39.64	0.16	4.40	0.99
4	Picat	2024-06-18	9	7 SAT, 2 UNSAT	90%	90%	43.67	0.47	4.85	43.86	0.47	4.87	1.00
5	Exact	2024-07-05	9	7 SAT, 2 UNSAT	90%	90%	49.06	0.10	5.45	49.24	0.11	5.47	1.00
6	Exact_no_SoPlex	2024-07-05	9	7 SAT, 2 UNSAT	90%	90%	83.18	0.38	9.24	83.48	0.38	9.28	1.00
7	Sat4j Partial RoundingSAT 2024	2024-07-12	9	7 SAT, 2 UNSAT	90%	90%	2258.51	3.52	250.95	2216.83	2.52	246.31	1.02
8	Sat4j Resolution Default	2024-07-07	3	1 SAT, 2 UNSAT	30%	30%	430.78	0.30	143.59	426.09	0.70	142.03	1.01
9	Sat4j Cutting Planes 2024	2024-07-12	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "decision problem, non-linear constraints, no UNSAT certificate" (DEC-NLC): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 10
Virtual Best Solver (VBS)			8	80%	100%	7.99	0.13	1.00	-	-	-	-
1	FiberSCIP	20 cores, 2024-08-12	8	80%	100%	24.64	2.65	3.08	3.35	0.38	0.42	7.35
2	FiberSCIP	2024-07-30	7	70%	88%	2.08	0.17	0.30	2.13	0.18	0.30	0.97
3	SCIP	2024-07-07	7	70%	88%	39.36	0.23	5.62	39.57	0.25	5.65	0.99
4	Picat	2024-06-18	7	70%	88%	43.65	0.71	6.24	43.84	0.71	6.26	1.00
5	Exact	2024-07-05	7	70%	88%	49.06	0.15	7.01	49.24	0.15	7.03	1.00
6	Exact_no_SoPlex	2024-07-05	7	70%	88%	83.17	1.06	11.88	83.47	1.07	11.92	1.00
7	Sat4j Partial RoundingSAT 2024	2024-07-12	7	70%	88%	2256.66	4.66	322.38	2215.56	2.74	316.51	1.02
8	Sat4j Resolution Default	2024-07-07	1	10%	13%	430.20	430.20	430.20	424.87	424.87	424.87	1.01
9	Sat4j Cutting Planes 2024	2024-07-12	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "decision problem, non-linear constraints, no UNSAT certificate" (DEC-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 10
Virtual Best Solver (VBS)			2	2 UNSAT	20%	100%	0.00	0.00	0.00	-	-	-	-
1	Exact	2024-07-05	2	2 UNSAT	20%	100%	0.00	0.00	0.00	0.00	0.00	0.00	0.91
2	Exact_no_SoPlex	2024-07-05	2	2 UNSAT	20%	100%	0.01	0.00	0.00	0.01	0.01	0.01	0.94
3	Picat	2024-06-18	2	2 UNSAT	20%	100%	0.02	0.01	0.01	0.02	0.01	0.01	0.74
4	FiberSCIP	2024-07-30	2	2 UNSAT	20%	100%	0.05	0.02	0.02	0.05	0.02	0.02	1.00
5	SCIP	2024-07-07	2	2 UNSAT	20%	100%	0.05	0.03	0.03	0.07	0.03	0.03	0.73
6	Sat4j Resolution Default	2024-07-07	2	2 UNSAT	20%	100%	0.58	0.29	0.29	1.22	0.61	0.61	0.48
7	FiberSCIP	20 cores, 2024-08-12	2	2 UNSAT	20%	100%	0.96	0.48	0.48	0.16	0.08	0.08	6.14
8	Sat4j Partial RoundingSAT 2024	2024-07-12	2	2 UNSAT	20%	100%	1.85	0.92	0.92	1.26	0.63	0.63	1.47
9	Sat4j Cutting Planes 2024	2024-07-12	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimization problem, linear constraints, no OPT/UNSAT certificate" (OPT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 478
Virtual Best Solver (VBS)			339	320 OPT, 19 UNSAT	71%	100%	228167.77	0.83	673.06	-	-	-	-
1	mixed-bag	2024-07-19	279	266 OPT, 13 UNSAT	58%	82%	35632.58	1.62	127.72	35756.13	1.62	128.16	1.00
2	SCIP	2024-07-07	263	248 OPT, 15 UNSAT	55%	78%	54056.12	1.25	205.54	54231.92	1.27	206.21	1.00
3	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	259	245 OPT, 14 UNSAT	54%	76%	118078.15	30.97	455.90	118452.58	31.05	457.35	1.00
4	IPBHS-GUROBI	2024-07-09	257	244 OPT, 13 UNSAT	54%	76%	46876.60	1.25	182.40	46966.35	1.27	182.75	1.00
5	RoundingSat	nolog (2)	256	243 OPT, 13 UNSAT	54%	76%	38128.55	1.31	148.94	38238.75	1.32	149.37	1.00
6	RoundingSat	log (3)	254	23 OPT, 218 OPTC, 13 UNSATC	53%	75%	48324.17	1.57	190.25	48953.61	2.28	192.73	0.99
7	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	254	241 OPT, 13 UNSAT	53%	75%	124959.95	30.55	491.97	125352.37	30.66	493.51	1.00
8	Exact	2024-07-05	252	238 OPT, 14 UNSAT	53%	74%	30596.03	2.08	121.41	30682.71	2.09	121.76	1.00
9	FiberSCIP	2024-07-30	245	230 OPT, 15 UNSAT	51%	72%	36787.71	0.86	150.15	36914.52	0.87	150.67	1.00
10	IPBHS-SCIP	2024-07-09	245	232 OPT, 13 UNSAT	51%	72%	36827.96	1.66	150.32	36937.41	1.67	150.76	1.00
11	FiberSCIP	20 cores, 2024-08-12	245	230 OPT, 15 UNSAT	51%	72%	418772.81	7.36	1709.28	25818.83	0.77	105.38	16.22
12	Exact_veripb2	2024-07-05	239	56 OPT, 169 OPTC, 1 UNSAT, 13 UNSATC	50%	71%	42700.07	1.45	178.66	45121.21	2.01	188.79	0.95
13	CASHWMaxSAT-DisjCad-SP	2024-07-10	235	222 OPT, 13 UNSAT	49%	69%	120975.89	18.28	514.79	121345.80	18.34	516.37	1.00
14	CASHWMaxSAT-DisjCad-S	2024-07-10	235	223 OPT, 12 UNSAT	49%	69%	140428.51	24.47	597.57	140848.80	24.55	599.36	1.00
15	CASHWMaxSAT-DisjCom-S	2024-07-10	232	220 OPT, 12 UNSAT	49%	68%	131631.98	17.06	567.38	132055.94	17.12	569.21	1.00
16	CASHWMaxSAT-DisjCom-SP	2024-07-10	231	218 OPT, 13 UNSAT	48%	68%	108042.58	10.40	467.72	108387.01	10.44	469.21	1.00
17	Exact_no_SoPlex	2024-07-05	226	213 OPT, 13 UNSAT	47%	67%	27536.46	2.22	121.84	27613.37	2.23	122.18	1.00
18	NaPS	1.03	208	196 OPT, 12 UNSAT	44%	61%	30843.63	8.19	148.29	30932.86	8.22	148.72	1.00
19	pb-oll-rs	2024-07-19	208	196 OPT, 12 UNSAT	44%	61%	31993.30	1.85	153.81	32082.98	1.86	154.25	1.00
20	NaPS-GM	1.03	204	191 OPT, 13 UNSAT	43%	60%	28779.42	5.52	141.08	28869.23	5.53	141.52	1.00
21	NaPS-PB16	1.02b5	203	190 OPT, 13 UNSAT	42%	60%	23216.77	4.61	114.37	23288.33	4.62	114.72	1.00
22	Picat	2024-06-18	201	187 OPT, 14 UNSAT	42%	59%	53407.18	14.17	265.71	53559.91	14.21	266.47	1.00
23	Exact_veripb2_no_SoPlex	2024-07-05	195	38 OPT, 144 OPTC, 1 UNSAT, 12 UNSATC	41%	58%	26458.25	1.87	135.68	27613.89	2.08	141.61	0.96
24	Sat4j Partial RoundingSAT 2024	2024-07-12	146	134 OPT, 12 UNSAT	31%	43%	38512.42	4.87	263.78	37792.20	2.57	258.85	1.02
25	Sat4j Cutting Planes 2024	2024-07-12	133	123 OPT, 10 UNSAT	28%	39%	23128.58	2.28	173.90	22603.29	1.31	169.95	1.02
26	Sat4j Resolution Default	2024-07-07	128	117 OPT, 11 UNSAT	27%	38%	13095.56	1.68	102.31	12879.64	0.96	100.62	1.02
27	Sat4j Resolution 2024	2024-07-07	126	2 OPT, 118 OPTC, 6 UNSATC	26%	37%	16952.14	2.71	134.54	16826.61	1.52	133.54	1.01
28	ParLS-PBO-1.1	2024-07-09	8	8 UNSAT	2%	2%	0.45	0.05	0.06	0.46	0.05	0.06	0.98
29	DLS-PBO	2024-07-09	8	8 UNSAT	2%	2%	0.48	0.05	0.06	0.49	0.05	0.06	0.99
30	ParLS-PBO-1.1	20 cores, 2024-07-09	8	8 UNSAT	2%	2%	0.56	0.08	0.07	0.32	0.05	0.04	1.74
31	LS-MabSP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
32	LS-MabS	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
33	LS-MabP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
34	LS-Mab	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
35	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	Hybrid-LSMabSP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	Hybrid-LSMabP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimization problem, linear constraints, no OPT/UNSAT certificate" (OPT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 478
Virtual Best Solver (VBS)			320	320 OPT	67%	100%	111677.99	1.02	348.99	-	-	-	-
1	mixed-bag	2024-07-19	266	266 OPT	56%	83%	35451.45	1.84	133.28	35574.39	1.85	133.74	1.00
2	SCIP	2024-07-07	248	248 OPT	52%	78%	51337.19	1.34	207.00	51503.42	1.35	207.68	1.00
3	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	245	245 OPT	51%	77%	113512.90	39.78	463.32	113872.81	39.88	464.79	1.00
4	IPBHS-GUROBI	2024-07-09	244	244 OPT	51%	76%	46743.16	1.44	191.57	46832.45	1.50	191.94	1.00
5	RoundingSat	nolog (2)	243	243 OPT	51%	76%	38028.09	1.69	156.49	38137.93	1.71	156.95	1.00
6	RoundingSat	log (3)	241	23 OPT, 218 OPTC	50%	75%	48149.24	1.93	199.79	48775.10	2.36	202.39	0.99
7	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	241	241 OPT	50%	75%	121235.02	33.51	503.05	121615.80	33.62	504.63	1.00
8	Exact	2024-07-05	238	238 OPT	50%	74%	30479.72	2.48	128.07	30566.03	2.49	128.43	1.00
9	IPBHS-SCIP	2024-07-09	232	232 OPT	49%	73%	36586.61	1.91	157.70	36695.47	1.91	158.17	1.00
10	FiberSCIP	2024-07-30	230	230 OPT	48%	72%	34959.75	0.89	152.00	35081.59	0.90	152.53	1.00
11	FiberSCIP	20 cores, 2024-08-12	230	230 OPT	48%	72%	371333.34	8.12	1614.49	23272.75	0.82	101.19	15.96
12	Exact_veripb2	2024-07-05	225	56 OPT, 169 OPTC	47%	70%	42640.02	1.49	189.51	45058.61	2.31	200.26	0.95
13	CASHWMaxSAT-DisjCad-S	2024-07-10	223	223 OPT	47%	70%	138534.31	31.22	621.23	138948.02	31.31	623.09	1.00
14	CASHWMaxSAT-DisjCad-SP	2024-07-10	222	222 OPT	46%	69%	117463.71	21.81	529.12	117820.81	21.88	530.72	1.00
15	CASHWMaxSAT-DisjCom-S	2024-07-10	220	220 OPT	46%	69%	127967.96	22.01	581.67	128378.49	22.09	583.54	1.00
16	CASHWMaxSAT-DisjCom-SP	2024-07-10	218	218 OPT	46%	68%	104264.13	15.53	478.28	104595.71	15.58	479.80	1.00
17	Exact_no_SoPlex	2024-07-05	213	213 OPT	45%	67%	27407.00	2.58	128.67	27483.48	2.62	129.03	1.00
18	NaPS	1.03	196	196 OPT	41%	61%	29704.44	8.51	151.55	29790.31	8.56	151.99	1.00
19	pb-oll-rs	2024-07-19	196	196 OPT	41%	61%	31698.70	2.30	161.73	31787.44	2.30	162.18	1.00
20	NaPS-GM	1.03	191	191 OPT	40%	60%	27439.49	5.68	143.66	27525.80	5.70	144.11	1.00
21	NaPS-PB16	1.02b5	190	190 OPT	40%	59%	23043.68	4.76	121.28	23114.67	4.78	121.66	1.00
22	Picat	2024-06-18	187	187 OPT	39%	58%	50567.56	15.41	270.41	50712.00	15.46	271.19	1.00
23	Exact_veripb2_no_SoPlex	2024-07-05	182	38 OPT, 144 OPTC	38%	57%	26333.83	2.28	144.69	27486.62	2.46	151.03	0.96
24	Sat4j Partial RoundingSAT 2024	2024-07-12	134	134 OPT	28%	42%	34711.28	5.85	259.04	34024.46	3.48	253.91	1.02
25	Sat4j Cutting Planes 2024	2024-07-12	123	123 OPT	26%	38%	22691.26	3.57	184.48	22175.64	1.98	180.29	1.02
26	Sat4j Resolution 2024	2024-07-07	120	2 OPT, 118 OPTC	25%	38%	16946.19	3.20	141.22	16822.88	1.73	140.19	1.01
27	Sat4j Resolution Default	2024-07-07	117	117 OPT	24%	37%	13071.79	1.85	111.72	12861.32	1.02	109.93	1.02
28	LS-MabSP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
29	LS-MabS	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
30	LS-MabP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
31	LS-Mab	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
32	DLS-PBO	2024-07-09	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
33	ParLS-PBO-1.1	2024-07-09	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
34	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
35	Hybrid-LSMabSP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	Hybrid-LSMabP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	ParLS-PBO-1.1	20 cores, 2024-07-09	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimization problem, linear constraints, no OPT/UNSAT certificate" (OPT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 478
Virtual Best Solver (VBS)			19	19 UNSAT	4%	100%	116489.78	0.16	6131.04	-	-	-	-
1	FiberSCIP	2024-07-30	15	15 UNSAT	3%	79%	1827.96	0.42	121.86	1832.93	0.42	122.20	1.00
2	SCIP	2024-07-07	15	15 UNSAT	3%	79%	2718.94	0.41	181.26	2728.50	0.42	181.90	1.00
3	FiberSCIP	20 cores, 2024-08-12	15	15 UNSAT	3%	79%	47439.47	0.18	3162.63	2546.08	0.18	169.74	18.63
4	Exact_veripb2	2024-07-05	14	1 UNSAT, 13 UNSATC	3%	74%	60.04	0.21	4.29	62.60	0.23	4.47	0.96
5	Exact	2024-07-05	14	14 UNSAT	3%	74%	116.30	0.06	8.31	116.68	0.06	8.33	1.00
6	Picat	2024-06-18	14	14 UNSAT	3%	74%	2839.62	0.38	202.83	2847.91	0.38	203.42	1.00
7	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	14	14 UNSAT	3%	74%	4565.25	0.54	326.09	4579.77	0.55	327.13	1.00
8	RoundingSat	nolog (2)	13	13 UNSAT	3%	68%	100.47	0.08	7.73	100.82	0.08	7.76	1.00
9	Exact_veripb2_no_SoPlex	2024-07-05	13	1 UNSAT, 12 UNSATC	3%	68%	124.42	0.09	9.57	127.27	0.11	9.79	0.98
10	Exact_no_SoPlex	2024-07-05	13	13 UNSAT	3%	68%	129.46	0.06	9.96	129.89	0.06	9.99	1.00
11	IPBHS-GUROBI	2024-07-09	13	13 UNSAT	3%	68%	133.44	0.22	10.26	133.90	0.22	10.30	1.00
12	NaPS-PB16	1.02b5	13	13 UNSAT	3%	68%	173.09	0.17	13.31	173.66	0.17	13.36	1.00
13	RoundingSat	log (3)	13	13 UNSATC	3%	68%	174.93	0.10	13.46	178.51	0.20	13.73	0.98
14	mixed-bag	2024-07-19	13	13 UNSAT	3%	68%	181.13	0.10	13.93	181.74	0.11	13.98	1.00
15	IPBHS-SCIP	2024-07-09	13	13 UNSAT	3%	68%	241.35	0.30	18.57	241.95	0.30	18.61	1.00
16	NaPS-GM	1.03	13	13 UNSAT	3%	68%	1339.93	0.28	103.07	1343.43	0.28	103.34	1.00
17	CASHWMaxSAT-DisjCad-SP	2024-07-10	13	13 UNSAT	3%	68%	3512.17	0.37	270.17	3524.99	0.38	271.15	1.00
18	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	13	13 UNSAT	3%	68%	3724.93	0.31	286.53	3736.57	0.32	287.43	1.00
19	CASHWMaxSAT-DisjCom-SP	2024-07-10	13	13 UNSAT	3%	68%	3778.45	0.41	290.65	3791.30	0.42	291.64	1.00
20	pb-oll-rs	2024-07-19	12	12 UNSAT	3%	63%	294.60	0.09	24.55	295.54	0.09	24.63	1.00
21	NaPS	1.03	12	12 UNSAT	3%	63%	1139.20	0.16	94.93	1142.54	0.16	95.21	1.00
22	CASHWMaxSAT-DisjCad-S	2024-07-10	12	12 UNSAT	3%	63%	1894.19	0.15	157.85	1900.79	0.15	158.40	1.00
23	CASHWMaxSAT-DisjCom-S	2024-07-10	12	12 UNSAT	3%	63%	3664.02	0.23	305.34	3677.45	0.23	306.45	1.00
24	Sat4j Partial RoundingSAT 2024	2024-07-12	12	12 UNSAT	3%	63%	3801.13	1.81	316.76	3767.74	1.01	313.98	1.01
25	Sat4j Resolution Default	2024-07-07	11	11 UNSAT	2%	58%	23.77	0.62	2.16	18.32	0.38	1.67	1.30
26	Sat4j Cutting Planes 2024	2024-07-12	10	10 UNSAT	2%	53%	437.32	1.63	43.73	427.65	0.92	42.77	1.02
27	ParLS-PBO-1.1	2024-07-09	8	8 UNSAT	2%	42%	0.45	0.05	0.06	0.46	0.05	0.06	0.98
28	DLS-PBO	2024-07-09	8	8 UNSAT	2%	42%	0.48	0.05	0.06	0.49	0.05	0.06	0.99
29	ParLS-PBO-1.1	20 cores, 2024-07-09	8	8 UNSAT	2%	42%	0.56	0.08	0.07	0.32	0.05	0.04	1.74
30	Sat4j Resolution 2024	2024-07-07	6	6 UNSATC	1%	32%	5.94	1.06	0.99	3.73	0.65	0.62	1.59
31	LS-MabSP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
32	LS-MabS	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
33	LS-MabP	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
34	LS-Mab	2024-07-10	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
35	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	Hybrid-LSMabSP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	Hybrid-LSMabP+Exact	2024-07-11	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimization problem, non-linear constraints, no OPT/UNSAT certificate" (OPT-NLC): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 54
Virtual Best Solver (VBS)			37	37 OPT	69%	100%	6836.69	0.72	184.78	-	-	-	-
1	FiberSCIP	2024-07-30	37	37 OPT	69%	100%	7783.09	3.24	210.35	7809.68	3.26	211.07	1.00
2	SCIP	2024-07-07	37	37 OPT	69%	100%	7840.64	2.69	211.91	7863.88	2.71	212.54	1.00
3	FiberSCIP	20 cores, 2024-08-12	36	36 OPT	67%	97%	72379.92	25.54	2010.55	3745.66	1.57	104.05	19.32
4	Exact	2024-07-05	30	30 OPT	56%	81%	7736.91	0.35	257.90	7756.85	0.36	258.56	1.00
5	Exact_no_SoPlex	2024-07-05	27	27 OPT	50%	73%	768.39	0.20	28.46	770.43	0.20	28.53	1.00
6	Picat	2024-06-18	25	25 OPT	46%	68%	7391.82	1.41	295.67	7412.59	1.41	296.50	1.00
7	Sat4j Partial RoundingSAT 2024	2024-07-12	20	20 OPT	37%	54%	1968.40	16.62	98.42	1888.30	12.60	94.41	1.04
8	Sat4j Resolution Default	2024-07-07	19	19 OPT	35%	51%	1512.27	3.74	79.59	1478.45	2.05	77.81	1.02
9	Sat4j Cutting Planes 2024	2024-07-12	13	13 OPT	24%	35%	976.78	3.71	75.14	946.07	1.95	72.77	1.03

Category "optimization problem, non-linear constraints, no OPT/UNSAT certificate" (OPT-NLC): OPT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 54
Virtual Best Solver (VBS)			37	37 OPT	69%	100%	6836.69	0.72	184.78	-	-	-	-
1	FiberSCIP	2024-07-30	37	37 OPT	69%	100%	7783.09	3.24	210.35	7809.68	3.26	211.07	1.00
2	SCIP	2024-07-07	37	37 OPT	69%	100%	7840.64	2.69	211.91	7863.88	2.71	212.54	1.00
3	FiberSCIP	20 cores, 2024-08-12	36	36 OPT	67%	97%	72379.92	25.54	2010.55	3745.66	1.57	104.05	19.32
4	Exact	2024-07-05	30	30 OPT	56%	81%	7736.91	0.35	257.90	7756.85	0.36	258.56	1.00
5	Exact_no_SoPlex	2024-07-05	27	27 OPT	50%	73%	768.39	0.20	28.46	770.43	0.20	28.53	1.00
6	Picat	2024-06-18	25	25 OPT	46%	68%	7391.82	1.41	295.67	7412.59	1.41	296.50	1.00
7	Sat4j Partial RoundingSAT 2024	2024-07-12	20	20 OPT	37%	54%	1968.40	16.62	98.42	1888.30	12.60	94.41	1.04
8	Sat4j Resolution Default	2024-07-07	19	19 OPT	35%	51%	1512.27	3.74	79.59	1478.45	2.05	77.81	1.02
9	Sat4j Cutting Planes 2024	2024-07-12	13	13 OPT	24%	35%	976.78	3.71	75.14	946.07	1.95	72.77	1.03

Category "optimization problem, non-linear constraints, no OPT/UNSAT certificate" (OPT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 54
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	Sat4j Partial RoundingSAT 2024	2024-07-12	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	Sat4j Cutting Planes 2024	2024-07-12	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	Sat4j Resolution Default	2024-07-07	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	SCIP	2024-07-07	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
5	Exact	2024-07-05	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
6	Exact_no_SoPlex	2024-07-05	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
7	Picat	2024-06-18	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
8	FiberSCIP	2024-07-30	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
9	FiberSCIP	20 cores, 2024-08-12	0	0%	-	0.00	0.00	-	0.00	0.00	-	-

Ranking on number of best answers found (incomplete solver point of view)

Number of best answers found with no further time limit

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within the global time limit. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

Category "optimization problem, linear constraints, no OPT/UNSAT certificate" (OPT-LIN): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 478
Virtual Best Solver (VBS)			474	99%	100%
1	mixed-bag	2024-07-19	281	59%	59%	42961.0781909999
2	SCIP	2024-07-07	279	58%	59%	61037.027718
3	FiberSCIP	20 cores, 2024-08-12	278	58%	59%	189506.249527
4	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	272	57%	57%	165189.05262
5	Exact	2024-07-05	269	56%	57%	42474.054323
6	RoundingSat	nolog (2)	268	56%	57%	81549.619175
7	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	266	56%	56%	168498.108772
8	RoundingSat	log (3)	266	56%	56%	91906.237005
9	FiberSCIP	2024-07-30	258	54%	54%	83669.170396
10	IPBHS-GUROBI	2024-07-09	257	54%	54%	46957.630398
11	Exact_veripb2	2024-07-05	252	53%	53%	40914.26483
12	CASHWMaxSAT-DisjCad-S	2024-07-10	249	52%	53%	151144.304199
13	CASHWMaxSAT-DisjCad-SP	2024-07-10	247	52%	52%	134634.623938
14	IPBHS-SCIP	2024-07-09	245	51%	52%	36924.71939
15	CASHWMaxSAT-DisjCom-SP	2024-07-10	241	50%	51%	112732.91174
16	CASHWMaxSAT-DisjCom-S	2024-07-10	240	50%	51%	128121.993289
17	Exact_no_SoPlex	2024-07-05	240	50%	51%	29296.989202
18	NaPS	1.03	210	44%	44%	22016.396453
19	pb-oll-rs	2024-07-19	209	44%	44%	35687.801699
20	Exact_veripb2_no_SoPlex	2024-07-05	207	43%	44%	28313.284932
21	NaPS-GM	1.03	206	43%	43%	20265.681026
22	NaPS-PB16	1.02b5	203	42%	43%	23269.629537
23	Picat	2024-06-18	201	42%	42%	40351.180195
24	Sat4j Partial RoundingSAT 2024	2024-07-12	149	31%	31%	28873.682386
25	Sat4j Resolution Default	2024-07-07	137	29%	29%	9576.831548
26	Sat4j Cutting Planes 2024	2024-07-12	135	28%	28%	13448.755712
27	Sat4j Resolution 2024	2024-07-07	133	28%	28%	9666.384586
28	ParLS-PBO-1.1	20 cores, 2024-07-09	80	17%	17%	58866.338403
29	LS-MabP	2024-07-10	54	11%	11%	20866.91
30	ParLS-PBO-1.1	2024-07-09	54	11%	11%	16018.075724
31	DLS-PBO	2024-07-09	53	11%	11%	12562.411813
32	LS-Mab	2024-07-10	52	11%	11%	14768.58
33	Hybrid-LSMabP+Exact	2024-07-11	51	11%	11%	183413.37
34	LS-MabSP	2024-07-10	51	11%	11%	88582.43
35	Hybrid-LSMabSP+Exact	2024-07-11	48	10%	10%	172622.18
36	LS-MabS	2024-07-10	48	10%	10%	78964.3
37	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	47	10%	10%	164432.03

Category "optimization problem, non-linear constraints, no OPT/UNSAT certificate" (OPT-NLC): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 54
Virtual Best Solver (VBS)			54	100%	100%
1	FiberSCIP	20 cores, 2024-08-12	52	96%	96%	61323.67
2	FiberSCIP	2024-07-30	39	72%	72%	15006.53
3	SCIP	2024-07-07	37	69%	69%	3263.85
4	Exact	2024-07-05	30	56%	56%	1645.85
5	Exact_no_SoPlex	2024-07-05	28	52%	52%	1522.57
6	Picat	2024-06-18	25	46%	46%	6163.49
7	Sat4j Partial RoundingSAT 2024	2024-07-12	20	37%	37%	723.5
8	Sat4j Resolution Default	2024-07-07	19	35%	35%	653.7
9	Sat4j Cutting Planes 2024	2024-07-12	13	24%	24%	594.93

Number of best answers found within 2 times the VBS time

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 2 times the VBS time. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

Category "optimization problem, linear constraints, no OPT/UNSAT certificate" (OPT-LIN): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 478
Virtual Best Solver (VBS)			474	99%	100%
1	Exact_no_SoPlex	2024-07-05	79	17%	17%	7870.56
2	ParLS-PBO-1.1	20 cores, 2024-07-09	66	14%	14%	57861.98
3	Exact	2024-07-05	66	14%	14%	14354.53
4	RoundingSat	nolog (2)	50	10%	11%	4156.64
5	pb-oll-rs	2024-07-19	44	9%	9%	980.99
6	SCIP	2024-07-07	38	8%	8%	19612.84
7	FiberSCIP	20 cores, 2024-08-12	37	8%	8%	37479.19
8	FiberSCIP	2024-07-30	34	7%	7%	22578.9
9	Exact_veripb2	2024-07-05	34	7%	7%	5954.39
10	CASHWMaxSAT-DisjCad-S	2024-07-10	32	7%	7%	6608.7
11	Exact_veripb2_no_SoPlex	2024-07-05	29	6%	6%	2835.27
12	CASHWMaxSAT-DisjCom-S	2024-07-10	28	6%	6%	7665.07
13	CASHWMaxSAT-DisjCom-SP	2024-07-10	27	6%	6%	6912.2
14	CASHWMaxSAT-DisjCad-SP	2024-07-10	27	6%	6%	3805.86
15	IPBHS-GUROBI	2024-07-09	27	6%	6%	3053.47
16	RoundingSat	log (3)	20	4%	4%	11489.34
17	mixed-bag	2024-07-19	20	4%	4%	3638.58
18	LS-Mab	2024-07-10	17	4%	4%	11353.92
19	NaPS-PB16	1.02b5	16	3%	3%	17.94
20	IPBHS-SCIP	2024-07-09	16	3%	3%	1.93
21	NaPS	1.03	15	3%	3%	229.96
22	NaPS-GM	1.03	15	3%	3%	4.52
23	ParLS-PBO-1.1	2024-07-09	13	3%	3%	1462.69
24	LS-MabP	2024-07-10	12	3%	3%	12962.33
25	DLS-PBO	2024-07-09	11	2%	2%	3603
26	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	9	2%	2%	7287.96
27	LS-MabSP	2024-07-10	7	1%	1%	11325.16
28	LS-MabS	2024-07-10	7	1%	1%	6949.49
29	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	7	1%	1%	177.97
30	Picat	2024-06-18	7	1%	1%	0.05
31	Sat4j Resolution Default	2024-07-07	6	1%	1%	2822.95
32	Sat4j Resolution 2024	2024-07-07	4	1%	1%	1334.26
33	Hybrid-LSMabP+Exact	2024-07-11	1	0%	0%	3596.02

Category "optimization problem, non-linear constraints, no OPT/UNSAT certificate" (OPT-NLC): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 54
Virtual Best Solver (VBS)			54	100%	100%
1	FiberSCIP	20 cores, 2024-08-12	26	48%	48%	58650.04
2	Exact_no_SoPlex	2024-07-05	16	30%	30%	1421.88
3	SCIP	2024-07-07	16	30%	30%	1407.81
4	Exact	2024-07-05	16	30%	30%	4.06
5	FiberSCIP	2024-07-30	9	17%	17%	8064.75
6	Picat	2024-06-18	1	2%	2%	0

Number of best answers found within 4 times the VBS time

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 4 times the VBS time. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

Category "optimization problem, linear constraints, no OPT/UNSAT certificate" (OPT-LIN): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 478
Virtual Best Solver (VBS)			474	99%	100%
1	Exact_no_SoPlex	2024-07-05	101	21%	21%	10443.11
2	Exact	2024-07-05	97	20%	20%	19073.96
3	ParLS-PBO-1.1	20 cores, 2024-07-09	67	14%	14%	57866.25
4	RoundingSat	nolog (2)	63	13%	13%	4674.86
5	pb-oll-rs	2024-07-19	63	13%	13%	3985.29
6	Exact_veripb2	2024-07-05	57	12%	12%	6504.91
7	FiberSCIP	20 cores, 2024-08-12	53	11%	11%	45441.17
8	SCIP	2024-07-07	53	11%	11%	21541.99
9	FiberSCIP	2024-07-30	51	11%	11%	25141.72
10	IPBHS-GUROBI	2024-07-09	50	10%	11%	6737.06
11	Exact_veripb2_no_SoPlex	2024-07-05	45	9%	9%	3116.21
12	CASHWMaxSAT-DisjCad-S	2024-07-10	39	8%	8%	6619.3
13	RoundingSat	log (3)	37	8%	8%	17307.18
14	CASHWMaxSAT-DisjCad-SP	2024-07-10	37	8%	8%	9665.63
15	mixed-bag	2024-07-19	36	8%	8%	6646.8
16	CASHWMaxSAT-DisjCom-SP	2024-07-10	34	7%	7%	7125.08
17	CASHWMaxSAT-DisjCom-S	2024-07-10	33	7%	7%	7715.89
18	IPBHS-SCIP	2024-07-09	29	6%	6%	3551.4
19	NaPS-GM	1.03	29	6%	6%	323.75
20	NaPS	1.03	28	6%	6%	484.57
21	NaPS-PB16	1.02b5	23	5%	5%	147.28
22	LS-Mab	2024-07-10	20	4%	4%	11736.76
23	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	20	4%	4%	11125.72
24	LS-MabP	2024-07-10	18	4%	4%	13692.48
25	DLS-PBO	2024-07-09	18	4%	4%	3651.79
26	ParLS-PBO-1.1	2024-07-09	18	4%	4%	1465.35
27	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	16	3%	3%	3849.36
28	Sat4j Resolution Default	2024-07-07	14	3%	3%	2896.4
29	Sat4j Resolution 2024	2024-07-07	11	2%	2%	1375.14
30	LS-MabSP	2024-07-10	10	2%	2%	14169.33
31	Picat	2024-06-18	9	2%	2%	57.3
32	LS-MabS	2024-07-10	8	2%	2%	7168.44
33	Hybrid-LSMabP+Exact	2024-07-11	3	1%	1%	10788.38
34	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	2	0%	0%	7192.25
35	Hybrid-LSMabSP+Exact	2024-07-11	1	0%	0%	3596.19

Category "optimization problem, non-linear constraints, no OPT/UNSAT certificate" (OPT-NLC): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 54
Virtual Best Solver (VBS)			54	100%	100%
1	FiberSCIP	20 cores, 2024-08-12	32	59%	59%	59038.76
2	Exact_no_SoPlex	2024-07-05	22	41%	41%	1427.58
3	Exact	2024-07-05	22	41%	41%	24.08
4	SCIP	2024-07-07	19	35%	35%	2329.63
5	FiberSCIP	2024-07-30	14	26%	26%	10633.55
6	Picat	2024-06-18	2	4%	4%	2.47
7	Sat4j Cutting Planes 2024	2024-07-12	1	2%	2%	0.44
8	Sat4j Partial RoundingSAT 2024	2024-07-12	1	2%	2%	0.43

Number of best answers found within 8 times the VBS time

This section presents the ranking of the different solvers. Solvers are ranked on the number of instances for which they were able to find the best solution obtained by one the solvers within 8 times the VBS time. Ties are broken using the cumulated T1 time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

Category "optimization problem, linear constraints, no OPT/UNSAT certificate" (OPT-LIN): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 478
Virtual Best Solver (VBS)			474	99%	100%
1	Exact_no_SoPlex	2024-07-05	116	24%	24%	11460.14
2	Exact	2024-07-05	113	24%	24%	21320.53
3	RoundingSat	nolog (2)	90	19%	19%	5489.61000000001
4	SCIP	2024-07-07	75	16%	16%	22980.62
5	pb-oll-rs	2024-07-19	75	16%	16%	4800.75
6	FiberSCIP	20 cores, 2024-08-12	73	15%	15%	46043.21
7	Exact_veripb2	2024-07-05	73	15%	15%	7992.69
8	FiberSCIP	2024-07-30	70	15%	15%	28398.3
9	ParLS-PBO-1.1	20 cores, 2024-07-09	69	14%	15%	58479.26
10	IPBHS-GUROBI	2024-07-09	66	14%	14%	6900.25
11	Exact_veripb2_no_SoPlex	2024-07-05	61	13%	13%	6389.99
12	RoundingSat	log (3)	57	12%	12%	18731.91
13	mixed-bag	2024-07-19	54	11%	11%	11275.51
14	NaPS	1.03	54	11%	11%	2134.25
15	CASHWMaxSAT-DisjCom-SP	2024-07-10	49	10%	10%	7326.69
16	CASHWMaxSAT-DisjCad-S	2024-07-10	49	10%	10%	7316.48
17	IPBHS-SCIP	2024-07-09	48	10%	10%	3809.92
18	CASHWMaxSAT-DisjCad-SP	2024-07-10	47	10%	10%	10432.33
19	CASHWMaxSAT-DisjCom-S	2024-07-10	45	9%	9%	7743.24
20	NaPS-GM	1.03	44	9%	9%	1608.63
21	NaPS-PB16	1.02b5	32	7%	7%	2922.93
22	Hybrid-CASHWMaxSATDisjComS+Exact	2024-07-11	29	6%	6%	3872.92
23	LS-Mab	2024-07-10	28	6%	6%	11763.38
24	Hybrid-CASHWMaxSATDisjCadSP+Exact	2024-07-11	27	6%	6%	11340.71
25	DLS-PBO	2024-07-09	27	6%	6%	4781.52
26	LS-MabP	2024-07-10	25	5%	5%	14206.41
27	ParLS-PBO-1.1	2024-07-09	25	5%	5%	4794.77
28	Sat4j Resolution Default	2024-07-07	19	4%	4%	2915.64
29	Sat4j Resolution 2024	2024-07-07	18	4%	4%	1392.83
30	Picat	2024-06-18	17	4%	4%	1625.5
31	LS-MabSP	2024-07-10	10	2%	2%	14169.33
32	LS-MabS	2024-07-10	8	2%	2%	7168.44
33	Hybrid-LSMabP+Exact	2024-07-11	4	1%	1%	14384.55
34	Hybrid-LSMabSP+CASHWMaxSATDisjCom	2024-07-11	3	1%	1%	10788.39
35	Hybrid-LSMabSP+Exact	2024-07-11	2	0%	0%	7192.38
36	Sat4j Partial RoundingSAT 2024	2024-07-12	2	0%	0%	62.01
37	Sat4j Cutting Planes 2024	2024-07-12	2	0%	0%	0.67

Category "optimization problem, non-linear constraints, no OPT/UNSAT certificate" (OPT-NLC): answers

Rank	Solver	Version	Number of best solutions found	% of all instances	% of VBS	Cumulated time
Total number of instances in the category: 54
Virtual Best Solver (VBS)			54	100%	100%
1	FiberSCIP	20 cores, 2024-08-12	35	65%	65%	59044.63
2	Exact_no_SoPlex	2024-07-05	24	44%	44%	1433.3
3	Exact	2024-07-05	23	43%	43%	89.55
4	SCIP	2024-07-07	21	39%	39%	2335.18
5	FiberSCIP	2024-07-30	18	33%	33%	11723.9
6	Picat	2024-06-18	3	6%	6%	4.05
7	Sat4j Cutting Planes 2024	2024-07-12	2	4%	4%	2.34
8	Sat4j Resolution Default	2024-07-07	1	2%	2%	0.87
9	Sat4j Partial RoundingSAT 2024	2024-07-12	1	2%	2%	0.43