PB'11 competition: satisfaction and optimization track: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 481
Virtual Best Solver (VBS)			448	191 SAT, 257 UNSAT	93%	100%	12868.51	0.12	28.72	-	-	-	-
1	borg	pb-dec-11.04.03	431	183 SAT, 248 UNSAT	90%	96%	10281.04	1.93	23.85	10957.42	3.08	25.42	0.94
2	Sat4j Res//CP	2.3.0	420	183 SAT, 237 UNSAT	87%	94%	51588.82	5.47	122.83	27980.84	2.63	66.62	1.84
3	bsolo	3.2	416	179 SAT, 237 UNSAT	86%	93%	58921.79	8.33	141.64	58922.02	8.33	141.64	1.00
4	wbo	1.6	394	180 SAT, 214 UNSAT	82%	88%	17024.67	1.45	43.21	17025.89	1.45	43.21	1.00
5	Sat4j Resolution	2.3.0	392	184 SAT, 208 UNSAT	81%	88%	33537.21	4.07	85.55	33199.22	3.42	84.69	1.01
6	SCIP spx E_2	2011-06-10 (fixed)	384	149 SAT, 235 UNSAT	80%	86%	46478.58	1.28	121.04	46478.01	1.28	121.04	1.00
7	SCIP spx 2	2011-06-10 (fixed)	383	148 SAT, 235 UNSAT	80%	85%	44742.45	1.34	116.82	44741.93	1.35	116.82	1.00
8	clasp	2.0-R4191	380	168 SAT, 212 UNSAT	79%	85%	12375.19	0.63	32.57	12376.43	0.63	32.57	1.00
9	MinisatID	2.5.2 (fixed)	368	169 SAT, 199 UNSAT	77%	82%	19281.13	5.20	52.39	19281.13	5.20	52.39	1.00
10	MinisatID	2.5.2-gmp (fixed)	362	165 SAT, 197 UNSAT	75%	81%	37474.45	20.60	103.52	37474.12	20.60	103.52	1.00
11	Sat4j CuttingPlanes	2.3.0	242	114 SAT, 128 UNSAT	50%	54%	22807.02	2.22	94.24	22539.31	1.04	93.14	1.01

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 481
Virtual Best Solver (VBS)			191	40%	100%	5863.78	0.09	30.70	-	-	-	-
1	Sat4j Resolution	2.3.0	184	38%	96%	5265.59	1.89	28.62	5145.79	1.21	27.97	1.02
2	borg	pb-dec-11.04.03	183	38%	96%	3819.10	1.52	20.87	4257.53	1.91	23.27	0.90
3	Sat4j Res//CP	2.3.0	183	38%	96%	6865.66	3.90	37.52	3513.53	1.67	19.20	1.95
4	wbo	1.6	180	37%	94%	2504.41	0.32	13.91	2505.05	0.32	13.92	1.00
5	bsolo	3.2	179	37%	94%	7297.97	2.52	40.77	7298.52	2.52	40.77	1.00
6	MinisatID	2.5.2 (fixed)	169	35%	88%	5498.95	1.27	32.54	5498.95	1.27	32.54	1.00
7	clasp	2.0-R4191	168	35%	88%	7648.76	0.15	45.53	7649.40	0.15	45.53	1.00
8	MinisatID	2.5.2-gmp (fixed)	165	34%	86%	3892.70	5.28	23.59	3892.90	5.28	23.59	1.00
9	SCIP spx E_2	2011-06-10 (fixed)	149	31%	78%	19631.44	3.43	131.75	19631.19	3.44	131.75	1.00
10	SCIP spx 2	2011-06-10 (fixed)	148	31%	77%	17871.17	3.19	120.75	17871.13	3.19	120.75	1.00
11	Sat4j CuttingPlanes	2.3.0	114	24%	60%	12424.65	5.11	108.99	12260.76	3.40	107.55	1.01

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 481
Virtual Best Solver (VBS)			257	53%	100%	7004.74	0.17	27.26	-	-	-	-
1	borg	pb-dec-11.04.03	248	52%	96%	6461.93	5.10	26.06	6699.88	5.81	27.02	0.96
2	Sat4j Res//CP	2.3.0	237	49%	92%	44723.16	11.94	188.71	24467.31	5.60	103.24	1.83
3	bsolo	3.2	237	49%	92%	51623.82	24.96	217.82	51623.50	24.96	217.82	1.00
4	SCIP spx E_2	2011-06-10 (fixed)	235	49%	91%	26847.14	0.59	114.24	26846.82	0.59	114.24	1.00
5	SCIP spx 2	2011-06-10 (fixed)	235	49%	91%	26871.28	0.64	114.35	26870.80	0.64	114.34	1.00
6	wbo	1.6	214	44%	83%	14520.26	2.71	67.85	14520.84	2.71	67.85	1.00
7	clasp	2.0-R4191	212	44%	82%	4726.43	0.90	22.29	4727.03	0.90	22.30	1.00
8	Sat4j Resolution	2.3.0	208	43%	81%	28271.62	9.08	135.92	28053.43	8.37	134.87	1.01
9	MinisatID	2.5.2 (fixed)	199	41%	77%	13782.18	9.62	69.26	13782.17	9.62	69.26	1.00
10	MinisatID	2.5.2-gmp (fixed)	197	41%	77%	33581.75	32.70	170.47	33581.22	32.70	170.46	1.00
11	Sat4j CuttingPlanes	2.3.0	128	27%	50%	10382.37	1.00	81.11	10278.55	0.53	80.30	1.01

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 100
Virtual Best Solver (VBS)			76	55 SAT, 21 UNSAT	76%	100%	2063.40	0.22	27.15	-	-	-	-
1	SCIP spx E_2	2011-06-10 (fixed)	75	55 SAT, 20 UNSAT	75%	99%	8178.76	13.76	109.05	8178.67	13.76	109.05	1.00
2	SCIP spx 2	2011-06-10 (fixed)	74	54 SAT, 20 UNSAT	74%	97%	6518.32	6.42	88.09	6518.47	6.42	88.09	1.00
3	borg	pb-dec-11.04.03	73	53 SAT, 20 UNSAT	73%	96%	10603.89	0.88	145.26	10569.77	1.19	144.79	1.00
4	Sat4j CuttingPlanes	2.3.0	65	50 SAT, 15 UNSAT	65%	86%	446.59	1.87	6.87	388.60	0.86	5.98	1.15
5	Sat4j Res//CP	2.3.0	65	50 SAT, 15 UNSAT	65%	86%	792.28	2.82	12.19	396.23	1.34	6.10	2.00
6	clasp	2.0-R4191	64	49 SAT, 15 UNSAT	64%	84%	3517.38	0.09	54.96	3517.66	0.09	54.96	1.00
7	bsolo	3.2	62	47 SAT, 15 UNSAT	62%	82%	3615.09	3.03	58.31	3615.27	3.03	58.31	1.00
8	Sat4j Resolution	2.3.0	27	12 SAT, 15 UNSAT	27%	36%	3673.64	5.41	136.06	3656.90	4.65	135.44	1.00
9	MinisatID	2.5.2 (fixed)	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
10	MinisatID	2.5.2-gmp (fixed)	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 100
Virtual Best Solver (VBS)			55	55%	100%	234.89	0.21	4.27	-	-	-	-
1	SCIP spx E_2	2011-06-10 (fixed)	55	55%	100%	8084.85	26.62	147.00	8084.75	26.62	147.00	1.00
2	SCIP spx 2	2011-06-10 (fixed)	54	54%	98%	4656.35	6.42	86.23	4656.52	6.42	86.23	1.00
3	borg	pb-dec-11.04.03	53	53%	96%	4211.29	0.88	79.46	4202.82	1.17	79.30	1.00
4	Sat4j CuttingPlanes	2.3.0	50	50%	91%	270.45	2.02	5.41	220.69	0.91	4.41	1.23
5	Sat4j Res//CP	2.3.0	50	50%	91%	463.95	2.86	9.28	224.98	1.35	4.50	2.06
6	clasp	2.0-R4191	49	49%	89%	3461.65	0.13	70.65	3461.77	0.13	70.65	1.00
7	bsolo	3.2	47	47%	85%	2476.44	3.64	52.69	2476.50	3.64	52.69	1.00
8	Sat4j Resolution	2.3.0	12	12%	22%	1877.13	10.98	156.43	1868.66	10.50	155.72	1.00
9	MinisatID	2.5.2 (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
10	MinisatID	2.5.2-gmp (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 100
Virtual Best Solver (VBS)			21	21%	100%	1828.51	0.57	87.07	-	-	-	-
1	SCIP spx E_2	2011-06-10 (fixed)	20	20%	95%	93.91	0.30	4.70	93.92	0.30	4.70	1.00
2	SCIP spx 2	2011-06-10 (fixed)	20	20%	95%	1861.98	10.22	93.10	1861.95	10.22	93.10	1.00
3	borg	pb-dec-11.04.03	20	20%	95%	6392.60	6.27	319.63	6366.96	7.83	318.35	1.00
4	clasp	2.0-R4191	15	15%	71%	55.73	0.00	3.72	55.89	0.02	3.73	1.00
5	Sat4j CuttingPlanes	2.3.0	15	15%	71%	176.13	0.28	11.74	167.91	0.22	11.19	1.05
6	Sat4j Res//CP	2.3.0	15	15%	71%	328.33	0.58	21.89	171.25	0.65	11.42	1.92
7	bsolo	3.2	15	15%	71%	1138.65	0.00	75.91	1138.77	0.02	75.92	1.00
8	Sat4j Resolution	2.3.0	15	15%	71%	1796.51	0.31	119.77	1788.23	0.22	119.22	1.00
9	MinisatID	2.5.2 (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
10	MinisatID	2.5.2-gmp (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 557
Virtual Best Solver (VBS)			213	154 OPT, 59 UNSAT	38%	100%	9573.55	2.48	44.95	-	-	-	-
1	Sat4j Res//CP	2.3.0	208	149 OPT, 59 UNSAT	37%	98%	13907.35	3.52	66.86	8297.48	2.93	39.89	1.68
2	Sat4j Resolution	2.3.0	201	144 OPT, 57 UNSAT	36%	94%	10070.77	2.63	50.10	9894.98	1.75	49.23	1.02
3	Sat4j CuttingPlanes	2.3.0	175	116 OPT, 59 UNSAT	31%	82%	10253.32	2.30	58.59	10077.46	1.38	57.59	1.02
4	MinisatID	2.5.2-gmp (fixed)	117	60 OPT, 57 UNSAT	21%	55%	23089.88	3.58	197.35	23089.54	3.58	197.35	1.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 557
Virtual Best Solver (VBS)			154	28%	100%	9460.02	4.25	61.43	-	-	-	-
1	Sat4j Res//CP	2.3.0	149	27%	97%	13719.20	7.33	92.08	8167.71	6.07	54.82	1.68
2	Sat4j Resolution	2.3.0	144	26%	94%	9962.32	5.90	69.18	9814.40	4.84	68.16	1.02
3	Sat4j CuttingPlanes	2.3.0	116	21%	75%	8943.45	4.00	77.10	8806.65	2.91	75.92	1.02
4	MinisatID	2.5.2-gmp (fixed)	60	11%	39%	20756.40	111.05	345.94	20756.07	111.14	345.93	1.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 557
Virtual Best Solver (VBS)			59	11%	100%	113.53	1.01	1.92	-	-	-	-
1	Sat4j Res//CP	2.3.0	59	11%	100%	188.15	1.73	3.19	129.77	1.20	2.20	1.45
2	Sat4j CuttingPlanes	2.3.0	59	11%	100%	1309.87	1.36	22.20	1270.81	0.68	21.54	1.03
3	Sat4j Resolution	2.3.0	57	10%	97%	108.44	1.07	1.90	80.58	0.58	1.41	1.35
4	MinisatID	2.5.2-gmp (fixed)	57	10%	97%	2333.48	1.10	40.94	2333.46	1.10	40.94	1.00

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 731
Virtual Best Solver (VBS)			494	459 OPT, 35 UNSAT	68%	100%	27524.70	0.54	55.72	-	-	-	-
1	SCIP spx E_2	2011-06-10 (fixed)	409	374 OPT, 35 UNSAT	56%	83%	51061.22	4.28	124.84	51055.05	4.28	124.83	1.00
2	SCIP spx 2	2011-06-10 (fixed)	409	374 OPT, 35 UNSAT	56%	83%	52208.54	4.23	127.65	52208.03	4.23	127.65	1.00
3	pwbo	1.1	383	350 OPT, 33 UNSAT	52%	78%	25328.59	1.07	66.13	12714.50	0.69	33.20	1.99
4	bsolo	3.2	350	316 OPT, 34 UNSAT	48%	71%	35382.65	2.88	101.09	35383.54	2.88	101.10	1.00
5	Sat4j Res//CP	2.3.0	329	295 OPT, 34 UNSAT	45%	67%	21642.59	3.24	65.78	13140.56	2.73	39.94	1.65
6	clasp	2.0-R4191	320	286 OPT, 34 UNSAT	44%	65%	11896.81	0.40	37.18	11898.45	0.40	37.18	1.00
7	Sat4j Resolution	2.3.0	316	282 OPT, 34 UNSAT	43%	64%	14889.11	2.13	47.12	14623.56	1.18	46.28	1.02
8	MinisatID	2.5.2 (fixed)	288	256 OPT, 32 UNSAT	39%	58%	31859.77	5.74	110.62	31859.28	5.74	110.62	1.00
9	Sat4j CuttingPlanes	2.3.0	270	240 OPT, 30 UNSAT	37%	55%	24809.58	2.14	91.89	24492.17	1.03	90.71	1.01
10	wbo	1.6	269	236 OPT, 33 UNSAT	37%	54%	18212.75	0.88	67.71	18213.68	0.88	67.71	1.00
11	MinisatID	2.5.2-gmp (fixed)	262	232 OPT, 30 UNSAT	36%	53%	36343.88	9.04	138.72	36345.50	9.04	138.72	1.00

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 731
Virtual Best Solver (VBS)			459	63%	100%	27261.29	0.72	59.39	-	-	-	-
1	SCIP spx E_2	2011-06-10 (fixed)	374	51%	81%	50199.41	4.89	134.22	50193.23	4.90	134.21	1.00
2	SCIP spx 2	2011-06-10 (fixed)	374	51%	81%	51348.81	4.95	137.30	51348.30	4.95	137.29	1.00
3	pwbo	1.1	350	48%	76%	24711.03	1.32	70.60	12404.91	0.90	35.44	1.99
4	bsolo	3.2	316	43%	69%	34517.45	6.60	109.23	34518.17	6.60	109.23	1.00
5	Sat4j Res//CP	2.3.0	295	40%	64%	20982.25	3.59	71.13	12798.43	2.93	43.38	1.64
6	clasp	2.0-R4191	286	39%	62%	11784.44	0.51	41.20	11785.82	0.52	41.21	1.00
7	Sat4j Resolution	2.3.0	282	39%	61%	14566.30	2.38	51.65	14317.92	1.36	50.77	1.02
8	MinisatID	2.5.2 (fixed)	256	35%	56%	29555.22	8.20	115.45	29554.78	8.20	115.45	1.00
9	Sat4j CuttingPlanes	2.3.0	240	33%	52%	24787.01	2.57	103.28	24479.56	1.49	102.00	1.01
10	wbo	1.6	236	32%	51%	17500.77	1.10	74.16	17501.55	1.11	74.16	1.00
11	MinisatID	2.5.2-gmp (fixed)	232	32%	51%	36301.53	20.21	156.47	36303.13	20.21	156.48	1.00

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 731
Virtual Best Solver (VBS)			35	5%	100%	263.40	0.01	7.53	-	-	-	-
1	SCIP spx 2	2011-06-10 (fixed)	35	5%	100%	859.73	0.34	24.56	859.73	0.34	24.56	1.00
2	SCIP spx E_2	2011-06-10 (fixed)	35	5%	100%	861.81	0.31	24.62	861.82	0.31	24.62	1.00
3	clasp	2.0-R4191	34	5%	97%	112.38	0.01	3.31	112.63	0.02	3.31	1.00
4	Sat4j Resolution	2.3.0	34	5%	97%	322.81	0.82	9.49	305.64	0.45	8.99	1.06
5	Sat4j Res//CP	2.3.0	34	5%	97%	660.34	0.98	19.42	342.13	0.86	10.06	1.93
6	bsolo	3.2	34	5%	97%	865.19	0.06	25.45	865.37	0.06	25.45	1.00
7	pwbo	1.1	33	5%	94%	617.56	0.03	18.71	309.60	0.03	9.38	1.99
8	wbo	1.6	33	5%	94%	711.98	0.06	21.58	712.14	0.07	21.58	1.00
9	MinisatID	2.5.2 (fixed)	32	4%	91%	2304.56	0.25	72.02	2304.51	0.25	72.02	1.00
10	Sat4j CuttingPlanes	2.3.0	30	4%	86%	22.57	0.85	0.75	12.60	0.46	0.42	1.79
11	MinisatID	2.5.2-gmp (fixed)	30	4%	86%	42.35	0.38	1.41	42.38	0.38	1.41	1.00

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 409
Virtual Best Solver (VBS)			298	298 OPT	73%	100%	5387.76	0.09	18.08	-	-	-	-
1	SCIP spx E_2	2011-06-10 (fixed)	297	297 OPT	73%	100%	6442.90	0.13	21.69	6442.97	0.13	21.69	1.00
2	SCIP spx 2	2011-06-10 (fixed)	294	294 OPT	72%	99%	5240.65	0.14	17.83	5240.81	0.14	17.83	1.00
3	clasp	2.0-R4191-patched (fixed)	280	280 OPT	68%	94%	9225.64	1.02	32.95	9225.77	1.02	32.95	1.00
4	Sat4j Resolution	2.3.0	277	277 OPT	68%	93%	18696.59	2.95	67.50	18465.85	2.30	66.66	1.01
5	Sat4j Res//CP	2.3.0	274	274 OPT	67%	92%	34407.38	7.48	125.57	19080.46	3.77	69.64	1.80
6	bsolo	3.2	234	234 OPT	57%	79%	29076.06	2.58	124.26	29076.57	2.58	124.26	1.00
7	Sat4j CuttingPlanes	2.3.0	120	120 OPT	29%	40%	9357.40	4.11	77.98	9218.19	2.82	76.82	1.02
8	MinisatID	2.5.2 (fixed)	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
9	MinisatID	2.5.2-gmp (fixed)	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 409
Virtual Best Solver (VBS)			298	73%	100%	5387.76	0.09	18.08	-	-	-	-
1	SCIP spx E_2	2011-06-10 (fixed)	297	73%	100%	6442.90	0.13	21.69	6442.97	0.13	21.69	1.00
2	SCIP spx 2	2011-06-10 (fixed)	294	72%	99%	5240.65	0.14	17.83	5240.81	0.14	17.83	1.00
3	clasp	2.0-R4191-patched (fixed)	280	68%	94%	9225.64	1.02	32.95	9225.77	1.02	32.95	1.00
4	Sat4j Resolution	2.3.0	277	68%	93%	18696.59	2.95	67.50	18465.85	2.30	66.66	1.01
5	Sat4j Res//CP	2.3.0	274	67%	92%	34407.38	7.48	125.57	19080.46	3.77	69.64	1.80
6	bsolo	3.2	234	57%	79%	29076.06	2.58	124.26	29076.57	2.58	124.26	1.00
7	Sat4j CuttingPlanes	2.3.0	120	29%	40%	9357.40	4.11	77.98	9218.19	2.82	76.82	1.02
8	MinisatID	2.5.2 (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
9	MinisatID	2.5.2-gmp (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 409
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	Sat4j Res//CP	2.3.0	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	Sat4j CuttingPlanes	2.3.0	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	Sat4j Resolution	2.3.0	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	bsolo	3.2	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
5	SCIP spx 2	2011-06-10 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
6	SCIP spx E_2	2011-06-10 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
7	MinisatID	2.5.2 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
8	clasp	2.0-R4191-patched (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
9	MinisatID	2.5.2-gmp (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-