PB'12 competition: WBO track: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 238
Virtual Best Solver (VBS)			133	133 MOPT	56%	100%	6779.72	0.00	50.98	-	-	-	-
1	Sat4j PB	2012-05-28	128	128 MOPT	54%	96%	5042.69	3.26	39.40	4816.74	1.46	37.63	1.05
2	toysat	2012-05-17	101	101 MOPT	42%	76%	25947.51	103.07	256.91	25953.00	103.09	256.96	1.00
3	wbo2sat	2012-05-19	61	61 MOPT	26%	46%	17631.71	54.03	289.04	17648.10	54.05	289.31	1.00
4	wbo2satCp2	2012-05-19	61	61 MOPT	26%	46%	17987.50	63.66	294.88	18008.74	63.68	295.23	1.00
5	npSolver	1.0 (fixed)	46	46 MOPT	19%	35%	8661.86	14.17	188.30	8692.03	14.18	188.96	1.00
6	npSolver	inc (fixed)	46	46 MOPT	19%	35%	8977.25	14.65	195.16	9004.78	14.66	195.76	1.00
7	npSolver	inc-topDown (fixed)	27	27 MOPT	11%	20%	2543.66	23.29	94.21	2549.51	23.28	94.43	1.00
8	npSolver	inc-topdown-quickBound (fixed)	11	11 MOPT	5%	8%	604.92	11.73	54.99	609.74	11.93	55.43	0.99

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 238
Virtual Best Solver (VBS)			133	56%	100%	6779.72	0.00	50.98	-	-	-	-
1	Sat4j PB	2012-05-28	128	54%	96%	5042.69	3.26	39.40	4816.74	1.46	37.63	1.05
2	toysat	2012-05-17	101	42%	76%	25947.51	103.07	256.91	25953.00	103.09	256.96	1.00
3	wbo2sat	2012-05-19	61	26%	46%	17631.71	54.03	289.04	17648.10	54.05	289.31	1.00
4	wbo2satCp2	2012-05-19	61	26%	46%	17987.50	63.66	294.88	18008.74	63.68	295.23	1.00
5	npSolver	1.0 (fixed)	46	19%	35%	8661.86	14.17	188.30	8692.03	14.18	188.96	1.00
6	npSolver	inc (fixed)	46	19%	35%	8977.25	14.65	195.16	9004.78	14.66	195.76	1.00
7	npSolver	inc-topDown (fixed)	27	11%	20%	2543.66	23.29	94.21	2549.51	23.28	94.43	1.00
8	npSolver	inc-topdown-quickBound (fixed)	11	5%	8%	604.92	11.73	54.99	609.74	11.93	55.43	0.99

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 238
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	toysat	2012-05-17	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	wbo2sat	2012-05-19	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	wbo2satCp2	2012-05-19	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	Sat4j PB	2012-05-28	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
5	npSolver	inc-topdown-quickBound (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
6	npSolver	inc-topDown (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
7	npSolver	inc (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
8	npSolver	1.0 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 276
Virtual Best Solver (VBS)			270	269 MOPT, 1 UNSAT	98%	100%	3410.97	7.19	12.63	-	-	-	-
1	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	248	247 MOPT, 1 UNSAT	90%	92%	31988.35	3.58	128.99	31994.28	3.58	129.01	1.00
2	clasp	2.0.6-R5325	210	209 MOPT, 1 UNSAT	76%	78%	5343.11	0.14	25.44	5341.31	0.14	25.43	1.00
3	Sat4j PB	2012-05-28	208	207 MOPT, 1 UNSAT	75%	77%	10807.85	2.28	51.96	10568.93	0.97	50.81	1.02
4	toysat	2012-05-17	199	198 MOPT, 1 UNSAT	72%	74%	18066.87	17.10	90.79	18070.81	17.11	90.81	1.00
5	npSolver	1.0 (fixed)	195	195 MOPT	71%	72%	25187.37	14.15	129.17	25200.52	14.17	129.23	1.00
6	wbo2sat	2012-05-19	194	194 MOPT	70%	72%	12943.43	16.29	66.72	12982.90	16.32	66.92	1.00
7	wbo2satCp2	2012-05-19	194	194 MOPT	70%	72%	13821.17	16.40	71.24	13837.55	16.51	71.33	1.00
8	pwbo	2.01	194	193 MOPT, 1 UNSAT	70%	72%	23741.85	0.68	122.38	11899.35	0.39	61.34	2.00
9	npSolver	inc (fixed)	193	193 MOPT	70%	71%	21605.58	13.77	111.95	21616.49	13.83	112.00	1.00
10	pwbo	2.02	193	192 MOPT, 1 UNSAT	70%	71%	25485.56	0.68	132.05	12773.07	0.45	66.18	2.00
11	wbo	1.71	181	180 MOPT, 1 UNSAT	66%	67%	24057.27	0.24	132.91	24064.40	0.23	132.95	1.00
12	wbo	1.72	181	180 MOPT, 1 UNSAT	66%	67%	24223.39	0.24	133.83	24231.98	0.24	133.88	1.00
13	npSolver	inc-topDown (fixed)	152	152 MOPT	55%	56%	46515.93	17.90	306.03	46538.51	18.11	306.17	1.00
14	npSolver	inc-topdown-quickBound (fixed)	60	60 MOPT	22%	22%	2001.03	0.07	33.35	2001.88	0.07	33.36	1.00

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 276
Virtual Best Solver (VBS)			269	97%	100%	3403.78	0.00	12.65	-	-	-	-
1	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	247	89%	92%	31913.78	3.57	129.21	31919.67	3.57	129.23	1.00
2	clasp	2.0.6-R5325	209	76%	78%	5335.91	0.13	25.53	5334.11	0.13	25.52	1.00
3	Sat4j PB	2012-05-28	207	75%	77%	10739.63	2.26	51.88	10501.90	0.96	50.73	1.02
4	toysat	2012-05-17	198	72%	74%	17248.97	16.36	87.12	17252.72	16.37	87.13	1.00
5	npSolver	1.0 (fixed)	195	71%	72%	25187.37	14.15	129.17	25200.52	14.17	129.23	1.00
6	wbo2sat	2012-05-19	194	70%	72%	12943.43	16.29	66.72	12982.90	16.32	66.92	1.00
7	wbo2satCp2	2012-05-19	194	70%	72%	13821.17	16.40	71.24	13837.55	16.51	71.33	1.00
8	npSolver	inc (fixed)	193	70%	72%	21605.58	13.77	111.95	21616.49	13.83	112.00	1.00
9	pwbo	2.01	193	70%	72%	23336.66	0.68	120.92	11696.49	0.39	60.60	2.00
10	pwbo	2.02	192	70%	71%	25099.22	0.68	130.73	12579.77	0.45	65.52	2.00
11	wbo	1.71	180	65%	67%	23876.63	0.23	132.65	23883.68	0.22	132.69	1.00
12	wbo	1.72	180	65%	67%	24060.35	0.23	133.67	24068.93	0.23	133.72	1.00
13	npSolver	inc-topDown (fixed)	152	55%	57%	46515.93	17.90	306.03	46538.51	18.11	306.17	1.00
14	npSolver	inc-topdown-quickBound (fixed)	60	22%	22%	2001.03	0.07	33.35	2001.88	0.07	33.36	1.00

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 276
Virtual Best Solver (VBS)			1	0%	100%	7.19	7.19	7.19	-	-	-	-
1	clasp	2.0.6-R5325	1	0%	100%	7.19	7.19	7.19	7.20	7.20	7.20	1.00
2	Sat4j PB	2012-05-28	1	0%	100%	68.22	68.22	68.22	67.03	67.03	67.03	1.02
3	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	1	0%	100%	74.58	74.58	74.58	74.61	74.61	74.61	1.00
4	wbo	1.72	1	0%	100%	163.04	163.04	163.04	163.05	163.05	163.05	1.00
5	wbo	1.71	1	0%	100%	180.64	180.64	180.64	180.73	180.73	180.73	1.00
6	pwbo	2.02	1	0%	100%	386.34	386.34	386.34	193.30	193.30	193.30	2.00
7	pwbo	2.01	1	0%	100%	405.19	405.19	405.19	202.86	202.86	202.86	2.00
8	toysat	2012-05-17	1	0%	100%	817.91	817.91	817.91	818.09	818.09	818.09	1.00
9	wbo2sat	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
10	wbo2satCp2	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
11	npSolver	inc-topdown-quickBound (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
12	npSolver	inc-topDown (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
13	npSolver	inc (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
14	npSolver	1.0 (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 46
Virtual Best Solver (VBS)			40	40 MOPT	87%	100%	4529.44	0.00	113.24	-	-	-	-
1	Sat4j PB	2012-05-28	39	39 MOPT	85%	98%	3914.22	3.91	100.36	3823.73	2.44	98.04	1.02
2	toysat	2012-05-17	30	30 MOPT	65%	75%	10061.11	264.25	335.37	10063.30	264.31	335.44	1.00
3	npSolver	inc (fixed)	7	7 MOPT	15%	18%	1677.44	71.92	239.63	1680.18	72.15	240.03	1.00
4	npSolver	1.0 (fixed)	7	7 MOPT	15%	18%	1906.30	69.54	272.33	1906.75	69.55	272.39	1.00
5	npSolver	inc-topdown-quickBound	5	5 MOPT	11%	13%	264.89	16.60	52.98	266.87	17.70	53.37	0.99
6	npSolver	inc-topDown (fixed)	4	4 MOPT	9%	10%	568.25	62.83	142.06	569.27	63.30	142.32	1.00
7	npSolver	inc-topdown-quickBound (fixed)	3	3 MOPT	7%	8%	133.78	7.95	44.59	141.04	7.97	47.01	0.95

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 46
Virtual Best Solver (VBS)			40	87%	100%	4529.44	0.00	113.24	-	-	-	-
1	Sat4j PB	2012-05-28	39	85%	98%	3914.22	3.91	100.36	3823.73	2.44	98.04	1.02
2	toysat	2012-05-17	30	65%	75%	10061.11	264.25	335.37	10063.30	264.31	335.44	1.00
3	npSolver	inc (fixed)	7	15%	18%	1677.44	71.92	239.63	1680.18	72.15	240.03	1.00
4	npSolver	1.0 (fixed)	7	15%	18%	1906.30	69.54	272.33	1906.75	69.55	272.39	1.00
5	npSolver	inc-topdown-quickBound	5	11%	13%	264.89	16.60	52.98	266.87	17.70	53.37	0.99
6	npSolver	inc-topDown (fixed)	4	9%	10%	568.25	62.83	142.06	569.27	63.30	142.32	1.00
7	npSolver	inc-topdown-quickBound (fixed)	3	7%	8%	133.78	7.95	44.59	141.04	7.97	47.01	0.95

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 46
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	toysat	2012-05-17	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	npSolver	inc-topdown-quickBound	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	Sat4j PB	2012-05-28	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	npSolver	inc-topdown-quickBound (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
5	npSolver	inc-topDown (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
6	npSolver	inc (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
7	npSolver	1.0 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 133
Virtual Best Solver (VBS)			133	133 MOPT	100%	100%	1976.96	0.00	14.86	-	-	-	-
1	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	127	127 MOPT	95%	95%	22308.29	0.86	175.66	22264.74	0.86	175.31	1.00
2	Sat4j PB	2012-05-28	97	97 MOPT	73%	73%	8580.48	22.70	88.46	8413.13	20.70	86.73	1.02
3	clasp	2.0.6-R5325	95	95 MOPT	71%	71%	3929.31	5.61	41.36	3927.15	4.78	41.34	1.00
4	toysat	2012-05-17	92	92 MOPT	69%	69%	22308.85	59.18	242.49	22313.02	59.20	242.53	1.00
5	npSolver	1.0	90	90 MOPT	68%	68%	16133.61	42.30	179.26	16165.46	42.75	179.62	1.00
6	npSolver	inc	89	89 MOPT	67%	67%	13775.26	40.46	154.78	13824.22	40.51	155.33	1.00
7	wbo2satCp2	2012-05-19	89	89 MOPT	67%	67%	13795.73	43.65	155.01	13810.24	42.13	155.17	1.00
8	wbo2sat	2012-05-19	87	87 MOPT	65%	65%	10510.79	38.92	120.81	10544.35	39.00	121.20	1.00
9	npSolver	1.0 (fixed)	84	84 MOPT	63%	63%	18974.29	142.12	225.88	18979.80	142.15	225.95	1.00
10	npSolver	inc (fixed)	84	84 MOPT	63%	63%	19134.37	146.00	227.79	19139.77	146.04	227.85	1.00
11	wbo	1.72	82	82 MOPT	62%	62%	24681.76	21.85	301.00	24691.31	21.92	301.11	1.00
12	wbo	1.71	82	82 MOPT	62%	62%	24833.47	21.97	302.85	24842.19	22.03	302.95	1.00
13	pwbo	2.01	73	73 MOPT	55%	55%	17201.95	33.09	235.64	8610.61	16.59	117.95	2.00
14	pwbo	2.02	71	71 MOPT	53%	53%	13956.68	35.00	196.57	6986.04	17.55	98.39	2.00
15	npSolver	inc-topDown	70	70 MOPT	53%	53%	32009.60	99.73	457.28	32057.72	100.11	457.97	1.00
16	npSolver	inc-topDown (fixed)	51	51 MOPT	38%	38%	27706.75	357.36	543.27	27426.59	357.45	537.78	1.01
17	npSolver	inc-topdown-quickBound	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
18	npSolver	inc-topdown-quickBound (fixed)	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 133
Virtual Best Solver (VBS)			133	100%	100%	1976.96	0.00	14.86	-	-	-	-
1	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	127	95%	95%	22308.29	0.86	175.66	22264.74	0.86	175.31	1.00
2	Sat4j PB	2012-05-28	97	73%	73%	8580.48	22.70	88.46	8413.13	20.70	86.73	1.02
3	clasp	2.0.6-R5325	95	71%	71%	3929.31	5.61	41.36	3927.15	4.78	41.34	1.00
4	toysat	2012-05-17	92	69%	69%	22308.85	59.18	242.49	22313.02	59.20	242.53	1.00
5	npSolver	1.0	90	68%	68%	16133.61	42.30	179.26	16165.46	42.75	179.62	1.00
6	npSolver	inc	89	67%	67%	13775.26	40.46	154.78	13824.22	40.51	155.33	1.00
7	wbo2satCp2	2012-05-19	89	67%	67%	13795.73	43.65	155.01	13810.24	42.13	155.17	1.00
8	wbo2sat	2012-05-19	87	65%	65%	10510.79	38.92	120.81	10544.35	39.00	121.20	1.00
9	npSolver	1.0 (fixed)	84	63%	63%	18974.29	142.12	225.88	18979.80	142.15	225.95	1.00
10	npSolver	inc (fixed)	84	63%	63%	19134.37	146.00	227.79	19139.77	146.04	227.85	1.00
11	wbo	1.72	82	62%	62%	24681.76	21.85	301.00	24691.31	21.92	301.11	1.00
12	wbo	1.71	82	62%	62%	24833.47	21.97	302.85	24842.19	22.03	302.95	1.00
13	pwbo	2.01	73	55%	55%	17201.95	33.09	235.64	8610.61	16.59	117.95	2.00
14	pwbo	2.02	71	53%	53%	13956.68	35.00	196.57	6986.04	17.55	98.39	2.00
15	npSolver	inc-topDown	70	53%	53%	32009.60	99.73	457.28	32057.72	100.11	457.97	1.00
16	npSolver	inc-topDown (fixed)	51	38%	38%	27706.75	357.36	543.27	27426.59	357.45	537.78	1.01
17	npSolver	inc-topdown-quickBound	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
18	npSolver	inc-topdown-quickBound (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 133
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	clasp	2.0.6-R5325	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	toysat	2012-05-17	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	pwbo	2.01	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	wbo	1.71	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
5	npSolver	1.0	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
6	npSolver	inc	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
7	npSolver	inc-topDown	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
8	npSolver	inc-topdown-quickBound	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
9	wbo2sat	2012-05-19	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
10	wbo2satCp2	2012-05-19	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
11	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
12	Sat4j PB	2012-05-28	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
13	wbo	1.72	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
14	pwbo	2.02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
15	npSolver	inc-topdown-quickBound (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
16	npSolver	inc-topDown (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
17	npSolver	inc (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
18	npSolver	1.0 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-