PB'12 competition: satisfaction and optimization track: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "no optimisation, big integers, linear constraints" (DEC-BIGINT-LIN): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 14
Virtual Best Solver (VBS)			14	13 SAT, 1 UNSAT	100%	100%	238.28	0.41	17.02	-	-	-	-
1	PB12: minisatp	1.0-2-g022594c	13	12 SAT, 1 UNSAT	93%	93%	2401.48	12.75	184.73	2402.00	12.75	184.77	1.00
2	PB07: minisat+	1.14	12	12 SAT	86%	86%	3727.14	192.18	310.59	3727.83	192.25	310.65	1.00
3	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	10	10 SAT	71%	71%	1772.83	14.28	177.28	934.62	7.14	93.46	1.90
4	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	9	9 SAT	64%	64%	2714.20	25.19	301.58	2714.76	25.19	301.64	1.00
5	PB11: Sat4j Res//CP	2.3.0	8	8 SAT	57%	57%	830.42	10.41	103.80	428.33	4.92	53.54	1.94
6	PB07: bsolo	3.0.17	7	7 SAT	50%	50%	188.23	0.49	26.89	188.29	0.50	26.90	1.00
7	SAT4J PB specific settings	2.3.2 snapshot	6	6 SAT	43%	43%	157.12	6.50	26.19	153.70	5.99	25.62	1.02
8	Sat 4j PB Resolution	2.3.2 Snapshot	6	6 SAT	43%	43%	1303.88	216.39	217.31	1293.62	214.59	215.60	1.01
9	npSolver	inc-topdown-quickBound (fixed)	6	6 SAT	43%	43%	2373.62	176.72	395.60	2374.00	176.75	395.67	1.00
10	npSolver	1.0 (fixed)	6	6 SAT	43%	43%	2378.06	177.49	396.34	2378.49	177.53	396.41	1.00
11	npSolver	inc (fixed)	6	6 SAT	43%	43%	2407.64	171.12	401.27	2408.07	171.15	401.35	1.00
12	npSolver	inc-topDown (fixed)	6	6 SAT	43%	43%	2415.40	171.49	402.57	2415.83	171.52	402.64	1.00
13	SAT 4j PB RES // CP	2.3.2 Snapshot	5	5 SAT	36%	36%	382.60	3.24	76.52	191.66	1.16	38.33	2.00
14	PB07: SAT4JPseudoResolution	2007-03-23	4	4 SAT	29%	29%	17.20	0.43	4.30	16.36	0.27	4.09	1.05
15	PB09: SAT4J Pseudo Resolution	2.1.1	3	3 SAT	21%	21%	1187.15	3.28	395.72	1183.91	2.96	394.64	1.00
16	toysat	2012-06-01	1	1 SAT	7%	7%	12.87	12.87	12.87	12.87	12.87	12.87	1.00
17	toysat	2012-05-17	1	1 SAT	7%	7%	13.02	13.02	13.02	13.02	13.02	13.02	1.00
18	pb2satCp2	2012-05-19	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
19	pb2sat	2012-05-19	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
20	npSolver	inc-topDown	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
21	npSolver	inc	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
22	npSolver	1.0	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
23	npSolver	inc-topdown-quickBound	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
24	PBPASSolver-CARD.DP	2012-05-28	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
25	PBPASSolver-CARD.SN	2012-05-28	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, big integers, linear constraints" (DEC-BIGINT-LIN): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 14
Virtual Best Solver (VBS)			13	93%	100%	238.09	0.42	18.31	-	-	-	-
1	PB12: minisatp	1.0-2-g022594c	12	86%	92%	2401.29	14.31	200.11	2401.81	14.32	200.15	1.00
2	PB07: minisat+	1.14	12	86%	92%	3727.14	192.18	310.59	3727.83	192.25	310.65	1.00
3	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	10	71%	77%	1772.83	14.28	177.28	934.62	7.14	93.46	1.90
4	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	9	64%	69%	2714.20	25.19	301.58	2714.76	25.19	301.64	1.00
5	PB11: Sat4j Res//CP	2.3.0	8	57%	62%	830.42	10.41	103.80	428.33	4.92	53.54	1.94
6	PB07: bsolo	3.0.17	7	50%	54%	188.23	0.49	26.89	188.29	0.50	26.90	1.00
7	SAT4J PB specific settings	2.3.2 snapshot	6	43%	46%	157.12	6.50	26.19	153.70	5.99	25.62	1.02
8	Sat 4j PB Resolution	2.3.2 Snapshot	6	43%	46%	1303.88	216.39	217.31	1293.62	214.59	215.60	1.01
9	npSolver	inc-topdown-quickBound (fixed)	6	43%	46%	2373.62	176.72	395.60	2374.00	176.75	395.67	1.00
10	npSolver	1.0 (fixed)	6	43%	46%	2378.06	177.49	396.34	2378.49	177.53	396.41	1.00
11	npSolver	inc (fixed)	6	43%	46%	2407.64	171.12	401.27	2408.07	171.15	401.35	1.00
12	npSolver	inc-topDown (fixed)	6	43%	46%	2415.40	171.49	402.57	2415.83	171.52	402.64	1.00
13	SAT 4j PB RES // CP	2.3.2 Snapshot	5	36%	38%	382.60	3.24	76.52	191.66	1.16	38.33	2.00
14	PB07: SAT4JPseudoResolution	2007-03-23	4	29%	31%	17.20	0.43	4.30	16.36	0.27	4.09	1.05
15	PB09: SAT4J Pseudo Resolution	2.1.1	3	21%	23%	1187.15	3.28	395.72	1183.91	2.96	394.64	1.00
16	toysat	2012-06-01	1	7%	8%	12.87	12.87	12.87	12.87	12.87	12.87	1.00
17	toysat	2012-05-17	1	7%	8%	13.02	13.02	13.02	13.02	13.02	13.02	1.00
18	pb2satCp2	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
19	pb2sat	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
20	npSolver	inc-topDown	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
21	npSolver	inc	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
22	npSolver	1.0	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
23	npSolver	inc-topdown-quickBound	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
24	PBPASSolver-CARD.DP	2012-05-28	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
25	PBPASSolver-CARD.SN	2012-05-28	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, big integers, linear constraints" (DEC-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 14
Virtual Best Solver (VBS)			1	7%	100%	0.19	0.19	0.19	-	-	-	-
1	PB12: minisatp	1.0-2-g022594c	1	7%	100%	0.19	0.19	0.19	0.19	0.19	0.19	0.99
2	SAT 4j PB RES // CP	2.3.2 Snapshot	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
3	Sat 4j PB Resolution	2.3.2 Snapshot	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
4	pb2satCp2	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
5	pb2sat	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
6	npSolver	inc-topDown	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
7	npSolver	inc	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
8	npSolver	1.0	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
9	npSolver	inc-topdown-quickBound	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
10	toysat	2012-05-17	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
11	SAT4J PB specific settings	2.3.2 snapshot	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
12	PBPASSolver-CARD.DP	2012-05-28	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
13	PBPASSolver-CARD.SN	2012-05-28	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
14	PB07: minisat+	1.14	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
15	toysat	2012-06-01	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
16	PB07: bsolo	3.0.17	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
17	PB07: SAT4JPseudoResolution	2007-03-23	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
18	PB09: SAT4J Pseudo Resolution	2.1.1	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
19	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
20	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
21	PB11: Sat4j Res//CP	2.3.0	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
22	npSolver	inc-topDown (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
23	npSolver	inc (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
24	npSolver	1.0 (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
25	npSolver	inc-topdown-quickBound (fixed)	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 355
Virtual Best Solver (VBS)			284	156 SAT, 128 UNSAT	80%	100%	17716.96	0.09	62.38	-	-	-	-
1	SAT 4j PB RES // CP	2.3.2 Snapshot	249	130 SAT, 119 UNSAT	70%	88%	20712.25	4.34	83.18	10523.82	1.69	42.26	1.97
2	PB11: Sat4j Res//CP	2.3.0	247	129 SAT, 118 UNSAT	70%	87%	29213.16	4.27	118.27	15226.69	1.84	61.65	1.92
3	clasp	2.0.6-R5325 (dec)	246	149 SAT, 97 UNSAT	69%	87%	16238.93	0.79	66.01	16242.88	0.79	66.03	1.00
4	PB11: borg	pb-dec-11.04.03	243	124 SAT, 119 UNSAT	68%	86%	9046.44	1.54	37.23	9825.20	1.83	40.43	0.92
5	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	243	129 SAT, 114 UNSAT	68%	86%	29282.66	4.12	120.50	15116.66	2.17	62.21	1.94
6	PB10: borg-pb	10.05.30	241	129 SAT, 112 UNSAT	68%	85%	27049.74	1.68	112.24	25824.18	2.19	107.15	1.05
7	PB07: Pueblo	1.4	240	128 SAT, 112 UNSAT	68%	85%	14689.13	2.35	61.20	14697.31	2.35	61.24	1.00
8	bsolo	3.2	238	123 SAT, 115 UNSAT	67%	84%	32061.27	2.31	134.71	32069.57	2.31	134.75	1.00
9	Sat 4j PB Resolution	2.3.2 Snapshot	231	140 SAT, 91 UNSAT	65%	81%	23379.18	2.35	101.21	23022.54	1.73	99.66	1.02
10	pwbo	2.0	228	134 SAT, 94 UNSAT	64%	80%	25291.30	1.29	110.93	25299.75	1.30	110.96	1.00
11	PB07: PB-clasp	2007-04-10	226	133 SAT, 93 UNSAT	64%	80%	21931.14	2.30	97.04	21814.83	2.22	96.53	1.01
12	wbo	1.72	226	131 SAT, 95 UNSAT	64%	80%	24929.27	1.18	110.31	24937.51	1.18	110.34	1.00
13	pwbo	2.02	226	131 SAT, 95 UNSAT	64%	80%	25122.90	1.13	111.16	25126.80	1.13	111.18	1.00
14	wbo	1.7	225	131 SAT, 94 UNSAT	63%	79%	22817.96	1.03	101.41	22825.84	1.03	101.45	1.00
15	npSolver	1.0 (fixed)	222	125 SAT, 97 UNSAT	63%	78%	23804.93	3.72	107.23	23816.40	3.81	107.28	1.00
16	PB07: minisat+	1.14	221	130 SAT, 91 UNSAT	62%	78%	29690.60	13.23	134.35	29677.86	13.24	134.29	1.00
17	PB12: minisatp	1.0-2-g022594c	220	129 SAT, 91 UNSAT	62%	77%	25319.13	6.32	115.09	25321.18	6.18	115.10	1.00
18	SAT4J PB specific settings	2.3.2 snapshot	205	114 SAT, 91 UNSAT	58%	72%	15562.57	2.91	75.91	15291.72	2.13	74.59	1.02
19	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	203	90 SAT, 113 UNSAT	57%	71%	21845.41	1.35	107.61	21849.55	1.35	107.63	1.00
20	toysat	2012-06-01	198	114 SAT, 84 UNSAT	56%	70%	11058.81	5.83	55.85	11028.39	5.84	55.70	1.00
21	PB07: bsolo	3.0.17	198	113 SAT, 85 UNSAT	56%	70%	26229.56	4.92	132.47	26236.57	4.93	132.51	1.00
22	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	197	84 SAT, 113 UNSAT	55%	69%	21832.01	1.23	110.82	21830.94	1.23	110.82	1.00
23	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	196	83 SAT, 113 UNSAT	55%	69%	21931.23	1.16	111.89	21898.52	1.16	111.73	1.00
24	PB09: SAT4J Pseudo Resolution	2.1.1	194	106 SAT, 88 UNSAT	55%	68%	13270.92	2.45	68.41	13046.49	1.84	67.25	1.02
25	PB11: SCIP spx E_2	2011-06-10 (fixed)	194	81 SAT, 113 UNSAT	55%	68%	18869.36	1.12	97.26	18863.35	1.12	97.23	1.00
26	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	194	81 SAT, 113 UNSAT	55%	68%	25244.17	1.35	130.12	25249.01	1.35	130.15	1.00
27	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	193	81 SAT, 112 UNSAT	54%	68%	15026.12	1.38	77.86	15029.21	1.38	77.87	1.00
28	PB07: SAT4JPseudoResolution	2007-03-23	192	103 SAT, 89 UNSAT	54%	68%	14098.26	2.72	73.43	13850.42	2.13	72.14	1.02
29	PB10: pb_cplex	2010-06-29	192	88 SAT, 104 UNSAT	54%	68%	17652.40	0.24	91.94	7615.46	0.24	39.66	2.32
30	npSolver	inc-topdown-quickBound (fixed)	182	85 SAT, 97 UNSAT	51%	64%	22022.99	3.35	121.01	22037.55	3.38	121.09	1.00
31	npSolver	inc-topDown (fixed)	181	84 SAT, 97 UNSAT	51%	64%	20300.04	3.24	112.15	20313.75	3.24	112.23	1.00
32	npSolver	inc (fixed)	181	85 SAT, 96 UNSAT	51%	64%	20513.91	3.43	113.34	20533.02	3.43	113.44	1.00
33	PBPASSolver-CARD.SN	2012-05-28	165	98 SAT, 67 UNSAT	46%	58%	42711.65	136.00	258.86	42741.91	136.09	259.04	1.00
34	PBPASSolver-CARD.DP	2012-05-28	144	79 SAT, 65 UNSAT	41%	51%	32193.88	134.76	223.57	31673.93	134.30	219.96	1.02
35	pb2satCp2	2012-05-19	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	pb2sat	2012-05-19	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	npSolver	inc-topDown	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
38	npSolver	inc	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
39	npSolver	1.0	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
40	npSolver	inc-topdown-quickBound	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 355
Virtual Best Solver (VBS)			156	44%	100%	12673.28	0.37	81.24	-	-	-	-
1	clasp	2.0.6-R5325 (dec)	149	42%	96%	11131.85	1.33	74.71	11134.70	1.34	74.73	1.00
2	Sat 4j PB Resolution	2.3.2 Snapshot	140	39%	90%	15752.03	3.00	112.51	15479.86	1.97	110.57	1.02
3	pwbo	2.0	134	38%	86%	13819.43	0.91	103.13	13823.57	0.91	103.16	1.00
4	PB07: PB-clasp	2007-04-10	133	37%	85%	17073.75	2.53	128.37	16968.44	2.27	127.58	1.01
5	wbo	1.7	131	37%	84%	11879.60	0.87	90.68	11884.16	0.87	90.72	1.00
6	wbo	1.72	131	37%	84%	11999.18	0.91	91.60	12003.31	0.91	91.63	1.00
7	pwbo	2.02	131	37%	84%	12002.80	0.91	91.62	12007.31	0.91	91.66	1.00
8	SAT 4j PB RES // CP	2.3.2 Snapshot	130	37%	83%	14513.99	5.20	111.65	7418.93	2.29	57.07	1.96
9	PB07: minisat+	1.14	130	37%	83%	19037.37	14.38	146.44	19021.88	14.44	146.32	1.00
10	PB10: borg-pb	10.05.30	129	36%	83%	10510.60	1.43	81.48	10111.88	2.17	78.39	1.04
11	PB11: Sat4j Res//CP	2.3.0	129	36%	83%	13772.36	4.67	106.76	7025.03	2.72	54.46	1.96
12	PB12: minisatp	1.0-2-g022594c	129	36%	83%	14567.97	6.95	112.93	14567.54	6.38	112.93	1.00
13	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	129	36%	83%	18402.80	5.25	142.66	9357.39	3.23	72.54	1.97
14	PB07: Pueblo	1.4	128	36%	82%	8111.17	2.70	63.37	8115.58	2.74	63.40	1.00
15	npSolver	1.0 (fixed)	125	35%	80%	17510.71	5.18	140.09	17519.03	5.19	140.15	1.00
16	PB11: borg	pb-dec-11.04.03	124	35%	79%	5312.01	1.89	42.84	6076.40	2.74	49.00	0.87
17	bsolo	3.2	123	35%	79%	15143.66	5.48	123.12	15149.28	5.48	123.16	1.00
18	SAT4J PB specific settings	2.3.2 snapshot	114	32%	73%	7862.84	2.50	68.97	7690.29	1.98	67.46	1.02
19	toysat	2012-06-01	114	32%	73%	7946.51	5.83	69.71	7941.28	5.83	69.66	1.00
20	PB07: bsolo	3.0.17	113	32%	72%	7683.30	3.12	67.99	7686.55	3.13	68.02	1.00
21	PB09: SAT4J Pseudo Resolution	2.1.1	106	30%	68%	8879.50	2.55	83.77	8721.03	1.92	82.27	1.02
22	PB07: SAT4JPseudoResolution	2007-03-23	103	29%	66%	6186.41	2.85	60.06	6042.53	2.35	58.67	1.02
23	PBPASSolver-CARD.SN	2012-05-28	98	28%	63%	29780.93	135.24	303.89	29824.54	135.43	304.33	1.00
24	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	90	25%	58%	13112.48	6.70	145.69	13114.92	6.71	145.72	1.00
25	PB10: pb_cplex	2010-06-29	88	25%	56%	2297.39	0.86	26.11	1267.89	0.75	14.41	1.81
26	npSolver	inc-topdown-quickBound (fixed)	85	24%	54%	15650.16	5.79	184.12	15655.06	5.79	184.18	1.00
27	npSolver	inc (fixed)	85	24%	54%	15800.09	5.69	185.88	15809.56	5.70	185.99	1.00
28	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	84	24%	54%	13006.33	3.46	154.84	13003.43	3.46	154.80	1.00
29	npSolver	inc-topDown (fixed)	84	24%	54%	13863.37	5.65	165.04	13866.97	5.67	165.08	1.00
30	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	83	23%	53%	12023.47	3.42	144.86	11988.79	3.42	144.44	1.00
31	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	81	23%	52%	3719.02	7.87	45.91	3719.94	7.88	45.93	1.00
32	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	81	23%	52%	9391.95	4.00	115.95	9393.75	4.00	115.97	1.00
33	PB11: SCIP spx E_2	2011-06-10 (fixed)	81	23%	52%	11127.51	3.75	137.38	11119.85	3.76	137.28	1.00
34	PBPASSolver-CARD.DP	2012-05-28	79	22%	51%	21605.34	133.34	273.49	21111.58	133.91	267.24	1.02
35	pb2satCp2	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	pb2sat	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	npSolver	inc-topDown	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
38	npSolver	inc	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
39	npSolver	1.0	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
40	npSolver	inc-topdown-quickBound	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, linear constraints" (DEC-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 355
Virtual Best Solver (VBS)			128	36%	100%	5043.69	0.07	39.40	-	-	-	-
1	PB11: borg	pb-dec-11.04.03	119	34%	93%	3734.43	1.21	31.38	3748.80	1.44	31.50	1.00
2	SAT 4j PB RES // CP	2.3.2 Snapshot	119	34%	93%	6198.26	2.52	52.09	3104.89	1.21	26.09	2.00
3	PB11: Sat4j Res//CP	2.3.0	118	33%	92%	15440.79	2.11	130.85	8201.66	1.09	69.51	1.88
4	bsolo	3.2	115	32%	90%	16917.62	0.96	147.11	16920.28	0.96	147.13	1.00
5	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	114	32%	89%	10879.86	2.00	95.44	5759.27	1.53	50.52	1.89
6	PB11: SCIP spx E_2	2011-06-10 (fixed)	113	32%	88%	7741.85	0.34	68.51	7743.51	0.34	68.53	1.00
7	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	113	32%	88%	8732.93	0.53	77.28	8734.63	0.53	77.30	1.00
8	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	113	32%	88%	8825.68	0.44	78.10	8827.51	0.44	78.12	1.00
9	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	113	32%	88%	9907.76	0.41	87.68	9909.73	0.41	87.70	1.00
10	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	113	32%	88%	15852.22	0.37	140.29	15855.26	0.38	140.31	1.00
11	PB07: Pueblo	1.4	112	32%	88%	6577.95	1.13	58.73	6581.72	1.13	58.77	1.00
12	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	112	32%	88%	11307.10	0.45	100.96	11309.27	0.45	100.98	1.00
13	PB10: borg-pb	10.05.30	112	32%	88%	16539.14	2.01	147.67	15712.30	2.26	140.29	1.05
14	PB10: pb_cplex	2010-06-29	104	29%	81%	15355.01	0.14	147.64	6347.58	0.15	61.03	2.42
15	clasp	2.0.6-R5325 (dec)	97	27%	76%	5107.08	0.56	52.65	5108.18	0.56	52.66	1.00
16	npSolver	1.0 (fixed)	97	27%	76%	6294.23	1.43	64.89	6297.37	1.52	64.92	1.00
17	npSolver	inc-topdown-quickBound (fixed)	97	27%	76%	6372.83	1.37	65.70	6382.49	1.43	65.80	1.00
18	npSolver	inc-topDown (fixed)	97	27%	76%	6436.68	1.37	66.36	6446.77	1.43	66.46	1.00
19	npSolver	inc (fixed)	96	27%	75%	4713.83	1.35	49.10	4723.47	1.53	49.20	1.00
20	wbo	1.72	95	27%	74%	12930.10	2.47	136.11	12934.20	2.48	136.15	1.00
21	pwbo	2.02	95	27%	74%	13120.09	2.44	138.11	13119.49	2.44	138.10	1.00
22	wbo	1.7	94	26%	73%	10938.36	2.40	116.37	10941.68	2.40	116.40	1.00
23	pwbo	2.0	94	26%	73%	11471.87	2.56	122.04	11476.18	2.56	122.09	1.00
24	PB07: PB-clasp	2007-04-10	93	26%	73%	4857.40	2.25	52.23	4846.39	2.14	52.11	1.00
25	Sat 4j PB Resolution	2.3.2 Snapshot	91	26%	71%	7627.14	2.13	83.81	7542.67	1.70	82.89	1.01
26	SAT4J PB specific settings	2.3.2 snapshot	91	26%	71%	7699.73	3.37	84.61	7601.44	2.28	83.53	1.01
27	PB07: minisat+	1.14	91	26%	71%	10653.23	5.23	117.07	10655.98	5.23	117.10	1.00
28	PB12: minisatp	1.0-2-g022594c	91	26%	71%	10751.16	3.97	118.14	10753.64	3.97	118.17	1.00
29	PB07: SAT4JPseudoResolution	2007-03-23	89	25%	70%	7911.85	2.53	88.90	7807.88	2.05	87.73	1.01
30	PB09: SAT4J Pseudo Resolution	2.1.1	88	25%	69%	4391.41	2.40	49.90	4325.45	1.76	49.15	1.02
31	PB07: bsolo	3.0.17	85	24%	66%	18546.26	11.46	218.19	18550.02	11.48	218.24	1.00
32	toysat	2012-06-01	84	24%	66%	3112.30	5.83	37.05	3087.10	5.84	36.75	1.01
33	PBPASSolver-CARD.SN	2012-05-28	67	19%	52%	12930.72	136.64	193.00	12917.37	137.19	192.80	1.00
34	PBPASSolver-CARD.DP	2012-05-28	65	18%	51%	10588.54	138.73	162.90	10562.35	137.32	162.50	1.00
35	pb2satCp2	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	pb2sat	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	npSolver	inc-topDown	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
38	npSolver	inc	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
39	npSolver	1.0	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
40	npSolver	inc-topdown-quickBound	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 30
Virtual Best Solver (VBS)			25	16 SAT, 9 UNSAT	83%	100%	4574.81	0.21	182.99	-	-	-	-
1	PB10: pb_cplex	2010-06-29	24	15 SAT, 9 UNSAT	80%	96%	4623.78	0.62	192.66	1405.94	0.62	58.58	3.29
2	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	23	16 SAT, 7 UNSAT	77%	92%	3024.43	5.44	131.50	3025.03	5.44	131.52	1.00
3	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	22	16 SAT, 6 UNSAT	73%	88%	731.90	8.34	33.27	732.14	8.34	33.28	1.00
4	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	22	16 SAT, 6 UNSAT	73%	88%	1783.10	13.76	81.05	1783.49	13.76	81.07	1.00
5	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	22	16 SAT, 6 UNSAT	73%	88%	2590.34	16.76	117.74	2590.91	16.77	117.77	1.00
6	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	22	16 SAT, 6 UNSAT	73%	88%	3010.31	7.27	136.83	3010.92	7.27	136.86	1.00
7	PB11: SCIP spx E_2	2011-06-10 (fixed)	22	16 SAT, 6 UNSAT	73%	88%	3339.95	23.22	151.82	3340.58	23.23	151.84	1.00
8	PB11: borg	pb-dec-11.04.03	21	15 SAT, 6 UNSAT	70%	84%	1320.33	0.98	62.87	1320.66	1.75	62.89	1.00
9	SAT4J PB specific settings	2.3.2 snapshot	21	16 SAT, 5 UNSAT	70%	84%	1838.01	2.30	87.52	1809.08	1.15	86.15	1.02
10	npSolver	1.0 (fixed)	20	15 SAT, 5 UNSAT	67%	80%	80.24	0.90	4.01	80.77	0.90	4.04	0.99
11	PB07: minisat+	1.14	20	15 SAT, 5 UNSAT	67%	80%	83.41	1.50	4.17	83.45	1.50	4.17	1.00
12	PB07: PB-clasp	2007-04-10	20	15 SAT, 5 UNSAT	67%	80%	193.92	2.96	9.70	192.29	2.82	9.61	1.01
13	SAT 4j PB RES // CP	2.3.2 Snapshot	20	15 SAT, 5 UNSAT	67%	80%	214.81	4.80	10.74	100.52	1.92	5.03	2.14
14	PB11: Sat4j Res//CP	2.3.0	20	15 SAT, 5 UNSAT	67%	80%	301.21	3.53	15.06	149.68	1.60	7.48	2.01
15	clasp	2.0.6-R5325 (dec)	20	15 SAT, 5 UNSAT	67%	80%	596.09	2.96	29.80	596.25	2.96	29.81	1.00
16	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	20	15 SAT, 5 UNSAT	67%	80%	611.19	11.57	30.56	460.19	6.18	23.01	1.33
17	PB07: Pueblo	1.4	20	15 SAT, 5 UNSAT	67%	80%	883.92	0.35	44.20	884.12	0.35	44.21	1.00
18	toysat	2012-06-01	20	15 SAT, 5 UNSAT	67%	80%	971.81	0.20	48.59	972.07	0.20	48.60	1.00
19	toysat	2012-05-17	20	15 SAT, 5 UNSAT	67%	80%	979.18	0.20	48.96	979.40	0.20	48.97	1.00
20	PB09: SAT4J Pseudo Resolution	2.1.1	20	15 SAT, 5 UNSAT	67%	80%	1416.28	4.66	70.81	1395.61	3.71	69.78	1.01
21	pwbo	2.0	20	15 SAT, 5 UNSAT	67%	80%	5034.44	2.16	251.72	5035.94	2.16	251.80	1.00
22	npSolver	inc-topdown-quickBound (fixed)	19	14 SAT, 5 UNSAT	63%	76%	78.85	0.91	4.15	79.24	0.91	4.17	1.00
23	npSolver	inc-topDown (fixed)	19	14 SAT, 5 UNSAT	63%	76%	79.78	0.94	4.20	80.35	0.94	4.23	0.99
24	npSolver	inc (fixed)	19	14 SAT, 5 UNSAT	63%	76%	79.80	0.94	4.20	80.28	0.94	4.23	0.99
25	wbo	1.7	19	14 SAT, 5 UNSAT	63%	76%	2498.54	1.91	131.50	2498.99	1.91	131.53	1.00
26	pwbo	2.02	19	14 SAT, 5 UNSAT	63%	76%	2563.99	1.91	134.95	2564.93	1.91	135.00	1.00
27	wbo	1.72	19	14 SAT, 5 UNSAT	63%	76%	2574.54	1.91	135.50	2575.35	1.91	135.54	1.00
28	PB09: bsolo	3.1	18	13 SAT, 5 UNSAT	60%	72%	1200.37	2.92	66.69	1200.79	2.92	66.71	1.00
29	bsolo	3.2	18	13 SAT, 5 UNSAT	60%	72%	1433.99	11.08	79.67	1434.44	11.09	79.69	1.00
30	Sat 4j PB Resolution	2.3.2 Snapshot	16	11 SAT, 5 UNSAT	53%	64%	4460.15	30.57	278.76	4351.34	29.07	271.96	1.03
31	PB10: borg-pb	10.05.30	8	5 SAT, 3 UNSAT	27%	32%	1961.80	217.61	245.23	1870.30	207.92	233.79	1.05
32	PB07: bsolo	3.0.17	7	4 SAT, 3 UNSAT	23%	28%	201.96	6.86	28.85	202.06	6.87	28.87	1.00
33	pb2satCp2	2012-05-19	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
34	pb2sat	2012-05-19	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
35	npSolver	inc-topDown	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	npSolver	inc	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	npSolver	1.0	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
38	npSolver	inc-topdown-quickBound	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
39	PB12: minisatp	1.0-2-g022594c	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-
40	PB07: SAT4JPseudoResolution	2007-03-23	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): SAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 30
Virtual Best Solver (VBS)			16	53%	100%	97.66	0.14	6.10	-	-	-	-
1	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	16	53%	100%	601.90	9.29	37.62	602.09	9.30	37.63	1.00
2	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	16	53%	100%	950.04	4.61	59.38	950.24	4.62	59.39	1.00
3	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	16	53%	100%	1422.54	13.76	88.91	1422.85	13.76	88.93	1.00
4	SAT4J PB specific settings	2.3.2 snapshot	16	53%	100%	1739.50	2.50	108.72	1713.87	1.19	107.12	1.01
5	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	16	53%	100%	1873.43	16.76	117.09	1873.87	16.77	117.12	1.00
6	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	16	53%	100%	2990.80	23.04	186.93	2991.40	23.05	186.96	1.00
7	PB11: SCIP spx E_2	2011-06-10 (fixed)	16	53%	100%	3321.22	69.12	207.58	3321.83	69.14	207.61	1.00
8	toysat	2012-05-17	15	50%	94%	4.82	0.21	0.32	4.85	0.21	0.32	0.99
9	toysat	2012-06-01	15	50%	94%	4.97	0.21	0.33	4.99	0.21	0.33	1.00
10	PB10: pb_cplex	2010-06-29	15	50%	94%	16.78	0.51	1.12	16.80	0.51	1.12	1.00
11	npSolver	1.0 (fixed)	15	50%	94%	27.81	0.96	1.85	28.34	0.97	1.89	0.98
12	PB07: minisat+	1.14	15	50%	94%	54.80	1.71	3.65	54.83	1.71	3.66	1.00
13	PB11: borg	pb-dec-11.04.03	15	50%	94%	100.25	0.98	6.68	102.03	1.75	6.80	0.98
14	SAT 4j PB RES // CP	2.3.2 Snapshot	15	50%	94%	114.39	4.86	7.63	48.89	1.92	3.26	2.34
15	PB07: PB-clasp	2007-04-10	15	50%	94%	131.28	4.77	8.75	129.68	4.57	8.65	1.01
16	PB11: Sat4j Res//CP	2.3.0	15	50%	94%	158.95	3.55	10.60	76.19	1.60	5.08	2.09
17	PB09: SAT4J Pseudo Resolution	2.1.1	15	50%	94%	336.91	5.42	22.46	322.70	4.28	21.51	1.04
18	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	15	50%	94%	465.32	13.26	31.02	383.76	7.30	25.58	1.21
19	clasp	2.0.6-R5325 (dec)	15	50%	94%	574.64	3.25	38.31	574.78	3.25	38.32	1.00
20	PB07: Pueblo	1.4	15	50%	94%	869.73	0.36	57.98	869.92	0.36	57.99	1.00
21	pwbo	2.0	15	50%	94%	2863.81	2.31	190.92	2864.65	2.31	190.98	1.00
22	npSolver	inc-topDown (fixed)	14	47%	88%	25.95	1.74	1.85	26.27	1.74	1.88	0.99
23	npSolver	inc-topdown-quickBound (fixed)	14	47%	88%	26.78	1.62	1.91	26.96	1.62	1.93	0.99
24	npSolver	inc (fixed)	14	47%	88%	27.41	1.75	1.96	27.73	1.85	1.98	0.99
25	wbo	1.72	14	47%	88%	923.07	2.03	65.93	923.34	2.03	65.95	1.00
26	pwbo	2.02	14	47%	88%	944.20	2.03	67.44	944.75	2.04	67.48	1.00
27	wbo	1.7	14	47%	88%	946.38	2.03	67.60	946.69	2.03	67.62	1.00
28	PB09: bsolo	3.1	13	43%	81%	700.37	4.28	53.87	700.66	4.28	53.90	1.00
29	bsolo	3.2	13	43%	81%	848.82	18.46	65.29	849.14	18.49	65.32	1.00
30	Sat 4j PB Resolution	2.3.2 Snapshot	11	37%	69%	4401.74	136.15	400.16	4295.94	133.24	390.54	1.02
31	PB10: borg-pb	10.05.30	5	17%	31%	1084.76	216.77	216.95	1033.81	207.31	206.76	1.05
32	PB07: bsolo	3.0.17	4	13%	25%	201.94	31.22	50.49	202.03	31.24	50.51	1.00
33	pb2satCp2	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
34	pb2sat	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
35	npSolver	inc-topDown	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	npSolver	inc	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	npSolver	1.0	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
38	npSolver	inc-topdown-quickBound	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
39	PB12: minisatp	1.0-2-g022594c	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
40	PB07: SAT4JPseudoResolution	2007-03-23	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "no optimisation, small integers, non linear constraints" (DEC-SMALLINT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 30
Virtual Best Solver (VBS)			9	30%	100%	4477.15	0.83	497.46	-	-	-	-
1	PB10: pb_cplex	2010-06-29	9	30%	100%	4607.00	13.29	511.89	1389.14	11.40	154.35	3.32
2	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	7	23%	78%	2074.39	27.04	296.34	2074.79	27.04	296.40	1.00
3	PB11: SCIP spx E_2	2011-06-10 (fixed)	6	20%	67%	18.73	0.39	3.12	18.74	0.39	3.12	1.00
4	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	6	20%	67%	19.51	0.41	3.25	19.52	0.41	3.25	1.00
5	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	6	20%	67%	130.00	4.09	21.67	130.05	4.09	21.67	1.00
6	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	6	20%	67%	360.56	10.77	60.09	360.64	10.78	60.11	1.00
7	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	6	20%	67%	716.91	20.34	119.49	717.04	20.35	119.51	1.00
8	PB11: borg	pb-dec-11.04.03	6	20%	67%	1220.08	7.52	203.35	1218.63	7.99	203.11	1.00
9	PB07: Pueblo	1.4	5	17%	56%	14.19	0.00	2.84	14.20	0.01	2.84	1.00
10	clasp	2.0.6-R5325 (dec)	5	17%	56%	21.46	0.00	4.29	21.47	0.01	4.29	1.00
11	PB07: minisat+	1.14	5	17%	56%	28.61	0.01	5.72	28.62	0.01	5.72	1.00
12	npSolver	inc-topdown-quickBound (fixed)	5	17%	56%	52.08	0.03	10.42	52.28	0.03	10.46	1.00
13	npSolver	inc (fixed)	5	17%	56%	52.39	0.03	10.48	52.55	0.03	10.51	1.00
14	npSolver	1.0 (fixed)	5	17%	56%	52.42	0.04	10.48	52.43	0.04	10.49	1.00
15	npSolver	inc-topDown (fixed)	5	17%	56%	53.82	0.02	10.76	54.08	0.03	10.82	1.00
16	Sat 4j PB Resolution	2.3.2 Snapshot	5	17%	56%	58.42	0.28	11.68	55.40	0.21	11.08	1.05
17	PB07: PB-clasp	2007-04-10	5	17%	56%	62.64	0.02	12.53	62.61	0.02	12.52	1.00
18	SAT4J PB specific settings	2.3.2 snapshot	5	17%	56%	98.52	0.32	19.70	95.21	0.23	19.04	1.03
19	SAT 4j PB RES // CP	2.3.2 Snapshot	5	17%	56%	100.42	0.55	20.08	51.63	0.66	10.33	1.95
20	PB11: Sat4j Res//CP	2.3.0	5	17%	56%	142.26	0.59	28.45	73.49	0.65	14.70	1.94
21	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	5	17%	56%	145.87	0.63	29.17	76.43	1.17	15.29	1.91
22	PB09: bsolo	3.1	5	17%	56%	500.00	0.00	100.00	500.13	0.01	100.03	1.00
23	bsolo	3.2	5	17%	56%	585.17	0.00	117.03	585.30	0.01	117.06	1.00
24	toysat	2012-06-01	5	17%	56%	966.84	0.01	193.37	967.08	0.02	193.42	1.00
25	toysat	2012-05-17	5	17%	56%	974.35	0.01	194.87	974.56	0.01	194.91	1.00
26	PB09: SAT4J Pseudo Resolution	2.1.1	5	17%	56%	1079.37	0.35	215.87	1072.91	0.23	214.58	1.01
27	wbo	1.7	5	17%	56%	1552.15	0.00	310.43	1552.31	0.01	310.46	1.00
28	pwbo	2.02	5	17%	56%	1619.79	0.00	323.96	1620.18	0.01	324.04	1.00
29	wbo	1.72	5	17%	56%	1651.46	0.00	330.29	1652.01	0.01	330.40	1.00
30	pwbo	2.0	5	17%	56%	2170.64	0.00	434.13	2171.30	0.01	434.26	1.00
31	PB07: bsolo	3.0.17	3	10%	33%	0.02	0.01	0.01	0.03	0.01	0.01	0.54
32	PB10: borg-pb	10.05.30	3	10%	33%	877.04	437.67	292.35	836.49	416.20	278.83	1.05
33	pb2satCp2	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
34	pb2sat	2012-05-19	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
35	npSolver	inc-topDown	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
36	npSolver	inc	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
37	npSolver	1.0	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
38	npSolver	inc-topdown-quickBound	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
39	PB12: minisatp	1.0-2-g022594c	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-
40	PB07: SAT4JPseudoResolution	2007-03-23	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 416
Virtual Best Solver (VBS)			125	102 OPT, 23 UNSAT	30%	100%	19400.40	3.46	155.20	-	-	-	-
1	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	112	89 OPT, 23 UNSAT	27%	90%	21308.43	7.06	190.25	12632.51	6.46	112.79	1.69
2	PB11: Sat4j Res//CP	2.3.0	108	85 OPT, 23 UNSAT	26%	86%	14791.41	5.66	136.96	8057.37	3.87	74.61	1.84
3	SAT 4j PB RES // CP	2.3.2 Snapshot	103	80 OPT, 23 UNSAT	25%	82%	13299.06	4.16	129.12	7422.13	3.00	72.06	1.79
4	Sat 4j PB Resolution	2.3.2 Snapshot	97	74 OPT, 23 UNSAT	23%	78%	7759.30	2.81	79.99	7615.27	1.66	78.51	1.02
5	PB07: SAT4JPseudoResolution	2007-03-23	89	66 OPT, 23 UNSAT	21%	71%	8096.54	4.12	90.97	7955.89	2.78	89.39	1.02
6	PB09: SAT4J Pseudo Resolution	2.1.1	88	65 OPT, 23 UNSAT	21%	70%	6760.49	3.52	76.82	6606.17	2.26	75.07	1.02
7	SAT4J PB specific settings	2.3.2 snapshot	77	54 OPT, 23 UNSAT	19%	62%	13683.56	5.91	177.71	13533.54	4.68	175.76	1.01
8	toysat	2012-05-17	59	40 OPT, 19 UNSAT	14%	47%	12383.83	5.15	209.90	12386.10	5.15	209.93	1.00
9	PB12: minisatp	1.0-2-g022594c	38	15 OPT, 23 UNSAT	9%	30%	3489.43	0.09	91.83	3490.37	0.09	91.85	1.00
10	npSolver	1.0 (fixed)	37	18 OPT, 19 UNSAT	9%	30%	2111.93	1.85	57.08	2122.32	1.85	57.36	1.00
11	PB07: minisat+	1.14	36	13 OPT, 23 UNSAT	9%	29%	7899.89	0.59	219.44	7901.51	0.59	219.49	1.00
12	npSolver	inc-topDown (fixed)	34	15 OPT, 19 UNSAT	8%	27%	10637.36	0.89	312.86	10641.46	0.89	312.98	1.00
13	npSolver	1.0	33	17 OPT, 16 UNSAT	8%	26%	3829.85	0.23	116.06	3831.63	0.24	116.11	1.00
14	pb2sat	2012-05-19	31	15 OPT, 16 UNSAT	7%	25%	3813.18	0.16	123.01	3818.08	0.17	123.16	1.00
15	npSolver	inc (fixed)	30	11 OPT, 19 UNSAT	7%	24%	1456.15	0.19	48.54	1458.83	0.26	48.63	1.00
16	pb2satCp2	2012-05-19	29	13 OPT, 16 UNSAT	7%	23%	6853.68	0.16	236.33	6857.95	0.16	236.48	1.00
17	npSolver	inc-topdown-quickBound (fixed)	27	8 OPT, 19 UNSAT	6%	22%	2135.81	0.17	79.10	2141.52	0.18	79.32	1.00
18	npSolver	inc	25	9 OPT, 16 UNSAT	6%	20%	2162.68	0.20	86.51	2164.22	0.21	86.57	1.00
19	npSolver	inc-topDown	24	8 OPT, 16 UNSAT	6%	19%	4466.99	0.20	186.12	4205.20	0.20	175.22	1.06
20	npSolver	inc-topdown-quickBound	18	2 OPT, 16 UNSAT	4%	14%	1191.28	0.16	66.18	1191.88	0.18	66.22	1.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 416
Virtual Best Solver (VBS)			102	25%	100%	19395.21	5.41	190.15	-	-	-	-
1	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	89	21%	87%	21259.75	14.01	238.87	12595.99	10.78	141.53	1.69
2	PB11: Sat4j Res//CP	2.3.0	85	20%	83%	14746.07	11.33	173.48	8026.98	7.57	94.44	1.84
3	SAT 4j PB RES // CP	2.3.2 Snapshot	80	19%	78%	13249.81	6.37	165.62	7393.32	3.96	92.42	1.79
4	Sat 4j PB Resolution	2.3.2 Snapshot	74	18%	73%	7731.50	4.10	104.48	7600.16	2.75	102.70	1.02
5	PB07: SAT4JPseudoResolution	2007-03-23	66	16%	65%	8061.35	5.71	122.14	7935.55	4.44	120.24	1.02
6	PB09: SAT4J Pseudo Resolution	2.1.1	65	16%	64%	6727.63	6.93	103.50	6586.34	5.46	101.33	1.02
7	SAT4J PB specific settings	2.3.2 snapshot	54	13%	53%	13658.21	10.32	252.93	13519.02	8.94	250.35	1.01
8	toysat	2012-05-17	40	10%	39%	11102.31	40.89	277.56	11104.30	40.90	277.61	1.00
9	npSolver	1.0 (fixed)	18	4%	18%	1956.98	66.15	108.72	1960.35	66.65	108.91	1.00
10	npSolver	1.0	17	4%	17%	3827.40	89.53	225.14	3829.15	89.68	225.24	1.00
11	PB12: minisatp	1.0-2-g022594c	15	4%	15%	3483.57	74.49	232.24	3484.47	74.53	232.30	1.00
12	pb2sat	2012-05-19	15	4%	15%	3811.38	113.96	254.09	3816.26	114.09	254.42	1.00
13	npSolver	inc-topDown (fixed)	15	4%	15%	10482.37	825.36	698.82	10485.64	825.56	699.04	1.00
14	pb2satCp2	2012-05-19	13	3%	13%	6851.88	158.22	527.07	6856.11	158.26	527.39	1.00
15	PB07: minisat+	1.14	13	3%	13%	7814.47	489.62	601.11	7816.07	489.72	601.24	1.00
16	npSolver	inc (fixed)	11	3%	11%	1300.83	28.67	118.26	1301.98	28.69	118.36	1.00
17	npSolver	inc	9	2%	9%	2160.37	93.31	240.04	2161.87	93.36	240.21	1.00
18	npSolver	inc-topdown-quickBound (fixed)	8	2%	8%	1982.72	11.80	247.84	1985.38	11.94	248.17	1.00
19	npSolver	inc-topDown	8	2%	8%	4464.68	297.48	558.08	4202.86	297.69	525.36	1.06
20	npSolver	inc-topdown-quickBound	2	0%	2%	1188.84	594.42	594.42	1189.24	594.62	594.62	1.00

Category "optimisation, big integers, linear constraints" (OPT-BIGINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 416
Virtual Best Solver (VBS)			23	6%	100%	5.19	0.05	0.23	-	-	-	-
1	PB12: minisatp	1.0-2-g022594c	23	6%	100%	5.87	0.05	0.26	5.90	0.05	0.26	0.99
2	SAT4J PB specific settings	2.3.2 snapshot	23	6%	100%	25.35	0.87	1.10	14.52	0.47	0.63	1.75
3	Sat 4j PB Resolution	2.3.2 Snapshot	23	6%	100%	27.80	0.95	1.21	15.10	0.48	0.66	1.84
4	PB09: SAT4J Pseudo Resolution	2.1.1	23	6%	100%	32.87	0.98	1.43	19.83	0.50	0.86	1.66
5	PB07: SAT4JPseudoResolution	2007-03-23	23	6%	100%	35.18	1.20	1.53	20.35	0.57	0.88	1.73
6	PB11: Sat4j Res//CP	2.3.0	23	6%	100%	45.34	1.21	1.97	30.39	1.02	1.32	1.49
7	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	23	6%	100%	48.68	1.40	2.12	36.51	1.52	1.59	1.33
8	SAT 4j PB RES // CP	2.3.2 Snapshot	23	6%	100%	49.24	1.66	2.14	28.81	1.09	1.25	1.71
9	PB07: minisat+	1.14	23	6%	100%	85.41	0.49	3.71	85.44	0.49	3.71	1.00
10	npSolver	inc-topdown-quickBound (fixed)	19	5%	83%	153.10	0.17	8.06	156.15	0.17	8.22	0.98
11	npSolver	1.0 (fixed)	19	5%	83%	154.95	0.18	8.16	161.97	0.18	8.52	0.96
12	npSolver	inc-topDown (fixed)	19	5%	83%	154.99	0.16	8.16	155.81	0.17	8.20	0.99
13	npSolver	inc (fixed)	19	5%	83%	155.32	0.16	8.17	156.85	0.17	8.26	0.99
14	toysat	2012-05-17	19	5%	83%	1281.52	2.12	67.45	1281.80	2.12	67.46	1.00
15	pb2sat	2012-05-19	16	4%	70%	1.79	0.13	0.11	1.82	0.13	0.11	0.98
16	pb2satCp2	2012-05-19	16	4%	70%	1.79	0.13	0.11	1.83	0.13	0.11	0.98
17	npSolver	inc-topDown	16	4%	70%	2.31	0.14	0.14	2.34	0.14	0.15	0.99
18	npSolver	inc	16	4%	70%	2.31	0.14	0.14	2.35	0.14	0.15	0.98
19	npSolver	inc-topdown-quickBound	16	4%	70%	2.44	0.15	0.15	2.63	0.16	0.16	0.93
20	npSolver	1.0	16	4%	70%	2.45	0.15	0.15	2.48	0.15	0.16	0.99

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 657
Virtual Best Solver (VBS)			463	441 OPT, 22 UNSAT	70%	100%	35624.18	0.34	76.94	-	-	-	-
1	PB10: pb_cplex	2010-06-29	376	355 OPT, 21 UNSAT	57%	81%	42252.51	1.30	112.37	28911.31	1.17	76.89	1.46
2	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	351	329 OPT, 22 UNSAT	53%	76%	47853.16	4.65	136.33	47863.35	4.65	136.36	1.00
3	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	350	328 OPT, 22 UNSAT	53%	76%	46085.33	4.26	131.67	46095.38	4.26	131.70	1.00
4	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	349	327 OPT, 22 UNSAT	53%	75%	47349.52	5.09	135.67	47342.10	5.09	135.65	1.00
5	pwbo	2.02	342	323 OPT, 19 UNSAT	52%	74%	16723.85	1.24	48.90	8427.45	0.80	24.64	1.98
6	npSolver	1.0 (fixed)	337	315 OPT, 22 UNSAT	51%	73%	31582.38	6.46	93.72	31634.66	6.61	93.87	1.00
7	pb2sat	2012-05-19	331	309 OPT, 22 UNSAT	50%	71%	29784.69	8.20	89.98	29942.28	8.26	90.46	0.99
8	PB11: SCIP spx E_2	2011-06-10 (fixed)	331	309 OPT, 22 UNSAT	50%	71%	51186.60	2.68	154.64	51197.13	2.69	154.67	1.00
9	npSolver	1.0	325	307 OPT, 18 UNSAT	49%	70%	35383.02	13.70	108.87	35419.16	13.78	108.98	1.00
10	pb2satCp2	2012-05-19	314	292 OPT, 22 UNSAT	48%	68%	49877.24	8.08	158.84	51964.62	8.14	165.49	0.96
11	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	308	287 OPT, 21 UNSAT	47%	67%	33161.85	1.78	107.67	33169.14	1.79	107.69	1.00
12	bsolo	3.2	300	279 OPT, 21 UNSAT	46%	65%	29551.46	4.16	98.50	29558.49	4.16	98.53	1.00
13	clasp	2.0.6-R5325 (opt)	281	260 OPT, 21 UNSAT	43%	61%	11181.53	0.20	39.79	11183.96	0.20	39.80	1.00
14	PB11: Sat4j Res//CP	2.3.0	279	258 OPT, 21 UNSAT	42%	60%	16406.88	3.37	58.81	9198.32	2.51	32.97	1.78
15	PB12: minisatp	1.0-2-g022594c	279	257 OPT, 22 UNSAT	42%	60%	29660.80	4.90	106.31	29563.14	4.90	105.96	1.00
16	Sat 4j PB Resolution	2.3.2 Snapshot	278	257 OPT, 21 UNSAT	42%	60%	14402.80	2.53	51.81	14091.36	1.36	50.69	1.02
17	PB07: minisat+	1.14	277	255 OPT, 22 UNSAT	42%	60%	32349.13	7.77	116.78	32321.63	7.77	116.68	1.00
18	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	275	254 OPT, 21 UNSAT	42%	59%	19336.32	3.58	70.31	11123.49	4.15	40.45	1.74
19	SAT 4j PB RES // CP	2.3.2 Snapshot	275	253 OPT, 22 UNSAT	42%	59%	20762.40	4.01	75.50	11543.56	2.77	41.98	1.80
20	PB07: Pueblo	1.4	275	254 OPT, 21 UNSAT	42%	59%	21377.62	3.13	77.74	21373.13	3.13	77.72	1.00
21	wbo	1.72	265	245 OPT, 20 UNSAT	40%	57%	13184.16	0.68	49.75	13189.12	0.67	49.77	1.00
22	PB09: SAT4J Pseudo Resolution	2.1.1	265	244 OPT, 21 UNSAT	40%	57%	16366.81	2.37	61.76	16057.95	1.30	60.60	1.02
23	PB07: SAT4JPseudoResolution	2007-03-23	265	243 OPT, 22 UNSAT	40%	57%	18040.86	2.49	68.08	17763.57	1.34	67.03	1.02
24	toysat	2012-06-01	248	227 OPT, 21 UNSAT	38%	54%	22994.38	2.45	92.72	22990.54	2.46	92.70	1.00
25	toysat	2012-05-17	247	226 OPT, 21 UNSAT	38%	53%	22719.99	2.47	91.98	22725.19	2.47	92.00	1.00
26	PB07: PB-clasp	2007-04-10	206	186 OPT, 20 UNSAT	31%	44%	19037.86	1.20	92.42	19087.42	1.27	92.66	1.00
27	npSolver	inc	203	185 OPT, 18 UNSAT	31%	44%	12728.14	7.84	62.70	12756.29	7.88	62.84	1.00
28	npSolver	inc (fixed)	201	179 OPT, 22 UNSAT	31%	43%	10367.47	1.34	51.58	10373.93	1.35	51.61	1.00
29	npSolver	inc-topDown	199	181 OPT, 18 UNSAT	30%	43%	11748.89	4.23	59.04	11763.08	4.37	59.11	1.00
30	npSolver	inc-topDown (fixed)	192	170 OPT, 22 UNSAT	29%	41%	11448.32	0.65	59.63	11458.22	0.65	59.68	1.00
31	npSolver	inc-topdown-quickBound	183	165 OPT, 18 UNSAT	28%	40%	8504.23	4.55	46.47	8517.31	4.56	46.54	1.00
32	npSolver	inc-topdown-quickBound (fixed)	168	146 OPT, 22 UNSAT	26%	36%	8844.95	0.85	52.65	8852.86	0.85	52.70	1.00

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 657
Virtual Best Solver (VBS)			441	67%	100%	35435.46	0.39	80.35	-	-	-	-
1	PB10: pb_cplex	2010-06-29	355	54%	80%	41280.56	1.62	116.28	28455.95	1.21	80.16	1.45
2	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	329	50%	75%	46144.87	4.90	140.26	46154.71	4.92	140.29	1.00
3	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	328	50%	74%	44502.37	4.76	135.68	44512.05	4.76	135.71	1.00
4	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	327	50%	74%	46087.58	5.24	140.94	46097.25	5.24	140.97	1.00
5	pwbo	2.02	323	49%	73%	15212.79	1.49	47.10	7670.20	0.90	23.75	1.98
6	npSolver	1.0 (fixed)	315	48%	71%	30178.87	7.39	95.81	30227.95	7.39	95.96	1.00
7	pb2sat	2012-05-19	309	47%	70%	27613.02	9.16	89.36	27770.07	9.37	89.87	0.99
8	PB11: SCIP spx E_2	2011-06-10 (fixed)	309	47%	70%	49110.73	3.41	158.93	49120.83	3.42	158.97	1.00
9	npSolver	1.0	307	47%	70%	34784.22	14.96	113.30	34817.38	14.96	113.41	1.00
10	pb2satCp2	2012-05-19	292	44%	66%	47924.09	8.87	164.12	50010.96	8.95	171.27	0.96
11	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	287	44%	65%	31902.02	2.44	111.16	31909.03	2.45	111.18	1.00
12	bsolo	3.2	279	42%	63%	27660.93	7.41	99.14	27667.50	7.41	99.17	1.00
13	clasp	2.0.6-R5325 (opt)	260	40%	59%	10676.27	0.22	41.06	10678.57	0.23	41.07	1.00
14	PB11: Sat4j Res//CP	2.3.0	258	39%	59%	15723.09	3.50	60.94	8851.63	2.63	34.31	1.78
15	Sat 4j PB Resolution	2.3.2 Snapshot	257	39%	58%	14140.30	2.77	55.02	13842.01	1.52	53.86	1.02
16	PB12: minisatp	1.0-2-g022594c	257	39%	58%	26175.61	5.49	101.85	26077.14	5.49	101.47	1.00
17	PB07: minisat+	1.14	255	39%	58%	27636.05	9.52	108.38	27607.48	9.52	108.26	1.00
18	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	254	39%	58%	18623.52	3.76	73.32	10759.56	4.21	42.36	1.73
19	PB07: Pueblo	1.4	254	39%	58%	19817.66	3.22	78.02	19812.88	3.24	78.00	1.00
20	SAT 4j PB RES // CP	2.3.2 Snapshot	253	39%	57%	18310.00	4.23	72.37	10309.41	2.83	40.75	1.78
21	wbo	1.72	245	37%	56%	10860.18	0.75	44.33	10864.46	0.75	44.34	1.00
22	PB09: SAT4J Pseudo Resolution	2.1.1	244	37%	55%	14452.44	2.44	59.23	14156.87	1.40	58.02	1.02
23	PB07: SAT4JPseudoResolution	2007-03-23	243	37%	55%	17213.87	2.63	70.84	16953.99	1.37	69.77	1.02
24	toysat	2012-06-01	227	35%	51%	22085.05	2.66	97.29	22081.03	2.67	97.27	1.00
25	toysat	2012-05-17	226	34%	51%	21804.74	2.67	96.48	21809.73	2.67	96.50	1.00
26	PB07: PB-clasp	2007-04-10	186	28%	42%	18950.05	1.04	101.88	18999.43	1.11	102.15	1.00
27	npSolver	inc	185	28%	42%	12146.95	9.28	65.66	12174.38	9.36	65.81	1.00
28	npSolver	inc-topDown	181	28%	41%	10994.74	5.89	60.74	11007.93	5.82	60.82	1.00
29	npSolver	inc (fixed)	179	27%	41%	8952.91	1.69	50.02	8956.98	1.69	50.04	1.00
30	npSolver	inc-topDown (fixed)	170	26%	39%	10081.73	0.90	59.30	10085.89	0.91	59.33	1.00
31	npSolver	inc-topdown-quickBound	165	25%	37%	7749.42	5.78	46.97	7761.77	5.79	47.04	1.00
32	npSolver	inc-topdown-quickBound (fixed)	146	22%	33%	7420.99	1.38	50.83	7425.58	1.38	50.86	1.00

Category "optimisation, small integers, linear constraints" (OPT-SMALLINT-LIN): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 657
Virtual Best Solver (VBS)			22	3%	100%	188.72	0.02	8.58	-	-	-	-
1	PB07: SAT4JPseudoResolution	2007-03-23	22	3%	100%	826.99	1.08	37.59	809.58	0.54	36.80	1.02
2	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	22	3%	100%	1261.94	0.37	57.36	1244.85	0.38	56.58	1.01
3	npSolver	inc-topDown (fixed)	22	3%	100%	1366.59	0.14	62.12	1372.33	0.14	62.38	1.00
4	npSolver	1.0 (fixed)	22	3%	100%	1403.52	0.15	63.80	1406.70	0.15	63.94	1.00
5	npSolver	inc (fixed)	22	3%	100%	1414.56	0.14	64.30	1416.96	0.15	64.41	1.00
6	npSolver	inc-topdown-quickBound (fixed)	22	3%	100%	1423.96	0.14	64.73	1427.27	0.14	64.88	1.00
7	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	22	3%	100%	1582.96	0.32	71.95	1583.33	0.32	71.97	1.00
8	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	22	3%	100%	1708.29	0.29	77.65	1708.64	0.29	77.67	1.00
9	pb2satCp2	2012-05-19	22	3%	100%	1953.15	0.14	88.78	1953.65	0.14	88.80	1.00
10	PB11: SCIP spx E_2	2011-06-10 (fixed)	22	3%	100%	2075.87	0.38	94.36	2076.30	0.38	94.38	1.00
11	pb2sat	2012-05-19	22	3%	100%	2171.67	0.14	98.71	2172.20	0.14	98.74	1.00
12	SAT 4j PB RES // CP	2.3.2 Snapshot	22	3%	100%	2452.40	1.62	111.47	1234.14	0.99	56.10	1.99
13	PB12: minisatp	1.0-2-g022594c	22	3%	100%	3485.19	0.06	158.42	3486.00	0.06	158.45	1.00
14	PB07: minisat+	1.14	22	3%	100%	4713.08	0.40	214.23	4714.15	0.40	214.28	1.00
15	Sat 4j PB Resolution	2.3.2 Snapshot	21	3%	95%	262.50	0.94	12.50	249.35	0.47	11.87	1.05
16	clasp	2.0.6-R5325 (opt)	21	3%	95%	505.26	0.01	24.06	505.39	0.02	24.07	1.00
17	PB11: Sat4j Res//CP	2.3.0	21	3%	95%	683.79	1.02	32.56	346.69	0.89	16.51	1.97
18	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	21	3%	95%	712.80	1.14	33.94	363.93	1.38	17.33	1.96
19	toysat	2012-06-01	21	3%	95%	909.33	1.21	43.30	909.51	1.22	43.31	1.00
20	toysat	2012-05-17	21	3%	95%	915.25	1.20	43.58	915.46	1.20	43.59	1.00
21	PB10: pb_cplex	2010-06-29	21	3%	95%	971.95	0.76	46.28	455.36	0.76	21.68	2.13
22	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	21	3%	95%	1259.83	0.33	59.99	1260.11	0.33	60.01	1.00
23	PB07: Pueblo	1.4	21	3%	95%	1559.96	0.49	74.28	1560.25	0.49	74.30	1.00
24	bsolo	3.2	21	3%	95%	1890.53	0.07	90.03	1890.98	0.07	90.05	1.00
25	PB09: SAT4J Pseudo Resolution	2.1.1	21	3%	95%	1914.37	0.86	91.16	1901.09	0.43	90.53	1.01
26	PB07: PB-clasp	2007-04-10	20	3%	91%	87.81	1.32	4.39	87.99	1.32	4.40	1.00
27	wbo	1.72	20	3%	91%	2323.98	0.07	116.20	2324.65	0.07	116.23	1.00
28	pwbo	2.02	19	3%	86%	1511.06	0.06	79.53	757.25	0.07	39.86	2.00
29	npSolver	inc	18	3%	82%	581.19	0.16	32.29	581.91	0.17	32.33	1.00
30	npSolver	1.0	18	3%	82%	598.80	0.17	33.27	601.78	0.17	33.43	1.00
31	npSolver	inc-topDown	18	3%	82%	754.15	0.16	41.90	755.15	0.16	41.95	1.00
32	npSolver	inc-topdown-quickBound	18	3%	82%	754.82	0.16	41.93	755.54	0.17	41.97	1.00

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT+UNSAT answers

Rank	Solver	Version	Number of solved instances	Detail	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 124
Virtual Best Solver (VBS)			67	67 OPT	54%	100%	868.38	0.05	12.96	-	-	-	-
1	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	67	67 OPT	54%	100%	1025.31	0.19	15.30	1025.61	0.19	15.31	1.00
2	PB11: SCIP spx E_2	2011-06-10 (fixed)	67	67 OPT	54%	100%	1654.71	0.18	24.70	1646.53	0.18	24.58	1.00
3	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	66	66 OPT	53%	99%	3904.40	0.11	59.16	3905.18	0.11	59.17	1.00
4	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	65	65 OPT	52%	97%	1892.95	0.24	29.12	1893.42	0.24	29.13	1.00
5	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	65	65 OPT	52%	97%	3846.65	2.75	59.18	3847.51	2.75	59.19	1.00
6	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	62	62 OPT	50%	93%	4873.42	0.78	78.60	4874.38	0.79	78.62	1.00
7	npSolver	1.0 (fixed)	60	60 OPT	48%	90%	2553.18	1.40	42.55	2554.22	1.44	42.57	1.00
8	clasp	2.0.6-R5325 (opt)	60	60 OPT	48%	90%	4573.65	0.87	76.23	4574.53	0.87	76.24	1.00
9	npSolver	inc-topDown (fixed)	59	59 OPT	48%	88%	1775.42	0.94	30.09	1776.18	0.95	30.10	1.00
10	npSolver	inc (fixed)	59	59 OPT	48%	88%	2552.63	1.05	43.26	2553.64	1.05	43.28	1.00
11	PB07: minisat+	1.14	59	59 OPT	48%	88%	5326.90	0.05	90.29	5311.56	0.05	90.03	1.00
12	PB07: Pueblo	1.4	57	57 OPT	46%	85%	9322.87	3.20	163.56	9324.55	3.21	163.59	1.00
13	PB11: Sat4j Res//CP	2.3.0	56	56 OPT	45%	84%	9219.16	5.68	164.63	5172.78	3.23	92.37	1.78
14	Sat 4j PB Resolution	2.3.2 Snapshot	55	55 OPT	44%	82%	4350.37	1.31	79.10	4299.09	0.95	78.17	1.01
15	SAT 4j PB RES // CP	2.3.2 Snapshot	55	55 OPT	44%	82%	8144.16	4.66	148.08	4209.07	2.79	76.53	1.93
16	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	55	55 OPT	44%	82%	10823.44	5.72	196.79	6060.21	4.37	110.19	1.79
17	pwbo	2.02	54	54 OPT	44%	81%	3912.75	0.86	72.46	1957.64	0.43	36.25	2.00
18	pwbo	2.0	54	54 OPT	44%	81%	4094.12	0.59	75.82	2047.80	0.29	37.92	2.00
19	pb2satCp2	2012-05-19	52	52 OPT	42%	78%	8518.38	8.31	163.81	8532.37	8.80	164.08	1.00
20	pb2sat	2012-05-19	52	52 OPT	42%	78%	9504.18	9.72	182.77	9496.23	9.73	182.62	1.00
21	PB07: bsolo	3.0.17	47	47 OPT	38%	70%	3356.13	0.66	71.41	3356.99	0.67	71.43	1.00
22	PB09: SAT4J Pseudo Resolution	2.1.1	47	47 OPT	38%	70%	4531.17	1.08	96.41	4496.08	0.67	95.66	1.01
23	wbo	1.72	45	45 OPT	36%	67%	4775.54	0.45	106.12	4724.43	0.45	104.99	1.01
24	wbo	1.7	45	45 OPT	36%	67%	4778.26	0.45	106.18	4778.91	0.45	106.20	1.00
25	toysat	2012-05-17	45	45 OPT	36%	67%	11532.79	2.11	256.28	11535.03	2.11	256.33	1.00
26	bsolo	3.2	44	44 OPT	35%	66%	3840.63	0.91	87.29	3841.35	0.91	87.30	1.00
27	toysat	2012-06-01	44	44 OPT	35%	66%	8322.82	1.93	189.16	8324.50	1.93	189.19	1.00
28	PB07: SAT4JPseudoResolution	2007-03-23	38	38 OPT	31%	57%	2469.68	1.91	64.99	2432.74	1.53	64.02	1.02
29	PB07: PB-clasp	2007-04-10	37	37 OPT	30%	55%	8288.37	7.83	224.01	8290.65	7.93	224.07	1.00
30	npSolver	inc-topdown-quickBound (fixed)	31	31 OPT	25%	46%	1393.56	0.43	44.95	1394.00	0.44	44.97	1.00
31	SAT4J PB specific settings	2.3.2 snapshot	23	23 OPT	19%	34%	3804.22	0.59	165.40	3781.76	0.35	164.42	1.01
32	PB12: minisatp	1.0-2-g022594c	0		0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): OPT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 124
Virtual Best Solver (VBS)			67	54%	100%	868.38	0.05	12.96	-	-	-	-
1	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	67	54%	100%	1025.31	0.19	15.30	1025.61	0.19	15.31	1.00
2	PB11: SCIP spx E_2	2011-06-10 (fixed)	67	54%	100%	1654.71	0.18	24.70	1646.53	0.18	24.58	1.00
3	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	66	53%	99%	3904.40	0.11	59.16	3905.18	0.11	59.17	1.00
4	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	65	52%	97%	1892.95	0.24	29.12	1893.42	0.24	29.13	1.00
5	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	65	52%	97%	3846.65	2.75	59.18	3847.51	2.75	59.19	1.00
6	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	62	50%	93%	4873.42	0.78	78.60	4874.38	0.79	78.62	1.00
7	npSolver	1.0 (fixed)	60	48%	90%	2553.18	1.40	42.55	2554.22	1.44	42.57	1.00
8	clasp	2.0.6-R5325 (opt)	60	48%	90%	4573.65	0.87	76.23	4574.53	0.87	76.24	1.00
9	npSolver	inc-topDown (fixed)	59	48%	88%	1775.42	0.94	30.09	1776.18	0.95	30.10	1.00
10	npSolver	inc (fixed)	59	48%	88%	2552.63	1.05	43.26	2553.64	1.05	43.28	1.00
11	PB07: minisat+	1.14	59	48%	88%	5326.90	0.05	90.29	5311.56	0.05	90.03	1.00
12	PB07: Pueblo	1.4	57	46%	85%	9322.87	3.20	163.56	9324.55	3.21	163.59	1.00
13	PB11: Sat4j Res//CP	2.3.0	56	45%	84%	9219.16	5.68	164.63	5172.78	3.23	92.37	1.78
14	Sat 4j PB Resolution	2.3.2 Snapshot	55	44%	82%	4350.37	1.31	79.10	4299.09	0.95	78.17	1.01
15	SAT 4j PB RES // CP	2.3.2 Snapshot	55	44%	82%	8144.16	4.66	148.08	4209.07	2.79	76.53	1.93
16	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	55	44%	82%	10823.44	5.72	196.79	6060.21	4.37	110.19	1.79
17	pwbo	2.02	54	44%	81%	3912.75	0.86	72.46	1957.64	0.43	36.25	2.00
18	pwbo	2.0	54	44%	81%	4094.12	0.59	75.82	2047.80	0.29	37.92	2.00
19	pb2satCp2	2012-05-19	52	42%	78%	8518.38	8.31	163.81	8532.37	8.80	164.08	1.00
20	pb2sat	2012-05-19	52	42%	78%	9504.18	9.72	182.77	9496.23	9.73	182.62	1.00
21	PB07: bsolo	3.0.17	47	38%	70%	3356.13	0.66	71.41	3356.99	0.67	71.43	1.00
22	PB09: SAT4J Pseudo Resolution	2.1.1	47	38%	70%	4531.17	1.08	96.41	4496.08	0.67	95.66	1.01
23	wbo	1.72	45	36%	67%	4775.54	0.45	106.12	4724.43	0.45	104.99	1.01
24	wbo	1.7	45	36%	67%	4778.26	0.45	106.18	4778.91	0.45	106.20	1.00
25	toysat	2012-05-17	45	36%	67%	11532.79	2.11	256.28	11535.03	2.11	256.33	1.00
26	bsolo	3.2	44	35%	66%	3840.63	0.91	87.29	3841.35	0.91	87.30	1.00
27	toysat	2012-06-01	44	35%	66%	8322.82	1.93	189.16	8324.50	1.93	189.19	1.00
28	PB07: SAT4JPseudoResolution	2007-03-23	38	31%	57%	2469.68	1.91	64.99	2432.74	1.53	64.02	1.02
29	PB07: PB-clasp	2007-04-10	37	30%	55%	8288.37	7.83	224.01	8290.65	7.93	224.07	1.00
30	npSolver	inc-topdown-quickBound (fixed)	31	25%	46%	1393.56	0.43	44.95	1394.00	0.44	44.97	1.00
31	SAT4J PB specific settings	2.3.2 snapshot	23	19%	34%	3804.22	0.59	165.40	3781.76	0.35	164.42	1.01
32	PB12: minisatp	1.0-2-g022594c	0	0%	0%	0.00	0.00	-	0.00	0.00	-	-

Category "optimisation, small integers, non linear constraints" (OPT-SMALLINT-NLC): UNSAT answers

Rank	Solver	Version	Number of solved instances	% of all instances	% of VBS	Cumulated CPU time on solved instances	Median CPU time	Average CPU time per solved instance	Cumulated WC time on solved instances	Median WC time	Average WC time per solved instance	CPU/WC
Total number of instances in the category: 124
Virtual Best Solver (VBS)			0	0%	-	0.00	0.00	-	-	-	-	-
1	SAT 4j PB RES // CP	2.3.2 Snapshot	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
2	Sat 4j PB Resolution	2.3.2 Snapshot	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
3	SCIP spx	SCIP 2.1.1.4. with SoPlex 1.6.0.3 fixed	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
4	SCIP spx E	SCIP 2.1.1.4. Exp with SoPlex 1.6.0.3 fixed	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
5	SCIP spx standard	SCIP 2.1.1.4. with SoPlex 1.6.0.3 standard fixed	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
6	pb2satCp2	2012-05-19	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
7	pb2sat	2012-05-19	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
8	pwbo	2.0	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
9	wbo	1.7	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
10	toysat	2012-05-17	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
11	bsolo	3.2	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
12	clasp	2.0.6-R5325 (opt)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
13	SAT4J PB specific settings	2.3.2 snapshot	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
14	PB07: Pueblo	1.4	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
15	PB07: minisat+	1.14	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
16	PB12: minisatp	1.0-2-g022594c	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
17	toysat	2012-06-01	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
18	pwbo	2.02	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
19	wbo	1.72	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
20	PB07: PB-clasp	2007-04-10	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
21	PB07: bsolo	3.0.17	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
22	PB07: SAT4JPseudoResolution	2007-03-23	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
23	PB09: SCIPspx	SCIP 1.1.0.7 with SoPLEX 1.4.1(24.4.2009)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
24	PB09: SAT4J Pseudo Resolution	2.1.1	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
25	PB10: SAT4J PB RES // CP	2.2.0 2010-05-31	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
26	PB10: SCIPspx	SCIP 1.2.1.3 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
27	PB11: Sat4j Res//CP	2.3.0	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
28	PB11: SCIP spx E_2	2011-06-10 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
29	npSolver	inc-topDown (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
30	npSolver	inc (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
31	npSolver	1.0 (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-
32	npSolver	inc-topdown-quickBound (fixed)	0	0%	-	0.00	0.00	-	0.00	0.00	-	-