PB'10 competition: WBO track: ranking of solvers

This page presents the ranking of the different solvers. Solvers are ranked on the number of solved instances. Ties are broken by considering the minimum total solution time.

Solvers which gave a wrong answer in a category are excluded from the ranking in that category.

The Virtual Best Solver (VBS) is a theoretical construction which returns the best answer provided by one of the submitted solver. It corresponds to the 'Best result' column in the bench per bench view of the results. It can be considered as a meta solver which uses a perfect oracle to guess instantaneously which of the submitted solver will be the most efficient to solve the instance. Another way to look at it is to consider this VBS as a solver which would run all other solvers in parallel, bringing together all the solvers strenghs. This VBS is essentially the same notion as State Of The Art (SOTA) solver defined in "Evaluating general purpose automated theorem proving systems", Artificial Intelligence, Volume 131, Issues 1-2, September 2001, Pages 39-54 Geoff Sutcliffe, Christian Suttner. However, SOTA is a more general notion: the VBS can be seen as the least powerful SOTA solver.

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 263
Virtual Best Solver (VBS)113113 MOPT43%100%7809.1769.11
1SAT4J PB Resolution2.2.1113113 MOPT43%100%8465.9974.92
2PB/CT0.1 fixed7878 MOPT30%69%3487.9744.72

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): OPT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 263
Virtual Best Solver (VBS)11343%100%7809.1769.11
1SAT4J PB Resolution2.2.111343%100%8465.9974.92
2PB/CT0.1 fixed7830%69%3487.9744.72

Category "both soft and hard constraints, big integers, linear constraints" (PARTIAL-BIGINT-LIN): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 263
Virtual Best Solver (VBS)00%-0.00-
1SAT4J PB Resolution2.2.100%-0.00-
2PB/CT0.1 fixed00%-0.00-

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 536
Virtual Best Solver (VBS)531530 MOPT, 1 UNSAT99%100%35847.3167.51
1SAT4J PB Resolution2.2.1446445 MOPT, 1 UNSAT83%84%82870.29185.81
2pb_cplex2010-07-01428428 MOPT80%81%77686.77181.51
3PB/CT0.1 fixed375374 MOPT, 1 UNSAT70%71%101088.10269.57
4wbo1.4a373372 MOPT, 1 UNSAT70%70%66239.15177.58
5SCIPclpSCIP 1.2.1.2 with Clp 1.11.1 (Release Version) as LP solver297296 MOPT, 1 UNSAT55%56%23453.5678.97
6SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver282281 MOPT, 1 UNSAT53%53%16779.8059.50
7SCIPspxSCIP 1.2.1.2 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver269268 MOPT, 1 UNSAT50%51%6316.5923.48
8SCIPnoneSCIP 1.2.1.2 without any LP solver146145 MOPT, 1 UNSAT27%27%61282.55419.74

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): OPT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 536
Virtual Best Solver (VBS)53099%100%35783.3667.52
1SAT4J PB Resolution2.2.144583%84%82763.11185.98
2pb_cplex2010-07-0142880%81%77686.77181.51
3PB/CT0.1 fixed37470%71%100947.93269.91
4wbo1.4a37269%70%65934.50177.24
5SCIPclpSCIP 1.2.1.2 with Clp 1.11.1 (Release Version) as LP solver29655%56%23339.2478.85
6SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver28152%53%16715.8559.49
7SCIPspxSCIP 1.2.1.2 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver26850%51%6237.3323.27
8SCIPnoneSCIP 1.2.1.2 without any LP solver14527%27%61172.69421.88

Category "both soft and hard constraints, small integers, linear constraints" (PARTIAL-SMALLINT-LIN): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 536
Virtual Best Solver (VBS)10%100%63.9563.95
1SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver10%100%63.9563.95
2SCIPspxSCIP 1.2.1.2 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver10%100%79.2679.26
3SAT4J PB Resolution2.2.110%100%107.18107.18
4SCIPnoneSCIP 1.2.1.2 without any LP solver10%100%109.86109.86
5SCIPclpSCIP 1.2.1.2 with Clp 1.11.1 (Release Version) as LP solver10%100%114.31114.31
6PB/CT0.1 fixed10%100%140.17140.17
7wbo1.4a10%100%304.65304.65
8pb_cplex2010-07-0100%0%0.00-

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 46
Virtual Best Solver (VBS)3333 MOPT72%100%5194.34157.40
1SAT4J PB Resolution2.2.13333 MOPT72%100%5194.34157.40
2PB/CT0.100%0%0.00-
3PB/CT0.1 fixed00%0%0.00-
4SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver00%0%0.00-

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): OPT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 46
Virtual Best Solver (VBS)3372%100%5194.34157.40
1SAT4J PB Resolution2.2.13372%100%5194.34157.40
2PB/CT0.100%0%0.00-
3PB/CT0.1 fixed00%0%0.00-
4SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver00%0%0.00-

Category "only soft constraints, big integers, linear constraints" (SOFT-BIGINT-LIN): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 46
Virtual Best Solver (VBS)00%-0.00-
1PB/CT0.100%-0.00-
2SAT4J PB Resolution2.2.100%-0.00-
3PB/CT0.1 fixed00%-0.00-
4SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver00%-0.00-

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): OPT+UNSAT answers

Rank SolverVersion Number of
solved instances		Detail % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 201
Virtual Best Solver (VBS)201201 MOPT100%100%15821.1578.71
1wbo1.4a161161 MOPT80%80%25985.26161.40
2pb_cplex2010-07-01160160 MOPT80%80%34467.56215.42
3SAT4J PB Resolution2.2.1160160 MOPT80%80%39929.75249.56
4PB/CT0.1 fixed138138 MOPT69%69%56779.52411.45
5SCIPclpSCIP 1.2.1.2 with Clp 1.11.1 (Release Version) as LP solver113113 MOPT56%56%1483.4213.13
6SCIPspxSCIP 1.2.1.2 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver113113 MOPT56%56%2387.6021.13
7SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver113113 MOPT56%56%4635.2541.02
8SCIPnoneSCIP 1.2.1.2 without any LP solver2222 MOPT11%11%18954.85861.58

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): OPT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 201
Virtual Best Solver (VBS)201100%100%15821.1578.71
1wbo1.4a16180%80%25985.26161.40
2pb_cplex2010-07-0116080%80%34467.56215.42
3SAT4J PB Resolution2.2.116080%80%39929.75249.56
4PB/CT0.1 fixed13869%69%56779.52411.45
5SCIPclpSCIP 1.2.1.2 with Clp 1.11.1 (Release Version) as LP solver11356%56%1483.4213.13
6SCIPspxSCIP 1.2.1.2 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver11356%56%2387.6021.13
7SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver11356%56%4635.2541.02
8SCIPnoneSCIP 1.2.1.2 without any LP solver2211%11%18954.85861.58

Category "only soft constraints, small integers, linear constraints" (SOFT-SMALLINT-LIN): UNSAT answers

Rank SolverVersion Number of
solved instances		 % of all instances% of VBSCumulated CPU time
on solved instances		Average CPU time
per solved instance
Total number of instances in the category: 201
Virtual Best Solver (VBS)00%-0.00-
1wbo1.4a00%-0.00-
2SCIPspxSCIP 1.2.1.2 with SoPlex 1.4.2 (CVS Version 30.5.2010) as LP solver00%-0.00-
3SCIPclpSCIP 1.2.1.2 with Clp 1.11.1 (Release Version) as LP solver00%-0.00-
4SCIPnoneSCIP 1.2.1.2 without any LP solver00%-0.00-
5pb_cplex2010-07-0100%-0.00-
6SAT4J PB Resolution2.2.100%-0.00-
7PB/CT0.1 fixed00%-0.00-
8SCIPclpSCIP 1.2.1.3 with Clp 1.11.1 (Release Version) as LP solver00%-0.00-