PB'06 evaluation: solvers page (tabular format, detailed results)

Category "optimisation, big integers" (OPT-BIGINT)

Category "optimisation, medium integers" (OPT-MEDINT)

Category "optimisation, small integers" (OPT-SMALLINT)

Category "no optimisation, big integers" (SATUNSAT-BIGINT)

Category "no optimisation, medium integers" (SATUNSAT-MEDINT)

Category "no optimisation, small integers" (SATUNSAT-SMALLINT)

All categories for solvers registered in all categories

All categories and all solvers (regardless of their ability to support all categories)

This table summarizes the results in all categories for all solvers. Note however that some solvers may not have support for some categories. Therefore, the solvers may have run on a different number of benchmarks and this table should not be used to compare them.

Some explanations

• #Benchs: Number of benchs for this solver
• Solved: Total number of solved instances (this is the sum of the OPT and UNSAT columns)
• UNSAT.: solver proved unsatitisfiability ("s UNSATISFIABLE" was output)
• OPT: solver found an optimum solution ("s OPTIMUM FOUND" was output), the provided implicant satisfies each constraint and no solver gave a better solution
• SAT: solver found a solution ("s SATISFIABLE" was output) of a SAT/UNSAT intances and the provided implicant satisfies each constraint
• SAT (subopt.): solver found a solution ("s SATISFIABLE" was output) and the provided implicant satisfies each constraint but this may not be the optimal solution
• SAT (timeout): solver exceeded the time limit but was able to find a solution ("s SATISFIABLE" was output) and the provided implicant satisfies each constraint
• SAT (out of mem.): solver exceeded the memory limit but was able to find a solution ("s SATISFIABLE" was output) and the provided implicant satisfies each constraint
• UNKNOWN: solver couldn't decide ("s UNKNOWN" was output)
• UNKNOWN (timeout): solver exceeded the time limit and gave no answer. These runs are considered to give result UNKNOWN
• UNKNOWN (out of mem.): solver exceeded the memory limit and gave no answer. These runs are considered to give result UNKNOWN.
• UNKNOWN (exit code): solver didn't output a solution line and terminated with an unexpected exit code. These runs are considered to give result UNKNOWN.
• UNKNOWN (problem): No solution line was correctly recognized. These runs are considered to give result UNKNOWN.
• Sig. Caught: solver was terminated by a signal (SIGSEGV for example) and didn't output a solution line
• NO CERT.: solver answered SATISFIABLE but either didn't provide a certificate (the "v " line) or either gave a truncated certificate (which doesn't end in a new line)
• WRONG CERT.: solver gave an implicant but it appears that it doesn't satisfy every constraint
• WRONG OPT.: solver found an optimum solution ("s OPTIMUM FOUND" was output) but there exists an implicant which gives a better value to the objective function
• WRONG UNSAT.: solver proved unsatisfiability but was wrong ("s UNSATISFIABLE" was output)
• Misc: This is a internal check. It counts the runs that don't fall into one of the previous category. Should be 0 if categories are exhaustive and mutually exclusive. Will be negative if categories are not mutually exclusive and positive if categories are not exhaustive.