CSP 2008 Competition: results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory academic problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell exampleMeaning
AnswerSolver result
TT=...Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out
T1=...Time to get the best model (T1): this is the wall clock time (in seconds) ellapsed when the solver found its best model. This time is obtained from the 'o lines'. It is useful to compare both complete and incomplete solvers. It doesn't take into account the time to prove optimality of the solution.
Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

AbbreviationMeaning
TOTime Out
MOMem. Out (out of memory)

Meaning of the different colors:

ColorMeaning
textthe solver cannot handle this instance
textthe solver gave no answer
textthe solver gave a definitive answer (SAT or UNSAT)
textthe solver performed better than the other ones on that instance
textthe solver was ended by a signal
textthe solver gave an incomplete answer
textthe solver was disqualified in the category
textthe solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC
Abscon 112v4 ESAC
bpsolver 2008-06-27
casper zao
casper zito
choco2_dwdeg 2008-06-26
choco2_impwdeg 2008-06-26
Concrete + CPS4J 2008-05-30
Concrete + CSP4J - Tabu Engine 2008-05-30
Concrete + CSP4J - WMC Engine 2008-05-30
cpHydra k_10
cpHydra k_40
galac E
galac M
MDG-noprobe 2008-06-27
MDG-probe 2008-06-27
Minion/Tailor 2008-07-04
Mistral-option 1.314
Mistral-prime 1.313
SAT4J CSP 2008-06-13
spider 2008-06-14
Sugar v1.13+minisat
Sugar v1.13+picosat

Bench nameBest results
on this
instance
Abscon 112v4
AC
Abscon 112v4
ESAC
bpsolver
2008-06-27
casper
zao
casper
zito
choco2_dwdeg
2008-06-26
choco2_impwdeg
2008-06-26
Concrete + CPS4J
2008-05-30
Concrete + CSP4J - Tabu Engine
2008-05-30
Concrete + CSP4J - WMC Engine
2008-05-30
cpHydra
k_10
cpHydra
k_40
galac
E
galac
M
MDG-noprobe
2008-06-27
MDG-probe
2008-06-27
Minion/Tailor
2008-07-04
Mistral-option
1.314
Mistral-prime
1.313
SAT4J CSP
2008-06-13
spider
2008-06-14
Sugar
v1.13+minisat
Sugar
v1.13+picosat
csp/hanoi/
normalized-hanoi-3_ext.xml
SAT
TT=0.006

SAT
TT=0.611

CHECKS=164
NODES=6
SAT
TT=0.593

CHECKS=164
NODES=0
SAT
TT=0.027

SAT
TT=0.006

CHECKS=17
SAT
TT=0.009

CHECKS=17
SAT
TT=0.327

ASSIGNMENTS=1
SAT
TT=0.338

ASSIGNMENTS=1
SAT
TT=0.284

CHECKS=145
NODES=6
SAT
TT=0.273

CHECKS=165
NODES=0
SAT
TT=0.275

CHECKS=165
NODES=0
SAT
TT=0.02

ASSIGNMENTS=0
CHECKS=16
SAT
TT=0.017

ASSIGNMENTS=0
CHECKS=16
SAT
TT=0.962

SAT
TT=0.837

SAT
TT=0.007

ASSIGNMENTS=0
CHECKS=16
SAT
TT=0.007

ASSIGNMENTS=0
CHECKS=16
SAT
TT=0.269

SAT
TT=0.006

ASSIGNMENTS=0
CHECKS=16
SAT
TT=0.006

ASSIGNMENTS=0
CHECKS=16
SAT
TT=0.702

SAT
TT=0.01

ASSIGNMENTS=7
CHECKS=835
SAT
TT=0.462

SAT
TT=0.482

csp/hanoi/
normalized-hanoi-4_ext.xml
SAT
TT=0.007

SAT
TT=0.619

CHECKS=2211
NODES=14
SAT
TT=0.618

CHECKS=2210
NODES=0
SAT
TT=0.032

SAT
TT=0.018

CHECKS=49
SAT
TT=0.019

CHECKS=49
SAT
TT=0.37

ASSIGNMENTS=1
SAT
TT=0.365

ASSIGNMENTS=1
SAT
TT=0.329

CHECKS=2970
NODES=14
SAT
TT=0.305

CHECKS=3157
NODES=0
SAT
TT=0.305

CHECKS=3157
NODES=0
SAT
TT=0.021

ASSIGNMENTS=0
CHECKS=48
SAT
TT=0.018

ASSIGNMENTS=0
CHECKS=48
SAT
TT=1.742

SAT
TT=1.673

SAT
TT=0.008

ASSIGNMENTS=0
CHECKS=48
SAT
TT=0.007

ASSIGNMENTS=0
CHECKS=48
SAT
TT=0.301

SAT
TT=0.007

ASSIGNMENTS=0
CHECKS=48
SAT
TT=0.008

ASSIGNMENTS=0
CHECKS=48
SAT
TT=0.973

SAT
TT=0.009

ASSIGNMENTS=14
CHECKS=13167
SAT
TT=0.733

SAT
TT=0.618

csp/hanoi/
normalized-hanoi-5_ext.xml
SAT
TT=0.009

SAT
TT=0.673

CHECKS=30105
NODES=30
SAT
TT=1.231

CHECKS=30105
NODES=0
SAT
TT=0.066

SAT
TT=0.208

CHECKS=113
SAT
TT=0.215

CHECKS=113
SAT
TT=0.578

ASSIGNMENTS=1
SAT
TT=0.571

ASSIGNMENTS=1
SAT
TT=0.754

CHECKS=55621
NODES=30
SAT
TT=0.372

CHECKS=58513
NODES=0
SAT
TT=0.39

CHECKS=58513
NODES=0
SAT
TT=0.021

ASSIGNMENTS=0
CHECKS=112
SAT
TT=0.019

ASSIGNMENTS=0
CHECKS=112
SAT
TT=19.202

SAT
TT=14.622

SAT
TT=0.01

ASSIGNMENTS=0
CHECKS=112
SAT
TT=0.011

ASSIGNMENTS=0
CHECKS=112
? (problem)
TT=0.443

SAT
TT=0.009

ASSIGNMENTS=0
CHECKS=112
SAT
TT=0.009

ASSIGNMENTS=0
CHECKS=112
SAT
TT=1.543

SAT
TT=0.101

ASSIGNMENTS=31
CHECKS=528638
SAT
TT=1.397

SAT
TT=1.684

csp/hanoi/
normalized-hanoi-6_ext.xml
SAT
TT=0.041

SAT
TT=0.868

CHECKS=570583
NODES=62
SAT
TT=0.732

CHECKS=570583
NODES=0
SAT
TT=0.356

SAT
TT=5.463

CHECKS=241
SAT
TT=5.519

CHECKS=241
SAT
TT=3.422

ASSIGNMENTS=1
SAT
TT=3.392

ASSIGNMENTS=1
SAT
TT=5.704

CHECKS=1045260
NODES=62
SAT
TT=0.533

CHECKS=1096760
NODES=0
SAT
TT=0.55

CHECKS=1096760
NODES=0
SAT
TT=0.062

ASSIGNMENTS=0
CHECKS=240
SAT
TT=0.058

ASSIGNMENTS=0
CHECKS=240
SAT
TT=274.011

SAT
TT=271.254

SAT
TT=0.046

ASSIGNMENTS=0
CHECKS=240
SAT
TT=0.041

ASSIGNMENTS=0
CHECKS=240
? (problem)
TT=1.33

SAT
TT=0.048

ASSIGNMENTS=0
CHECKS=240
SAT
TT=0.044

ASSIGNMENTS=0
CHECKS=240
SAT
TT=4.14

SAT
TT=0.925

ASSIGNMENTS=62
CHECKS=5075380
SAT
TT=17.11

SAT
TT=16.641

csp/hanoi/
normalized-hanoi-7_ext.xml
SAT
TT=0.391

SAT
TT=1.042

CHECKS=10427700
NODES=126
SAT
TT=1.09

CHECKS=10427700
NODES=126
SAT
TT=5.435

SAT
TT=203.012

CHECKS=497
SAT
TT=203.165

CHECKS=497
SAT
TT=55.823

ASSIGNMENTS=1
SAT
TT=55.218

ASSIGNMENTS=1
SAT
TT=60.605

CHECKS=19969000
NODES=126
SAT
TT=1.075

CHECKS=20567700
NODES=0
SAT
TT=1.117

CHECKS=20567700
NODES=0
SAT
TT=0.409

ASSIGNMENTS=0
CHECKS=496
SAT
TT=0.442

ASSIGNMENTS=0
CHECKS=496
? (TO)
TT=1800.03

? (TO)
TT=1800.09

SAT
TT=0.425

ASSIGNMENTS=0
CHECKS=496
SAT
TT=0.391

ASSIGNMENTS=0
CHECKS=496
? (problem)
TT=5.34

SAT
TT=0.407

ASSIGNMENTS=0
CHECKS=496
SAT
TT=0.425

ASSIGNMENTS=0
CHECKS=496
SAT
TT=28.386

SAT
TT=30.896

ASSIGNMENTS=127
CHECKS=185321000
SAT
TT=475.201

SAT
TT=368.853

csp/langford/
normalized-langford-2-4-ext.xml
SAT
TT=0.005

SAT
TT=0.669

CHECKS=242
NODES=8
SAT
TT=0.704

CHECKS=242
NODES=0
SAT
TT=0.029

SAT
TT=0.011

CHECKS=3422
SAT
TT=0.011

CHECKS=2609
SAT
TT=0.395

ASSIGNMENTS=2
SAT
TT=0.412

ASSIGNMENTS=2
SAT
TT=0.315

CHECKS=270
NODES=10
SAT
TT=0.324

CHECKS=2762
NODES=55
SAT
TT=0.325

CHECKS=32159
NODES=349
SAT
TT=0.017

ASSIGNMENTS=2
CHECKS=200
SAT
TT=0.018

ASSIGNMENTS=2
CHECKS=200
SAT
TT=0.809

SAT
TT=0.812

SAT
TT=0.006

ASSIGNMENTS=2
CHECKS=200
SAT
TT=0.008

ASSIGNMENTS=2
CHECKS=192
SAT
TT=0.274

SAT
TT=0.005

ASSIGNMENTS=2
CHECKS=226
SAT
TT=0.008

ASSIGNMENTS=2
CHECKS=200
SAT
TT=0.715

SAT
TT=0.009

ASSIGNMENTS=10
CHECKS=1024
SAT
TT=0.503

SAT
TT=0.499

csp/langford/
normalized-langford-3-11-ext.xml
UNSAT
TT=1.162

UNSAT
TT=7.291

CHECKS=10053800
NODES=34361
UNSAT
TT=7.19

CHECKS=10075500
NODES=34337
UNSAT
TT=2.278

UNSAT
TT=77.72

CHECKS=38638200
UNSAT
TT=83.566

CHECKS=60527400
UNSAT
TT=2.831

ASSIGNMENTS=4009
UNSAT
TT=5.009

ASSIGNMENTS=7542
UNSAT
TT=9.355

CHECKS=3182420
NODES=19411
? (TO)
TT=1800.09

? (TO)
TT=1800.08

UNSAT
TT=1.203

ASSIGNMENTS=8266
CHECKS=7322500
UNSAT
TT=1.214

ASSIGNMENTS=8266
CHECKS=7322500
UNSAT
TT=13.894

UNSAT
TT=11.687

UNSAT
TT=1.174

ASSIGNMENTS=8211
CHECKS=7306140
UNSAT
TT=1.602

ASSIGNMENTS=11454
CHECKS=10053200
UNSAT
TT=128.27

UNSAT
TT=1.162

ASSIGNMENTS=8504
CHECKS=7670600
UNSAT
TT=1.168

ASSIGNMENTS=8266
CHECKS=7322500
UNSAT
TT=38.073

UNSAT
TT=37.46

ASSIGNMENTS=38939
CHECKS=51069000
UNSAT
TT=10.274

UNSAT
TT=12.742

csp/langford/
normalized-langford-3-9-ext.xml
SAT
TT=0.019

SAT
TT=0.954

CHECKS=123859
NODES=438
SAT
TT=0.977

CHECKS=140446
NODES=432
SAT
TT=0.088

SAT
TT=1.919

CHECKS=1011290
SAT
TT=0.121

CHECKS=61258
SAT
TT=0.808

ASSIGNMENTS=47
SAT
TT=0.815

ASSIGNMENTS=93
SAT
TT=0.555

CHECKS=5001
NODES=32
SAT
TT=0.784

CHECKS=1932420
NODES=4662
? (TO)
TT=1800.03

SAT
TT=0.04

ASSIGNMENTS=110
CHECKS=81976
SAT
TT=0.039

ASSIGNMENTS=110
CHECKS=81976
SAT
TT=1.94

SAT
TT=1.838

SAT
TT=0.03

ASSIGNMENTS=110
CHECKS=81548
SAT
TT=0.019

ASSIGNMENTS=21
CHECKS=14078
SAT
TT=0.61

SAT
TT=0.048

ASSIGNMENTS=249
CHECKS=236445
SAT
TT=0.028

ASSIGNMENTS=110
CHECKS=81976
SAT
TT=1.696

SAT
TT=0.144

ASSIGNMENTS=47
CHECKS=59638
SAT
TT=1.102

SAT
TT=1.25

csp/langford/
normalized-slangford-3-11-ext.xml
UNSAT
TT=1.246

UNSAT
TT=7.164

CHECKS=10089200
NODES=34311
UNSAT
TT=7.188

CHECKS=10107900
NODES=34344
UNSAT
TT=2.366

UNSAT
TT=291.228

CHECKS=132919000
UNSAT
TT=89.689

CHECKS=48258100
UNSAT
TT=3.31

ASSIGNMENTS=4019
UNSAT
TT=6.048

ASSIGNMENTS=7542
UNSAT
TT=9.424

CHECKS=3191300
NODES=19411
? (TO)
TT=1800.01

? (TO)
TT=1800.01

UNSAT
TT=1.36

ASSIGNMENTS=8213
CHECKS=7771110
UNSAT
TT=1.246

ASSIGNMENTS=8213
CHECKS=7771110
UNSAT
TT=13.656

UNSAT
TT=19.671

UNSAT
TT=1.345

ASSIGNMENTS=8273
CHECKS=7820000
UNSAT
TT=1.852

ASSIGNMENTS=11450
CHECKS=10681500
UNSAT
TT=145.631

UNSAT
TT=1.397

ASSIGNMENTS=8465
CHECKS=8144540
UNSAT
TT=1.31

ASSIGNMENTS=8213
CHECKS=7771110
UNSAT
TT=37.618

UNSAT
TT=41.949

ASSIGNMENTS=50339
CHECKS=65965600
UNSAT
TT=11.155

UNSAT
TT=14.969

Some statistics...

Abscon 112v4
AC
Abscon 112v4
ESAC
bpsolver
2008-06-27
casper
zao
casper
zito
choco2_dwdeg
2008-06-26
choco2_impwdeg
2008-06-26
Concrete + CPS4J
2008-05-30
Concrete + CSP4J - Tabu Engine
2008-05-30
Concrete + CSP4J - WMC Engine
2008-05-30
cpHydra
k_10
cpHydra
k_40
galac
E
galac
M
MDG-noprobe
2008-06-27
MDG-probe
2008-06-27
Minion/Tailor
2008-07-04
Mistral-option
1.314
Mistral-prime
1.313
SAT4J CSP
2008-06-13
spider
2008-06-14
Sugar
v1.13+minisat
Sugar
v1.13+picosat
Number of times the solver is able to give the best known answer99999999709988996999999
Number of times the solver is able to give the best known answer from an incomplete solver point of view (i.e. without considering optimality proof)99999999709988996999999
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)00010000000100040520000
Number of times the solver is the best solver from an incomplete solver point of view (i.e. best known answer and best T1 time)00000000000000000000000