CSP 2008 Competition: results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell exampleMeaning
AnswerSolver result
TT=...Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out
T1=...Time to get the best model (T1): this is the wall clock time (in seconds) ellapsed when the solver found its best model. This time is obtained from the 'o lines'. It is useful to compare both complete and incomplete solvers. It doesn't take into account the time to prove optimality of the solution.
Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

AbbreviationMeaning
TOTime Out
MOMem. Out (out of memory)

Meaning of the different colors:

ColorMeaning
textthe solver cannot handle this instance
textthe solver gave no answer
textthe solver gave a definitive answer (SAT or UNSAT)
textthe solver performed better than the other ones on that instance
textthe solver was ended by a signal
textthe solver gave an incomplete answer
textthe solver was disqualified in the category
textthe solver gave a wrong answer

For better readability, you may choose to hide some solvers:
Abscon 112v4 AC
Abscon 112v4 ESAC
bpsolver 2008-06-27
casper zao
casper zito
choco2_dwdeg 2008-06-26
choco2_impwdeg 2008-06-26
Concrete + CPS4J 2008-05-30
Concrete + CSP4J - Tabu Engine 2008-05-30
Concrete + CSP4J - WMC Engine 2008-05-30
cpHydra k_10
cpHydra k_40
galac E
galac M
MDG-noprobe 2008-06-27
MDG-probe 2008-06-27
Minion/Tailor 2008-07-04
Mistral-option 1.314
Mistral-prime 1.313
SAT4J CSP 2008-06-13
spider 2008-06-14
Sugar v1.13+minisat
Sugar v1.13+picosat

Bench nameBest results
on this
instance
Abscon 112v4
AC
Abscon 112v4
ESAC
bpsolver
2008-06-27
casper
zao
casper
zito
choco2_dwdeg
2008-06-26
choco2_impwdeg
2008-06-26
Concrete + CPS4J
2008-05-30
Concrete + CSP4J - Tabu Engine
2008-05-30
Concrete + CSP4J - WMC Engine
2008-05-30
cpHydra
k_10
cpHydra
k_40
galac
E
galac
M
MDG-noprobe
2008-06-27
MDG-probe
2008-06-27
Minion/Tailor
2008-07-04
Mistral-option
1.314
Mistral-prime
1.313
SAT4J CSP
2008-06-13
spider
2008-06-14
Sugar
v1.13+minisat
Sugar
v1.13+picosat
csp/driver/
normalized-driverlogw-01c-sat_ext.xml
SAT
TT=0.01

SAT
TT=0.771

CHECKS=1271
NODES=71
SAT
TT=0.736

CHECKS=1242
NODES=0
SAT
TT=0.047

SAT
TT=0.018

CHECKS=4665
SAT
TT=0.016

CHECKS=3190
SAT
TT=0.509

ASSIGNMENTS=24
SAT
TT=0.518

ASSIGNMENTS=24
SAT
TT=0.378

CHECKS=733
NODES=71
SAT
TT=0.412

CHECKS=3422
NODES=31
SAT
TT=0.389

CHECKS=6207
NODES=19
SAT
TT=0.023

ASSIGNMENTS=24
CHECKS=535
SAT
TT=0.023

ASSIGNMENTS=24
CHECKS=535
SAT
TT=0.998

SAT
TT=1.01

SAT
TT=0.01

ASSIGNMENTS=24
CHECKS=535
SAT
TT=0.011

ASSIGNMENTS=20
CHECKS=512
SAT
TT=0.386

SAT
TT=0.01

ASSIGNMENTS=23
CHECKS=460
SAT
TT=0.01

ASSIGNMENTS=24
CHECKS=535
SAT
TT=0.844

SAT
TT=0.019

ASSIGNMENTS=71
CHECKS=1442
SAT
TT=0.699

SAT
TT=0.688

csp/driver/
normalized-driverlogw-02c-sat_ext.xml
SAT
TT=0.136

SAT
TT=2.307

CHECKS=1742190
NODES=4000
SAT
TT=3.196

CHECKS=2504230
NODES=4111
SAT
TT=0.477

SAT
TT=4.127

CHECKS=3171710
SAT
TT=1.333

CHECKS=1153110
SAT
TT=1.573

ASSIGNMENTS=65
SAT
TT=1.774

ASSIGNMENTS=65
SAT
TT=3.114

CHECKS=35400
NODES=423
SAT
TT=5.566

CHECKS=9618040
NODES=59971
SAT
TT=4.35

CHECKS=7845390
NODES=3763
SAT
TT=0.139

ASSIGNMENTS=180
CHECKS=117043
SAT
TT=0.142

ASSIGNMENTS=180
CHECKS=117043
SAT
TT=2.919

SAT
TT=3.028

SAT
TT=0.143

ASSIGNMENTS=201
CHECKS=150798
SAT
TT=0.781

ASSIGNMENTS=9610
CHECKS=2701420
SAT
TT=1.352

SAT
TT=0.447

ASSIGNMENTS=1308
CHECKS=1602120
SAT
TT=0.136

ASSIGNMENTS=180
CHECKS=117043
SAT
TT=3.679

SAT
TT=0.897

ASSIGNMENTS=1248
CHECKS=609322
SAT
TT=1.567

SAT
TT=1.728

csp/driver/
normalized-driverlogw-04c-sat_ext.xml
SAT
TT=0.118

SAT
TT=1.635

CHECKS=420610
NODES=884
SAT
TT=1.865

CHECKS=563609
NODES=627
SAT
TT=0.52

? (TO)
TT=1800.05

SAT
TT=1.208

CHECKS=1017290
SAT
TT=1.173

ASSIGNMENTS=31
SAT
TT=1.18

ASSIGNMENTS=31
SAT
TT=2.69

CHECKS=24260
NODES=292
SAT
TT=2.716

CHECKS=479876
NODES=2435
SAT
TT=2.669

CHECKS=851220
NODES=355
SAT
TT=0.129

ASSIGNMENTS=95
CHECKS=64460
SAT
TT=0.131

ASSIGNMENTS=95
CHECKS=64460
SAT
TT=2.928

SAT
TT=2.808

SAT
TT=0.124

ASSIGNMENTS=97
CHECKS=66005
SAT
TT=0.145

ASSIGNMENTS=458
CHECKS=150402
SAT
TT=1.136

SAT
TT=0.285

ASSIGNMENTS=591
CHECKS=754943
SAT
TT=0.118

ASSIGNMENTS=95
CHECKS=64460
SAT
TT=4.484

SAT
TT=0.415

ASSIGNMENTS=539
CHECKS=390557
SAT
TT=1.495

SAT
TT=1.625

csp/driver/
normalized-driverlogw-05c-sat_ext.xml
SAT
TT=0.174

SAT
TT=1.939

CHECKS=517412
NODES=1333
SAT
TT=1.71

CHECKS=216933
NODES=0
SAT
TT=0.733

SAT
TT=18.462

CHECKS=10742900
SAT
TT=3.754

CHECKS=2207510
SAT
TT=3.474

ASSIGNMENTS=95
SAT
TT=3.633

ASSIGNMENTS=95
SAT
TT=5.845

CHECKS=38628
NODES=378
SAT
TT=8.709

CHECKS=10323200
NODES=41956
SAT
TT=6

CHECKS=1051790
NODES=289
SAT
TT=0.183

ASSIGNMENTS=74
CHECKS=59750
SAT
TT=0.179

ASSIGNMENTS=74
CHECKS=59750
SAT
TT=3.677

SAT
TT=3.603

SAT
TT=0.196

ASSIGNMENTS=117
CHECKS=133099
SAT
TT=0.736

ASSIGNMENTS=5825
CHECKS=2512870
SAT
TT=1.874

SAT
TT=0.603

ASSIGNMENTS=873
CHECKS=1787770
SAT
TT=0.174

ASSIGNMENTS=74
CHECKS=59750
SAT
TT=4.455

SAT
TT=0.927

ASSIGNMENTS=1279
CHECKS=418105
SAT
TT=2.115

SAT
TT=1.861

csp/driver/
normalized-driverlogw-08cc-sat_ext.xml
SAT
TT=0.359

SAT
TT=2.762

CHECKS=2027910
NODES=2613
SAT
TT=3.832

CHECKS=4430730
NODES=2706
SAT
TT=1.308

SAT
TT=30.815

CHECKS=15004800
SAT
TT=6.917

CHECKS=2820000
SAT
TT=6.203

ASSIGNMENTS=76
SAT
TT=6.139

ASSIGNMENTS=76
SAT
TT=13.984

CHECKS=76029
NODES=489
SAT
TT=17.597

CHECKS=13907500
NODES=47630
SAT
TT=20.227

CHECKS=20735900
NODES=5463
SAT
TT=0.365

ASSIGNMENTS=174
CHECKS=476259
SAT
TT=0.367

ASSIGNMENTS=174
CHECKS=476259
SAT
TT=5.484

SAT
TT=4.615

SAT
TT=0.53

ASSIGNMENTS=470
CHECKS=1122820
SAT
TT=2.093

ASSIGNMENTS=9657
CHECKS=6612560
SAT
TT=3.392

SAT
TT=1.383

ASSIGNMENTS=1191
CHECKS=4009560
SAT
TT=0.359

ASSIGNMENTS=174
CHECKS=476259
SAT
TT=5.288

SAT
TT=2.164

ASSIGNMENTS=1141
CHECKS=648302
SAT
TT=2.208

SAT
TT=2.282

csp/driver/
normalized-driverlogw-08c-sat_ext.xml
SAT
TT=0.351

SAT
TT=3.349

CHECKS=3976790
NODES=3591
SAT
TT=4.645

CHECKS=5487000
NODES=3194
SAT
TT=1.269

SAT
TT=31.053

CHECKS=17547300
SAT
TT=5.419

CHECKS=2209600
SAT
TT=6.107

ASSIGNMENTS=76
SAT
TT=6.154

ASSIGNMENTS=76
SAT
TT=14.138

CHECKS=76056
NODES=489
SAT
TT=57.988

CHECKS=134653000
NODES=490165
SAT
TT=15.427

CHECKS=6441340
NODES=2234
SAT
TT=0.366

ASSIGNMENTS=174
CHECKS=476259
SAT
TT=0.363

ASSIGNMENTS=174
CHECKS=476259
SAT
TT=5.097

SAT
TT=4.959

SAT
TT=0.516

ASSIGNMENTS=470
CHECKS=1122820
SAT
TT=2.111

ASSIGNMENTS=9682
CHECKS=6424930
SAT
TT=3.491

SAT
TT=1.407

ASSIGNMENTS=1191
CHECKS=4009560
SAT
TT=0.351

ASSIGNMENTS=174
CHECKS=476259
SAT
TT=5.217

SAT
TT=2.209

ASSIGNMENTS=1141
CHECKS=648251
SAT
TT=2.264

SAT
TT=2.23

csp/driver/
normalized-driverlogw-09-sat_ext.xml
SAT
TT=2.291

SAT
TT=152.666

CHECKS=35571000
NODES=27204
SAT
TT=143.575

CHECKS=33533600
NODES=23806
SAT
TT=42.351

SAT
TT=80.622

CHECKS=34515400
SAT
TT=93.872

CHECKS=41022300
SAT
TT=15.952

ASSIGNMENTS=345
SAT
TT=17.887

ASSIGNMENTS=345
SAT
TT=141.088

CHECKS=1964450
NODES=6371
? (TO)
TT=1800.06

? (TO)
TT=1800.06

SAT
TT=34.14

ASSIGNMENTS=28499
CHECKS=36491500
SAT
TT=37.224

ASSIGNMENTS=28499
CHECKS=36491500
SAT
TT=10.527

SAT
TT=9.623

SAT
TT=2.291

ASSIGNMENTS=1581
CHECKS=5209070
SAT
TT=7.406

ASSIGNMENTS=14916
CHECKS=17398700
SAT
TT=8.216

SAT
TT=6.433

ASSIGNMENTS=2328
CHECKS=14564800
SAT
TT=2.975

ASSIGNMENTS=2280
CHECKS=6973980
SAT
TT=6.641

SAT
TT=39.181

ASSIGNMENTS=22420
CHECKS=20207800
SAT
TT=3.539

SAT
TT=3.934

Some statistics...

Abscon 112v4
AC
Abscon 112v4
ESAC
bpsolver
2008-06-27
casper
zao
casper
zito
choco2_dwdeg
2008-06-26
choco2_impwdeg
2008-06-26
Concrete + CPS4J
2008-05-30
Concrete + CSP4J - Tabu Engine
2008-05-30
Concrete + CSP4J - WMC Engine
2008-05-30
cpHydra
k_10
cpHydra
k_40
galac
E
galac
M
MDG-noprobe
2008-06-27
MDG-probe
2008-06-27
Minion/Tailor
2008-07-04
Mistral-option
1.314
Mistral-prime
1.313
SAT4J CSP
2008-06-13
spider
2008-06-14
Sugar
v1.13+minisat
Sugar
v1.13+picosat
Number of times the solver is able to give the best known answer77767777607777777777777
Number of times the solver is able to give the best known answer from an incomplete solver point of view (i.e. without considering optimality proof)77767777607777777777777
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)00000000000000200060000
Number of times the solver is the best solver from an incomplete solver point of view (i.e. best known answer and best T1 time)00000000000000000000000