MAX-CSP 2006 Competition: results by benchmark

Results by benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

This page displays the results of the different solvers for each benchmark for category binary constraints in extension (2-ARY-EXT), subcategory real world problems

REMINDER

Keep in mind that the 'Best result' columns only provide the best result given by one of the solvers. This 'Best result' may be wrong in case of an UNSATISFIABLE or OPTIMUM FOUND answer (because there's no efficient way to check these answers).

Description of a cell contents:

Cell exampleMeaning
AnswerSolver result
f=...value of the objective function for the model reported by the solver
TT=...Total Time (TT): this is the CPU time (in seconds) used by the solver until termination. This time is only meaningful for complete solvers because incomplete solvers will always run until they time out
Remember that CPU time and wall clock time are two very different notions. The CPU time represents the time during which the instructions of the solver were executed by the processor. The wall clock time represents how much time ellapsed on the clock. For a same event, the CPU time may be either smaller or greater than the wall clock time depending on the number of threads of execution and the number of processors.

Meaning of some abbreviations:

AbbreviationMeaning
f=...Value of the objective function
TOTime Out
MOMem. Out (out of memory)

Meaning of the different colors:

ColorMeaning
textthe solver cannot handle this instance
textthe solver gave no answer
textthe solver could give an answer (SAT)
textthe solver gave a definitive answer (OPTIMUM FOUND or UNSAT)
textthe solver performed better than the other ones on that instance
textthe solver was ended by a signal
textthe solver gave an incomplete answer
textthe solver was disqualified in the category
textthe solver gave a wrong answer

For better readability, you may choose to hide some solvers:
AbsconMax 109 EPFC
AbsconMax 109 PFC
aolibdvo 2007-01-17
aolibpvo 2007-01-17
CSP4J - MaxCSP 2006-12-19
toolbar 2007-01-12
Toolbar_BTD 2007-01-12
Toolbar_MaxSat 2007-01-19
Toulbar2 2007-01-12

Bench nameBest results
on this
instance		AbsconMax
109 EPFC		AbsconMax
109 PFC		aolibdvo
2007-01-17		aolibpvo
2007-01-17		CSP4J - MaxCSP
2006-12-19		toolbar
2007-01-12		Toolbar_BTD
2007-01-12		Toolbar_MaxSat
2007-01-19		Toulbar2
2007-01-12
MaxCSP/celar/graphs/
graphw-05_ext.xml		 MOPT
f=1029
TT=30.904

 OPTIMUM
f=1029
TT=45.638
T1=1.92
 OPTIMUM
f=1029
TT=73.691
T1=1.81
 ? (TO)

TT=2400.09

 MSAT (TO)
f=889
TT=2400.12

 ? (problem)

TT=2.364

NODES=0
 ? (TO)

TT=2400.12

 ? (TO)

TT=2400.03

 ? (exit code)

TT=1.489

 OPTIMUM
f=1029
TT=30.904
MaxCSP/celar/graphs/
graphw-06_ext.xml		 MSAT TO
f=1960
TT=2400.05

 MSAT (TO)
f=1960
TT=2400.05
T1=5.51
 MSAT (TO)
f=1960
TT=2400.1
T1=5.68
 ? (TO)

TT=2400.02

 ? (TO)

TT=2400.03

 ? (problem)

TT=4.411

NODES=0
 ? (TO)

TT=2400.07

 ? (TO)

TT=2400.02

 ? (exit code)

TT=2.586

 ? (TO)

TT=2400.1
MaxCSP/celar/graphs/
graphw-07_ext.xml		 MOPT
f=1861
TT=151.83

 MSAT (TO)
f=1852
TT=2400.14

 MSAT (TO)
f=1852
TT=2400.17

 OPTIMUM
f=1861
TT=151.83

 OPTIMUM
f=1861
TT=354.404

 ? (problem)

TT=2.275

NODES=0
 OPTIMUM
f=1861
TT=540.8

 OPTIMUM
f=1861
TT=537.458

 ? (exit code)

TT=0.761

 OPTIMUM
f=1861
TT=542.955
MaxCSP/celar/graphs/
graphw-11_ext.xml		 MSAT TO
f=3390
TT=2400.14

 MSAT (TO)
f=3390
TT=2400.14
T1=12.91
 MSAT (TO)
f=3390
TT=2400.14
T1=13.78
 ? (TO)

TT=2400.09

 ? (TO)

TT=2400.11

 ? (problem)

TT=7.321

NODES=0
 ? (TO)

TT=2400.04

 ? (TO)

TT=2400.05

 ? (exit code)

TT=2.645

 ? (TO)

TT=2400.13
MaxCSP/celar/graphs/
graphw-12_ext.xml		 MOPT
f=3571
TT=35.966

 MSAT (TO)
f=3534
TT=2400.09

 MSAT (TO)
f=3534
TT=2400.09

 OPTIMUM
f=3571
TT=166.988

 OPTIMUM
f=3571
TT=35.966

 ? (problem)

TT=4.539

NODES=0
 OPTIMUM
f=3571
TT=970.172

 OPTIMUM
f=3571
TT=915.109

 ? (exit code)

TT=4.116

 OPTIMUM
f=3571
TT=979.385
MaxCSP/celar/graphs/
graphw-13_ext.xml		 MSAT TO
f=4762
TT=2400.07

 MSAT (TO)
f=4762
TT=2400.07
T1=13.76
 MSAT (TO)
f=4762
TT=2400.14
T1=13.94
 ? (TO)

TT=2400.1

 ? (TO)

TT=2400.07

 ? (problem)

TT=10.478

NODES=0
 ? (TO)

TT=2400.01

 ? (TO)

TT=2400.01

 ? (exit code)

TT=3.805

 ? (TO)

TT=2400.03
MaxCSP/celar/scens/
scenw-06-16_ext.xml		 MOPT
f=1197
TT=57.742

 MSAT (TO)
f=1197
TT=2400.08
T1=2.19
 MSAT (TO)
f=1197
TT=2400.11
T1=2.35
 MSAT (TO)
f=1105
TT=2400.02

 MSAT (TO)
f=1170
TT=2400.05

 ? (problem)

TT=1.473

NODES=0
 OPTIMUM
f=1197
TT=1398.28

 ? (TO)

TT=2400.01

 ? (MO)

TT=1.07

 OPTIMUM
f=1197
TT=57.742
MaxCSP/celar/scens/
scenw-06-18_ext.xml		 MOPT
f=1197
TT=44.126

 MSAT (TO)
f=1194
TT=2400.07
T1=14.05
 MSAT (TO)
f=1194
TT=2400.17
T1=13.73
 MSAT (TO)
f=1183
TT=2400.1

 ? (TO)
f=1166
TT=2400.03

 ? (problem)

TT=1.41

NODES=0
 OPTIMUM
f=1197
TT=326.138

 OPTIMUM
f=1197
TT=763.838

 ? (MO)

TT=1.72

 OPTIMUM
f=1197
TT=44.126
MaxCSP/celar/scens/
scenw-06-20_ext.xml		 MOPT
f=1191
TT=46.911

 MSAT (TO)
f=1188
TT=2400.11
T1=35.3
 MSAT (TO)
f=1188
TT=2400.11
T1=35.14
 ? (TO)
f=1164
TT=2400.03

 MSAT (TO)
f=1171
TT=2400.02

 ? (problem)

TT=1.301

NODES=0
 OPTIMUM
f=1191
TT=789.441

 OPTIMUM
f=1191
TT=481.917

 ? (MO)

TT=1.99

 OPTIMUM
f=1191
TT=46.911
MaxCSP/celar/scens/
scenw-06-22_ext.xml		 MOPT
f=1188
TT=27.938

 OPTIMUM
f=1188
TT=2243.7
T1=18.04
 MSAT (TO)
f=1188
TT=2400.06
T1=16.57
 OPTIMUM
f=1188
TT=1761.94

 MSAT (TO)
f=1178
TT=2400.02

 ? (problem)

TT=1.131

NODES=0
 OPTIMUM
f=1188
TT=41.959

 OPTIMUM
f=1188
TT=268.143

 ? (MO)

TT=2

 OPTIMUM
f=1188
TT=27.938
Bench nameBest results
on this
instance		AbsconMax
109 EPFC		AbsconMax
109 PFC		aolibdvo
2007-01-17		aolibpvo
2007-01-17		CSP4J - MaxCSP
2006-12-19		toolbar
2007-01-12		Toolbar_BTD
2007-01-12		Toolbar_MaxSat
2007-01-19		Toulbar2
2007-01-12
MaxCSP/celar/scens/
scenw-06-24_ext.xml		 MOPT
f=1181
TT=29.235

 MSAT (TO)
f=1176
TT=2400.13
T1=10.21
 MSAT (TO)
f=1176
TT=2400.09
T1=10.14
 OPTIMUM
f=1181
TT=1208.25

 MSAT (TO)
f=1163
TT=2400.05

 ? (problem)

TT=1.05

NODES=0
 OPTIMUM
f=1181
TT=46.449

 OPTIMUM
f=1181
TT=338.6

 ? (MO)

TT=2.2

 OPTIMUM
f=1181
TT=29.235
MaxCSP/celar/scens/
scenw-06-30_ext.xml		 MOPT
f=1157
TT=20.451

 OPTIMUM
f=1157
TT=1879.89
T1=1816.52
 OPTIMUM
f=1157
TT=2057.36
T1=2030.12
 OPTIMUM
f=1157
TT=477.013

 OPTIMUM
f=1157
TT=53.797

 ? (problem)

TT=0.783

NODES=0
 OPTIMUM
f=1157
TT=22.986

 OPTIMUM
f=1157
TT=76.281

 ? (TO)

TT=2400.06

 OPTIMUM
f=1157
TT=20.451
MaxCSP/celar/
scens/
scenw-06-30reduc_ext.xml		 MOPT
f=397
TT=1.478

 OPTIMUM
f=397
TT=38.739

 OPTIMUM
f=397
TT=42.018

 OPTIMUM
f=397
TT=8.663
T1=8.66
 OPTIMUM
f=397
TT=1.478
T1=1.47
 MSAT (TO)
f=395
TT=2400.01

 OPTIMUM
f=397
TT=15.469

 OPTIMUM
f=397
TT=16.106

 ? (TO)

TT=2400.04

 OPTIMUM
f=397
TT=15.321
MaxCSP/celar/
scens/
scenw-06_ext.xml		 MOPT
f=1192
TT=605.8

 MSAT (TO)
f=1184
TT=2400.14
T1=14.8
 MSAT (TO)
f=1184
TT=2400.08
T1=13.71
 ? (TO)

TT=2400.02

 MSAT (TO)
f=1139
TT=2400.08

 ? (problem)

TT=2.761

NODES=0
 ? (TO)

TT=2400.05

 ? (TO)

TT=2400.02

 ? (exit code)

TT=0.871

 OPTIMUM
f=1192
TT=605.8
MaxCSP/celar/scens/
scenw-07_10000_30r_ext.xml		 MOPT
f=1228
TT=269.634

 MSAT (TO)
f=1228
TT=2400.04
T1=13.83
 MSAT (TO)
f=1228
TT=2400.07
T1=14
 ? (TO)

TT=2400.02

 Signal

TT=1427.63

 ? (problem)

TT=0.864

NODES=0
 MSAT (TO)
f=1196
TT=2400.1

 ? (TO)

TT=2400.01

 ? (MO)

TT=2.11

 OPTIMUM
f=1228
TT=269.634
MaxCSP/celar/
scens/
scenw-07_ext.xml		 MSAT TO
f=2589
TT=2400.12

 MSAT (TO)
f=2589
TT=2400.12
T1=70.35
 MSAT (TO)
f=2589
TT=2400.13
T1=68.56
 ? (TO)

TT=2400.1

 ? (TO)

TT=2400.11

 ? (problem)

TT=6.308

NODES=0
 ? (TO)

TT=2400.07

 ? (TO)

TT=2400.04

 ? (exit code)

TT=1.935

 ? (TO)

TT=2400.1
MaxCSP/celar/
scens/
scenw-08_ext.xml		 MSAT TO
f=5085
TT=2400.09

 MSAT (TO)
f=5085
TT=2400.15
T1=21.98
 MSAT (TO)
f=5085
TT=2400.09
T1=22.75
 ? (TO)

TT=2400.11

 ? (TO)

TT=2400.04

 ? (problem)

TT=9.999

NODES=0
 ? (TO)

TT=2400.07

 ? (TO)

TT=2400.01

 ? (exit code)

TT=3.676

 ? (TO)

TT=2400.05
MaxCSP/celar/
scens/
scenw-09_ext.xml		 MOPT
f=3204
TT=2.004

 MSAT (TO)
f=3194
TT=2400.18

 MSAT (TO)
f=3194
TT=2400.1

 OPTIMUM
f=3204
TT=2.62

 OPTIMUM
f=3204
TT=2.004

 ? (problem)

TT=3.24

NODES=0
 OPTIMUM
f=3204
TT=1665.78

 OPTIMUM
f=3204
TT=1597.39

 ? (exit code)

TT=1.027

 OPTIMUM
f=3204
TT=1566.98
MaxCSP/celar/
scens/
scenw-10_ext.xml		 MOPT
f=3204
TT=2.375

 MSAT (TO)
f=3194
TT=2400.13

 MSAT (TO)
f=3194
TT=2400.11

 OPTIMUM
f=3204
TT=2.614

 OPTIMUM
f=3204
TT=2.375

 ? (problem)

TT=3.264

NODES=0
 OPTIMUM
f=3204
TT=1573.81

 OPTIMUM
f=3204
TT=1572.43

 ? (exit code)

TT=1.08

 OPTIMUM
f=3204
TT=1647.02
MaxCSP/celar/
subs6/
scenw-6-sub0_ext.xml		 MOPT
f=201
TT=3.104

 OPTIMUM
f=201
TT=46.977
T1=0.87
 OPTIMUM
f=201
TT=48.636
T1=0.88
 OPTIMUM
f=201
TT=3.226

 OPTIMUM
f=201
TT=3.104

 MSAT (TO)
f=195
TT=2400.04

 OPTIMUM
f=201
TT=35.688

 OPTIMUM
f=201
TT=36.04

 Wrong Opt.
f=72
TT=20.044

 OPTIMUM
f=201
TT=35.725
Bench nameBest results
on this
instance		AbsconMax
109 EPFC		AbsconMax
109 PFC		aolibdvo
2007-01-17		aolibpvo
2007-01-17		CSP4J - MaxCSP
2006-12-19		toolbar
2007-01-12		Toolbar_BTD
2007-01-12		Toolbar_MaxSat
2007-01-19		Toulbar2
2007-01-12
MaxCSP/celar/
subs6/
scenw-6-sub1-24_ext.xml		 MOPT
f=282
TT=0.744

 OPTIMUM
f=282
TT=94.254
T1=0.82
 OPTIMUM
f=282
TT=58.993
T1=0.83
 OPTIMUM
f=282
TT=0.744

 OPTIMUM
f=282
TT=0.746

 MSAT (TO)
f=278
TT=2400.04

 OPTIMUM
f=282
TT=35.939

 OPTIMUM
f=282
TT=34.018

 ? (TO)

TT=2400.05

 OPTIMUM
f=282
TT=36.662
MaxCSP/celar/
subs6/
scenw-6-sub1_ext.xml		 MOPT
f=278
TT=43.165

 OPTIMUM
f=278
TT=549.88
T1=1.22
 OPTIMUM
f=278
TT=459.053
T1=1.23
 OPTIMUM
f=278
TT=43.45

 OPTIMUM
f=278
TT=43.165

 MSAT (TO)
f=274
TT=2400.02

 OPTIMUM
f=278
TT=60.261

 OPTIMUM
f=278
TT=59.929

 Wrong Opt.
f=88
TT=39.095

 OPTIMUM
f=278
TT=59.741
MaxCSP/celar/
subs6/
scenw-6-sub2_ext.xml		 MOPT
f=331
TT=0

 OPTIMUM
f=331
TT=704.054
T1=3.35
 OPTIMUM
f=331
TT=563.798
T1=3.15
 OPTIMUM
f=331
TT=73.985

 OPTIMUM
f=331
TT=75.653

 MSAT (TO)
f=327
TT=2400.05

 OPTIMUM
f=331
TT=0

 OPTIMUM
f=331
TT=72.151

 Wrong Opt.
f=106
TT=47.055

 OPTIMUM
f=331
TT=69.531
MaxCSP/celar/
subs6/
scenw-6-sub3_ext.xml		 MOPT
f=397
TT=71.146

 OPTIMUM
f=397
TT=1215.83
T1=3.31
 OPTIMUM
f=397
TT=1137.78
T1=3.59
 OPTIMUM
f=397
TT=1301.56

 OPTIMUM
f=397
TT=1690.82

 MSAT (TO)
f=393
TT=2400

 OPTIMUM
f=397
TT=73.715

 OPTIMUM
f=397
TT=72.624

 ? (MO)

TT=2.09

 OPTIMUM
f=397
TT=71.146
MaxCSP/celar/
subs6/
scenw-6-sub4-20_ext.xml		 MOPT
f=454
TT=2.138

 OPTIMUM
f=454
TT=444.631
T1=21.71
 OPTIMUM
f=454
TT=503.605
T1=21.93
 OPTIMUM
f=454
TT=2.138

 OPTIMUM
f=454
TT=2.167

 MSAT (TO)
f=451
TT=2400.09

 OPTIMUM
f=454
TT=27.349

 OPTIMUM
f=454
TT=27.49

 ? (TO)

TT=2400.06

 OPTIMUM
f=454
TT=28.039
MaxCSP/celar/
subs6/
scenw-6-sub4_ext.xml		 MOPT
f=453
TT=63.575

 OPTIMUM
f=453
TT=1498.71
T1=1.75
 OPTIMUM
f=453
TT=1813.82
T1=1.73
 MSAT (TO)
f=433
TT=2400.1

 MSAT (TO)
f=433
TT=2400.05

 MSAT (TO)
f=441
TT=2400.1

 OPTIMUM
f=453
TT=67.969

 OPTIMUM
f=453
TT=63.575

 ? (MO)

TT=1.09

 OPTIMUM
f=453
TT=66.724
MaxCSP/celar/
subs7/
scenw-7-sub0_ext.xml		 MOPT
f=184
TT=8.641

 OPTIMUM
f=184
TT=32.47
T1=1.23
 OPTIMUM
f=184
TT=32.506
T1=1.22
 OPTIMUM
f=184
TT=18.252

 OPTIMUM
f=184
TT=8.641

 ? (problem)

TT=0.877

NODES=0
 OPTIMUM
f=184
TT=26.969

 OPTIMUM
f=184
TT=27.805

 Wrong Opt.
f=68
TT=28.229

 OPTIMUM
f=184
TT=26.225
MaxCSP/celar/
subs7/
scenw-7-sub1-20_ext.xml		 MOPT
f=284
TT=0.129

 OPTIMUM
f=284
TT=42.371
T1=0.76
 OPTIMUM
f=284
TT=36.632
T1=0.76
 OPTIMUM
f=284
TT=0.129

 OPTIMUM
f=284
TT=0.133

 MSAT (TO)
f=282
TT=2400.01

 OPTIMUM
f=284
TT=28.634

 OPTIMUM
f=284
TT=29.849

 ? (TO)

TT=2400.09

 OPTIMUM
f=284
TT=29.584
MaxCSP/celar/
subs7/
scenw-7-sub1_ext.xml		 MOPT
f=282
TT=46.85

 OPTIMUM
f=282
TT=131.922
T1=0.84
 OPTIMUM
f=282
TT=127.754
T1=0.84
 OPTIMUM
f=282
TT=47.126

 OPTIMUM
f=282
TT=46.85

 MSAT (TO)
f=282
TT=2400.04

 OPTIMUM
f=282
TT=64.582

 OPTIMUM
f=282
TT=64.706

 Wrong Opt.
f=92
TT=41.985

 OPTIMUM
f=282
TT=64.047
MaxCSP/celar/
subs7/
scenw-7-sub2_ext.xml		 MOPT
f=335
TT=67.074

 OPTIMUM
f=335
TT=405.458
T1=0.8
 OPTIMUM
f=335
TT=394.991
T1=0.8
 OPTIMUM
f=335
TT=72.645

 OPTIMUM
f=335
TT=76.991

 MSAT (TO)
f=335
TT=2400.04

 OPTIMUM
f=335
TT=68.268

 OPTIMUM
f=335
TT=67.074

 Wrong Opt.
f=106
TT=51.35

 OPTIMUM
f=335
TT=68.253
Bench nameBest results
on this
instance		AbsconMax
109 EPFC		AbsconMax
109 PFC		aolibdvo
2007-01-17		aolibpvo
2007-01-17		CSP4J - MaxCSP
2006-12-19		toolbar
2007-01-12		Toolbar_BTD
2007-01-12		Toolbar_MaxSat
2007-01-19		Toulbar2
2007-01-12
MaxCSP/celar/
subs7/
scenw-7-sub3_ext.xml		 MOPT
f=401
TT=68.71

 OPTIMUM
f=401
TT=886.74
T1=1.25
 OPTIMUM
f=401
TT=1047.05
T1=1.26
 OPTIMUM
f=401
TT=193.087

 OPTIMUM
f=401
TT=193.497

 MSAT (TO)
f=392
TT=2400.08

 OPTIMUM
f=401
TT=70.435

 OPTIMUM
f=401
TT=71.111

 ? (MO)

TT=2

 OPTIMUM
f=401
TT=68.71
MaxCSP/celar/
subs7/
scenw-7-sub4-22_ext.xml		 MOPT
f=454
TT=21.038

 OPTIMUM
f=454
TT=168.29
T1=27.69
 OPTIMUM
f=454
TT=227.251
T1=25.53
 OPTIMUM
f=454
TT=21.038

 OPTIMUM
f=454
TT=25.95

 MSAT (TO)
f=441
TT=2400.09

 OPTIMUM
f=454
TT=34.673

 OPTIMUM
f=454
TT=37.147

 ? (TO)

TT=2400.02

 OPTIMUM
f=454
TT=37.036
MaxCSP/celar/
subs7/
scenw-7-sub4_ext.xml		 MOPT
f=457
TT=66.872

 OPTIMUM
f=457
TT=1705.85
T1=17.66
 OPTIMUM
f=457
TT=1758.18
T1=17.71
 OPTIMUM
f=457
TT=1334.59

 OPTIMUM
f=457
TT=1460.01

 MSAT (TO)
f=434
TT=2400.1

 OPTIMUM
f=457
TT=73.678

 OPTIMUM
f=457
TT=74.48

 ? (MO)

TT=1.08

 OPTIMUM
f=457
TT=66.872
MaxCSP/spot5/
spot5-29_ext.xml		 MOPT
f=415
TT=7.839

 MSAT (TO)
f=414
TT=2400.07
T1=13.42
 MSAT (TO)
f=415
TT=2400.17
T1=383.11
 OPTIMUM
f=415
TT=16.892
T1=17.02
 ? (TO)

TT=2400.11

 ? (problem)

TT=0.499

NODES=0
 OPTIMUM
f=415
TT=871.13

 OPTIMUM
f=415
TT=7.839

 ? (TO)

TT=2400.13

 OPTIMUM
f=415
TT=376.068
MaxCSP/spot5/
spot5-8_ext.xml		 MOPT
f=14
TT=0.006

 OPTIMUM
f=14
TT=0.671
T1=0.53
 OPTIMUM
f=14
TT=0.663
T1=0.52
 OPTIMUM
f=14
TT=0.006
T1=0.01
 OPTIMUM
f=14
TT=0.006
T1=0.01
 ? (problem)

TT=0.281

NODES=0
 OPTIMUM
f=14
TT=1.184

 OPTIMUM
f=14
TT=1.184

 OPTIMUM
f=14
TT=0.22

 OPTIMUM
f=14
TT=1.18

Some statistics...

AbsconMax
109 EPFC		AbsconMax
109 PFC		aolibdvo
2007-01-17		aolibpvo
2007-01-17		CSP4J - MaxCSP
2006-12-19		toolbar
2007-01-12		Toolbar_BTD
2007-01-12		Toolbar_MaxSat
2007-01-19		Toulbar2
2007-01-12
Number of times the solver is able to give the best known answer2423232002726030
Number of times the solver is the best solver from a complete solver point of view (i.e. best known answer and best TT time)5569013012