Benchmarks selection
Last year, the steering committee decided of a random selection of instances, selecting (when possible) the same number of instances per domain. A domain is a set of similar instances. Basically, one directory in an archive equals one domain, but some directories must be merged into a single domain when they contain similar instances. At most 5% of the whole instance set can be selected per submitter. This is meant to avoid anyone to have too much influence on the competition results.
The classification obtained last year was kept (thanks again to Johannes Fichte). New instances submitted to PB25 were classified by the organizer according to the name of the directories.
The goal was to obtain between 300 and 600 instances in the DEC-LIN and OPT-LIN tracks. Given the number of domains, 10 instances were randomly selected per domain in all tracks, except in the DEC-LIN track where 15 instances per domain were selected. If a domain doesn't contain enough instances, they are all selected. Duplicate instances are removed. This test is performed by checking the md5 checksum of instances. This means that the check is purely syntactical. It doesn't identify that an instance where contraints are reordered or variables renamed can be the same as another instance.
The 5% limit per submitter was not hit except in the OPT-NLC track where few instances were available last year. In order to respect the selection limit, only 3 of the fresh instances were selected. Those instances will be more useful next year because the 5% limit won't be applied anymore.
The archive PB25-selection.tgz contains the source files used to perform the selection. The program was run on a system with glibc 2.40-26 (Fedora 41). The random seed used is 2025. The archive contains:
- runSelection: this is the script that runs the selection process in all categories. Running this script will generate *-selection.list files (as well as *-selection.sql files, which are used for registering the selection in the database). The *-selection.list files contain the list of selected instances.
- {dec-lin,dec-nlc,opt-lin,opt-nlc,wbo}.list: these are the list of available instances, grouped by domains, with the number of instances to select per domain
- md5.list: fingerprint of each intance
- select.cc: the program performing the random selection. It is derived from the one available on https://www.cril.univ-artois.fr/XCSP19/selection/. It adds support for removal of duplicates (based on the md5 fingerprint). This is the same program as last year, with the same defautlts: it was written in a rush and therefore is not well documented, not efficient and doesn't forgive any mistake in the input.
- savedSelection/: contains the selection actually used for the competition. Running the selection script should yield the same files as in this directory.
Selecting instances is a difficult subject. The selection made is not perfect and some mistakes were probably made. However, it is hopefully clear that the selection was automated, mainly random and largely independent of the submitted solvers.
Hardware used
The competition was run on the CRIL cluster. Two kind of nodes were available:
- 30 nodes bi-CPU Intel Xeon E5-2637 v4 3.5Ghz 4 cores, 128 GB RAM
- 5 nodes quad-CPU Intel Xeon Gold 6248 2.5Ghz 20 cores, 768 GB RAM
Resources limits
- Sequential solvers were run on the bi-CPU hosts and were allowed to use at most 1 h of CPU time, or at most 3700 s of wall clock time. They were allowed to use 31000 MiB of memory. Four instances of the same solver were run simultaneously on a host (2 solvers per CPU).
- Parallel solvers were run on the quad-CPU hosts and were allowed to use at most 20 h of CPU time, or at most 3700 s of wall clock time. They were allowed to use 195000 MiB of memory. Four instances of the same solver were run simultaneously on a host.
- veripb+cakepb was initially run with a 5 h CPU time limit (with a maximum of 18300 s of wall clock time) and a 48750 MiB memory limit. The jobs that exceeded those limits were later restarted with a 10 h CPU time limit (with a maximum of 36600 s of wall clock time) and a 95000 MiB memory limit. The output of VeriPB was sent directly to CakePB through a pipe (actually a FIFO, a.k.a. named pipes). The two process were running simultaneously.