Reproducibility of experimental results is crucial to foster an atmosphere of trustworthy, open, and reusable research. To improve and reward reproducibility, FormaliSE 2026 continues its Artifact Evaluation (AE) procedure. An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally makes them fully reproducible.
Submission of an artifact is optional but encouraged for all papers where it can support the results presented in the paper. Artifact review is single-anonymous (the paper corresponding to an artifact must still follow the double-anonymous submissions requirements) and will be conducted concurrently with the paper reviewing process. Artifacts will be handled by a separate Artifact Evaluation Committee, and the Artifact Evaluation process will be set up such that the anonymization of the corresponding papers will not be compromised. Accepted papers with a successfully evaluated artefact will be awarded the EAPLS Artifact Badges (https://eapls.org/eapls/artifact-badges/) that apply (among “Functional”, “Reusable”, and “Available”). Awarded badges are to be added to the camera-ready version of the paper.
Artifacts will be assessed with respect to their consistency with the results presented in the paper, their completeness, their documentation, and their ease of use. The Artifact Evaluation will include an initial check for technical issues; authors of artifacts may be contacted by email within the first two weeks after artifact submission to help resolve any technical problems that prevent the evaluation of an artifact, if necessary.
The results of an artifact evaluation will not be available to the reviewers of the corresponding paper; hence, they will not affect the paper’s acceptance decision. However, reviewers will know whether a paper has submitted any artifacts; this piece of information may be taken into account to decide whether the paper should be accepted. Thus, if there are justifiable reasons why a paper’s artifacts cannot be submitted, they should be pointed out in the paper so that the reviewers can appreciate them and adjust their expectations accordingly.
Call for Contributions
The reproducibility of experimental results is crucial to foster an atmosphere of trustworthy, open, and reusable research. To improve and reward reproducibility, FormaliSE 2027 continues its Artifact Evaluation (AE) procedure. A main goal of the artifact evaluation is to enable future researchers to more effectively build on and compare with previous work.
An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally makes them fully reproducible. Submission of an artifact is optional but encouraged for all papers where it can support the results presented in the paper.
Artifact review is single-blind (the paper corresponding to an artifact must still follow the double-blind submissions requirements) and will be conducted concurrently with the paper reviewing process. Artifacts will be reviewed by a separate Artifact Evaluation Committee (AEC). We will be attributing 3 badges, according to the EAPLS guidelines:
- Artifact functional: documented, consistent, complete, exercisable;
- Artifact reusable: exceeding functional, by being carefully documented and well-structured for reuse and repurposing, see below for details;
- Artifact available: available on a publicly accessible archival repository for permanent availability that provides a Digital Object Identifier (DOI).
Artifacts will be assessed with respect to their consistency with the results presented in the paper, their completeness, their documentation, and their ease of use.
The AEC will include an initial check for technical issues. Authors of artifacts will interact with the reviewers from dates to be announced, to help resolve any technical problems that prevent the evaluation of an artifact, if necessary.
Artifact submission guidelines
Submissions must be made via the following submission link: TBA
When the paper is submitted, authors should indicate that they plan to submit an artifact by registering via the above submission site.
A final artifact submission should consist of:
- an abstract that summarizes the artifact and explains its relation to the paper including:
- a URL from which a .zip file containing the artifact can be downloaded – we encourage you to provide a DOI – and
- if applicable, a description of any special requirements beyond a VM image (e.g., cloud-computing resources, certain software and hardware, including the CPU architecture and operating systems, number of cores, etc.), and,
- if you are aiming for a reusable badge, an explanation why you believe your artifact is reusable, and
- detailed specific instructions for an early light review that allows reviewers to: (1) verify that the artifact can properly run; and (2) perform a short evaluation of the artifact before the full evaluation and detect any difficulties, and
- the sha256 checksum of the zip file (to ensure consistency), which can be generated with:
- Linux: sha256sum
- Windows: CertUtil -hashfile
SHA256 - macOS: shasum -a 256
- Linux: sha256sum
- a .pdf file of the submitted paper.
Your artifact .zip file must contain the following elements.
-
The artifact, i.e., data, software, libraries, scripts, etc. required to replicate the results of your paper. Please prepare a Docker image. We discourage the use of a Virtual Machine (VM) image because supporting both Intel/AMD64 and Apple Silicon architectures is tricky. That said, in the absence of a Docker image, we will accept a VM image with a clear indication of the host machine’s architecture.
-
A license file. The license needs to allow the artifact evaluation chairs to download and distribute the artifact to the artifact evaluation committee members and the artifact evaluation committee members must be allowed to evaluate the artifact, e.g., use, execute, and modify the artifact for the purpose of artifact evaluation.
-
A readme text file that introduces the artifact to the user and guides the user through replication of your results. Ideally, it should consist of the following parts:
- Any additional requirements for running the artifact, such as hardware requirements, including CPU architecture, or additional proprietary software;
- The expected total runtime to run the experiments;
- Detailed and specific reproducibility instructions to setup and use the artifact to replicate the results in the paper; including an explanation which claims and results cannot be replicated and why.
If you are not in a position to prepare the artifact as above, please contact AEC chairs for an alternative arrangement before the submission deadline. For instance, if you cannot provide us with a VM that contains licensed software, e.g., MATLAB, please contact us, so we can find a solution.
Members of the AEC are asked to use submitted artifacts and papers for the sole purpose of evaluating the contribution associated with the artifact.
Evaluation Criteria
All artifacts are evaluated by the AEC. Each artifact will be reviewed by multiple committee members. Reviewers will read the paper and explore the artifact to evaluate how well the artifact supports the claims and results of the paper.
Criteria for the “functional” badge
The evaluation and the awarding of the functional badge is based on the following questions:
- Is the artifact documented, i.e., at minimum, an inventory of artifacts is included, and sufficient description to enable the artifacts to be exercised is included.
- Is the artifact consistent, i.e., relevant to the associated paper, significantly contributing to the generation of its main results?
- Is the artifact complete, i.e., rand as far as possible, are all components relevant to the associated paper included?
- Is the artifact runnable, i.e., can the software/scripts that generates the results in the associated paper be executed successfully, and can included data be accessed and appropriately manipulated?
Criteria for the “available” badge
To get the available badge, please upload your VM or Docker image to a permanent repository that provides a DOI, such as Zenodo, figshare, or Dryad, and use the DOI link provided in your artifact submission.
Additional criteria for the “reusable” badge
Artifacts seeking the “reusable” badge need to clear a significantly higher bar than functional artifacts. First, they must be available, i.e., receive an “available” badge. Second, we expect a higher level of quality during the evaluation of the functional level. Third, in addition to the criteria from the functional level, they are evaluated against the following criteria:
- Does the artifact have a license which allows reuse, repurposing, and which is easy to use?
- Are all dependencies and used libraries well documented and up to date?
- Does the artifact README explain in sufficient detail how the artifact can be used beyond the paper?
- Does the artifact provide documented interfaces for extensions, or is the artifact open source?
- Can the artifact be used in a different environment, e.g., built on another system, used outside of the VM image, etc.?
Suggestions for preparing the artifact
In the case of a Docker image, we recommend building images for multiple platforms, in particular for linux/amd64 and linux/arm64. The details of multi-platform builds are available, for instance, at https://docs.docker.com/build/building/multi-platform/.
We recommend that you compress the resulting Docker images to reduce the overall size of your artifact. Please use a compression that is compatible with Docker load, i.e., either gz or xz.
If it is not possible to build images for multiple platforms, for instance, due to some libraries, the required CPU architecture should be clarified in the abstract.
In the case of a VM image, please use VirtualBox and save the VM image as an Open Virtual Appliance (OVA) file. Since VirtualBox does not support cross-architecture VMs (as of Autumn 2025), the required CPU architectures should be clarified in the abstract if this option is chosen.
Make it simple for AEC members to exercise the artifact and reproduce the results of the paper via easy-to-use scripts and detailed instructions. When writing step-by-step instructions, assume the minimum expertise of users.
The artifact should run out of the box and not require the user to install any additional software. All required packages, where possible, should already be provided in the Docker (or VM) image.
For experiments that require a large amount of resources (time, memory, number of cores, etc.), we recommend indicating a subset of the results of the paper that can be reproduced with reasonably modest resources (w.r.t. RAM, number of cores, etc.) and in a reasonable amount of time. Please also include the full set of experiments (for reviewers with sufficient hardware or time), just make it optional.
In case the artifact cannot comply with some of the guidelines, please contact the AEC chairs before the AEC submission deadline. A common example is artifacts requiring restrictively licensed software such as MATLAB.