# Conformance Reports

## How this folder is structured

This folder stores all the conformance reports, for all the Gateway API Versions
and is structured as follows:

```text
|-- conformance/reports
|   |-- v1.0
|   |   |-- acme-operator
|   |   |   |-- README.md
|   |   |   |-- standard-v2.13-default-report.yaml
|   |-- v1.1
|   |   |-- acme-operator
|   |   |   |-- README.md
|   |   |   |-- standard-v2.14-default-report.yaml
|   |   |   |-- standard-v2.14-with-the-lot-report.yaml
|   |   |   |-- experimental-v2.14-with-the-lot-report.yaml
|   |   |-- umbrella-operator
|   |   |   |-- README.md
|   |   |   |-- standard-v1.8-default-report.yaml
```

## Rules

As represented above, each Gateway API version contains a set of folders, one for
each conformant implementation. The implementation is the owner of its folder and
can upload as many reports as it want and a mandatory README.md structured as follows:

```md
# Acme operator

General information about the Acme/operator project

## Table of contents

| API channel | Implementation version | Mode | Report |
|-------------|------------------------|------|--------|
|             |                        |      |        |
|             |                        |      |        |
|             |                        |      |        |

## Reproduce

Instructions on how to reproduce the claimed report.
```

### Table of Contents

The table of contents is supposed to contain one row for each submitted report and
is structured as follows:

- **API channel**: the channel of the Gateway API (standard or experimental). It
  MUST correspond to the `gatewayAPIChannel` field of the related report.
- **Implementation version**: the link to the GitHub/website page related to the
  release. The release value MUST always be a semver and correspond to the
  `implementation.version` field of the report.
- **Mode**: the operating mode of the implementation (the default is `standard`).
  It MUST correspond to the `mode` field specified in the related report.
  In case a mode different from `default` is declared in a report, the implementation
  needs to add information in the "reproduce" section (see below) on how to configure
  such a mode.
- **Report**: the link to the related report. It MUST be in the form of
  `[_version_ report](./report.yaml)`. The reports must be named according to the following
  pattern: `<API Channel>-<Implementation version>-<mode>-report.yaml`.

### Reproduce

The "Reproduce" sections MUST exist and contain the manual or automatic steps
to reproduce the results claimed by the uploaded conformance reports. In case
different implementation versions have different reproduction steps, this section
can have multiple sub-sections, each related to a specific or a subset of implementation
versions.

## Rules exceptions

This structure was introduced after the Gateway API release v1.0.0, while we started
gathering and storing conformance reports since Gateway API v0.7.1. Even if this
structure has been applied to all the Gateway API versions, it has been done in
a best-effort manner, without forcing the rules described above retroactively.
This is inevitable also because the `mode` and `gatewayAPIChannel` fields of the
reports have been introduced after v1.0.0. For these reasons, all the folders related
to Gateway API v1.0.0 or prior benefits from the following rules' exceptions:

- The fields `API channel` and `Mode` of the README.md's Table of Contents are replaced
  by an "x".
- The reports are named according to the pattern `<Implementation version>-report.yaml`.
- The implementation version can be different from a semver (for example, they can
  be commit hashes or pull request links).
- The "Reproduce" section is optional; implementations can decide whether to provide
  reproduction steps for those versions.

## Reports

The reports MUST be uploaded exactly as they have been created by the conformance
suite, without any modifications. The "Reproduce" section allows checking
any diff between the claimed report and the actual one.

### Reports rules

To be accepted, the reports uploaded **after** the Gateway API v1.1.0 release must
comply with the following rules:

- All the `implementation`-related fields must have a meaningful value assigned,
  in detail:
  - `organization`: can be an open source organization, an individual, a company,
    etc. Organizations can have multiple projects.
  - `project`: the name of the project. It must be unique within the same organization.
  - `contact`: it indicates the GitHub usernames of those who are responsible for
    maintaining this file, so they can be easily contacted when needed (e.g. for
    relevant release announcements regarding conformance, etc.). Optionally, it
    can be an email address or a support URL (e.g. GitHub new issue page).
  - `url`: it must be a valid url for a GitHub repository, or any other website with
    information related to the project.
  - `version`: it must be a snapshot of the project, which means it can be a commit,
    a tag, a release, or any other means of specifying a version. It can't be a
    branch name.
- `gatewayAPIVersion`: it must contain the version of the Gateway API used to run
  the conformance tests.
- `gatewayAPIChannel`: it must contain the channel of the Gateway API used to run
  the conformance tests (standard/experimental).
- `mode`: it is the mode of the implementation used to run the tests. If different
  from `default`, it must correspond to a specific setup of the implementation.
- Test result: in order to have a report valid to be accepted, all the profiles
  need to have the `result` fields (core and extended) set to `success` or `partial`.
  `partial` reports will be displayed separately from the `success` ones in the
  [implementations table][implementations-table].

  A `success` report implies **all** the following:
  - all the core conformance tests have been successfully run
  - all the tests related to the supported extended features have been successfully
    run.
  
  A `partial` report implies **at least one** of the following:
  - some core conformance tests have been skipped
  - some tests related to the supported extended features have been skipped
  - the conformance test run required some steps unexpected by the suite.

[implementations-table]: https://gateway-api.sigs.k8s.io/implementations/v1.1/
