> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# tests.json

> Learn how to write the `tests.json` config for your project

The `tests.json` file allows you to specify your Openlayer tests.

This allows you to fully control your project's tests with a single configuration file that
can be version-controlled, without needing to use the Openlayer UI to create and update tests.

This guide shows how you can write the `tests.json` for your project.

## Discoverability

First, it is important to signal to Openlayer that you have a test configuration file.

You can do so in the [testsPath](https://docs.openlayer.com/development/openlayer-json.md#testspath) field of your [openlayer.json](https://docs.openlayer.com/development/openlayer-json.md).
Make sure that you point to your `tests.json` file, as in:

```json openlayer.json theme={null}
{
  "testsPath": "tests.json",
  ...
}
```

## Structure

The `tests.json` file should contain an `array` of `Test` objects, each representing
a test in your project.

The `Test` objects have a set of **common attributes** and a set of attributes that
**depend on the `mode`** (i.e., if it is a development test or monitoring test).

To start from tests you authored in the UI, you can
[export one or more test configurations from the Results page](https://docs.openlayer.com/tests/manage-tests.md#export-test-configurations).
The downloaded `tests.json` contains the ready-to-use array, so you can check it
into your repository and use it with `openlayer push`.

You can also push the output of
[`openlayer tests --output tests.json`](https://docs.openlayer.com/api-reference/cli/commands/tests) as is. The
CLI wraps the tests in an object, `{"_meta": ..., "items": [...]}`, and Openlayer
reads the tests from its `items` array.

> [!TIP]
>
> The best way to write a test configuration is to **copy the examples** from
> the [documentation](https://docs.openlayer.com/tests/overview.md) and edit them to fit your use case. Each
> test has a documentation page with information about it and configuration
> examples. They are all listed [here](https://docs.openlayer.com/tests/overview.md).

### `Test` object common attributes

Common attributes apply regardless of the `mode`. Attributes marked as required must
always be present.

#### `name`

**Type**: `string`, required

The test name.

#### `description`

**Type**: `string`.

The test description.

#### `type`

**Type**: `string`, required.

The test type, which represents the test category. Must be one of `integrity`, `consistency`, or `performance`.

#### `subtype`

**Type**: `string`, required.

The test subtype, which identifies the test on the platform. Must be one of the valid `subtypes`.

#### `thresholds`

**Type**: `array` of `Threshold` objects, required.

The thresholds that

> [accordion] Threshold object attributes
>
> #### `insightName`
>
> **Type**: `string`, required.
>
> Name of the insight from which the test is based. Must be one of the valid insight names.
>
> ***
>
> #### `measurement`
>
> **Type**: `string`, required.
>
> Key from the `insightName` on top of which the threshold will be applied. Must be one of the valid measurement names.
>
> ***
>
> #### `operator`
>
> **Type**: `string`, required.
>
> Operator used to compare the `measurement` and `value`. Must be one of `is`, `>`, `>=`, `<`, `<=`.
>
> ***
>
> #### `value`
>
> **Type**: `number | string`, required.
>
> Threshold value.
>
> ***
>
> #### `insightParameters`
>
> **Type**: `object`.
>
> Parameters needed to compute the insight. Might be `null` depending on the insight.

#### `subpopulationFilters`

**Type**: `object`.

Filters that define the subpopulation.

#### `tags`

**Type**: `array` of `string`.

The names of the tags the test belongs to — the same tags you can add to a test in the
platform. Any name your project doesn't have yet is created on push, so you don't have
to create the tags in the UI first:

```json tests.json theme={null}
[
  {
    "name": "Mean quasi-exact match above 0.7",
    "tags": ["regression", "release-gate"],
    ...
  }
]
```

Tags are matched by exact name, so `Regression` and `regression` are two different tags.

How a push treats a test's tags depends on whether the field is present at all:

| In `tests.json` | Result |
| - | - |
| No `tags` field | The test keeps whatever tags it has on the platform. |
| `"tags": ["a", "b"]` | The test is tagged `a` and `b`, and nothing else. |
| `"tags": []` | The test's tags are removed. |

Because the field is only read when it's present, you can tag some tests from
`tests.json` and leave others to be tagged in the UI.

> [!TIP]
>
> [`openlayer tests`](https://docs.openlayer.com/api-reference/cli/commands/tests) writes the `tags` field
> for every test it exports, so you can start from the tests you already have on
> the platform and edit the tag names from there.

#### `syncId`

**Type**: `string`, required.

An id (UUID) that identifies the test. It is stable across commits — Openlayer uses it to
decide whether a push updates an existing test or creates a new one — so it is also the
handle you use to remove tests with
[`openlayer tests delete --sync-id`](https://docs.openlayer.com/api-reference/cli/commands/tests#openlayer-tests-delete).

#### `mode`

**Type**: `string`, required.

Defines to which mode the test applies to. Must be one of `development` or `monitoring`.

#### `evaluationScope`

**Type**: `string | null`.

Controls whether a monitoring test evaluates each trace or each matching step. Allowed
values are `trace`, `step`, and `null`. Omit the field, set it to `null`, or set it to
`trace` to evaluate whole traces. Set it to `step` to evaluate every matching step as its
own row and aggregate the step-level results.

Step scope is available only in monitoring mode and only for supported test subtypes and
measurements. It does not support semantic filters. Use `subpopulationFilters` on
**Step name** or **Step type** to narrow the steps to evaluate. You cannot change a test's
evaluation scope after creation.

Exports include `"evaluationScope": "step"` for step-scoped tests. For eligibility and
result behavior, see
[Evaluating whole traces or individual steps](https://docs.openlayer.com/tests/test-configuration.md#evaluating-whole-traces-or-individual-steps).

### `Test` object mode-specific attributes

The additional attributes you must specify for a test depend on its `mode`, i.e., if it is a
`development` or `monitoring` mode test.

> [tab] development
>
> #### `usesValidationDataset`
>
> **Type**: `bool`, required
>
> Indicates if the test uses the validation dataset.
>
> #### `usesTrainingDataset`
>
> **Type**: `bool`, required
>
> Indicates if the test uses the training dataset.
>
> #### `usesMlModel`
>
> **Type**: `bool`, required
>
> Indicates if the test uses model.

> [tab] monitoring
>
> #### `usesProductionData`
>
> **Type**: `bool`, required
>
> Must be `true` for monitoring tests. A test with `mode` set to `monitoring` but without
> `usesProductionData: true` fails validation.
>
> #### `evaluationWindow`
>
> **Type**: `integer`, required
>
> The test [evaluation window](https://docs.openlayer.com/monitoring/evaluation-and-delay-windows.md#evaluation-windows), in
> seconds. Must be a whole number of hours (a multiple of `3600`) between `3600` and `2592000`
> (30 days). For example, `86400` evaluates the last day of data.
>
> #### `delayWindow`
>
> **Type**: `integer`, required, default `0`.
>
> The test [delay window](https://docs.openlayer.com/monitoring/evaluation-and-delay-windows.md#delay-windows), in seconds.
> Must be a multiple of `3600`, up to `2592000` (30 days).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.