> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tests configuration

> Learn how to configure tests in Openlayer

Each Openlayer test can be tailored to your needs. You can choose the data to run on,
adjust thresholds, tweak parameters, and more.

Most options have sensible defaults, so setup is usually minimal.
This page walks you through all available configurations.

![Test configuration](https://openlayer-static-assets.s3.us-west-2.amazonaws.com/images/test_configuration.gif)

## Configurations

The configurations available are:

> [accordion] Data
>
> The "**Data**" section allows you to select the data on top of which this test will run.
>
> In [monitoring mode](https://docs.openlayer.com/monitoring/overview.md), it means choosing among your data
> sources (e.g., production data, staging data, etc.) In
> [development mode](https://docs.openlayer.com/development/overview.md), it means selecting among your datasets
> (e.g., training, validation, etc.)
>
> You can also apply filters to narrow the test to specific slices of the data, and
> turn on **Apply to new data sources** so the test covers data sources you connect
> later. When you fill in a filter, the value picker suggests values from your data —
> see [Filter value suggestions](#filter-value-suggestions).
>
> To test or filter on one key inside a structured input or output instead of the
> whole column, define a [custom column](https://docs.openlayer.com/monitoring/custom-columns.md) for it.
>
> For supported tests in monitoring mode, **Evaluate** controls whether the test
> evaluates each matching trace as one row or evaluates each matching step as its own
> row. **Whole trace** is the default. See
> [Evaluating whole traces or individual steps](#evaluating-whole-traces-or-individual-steps)
> for eligibility, filtering, and result behavior.
>
> In monitoring mode, the section also holds the **Evaluation windows** settings: the
> **Cadence** the test runs on (**Scheduled** or **Manual**), plus the
> **Evaluation delay**, **Include historical data in each run**, and **Anchor
> evaluation windows** toggles. See
> [Evaluation and delay windows](https://docs.openlayer.com/monitoring/evaluation-and-delay-windows.md) for details.

> [accordion] Parameters
>
> Some tests include configurable parameters that control their behavior.
> For example, a test such as the [PII detection test](https://docs.openlayer.com/tests/catalog/contains-p-i-i.md)
> lets you specify which type of PII to detect (e.g., credit card numbers, social security numbers, etc.)
>
> [Custom metrics](https://docs.openlayer.com/tests/custom-metrics.md) can also declare their own configurable
> parameters. See [Configurable parameters](https://docs.openlayer.com/tests/custom-metrics.md#configurable-parameters)
> for details on how to define them.
>
> Refer to the specific test page in the [Catalog](https://docs.openlayer.com/tests/overview.md) for details
> on its parameters.
>
> For [LLM-as-a-judge](https://docs.openlayer.com/tests/catalog/l-l-m-rubric-threshold.md) tests, the
> **Judge settings** hold the judge model and the sampling controls that bound how much
> data the judge scores: **Sample size (%)** (100 by default), **Data limit (rows)**
> (50 by default), and the **Sampling method** (for example, Random). You can set the
> same controls for every test in the project under **Settings** →
> **LLM-as-a-judge**.

> [accordion] Threshold
>
> The "**Threshold**" section allows you to define the condition for the test to pass.
>
> On the **Manual** tab, you pick a **Measurement**, an **Operator**, and a
> **Threshold** value — saying, for example, that the test passes if the metric is
> greater than 0.5.
>
> In [monitoring mode](https://docs.openlayer.com/monitoring/overview.md), most tests also offer an **Auto** tab.
> With automatic thresholds, Openlayer automatically learns the distribution of
> historical data and predicts upper and lower bounds for the metric. The test fails if
> the observed value falls outside the predicted range.
>
> Automatic thresholds are directional: choose **Both bounds** to fail on either side of
> the predicted range, or **Upper only** or **Lower only** to fail on one side only —
> useful when a move in a single direction is what concerns you, such as a rising cost.

> [accordion] Advanced settings
>
> The "**Advanced settings**" section holds the test metadata. Here's where you set
> the test's **Priority**.
>
> Cadence, delays, and anchoring live in the "**Data**" section — see
> [Evaluation and delay windows](https://docs.openlayer.com/monitoring/evaluation-and-delay-windows.md).

## Evaluating whole traces or individual steps

In monitoring mode, supported tests include an **Evaluate** setting:

* **Whole trace** evaluates each matching trace as one row. This is the default.
* **Each step** evaluates every span that matches the test's step filters as its own row.
  Each span receives its own score and explanation, and the test result aggregates those
  span-level results.

When you select **Each step**, filters are grouped into **Step** fields, including
**Step name** and **Step type**, and **Parent trace** fields. Use a step filter to narrow
which steps are evaluated; Openlayer displays a hint when no step filter does so. For tests
that require a column, you can evaluate the step's **Output** or full **Input**. Semantic
filters are not supported for step-scoped tests.

You choose the evaluation scope when you create the test and cannot change it later. To use
a different scope, create another test. Openlayer also rejects step scope in development
mode or for an unsupported test or metric. For **Metric** tests, the available scope follows
the selected measurement; when you edit a step-scoped Metric test, only measurements that
can evaluate steps are available.

### Supported tests and measurements

You can use **Each step** with Character length, String validation, Is JSON, JSON schema
validation, Is code, Contains valid URL, Special characters ratio for LLM projects,
Contains PII, Contains PHI, Prompt injection, LLM-as-a-judge, and Bias tests. RAGAS tests
support step scope only for metrics judged from the step's input and output.

For **Metric** tests, the following measurements support step scope:

* Latency: minimum, maximum, mean, median, p90, p95, and p99.
* Cost: maximum, mean, and total.
* Tokens: maximum, mean, and total.
* Correctness, harmfulness, coherence, conciseness, maliciousness, and answer relevancy.

Cost and token measurements evaluate only spans that record usage. Metrics that require
trace context or a ground truth remain whole-trace only: faithfulness, hallucination,
context utilization, context relevancy, context recall, answer correctness, exact match,
quasi-exact match, edit distance, semantic similarity, BLEU 1–4, and JSON score. The same
restriction applies to RAGAS metrics that require context or a ground truth.

### Reading step test results

The test's **Properties** panel shows **Evaluates: Each step** or **Evaluates: Whole trace**.
For a step-scoped test, the **Data** tab lists one span per row with its score, explanation,
and failure highlighting. You can filter these rows by **Step ID**. Opening a row selects
that step in the trace view.

In the trace view, the **Scores** panel lists the results for the selected record or step.
Results are labeled with the test, and explanations are available on hover. A results icon
in the trace tree identifies steps with scores.

## Filter value suggestions

When you filter a test on a column, the value picker suggests values Openlayer has already
seen in that column, so you can select a value instead of typing it from memory. Every
filter value picker can show these suggestions, not just
[semantic search filters](https://docs.openlayer.com/tests/semantic-search-filters.md), and the kind of comparison you
are making determines which values it suggests — whether you are matching on equality,
membership, contains, or an ordering comparison. The same suggestions appear when you
filter monitoring records: the **Records** table's filter prompts you to "Enter value...",
then lists matching values under "In the last 30 days" with each value's count beside it.

Each suggested value carries a count tag. Hover it to see how many rows matched, as in
"12 matches found in recent data", together with the note "Suggestions are drawn from a
recent sample of rows, so the full dataset may contain more matches." Read the count as a
signal of whether a value is worth filtering on rather than as an exact total.

Columns with many distinct values do not list suggestions up front. The picker asks you to
"Type to search values" instead, and looks up matching values as you type.

## Pausing tests

In [monitoring mode](https://docs.openlayer.com/monitoring/overview.md), you can pause tests when you don't want them to
compute results — for example, while you are replaying data or waiting on an upstream fix.
Open the overflow menu on a test row and select **Pause**.

While tests are paused, the tests list shows the banner "Tests are paused. You can still create
tests, but they will not compute results." Select **Unpause** in the banner to start receiving
test results again.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.