> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Azure AI Speech

> Learn how to monitor Azure AI Speech with Openlayer

![Azure AI Speech hero](https://mintcdn.com/openlayer-44/4cu8TDdOg1L6h2C1/images/integrations/azure_speech_hero.png?fit=max\&auto=format\&n=4cu8TDdOg1L6h2C1\&q=85\&s=2c4a863b9c881cb0ff09d74a0380c7ef)

Openlayer integrates with [Azure AI Speech](https://learn.microsoft.com/en-us/azure/ai-services/speech-service/overview),
Microsoft's service for speech-to-text, speech translation, and text-to-speech. Each
recognition, translation, or synthesis call made with the Azure Speech SDK becomes a step
in your Openlayer trace, with the recognized text or synthesized-audio statistics, latency,
the result reason, and failure details. You can optionally attach the audio itself.

> [!WARNING]
>
> Trace Azure AI Speech on your server only. Keep your Azure Speech key and your
> Openlayer API key in server-side configuration and out of browser or mobile
> code. For browser apps, issue short-lived Speech tokens from your backend and
> use `SpeechConfig.fromAuthorizationToken` in the browser.

## Monitoring Azure AI Speech

To use [monitoring mode](https://docs.openlayer.com/monitoring/overview.md), instrument your code to publish the Speech
calls your AI system makes to the Openlayer platform.

### Installation

Install Openlayer and the Azure Speech SDK:

```bash Python theme={null}
pip install openlayer azure-cognitiveservices-speech
```

```bash TypeScript theme={null}
npm install openlayer microsoft-cognitiveservices-speech-sdk
```

> [!NOTE]
>
> Azure AI Speech tracing requires `openlayer>=0.34.0` in Python and
> `openlayer>=0.32.0` in TypeScript. In TypeScript, the Azure Speech SDK is an
> optional peer dependency, so installing `openlayer` alone does not add it.

Set your credentials and inference pipeline in the environment:

```bash theme={null}
OPENLAYER_API_KEY=YOUR_OPENLAYER_API_KEY
OPENLAYER_INFERENCE_PIPELINE_ID=YOUR_OPENLAYER_INFERENCE_PIPELINE_ID
AZURE_SPEECH_KEY=YOUR_AZURE_SPEECH_KEY
AZURE_SPEECH_REGION=YOUR_AZURE_SPEECH_REGION
```

### Trace Speech calls

```python Python theme={null}
import os

import azure.cognitiveservices.speech as speechsdk

from openlayer.lib import init

# Auto-instruments every Speech client created afterwards
init()

speech_config = speechsdk.SpeechConfig(
    subscription=os.environ["AZURE_SPEECH_KEY"],
    region=os.environ["AZURE_SPEECH_REGION"],
)
speech_config.speech_recognition_language = "en-US"
speech_config.speech_synthesis_voice_name = "en-US-JennyNeural"

synthesizer = speechsdk.SpeechSynthesizer(
    speech_config=speech_config,
    audio_config=speechsdk.audio.AudioOutputConfig(filename="greeting.wav"),
)
synthesizer.speak_text("Hello! How can I help you today?")

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=speechsdk.audio.AudioConfig(filename="greeting.wav"),
)
result = recognizer.recognize_once()
print(result.text)
```

```typescript TypeScript theme={null}
import * as sdk from "microsoft-cognitiveservices-speech-sdk";
import { traceAzureSpeech } from "openlayer/lib/integrations/azureSpeechTracer";

const speechConfig = sdk.SpeechConfig.fromSubscription(
  process.env.AZURE_SPEECH_KEY ?? "",
  process.env.AZURE_SPEECH_REGION ?? "",
);
speechConfig.speechRecognitionLanguage = "en-US";
speechConfig.speechSynthesisVoiceName = "en-US-JennyNeural";

// Wrap each recognizer or synthesizer; the client is patched in place
const synthesizer = traceAzureSpeech(
  new sdk.SpeechSynthesizer(
    speechConfig,
    sdk.AudioConfig.fromAudioFileOutput("greeting.wav"),
  ),
);

// The Speech SDK is callback-based: wrap calls in a Promise to await them
const result = await new Promise<sdk.SpeechSynthesisResult>((resolve, reject) =>
  synthesizer.speakTextAsync(
    "Hello! How can I help you today?",
    resolve,
    reject,
  ),
);
synthesizer.close();
```

In Python, `init()` instruments every `SpeechRecognizer`, `TranslationRecognizer`, and
`SpeechSynthesizer` created afterwards. To trace a single client instead, pass it to
`trace_azure_speech(client)` from `openlayer.lib`. In TypeScript, wrap each client with
`traceAzureSpeech()`.

If your resource uses a custom endpoint (for example, an Azure AI Services
`*.cognitiveservices.azure.com` domain), create the configuration with
`SpeechConfig(subscription=..., endpoint=...)` in Python or
`SpeechConfig.fromEndpoint(new URL(endpoint), key)` in TypeScript. Tracing works the same
way.

### Trace a full voice turn

Wrap the function that handles a conversational turn so recognition and synthesis appear
as steps of the same trace:

```python Python theme={null}
from openlayer.lib import trace


@trace()
def voice_turn(audio_path: str) -> str:
    recognizer = speechsdk.SpeechRecognizer(
        speech_config=speech_config,
        audio_config=speechsdk.audio.AudioConfig(filename=audio_path),
    )
    heard = recognizer.recognize_once().text

    reply = f"You said: {heard}"
    speechsdk.SpeechSynthesizer(
        speech_config=speech_config, audio_config=None
    ).speak_text(reply)
    return reply


voice_turn("caller.wav")
```

```typescript TypeScript theme={null}
import * as fs from "fs";

import trace from "openlayer/lib/tracing/tracer";

function recognize(audioPath: string): Promise<sdk.SpeechRecognitionResult> {
  const recognizer = traceAzureSpeech(
    new sdk.SpeechRecognizer(
      speechConfig,
      sdk.AudioConfig.fromWavFileInput(fs.readFileSync(audioPath)),
    ),
  );
  return new Promise<sdk.SpeechRecognitionResult>((resolve, reject) =>
    recognizer.recognizeOnceAsync(resolve, reject),
  ).finally(() => recognizer.close());
}

function speak(text: string): Promise<sdk.SpeechSynthesisResult> {
  const replySynthesizer = traceAzureSpeech(
    new sdk.SpeechSynthesizer(speechConfig, null as any),
  );
  return new Promise<sdk.SpeechSynthesisResult>((resolve, reject) =>
    replySynthesizer.speakTextAsync(text, resolve, reject),
  ).finally(() => replySynthesizer.close());
}

const voiceTurn = trace(async function voiceTurn(
  audioPath: string,
): Promise<string> {
  const heard = await recognize(audioPath);
  const reply = `You said: ${heard.text}`;
  await speak(reply);
  return reply;
});

await voiceTurn("caller.wav");
```

> [card] [See full Python example](https://colab.research.google.com/github/openlayer-ai/openlayer-python/blob/main/examples/tracing/azure-speech/azure_speech_tracing.ipynb)

> [card] [See full TypeScript example](https://github.com/openlayer-ai/openlayer-ts/blob/main/examples/azure-speech-tracing.ts)

## Attach audio to traces

Audio is **not** sent to Openlayer by default. When you enable attachment uploads,
synthesized audio is attached to each synthesis step automatically. For recognition, pass
the audio you are transcribing explicitly, because the Speech SDK's `AudioConfig` does not
expose its source:

```python Python theme={null}
from openlayer.lib import init

init(attachment_upload_enabled=True)

recognizer = speechsdk.SpeechRecognizer(
    speech_config=speech_config,
    audio_config=speechsdk.audio.AudioConfig(filename="caller.wav"),
)
# `openlayer_audio` accepts a file path, bytes, or an Attachment
result = recognizer.recognize_once(openlayer_audio="caller.wav")
```

```typescript TypeScript theme={null}
import * as fs from "fs";

import { configure } from "openlayer/lib/tracing/tracer";

configure({ attachmentUploadEnabled: true });

const audio = fs.readFileSync("caller.wav");
// `inputAudio` accepts bytes, a file path, or an Attachment. It is attached
// to every recognition step of this recognizer.
const recognizer = traceAzureSpeech(
  new sdk.SpeechRecognizer(
    speechConfig,
    sdk.AudioConfig.fromWavFileInput(audio),
  ),
  { inputAudio: audio },
);
```

The audio is uploaded to Openlayer storage when the trace completes and appears as an audio
player on the step: recognition audio under **Inputs**, synthesized audio under **Output**.
A file path is read as bytes, so your local path is never recorded.

Synthesized audio is labeled by its actual encoding. MP3, Ogg, WebM, and WAV output keep
their format. Headerless PCM, µ-law, and A-law output (the `raw-*` formats) is wrapped in a
WAV container so it can be played. Formats without a playable container, such as raw Opus
frames, are attached as files that you can download.

> [!WARNING]
>
> Uploading audio sends that data to Openlayer. Enable it only when your privacy
> and security requirements allow it. See [Trace multimodal
> data](https://docs.openlayer.com/monitoring/multimodal.md) for how attachments are stored and displayed.

## Data captured

Speech calls appear on the **Data** page of your Openlayer data source as steps named
**Azure Speech Recognition**, **Azure Speech Translation**, or **Azure Speech Synthesis**,
with the `Azure_Speech` provider.

| Field | Recognition and translation | Synthesis |
| - | - | - |
| Inputs | Recognition language, translation target languages, optional audio | The text or SSML that was synthesized |
| Output | Recognized text, and translations for translation | Audio duration and size, and the audio when uploads are on |
| Model | Custom endpoint ID, or `speech-to-text` | The voice name, such as `en-US-Ava:DragonHDLatestNeural` |
| Model parameters | Non-secret settings such as region, language, and endpoint ID | Non-secret settings such as region, voice, and output format |
| Metadata | Result ID, reason, offset and duration, no-match or cancellation info | Result ID, reason, and cancellation info |

Azure reports most failures as canceled results rather than errors, so failed calls are
traced too, with the cancellation reason, error code, and error details.

Your Azure subscription key and authorization token are never recorded. Openlayer reads
only an allowlist of non-secret settings, and it redacts credentials from error details:
the Speech SDK can include your endpoint URL, with any token in its query string, in
connection errors.

Speech steps report 0 tokens, so their cost shows as \$0.

## Supported calls

| Client | Python | TypeScript |
| - | - | - |
| `SpeechRecognizer` and `TranslationRecognizer` | `recognize_once`, `recognize_once_async` | `recognizeOnceAsync` |
| `SpeechSynthesizer` | `speak_text`, `speak_ssml`, `speak_text_async`, `speak_ssml_async` | `speakTextAsync`, `speakSsmlAsync` |

Continuous recognition, streaming synthesis requests, and batch transcription are not
traced.

## Troubleshooting

> [accordion] No Speech steps appear in Openlayer
>
> In Python, call `init()` before creating your Speech clients, or pass each
> client to `trace_azure_speech()`. In TypeScript, wrap every recognizer and
> synthesizer with `traceAzureSpeech()`. Also check that `OPENLAYER_API_KEY`
> and `OPENLAYER_INFERENCE_PIPELINE_ID` are set.

> [accordion] Recognition and synthesis show up as separate traces
>
> Wrap the function that makes both calls with `@trace()` in Python or
> `trace()` in TypeScript. In TypeScript, `await` each Speech call inside that
> function so its step is recorded under the right parent.

> [accordion] The trace has no audio player
>
> Enable attachment uploads (`attachment_upload_enabled=True` in Python,
> `attachmentUploadEnabled: true` in TypeScript) before the trace is created.
> Recognition steps also need the audio passed explicitly, as
> `openlayer_audio` in Python or `inputAudio` in TypeScript.

> [accordion] The Azure Speech SDK cannot be resolved in TypeScript
>
> Install `microsoft-cognitiveservices-speech-sdk`. It is an optional peer
> dependency, so installing `openlayer` alone does not add it to your
> application.

> [accordion] Canceled and NoMatch results appear in my traces
>
> This is expected. Azure returns canceled and `NoMatch` outcomes as results,
> and Openlayer traces them so you can investigate the reason and latency.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.