Guide to using Azure AI Speech for speech recognition, text to speech, and building voice enabled agents with SDK examples, playgrounds, and Foundry integration
Welcome to this lesson on getting started with speech in Azure. This guide covers the fundamentals of Azure AI Speech: recognizing spoken language (speech-to-text), converting text into natural-sounding audio (text-to-speech), and building voice-capable AI agents that communicate with users in real time.We begin with speech recognition.Speech recognition enables an application to listen to spoken words and convert them into text the computer can process. Microsoft Foundry exposes three primary ways to work with Azure AI Speech recognition:
Approach
When to use it
Example
Playground
Fast experimentation and quality checks without writing code
Upload a recorded support call to evaluate transcription quality
Speech SDK
Integrate speech into production applications (files, streams, microphones)
A note-taking app that transcribes meeting audio in real time
Agent tool
Connect speech to an agent via an MCP server for conversational flows
Voice-first support agent that receives live audio and responds with speech
All three use the same Azure AI Speech recognition engine—Foundry supplies the tooling and UIs to try, configure, and integrate the service.Transcription configuration options
When transcribing audio, configure recognition to match your scenario. The most common options are:
Option
Purpose
Language selection
Specify the expected language (e.g., English (United States)) so the appropriate language model is used
Language identification
Enable detection when incoming audio may be in multiple languages or unknown
Speaker diarization
Tag different speakers in multi-person audio (meetings, interviews)
Output format
Choose between plain text or richer structured results with timestamps, confidence scores, and speaker tags
The diagram below illustrates a transcription UI with these controls and a sample transcript, showing how audio becomes text.
Key UI controls to notice: language selection, speaker diarization, language identification, and output format.Transcribing audio with the Speech SDK (Python)
Below is a concise example that uses the Azure Speech SDK for Python to transcribe a single audio file. It demonstrates the minimal steps required to authenticate, point to an audio source, and run a recognition request.
Import the Azure Speech SDK to access the SDK classes.
Create a SpeechConfig with subscription and endpoint so the SDK can authenticate and target the correct service.
Create an AudioConfig pointing to the audio source (here, record.wav).
Construct a SpeechRecognizer that ties together speech and audio configurations.
Call recognize_once_async() to perform a single recognition and read the result from result.text.
Continuous recognition (microphone)
For interactive or streaming scenarios—playgrounds, live transcription, and voice agents—use continuous recognition and event handlers. The sample below shows a typical continuous-recognition setup using the default system microphone.
# Continuous recognition (microphone) exampleimport osimport azure.cognitiveservices.speech as speechsdkendpoint_url = os.environ.get("AZURE_SPEECH_ENDPOINT", "https://your-foundry-endpoint/")speech_key = os.environ.get("AZURE_SPEECH_KEY", "your_api_key")speech_config = speechsdk.SpeechConfig(subscription=speech_key, endpoint=endpoint_url)# Use the default system microphoneaudio_config = speechsdk.audio.AudioConfig(use_default_microphone=True)speech_recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config)# Event handlers (handlers receive sender and event arguments)def recognized_handler(sender, evt): # evt.result contains the final recognized text for a segment print(f"Recognized: {evt.result.text}")def recognizing_handler(sender, evt): # evt.result contains intermediate partial results print(f"Recognizing: {evt.result.text}")speech_recognizer.recognized.connect(recognized_handler)speech_recognizer.recognizing.connect(recognizing_handler)speech_recognizer.start_continuous_recognition()print("Say something...")try: input("Press Enter to stop...\n")finally: speech_recognizer.stop_continuous_recognition()
Store keys and endpoints securely (for example, in environment variables or a secret store) and never check them into source control.
Speech synthesis (text-to-speech)
Speech synthesis converts text into spoken audio. You can use the Playground to experiment with voices, speaking rate, pitch, and SSML, or integrate synthesis into an app with the Speech SDK. You can also connect synthesis to an agent so model responses are spoken back to users.The UI below shows a speech synthesis demo: voice selection, tuning sliders (pitch, rate, volume), and a preview panel for the output text “Hello, welcome to Microsoft Foundry!”
Important synthesis controls:
Voice selection: choose neural voices with different accents and speaking styles.
Speaking rate, pitch, volume: tune delivery without changing the text.
Output format and sample rate: pick the audio format appropriate for playback or storage.
SSML (Speech Synthesis Markup Language): control pauses, emphasis, pronunciation, and speaking style.
A simple Python example that plays synthesized speech to the system default audio output:
# Text-to-speech example (play to default speaker)import azure.cognitiveservices.speech as speechsdkspeech_config = speechsdk.SpeechConfig(subscription="your_speech_key", endpoint="https://your-foundry-endpoint/")# Choose a neural voice (example name)speech_config.speech_synthesis_voice_name = "en-US-AvaNeural"# No AudioConfig required for default speaker output; provide one for file output insteadspeech_synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)result = speech_synthesizer.speak_text_async("My name is John").get()if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted: print("Speech synthesized successfully.")else: print(f"Speech synthesis failed: {result.reason}")
Combining recognition and synthesis with generative AI
By combining speech recognition, a generative AI model, and speech synthesis, you can create a continuous voice-enabled agent:
User speaks (live audio delivered to the agent).
Speech-to-text converts audio into text.
A generative AI model reasons and produces a text response (optionally using tools or external knowledge).
The response text is converted to speech and played back.
The loop continues for a natural conversational experience.
This flow supports multi-turn conversations so users and agents can speak back and forth.Creating speech-capable agents in Foundry
Microsoft Foundry enables you to create and configure voice-first agents. The “Let’s talk” start session experience shows a simple entry point for a speech session.
Speech services can be used inside Foundry or as standalone Azure resources. From a Foundry resource in the Azure portal you can find service endpoints and keys for speech-to-text, text-to-speech, translation, and other speech APIs.
Inside Foundry you can view deployments and services (for example, deployed models and speech services). The Deployments dashboard shows deployed models such as gpt-4.1-mini.
The Foundry Playgrounds provide an accessible way to experiment with speech-to-text, text-to-speech, translation, and other speech tools—supporting real-time transcription, language detection, speaker diarization, post-refinement, and drag-and-drop audio.
The playground also displays code snippets (for example, live streaming and continuous recognition) that you can adapt into your application code.Text-to-speech in the playground supports SSML and expressive features—whispering, shouting, sighs, and emotions—so you can craft nuanced spoken responses. The screenshot below reiterates the playground’s combined transcription and audio controls.
Creating a voice-first agent
To build a real-time speech agent in Foundry, go to Agents and create a new agent, selecting a voice-first experience. Agent configuration options include auto language detection, microphone settings, and attaching tools such as a speech MCP server or Blob Storage.
You can add tools from the Foundry catalog (for example, Azure Speech MCP Server or Azure Text to Speech) when your agent needs external services. The tools catalog shows available speech tools and other integrations.
Once the agent is running, users can have live spoken dialogs: the agent listens, transcribes, reasons with a model, and replies with synthesized speech. The example below shows a travel assistant voice session that responds to spoken requests with travel recommendations.
As sessions run, Foundry captures transcripts and retains conversational context so the agent can continue multi-turn dialogs—delivering a natural, interactive voice experience.Security and privacy considerations
Audio can contain personal or sensitive information. Ensure you follow your organization’s compliance and data retention policies, mask or redact PII where necessary, and configure logging and retention in line with regulations.
Next steps and references
If you want to experiment: use the Foundry Playgrounds for quick prototyping and to see sample code.
For integration: use the Speech SDK to add recognition and synthesis into applications (desktop, web, mobile).
For agents: create voice-first agents in Foundry and attach speech tools from the catalog.
With these concepts, code samples, and UI examples you can start experimenting with speech in Azure: try the Playground for exploration, use the Speech SDK for application integration, and connect speech to agents for real-time voice experiences.