> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Getting Started With Computer Vision in Azure
> Guide to Azure computer vision with multimodal models, text-to-image and text-to-video generation, Foundry model catalog, APIs, and developer-focused examples.
Now let's get started with computer vision in Azure.
This lesson explains how Azure enables applications to understand and work with visual information (images and video). We'll cover multimodal models (image + text), text-to-image and text-to-video generation, Foundry's model catalog and playground, and practical examples you can use in code.
## What is a multimodal model?
A multimodal model accepts more than one input modality (for example, images and text) and reasons across them. In practice this means you can send an image alongside a natural-language prompt in a single request, and the model will analyze both together.
Example: provide a photo of a banana and ask, “What desserts can I make with this?” The model recognizes the banana in the image and uses that visual context to generate text responses such as “banana ice cream,” “banana bread,” or “banana pancakes.”
You can supply images as URLs or as Base64-encoded data. Base64 converts binary image data into text so it can be embedded in an API request.
Azure provides multiple options for visual AI:
| Service | Use case | Link |
| - | - | - |
| Azure OpenAI | Multimodal deployments that accept images and text together | [https://learn.microsoft.com/en-us/azure/cognitive-services/openai/](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/) |
| Microsoft Foundry | Catalog and deployment environment where you can discover and test multimodal and generative models | [https://learn.microsoft.com/en-us/azure/ai-services/foundry/overview](https://learn.microsoft.com/en-us/azure/ai-services/foundry/overview) |
| Azure AI Vision | Specialized, task-specific computer vision capabilities (image analysis, OCR, document parsing) | [https://learn.microsoft.com/en-us/azure/ai-services/vision/overview](https://learn.microsoft.com/en-us/azure/ai-services/vision/overview) |
Key distinction: multimodal generative models reason about images and follow language instructions, while specialized vision services provide focused, task-specific APIs (for example, OCR or invoice parsing).
## Developer perspective: multimodal request example
In production you'll typically call the model directly from application code rather than using a web playground. A common pattern:
* Load a local image in binary mode.
* Base64-encode the image and embed it in a data URL that identifies the image MIME type.
* Call the Responses API (or an equivalent multimodal endpoint) and include both the input text and input image in the same user message.
* Parse and use the model's textual output.
Example (Python):
```python theme={null}
import base64
from openai import OpenAI
client = OpenAI(
base_url="foundry-endpoint",
api_key="api_key"
)
with open("./banana.png", "rb") as image_file:
image = base64.b64encode(image_file.read()).decode("utf-8")
response = client.responses.create(
model="model-deployment-name",
input=[{
"role": "user",
"content": [
{"type": "input_text", "text": "What desserts can I make with this?"},
{"type": "input_image", "image_url": f"data:image/png;base64,{image}"}
]
}]
)
# Print the first text output from the model
print(response.output[0].content[0].text)
```
Notes:
* Open image files with `"rb"` (read-binary) because images are binary data.
* In production, never hard-code API keys or endpoints; use environment variables, a secure secrets store, or Azure-managed identities.
Store API keys and endpoints in environment variables or a secure secrets store. Avoid committing credentials to source control.
Conceptual workflow:
* Load the image → encode the image → send prompt and image together → let the multimodal model analyze both → read the generated response.
Using a multimodal model often removes the need to build a separate image-recognition pipeline because the model reasons across visual and textual context directly.
## Text-to-image generation
Text-to-image generation produces a new image from a natural-language prompt (for example, “a hiker on a hill at sunrise”). Generative image models synthesize imagery based on learned patterns instead of searching for an existing photo.
Tips for better image generation prompts:
* Include time of day, camera angle, lighting, and art style if you want a specific look.
* Short prompts give broader, more creative results; longer prompts guide specificity.
## Text-to-video generation
Text-to-video extends text-to-image concepts to multiple frames. A text-to-video model generates short clips from a prompt, maintaining temporal consistency (motion, lighting changes, and camera perspective).
Video generation is more computationally intensive and requires models that can maintain coherence across frames.
## Foundry model catalog and the playground
In Microsoft Foundry you can explore models by inference task (image analysis, image classification, image-to-text, text-to-image, and video generation). Foundry lists available models and providers (MAI, FLUX, Mistral, Microsoft-hosted models) and includes a playground for testing: upload images, enter prompts, and try tasks such as OCR, invoice parsing, and visual question answering.
### Example: multimodal interaction in the playground
Deploy a multimodal model (for example, GPT-4o) and open it in the Foundry playground. The playground supports attaching images to chat inputs and asking questions like “How many dogs are in this image?” or “Which breed is that?” The model can reply with an answer and an uncertainty estimate (for example, “I can’t be 100% certain from the image alone, but it looks like a Golden Retriever”).
## Image generation in the Foundry playground and API
Foundry supports deploying image-generation models (subject to quota and regional availability). The playground shows example cURL or SDK calls that you can copy to run generation programmatically.
Example (bash/cURL):
```bash theme={null}
curl -X POST "https://aifoundry-ai901-01.services.ai.azure.com/mai/v1/images/generations" \
-H "Content-Type: application/json" \
-H "api-key: $AZURE_API_KEY" \
-d '{
"prompt": "A photograph of a red fox in an autumn forest",
"width": 1024,
"height": 1024,
"model": "MAI-Image-2.5-Flash"
}' | jq -r '.data[0].b64_json' | base64 --decode > generated_image.png
```
Replace `https://aifoundry-ai901-01.services.ai.azure.com` and `$AZURE_API_KEY` with your Foundry endpoint and API key. The command decodes the Base64 payload and writes `generated_image.png`.
## Video generation availability and quotas
Video and advanced image generation models usually require GPU-backed deployments (for example, NVIDIA A100 or H100) and are resource-intensive. Smaller subscriptions or demo accounts may lack GPU quota in certain regions. If you encounter quota limits:
* Try deploying in a different region.
* Request a quota increase via your Azure subscription.
* Use smaller models or lower-resolution outputs during development to minimize quota usage.
The general process is consistent: deploy a model → open the playground → provide natural-language instructions → retrieve generated media (image or video). Foundry enables running models programmatically and integrating them into applications.
## Links and references
* [Azure OpenAI Service documentation](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/)
* [Microsoft Foundry overview](https://learn.microsoft.com/en-us/azure/ai-services/foundry/overview)
* [Azure AI Vision overview](https://learn.microsoft.com/en-us/azure/ai-services/vision/overview)
* [Responses API guide](https://platform.openai.com/docs/guides/gpt/responses-api)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.