Getting Started With Computer Vision in Azure

Now let’s get started with computer vision in Azure. This lesson explains how Azure enables applications to understand and work with visual information (images and video). We’ll cover multimodal models (image + text), text-to-image and text-to-video generation, Foundry’s model catalog and playground, and practical examples you can use in code.

What is a multimodal model?

A multimodal model accepts more than one input modality (for example, images and text) and reasons across them. In practice this means you can send an image alongside a natural-language prompt in a single request, and the model will analyze both together. Example: provide a photo of a banana and ask, “What desserts can I make with this?” The model recognizes the banana in the image and uses that visual context to generate text responses such as “banana ice cream,” “banana bread,” or “banana pancakes.” You can supply images as URLs or as Base64-encoded data. Base64 converts binary image data into text so it can be embedded in an API request. Azure provides multiple options for visual AI:

Service	Use case	Link
Azure OpenAI	Multimodal deployments that accept images and text together	https://learn.microsoft.com/en-us/azure/cognitive-services/openai/
Microsoft Foundry	Catalog and deployment environment where you can discover and test multimodal and generative models	https://learn.microsoft.com/en-us/azure/ai-services/foundry/overview
Azure AI Vision	Specialized, task-specific computer vision capabilities (image analysis, OCR, document parsing)	https://learn.microsoft.com/en-us/azure/ai-services/vision/overview

Key distinction: multimodal generative models reason about images and follow language instructions, while specialized vision services provide focused, task-specific APIs (for example, OCR or invoice parsing).

An infographic titled "Multimodal Models for Image Analysis" that illustrates a workflow: an image plus a text prompt are sent in one API call to produce a text answer. It also lists features and platforms like Azure OpenAI, Microsoft Foundry, and Azure AI Vision.

Developer perspective: multimodal request example

In production you’ll typically call the model directly from application code rather than using a web playground. A common pattern:

Load a local image in binary mode.
Base64-encode the image and embed it in a data URL that identifies the image MIME type.
Call the Responses API (or an equivalent multimodal endpoint) and include both the input text and input image in the same user message.
Parse and use the model’s textual output.

Example (Python):

import base64
from openai import OpenAI

client = OpenAI(
    base_url="foundry-endpoint",
    api_key="api_key"
)

with open("./banana.png", "rb") as image_file:
    image = base64.b64encode(image_file.read()).decode("utf-8")

response = client.responses.create(
    model="model-deployment-name",
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "What desserts can I make with this?"},
            {"type": "input_image", "image_url": f"data:image/png;base64,{image}"}
        ]
    }]
)

# Print the first text output from the model
print(response.output[0].content[0].text)

Notes:

Open image files with "rb" (read-binary) because images are binary data.
In production, never hard-code API keys or endpoints; use environment variables, a secure secrets store, or Azure-managed identities.

Store API keys and endpoints in environment variables or a secure secrets store. Avoid committing credentials to source control.

Conceptual workflow:

Load the image → encode the image → send prompt and image together → let the multimodal model analyze both → read the generated response.

Using a multimodal model often removes the need to build a separate image-recognition pipeline because the model reasons across visual and textual context directly.

Text-to-image generation

Text-to-image generation produces a new image from a natural-language prompt (for example, “a hiker on a hill at sunrise”). Generative image models synthesize imagery based on learned patterns instead of searching for an existing photo. Tips for better image generation prompts:

Include time of day, camera angle, lighting, and art style if you want a specific look.
Short prompts give broader, more creative results; longer prompts guide specificity.

An infographic illustrating text-to-image generation: a text prompt ("A hiker on a hill at sunrise") goes into a Foundry model and outputs a simple illustration of two blue hills/mountains with a sun.

Text-to-video generation

Text-to-video extends text-to-image concepts to multiple frames. A text-to-video model generates short clips from a prompt, maintaining temporal consistency (motion, lighting changes, and camera perspective). Video generation is more computationally intensive and requires models that can maintain coherence across frames.

A slide illustrating a text-to-video pipeline: a text prompt ("A hiker on a hill at sunrise") is fed into a Foundry model which outputs a short filmstrip-style video showing stylized hills and a rising sun.

Foundry model catalog and the playground

In Microsoft Foundry you can explore models by inference task (image analysis, image classification, image-to-text, text-to-image, and video generation). Foundry lists available models and providers (MAI, FLUX, Mistral, Microsoft-hosted models) and includes a playground for testing: upload images, enter prompts, and try tasks such as OCR, invoice parsing, and visual question answering.

A screenshot of the Microsoft Foundry "Content Understanding" web playground showing a drag-and-drop area and a list of prebuilt analyzers (OCR/Read, Invoice, Receipt, Contract, etc.). The left sidebar shows navigation items like Agents, Deployments, Services, and Tools.

Example: multimodal interaction in the playground

Deploy a multimodal model (for example, GPT-4o) and open it in the Foundry playground. The playground supports attaching images to chat inputs and asking questions like “How many dogs are in this image?” or “Which breed is that?” The model can reply with an answer and an uncertainty estimate (for example, “I can’t be 100% certain from the image alone, but it looks like a Golden Retriever”).

A screenshot of the Microsoft Foundry/Azure AI "gpt-5.4-mini" playground and deployments interface. The chat panel shows the model replying that the attached image contains three dogs, likely Golden Retrievers.

Image generation in the Foundry playground and API

Foundry supports deploying image-generation models (subject to quota and regional availability). The playground shows example cURL or SDK calls that you can copy to run generation programmatically. Example (bash/cURL):

curl -X POST "https://aifoundry-ai901-01.services.ai.azure.com/mai/v1/images/generations" \
  -H "Content-Type: application/json" \
  -H "api-key: $AZURE_API_KEY" \
  -d '{
    "prompt": "A photograph of a red fox in an autumn forest",
    "width": 1024,
    "height": 1024,
    "model": "MAI-Image-2.5-Flash"
  }' | jq -r '.data[0].b64_json' | base64 --decode > generated_image.png

Replace https://aifoundry-ai901-01.services.ai.azure.com and $AZURE_API_KEY with your Foundry endpoint and API key. The command decodes the Base64 payload and writes generated_image.png.

A screenshot of a web dashboard (Microsoft Foundry / Azure AI) showing a grid of available AI model tiles and providers. A left sidebar shows filters and deployment options.

Video generation availability and quotas

Video and advanced image generation models usually require GPU-backed deployments (for example, NVIDIA A100 or H100) and are resource-intensive. Smaller subscriptions or demo accounts may lack GPU quota in certain regions. If you encounter quota limits:

Try deploying in a different region.
Request a quota increase via your Azure subscription.
Use smaller models or lower-resolution outputs during development to minimize quota usage.

The general process is consistent: deploy a model → open the playground → provide natural-language instructions → retrieve generated media (image or video). Foundry enables running models programmatically and integrating them into applications.

AI in Azure

Getting Started With AI

Getting Started With Generative AI and Agents

Getting Started With Text Analysis

Getting Started With AI Speech