AI Agents on Azure for AI-103: Responses API, Azure AI Agent Service, Function Calling, Code Interpreter, File Search, Agent Memory, Multi-Agent Orchestration, Semantic Kernel, Evaluation, and Building a Data Engineering Agent

Leave a Comment / AI/ML, Azure

Table of Contents

In the previous post, we built RAG pipelines that retrieve documents and generate grounded answers. But RAG answers questions — it does not take actions. An AI agent goes further: it reasons about what to do, uses tools to gather information or perform tasks, remembers previous interactions, and orchestrates multi-step workflows autonomously. This is the most forward-looking part of AI-103.

Analogy — A skilled assistant vs an encyclopedia. RAG is like having an encyclopedia that you can ask questions — it looks up information and tells you what it found. An agent is like having a skilled assistant: you say “check if our daily pipeline ran successfully, and if it failed, look up the error in the troubleshooting guide and create a Jira ticket.” The assistant reasons (check status → conditional logic → lookup → action), uses tools (monitoring API, knowledge base, Jira API), and remembers context across the conversation. The agent thinks, acts, and follows through.

What Is an AI Agent

An AI agent is a system that:
  1. REASONS about a task (understands what needs to be done)
  2. PLANS a sequence of steps (decides the approach)
  3. USES TOOLS to gather information or take actions
  4. OBSERVES results and adapts (iterates if needed)
  5. REMEMBERS context across interactions (conversation history)

  Agent = LLM + Tools + Memory + Reasoning Loop

The agent loop (ReAct pattern):
  1. User: "What was the revenue for Ontario in Q3?"
  2. Agent THINKS: "I need to query the database for Ontario Q3 revenue"
  3. Agent ACTS: calls query_database tool with SQL query
  4. Agent OBSERVES: tool returns $2.4M
  5. Agent THINKS: "I have the answer, let me format it"
  6. Agent RESPONDS: "Ontario revenue in Q3 was $2.4M"

  For complex tasks, the loop repeats:
  THINK → ACT → OBSERVE → THINK → ACT → OBSERVE → RESPOND

Agents vs RAG vs simple chat:
  Simple chat: question → model answers from training knowledge
  RAG: question → retrieve documents → model answers from documents
  Agent: question → model reasons → calls tools → observes results → responds

  Each level adds capability:
    Chat: knows general facts
    RAG: knows YOUR documents
    Agent: knows YOUR documents AND can take actions

The Responses API — The Foundation for Agents

The Responses API is Azure OpenAI’s latest API for building agents. It consolidates retrieval, reasoning, and action execution into a single API call.

Responses API vs Chat Completions API:

  Chat Completions (what we used for prompt engineering):
    - Stateless: each request is independent
    - No built-in tools (you implement function calling yourself)
    - No memory (you manage conversation history)
    - Best for: simple Q&A, RAG, one-shot tasks

  Responses API:
    - Stateful: manages conversation threads automatically
    - Built-in tools: web search, file search, code interpreter, functions
    - Built-in memory: thread-based conversation history
    - Multi-agent: native support for subagent delegation
    - Best for: complex tasks requiring tools, memory, and multi-step reasoning
# Simple agent using the Responses API
from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(),
    "https://ai.azure.com/.default"
)

client = AzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/openai/v1/",
    api_key=token_provider,
    default_query={"api-version": "preview"}
)

# Create a response with tool access
response = client.responses.create(
    model="gpt-4o",
    instructions="You are a data engineering assistant. Use tools when needed.",
    input="What is the current status of our daily pipeline?",
    tools=[
        {
            "type": "function",
            "function": {
                "name": "check_pipeline_status",
                "description": "Check the status of a data pipeline by name",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "pipeline_name": {
                            "type": "string",
                            "description": "Name of the pipeline to check"
                        }
                    },
                    "required": ["pipeline_name"]
                }
            }
        }
    ]
)

Azure AI Agent Service — Managed Agent Platform

Azure AI Agent Service is the managed platform in Foundry for hosting, scaling, and securing agents.

What Agent Service provides:
  - Managed hosting for agents (no infrastructure to manage)
  - Built-in tools: file search, code interpreter, Bing grounding
  - Function calling for custom tools
  - Thread management (conversation memory)
  - Multi-agent orchestration (agent-to-agent delegation)
  - Security: Entra ID, managed identity, Content Safety
  - Monitoring: Application Insights tracing
  - Scaling: automatic based on demand

Agent Service vs building from scratch:
  From scratch: you manage tool execution, memory, thread state, scaling
  Agent Service: platform manages everything, you define the agent behavior

Creating an agent in Agent Service:
  1. Foundry portal → Agents
  2. Configure:
     - Model: GPT-4o
     - Instructions: system prompt for the agent
     - Tools: select built-in + define custom functions
  3. Test in the Agents playground
  4. Deploy as endpoint

Tools — Giving Agents Capabilities

Tools are the extensions that let agents interact with the world beyond text generation.

Built-in tools in Azure AI Agent Service:

  1. FILE SEARCH
     Search across uploaded documents
     RAG built into the agent (no separate AI Search setup needed)
     Upload files → agent searches them when answering questions
     Best for: Q&A over a document library

  2. CODE INTERPRETER
     Executes Python code in a sandboxed environment
     Reads and writes files (CSV, JSON, Excel)
     Creates charts and visualizations
     Performs calculations, data analysis
     Best for: data analysis, chart generation, file processing

  3. BING GROUNDING
     Searches the web for current information
     Grounds responses in live web data
     Best for: questions about current events, real-time data

  4. FUNCTION CALLING (custom)
     Define your own tool functions
     Agent decides when to call them based on the conversation
     Your code executes the function and returns results
     Best for: database queries, API calls, system actions

  5. AZURE AI SEARCH
     Connect to your AI Search index for RAG
     Agent retrieves and uses indexed documents
     Best for: enterprise RAG with custom indexes

  6. AZURE FUNCTIONS (via MCP)
     Call serverless functions as tools
     Model Context Protocol (MCP) for standardized tool access
     Best for: complex integrations, behind-VNet services

Function Calling — Custom Tool Integration

Function calling lets agents invoke YOUR code. The agent decides when a function is needed, and your application executes it.

Analogy — A doctor ordering lab tests. The doctor (agent) talks to the patient (user), decides a blood test is needed (function call), writes the order (function name + parameters), the lab runs the test (your code executes), and results come back (function return value). The doctor then interprets the results and advises the patient. The doctor does not run the tests — they delegate to specialists (your functions).

# Define a tool for the agent
tools = [
    {
        "type": "function",
        "function": {
            "name": "query_pipeline_status",
            "description": "Query the status of a data pipeline run",
            "parameters": {
                "type": "object",
                "properties": {
                    "pipeline_name": {
                        "type": "string",
                        "description": "Name of the pipeline"
                    },
                    "date": {
                        "type": "string",
                        "description": "Date to check (YYYY-MM-DD format)"
                    }
                },
                "required": ["pipeline_name"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_error_details",
            "description": "Get error details for a failed pipeline run",
            "parameters": {
                "type": "object",
                "properties": {
                    "run_id": {
                        "type": "string",
                        "description": "The pipeline run ID"
                    }
                },
                "required": ["run_id"]
            }
        }
    }
]

# The agent loop:
# 1. Agent receives user question
# 2. Agent decides to call query_pipeline_status
# 3. YOUR CODE executes the function (calls ADF API, Databricks API, etc.)
# 4. You return the result to the agent
# 5. Agent may call get_error_details if the pipeline failed
# 6. Agent formulates a response using all gathered information
Function calling flow:

  User: "Did the daily sales pipeline run successfully today?"

  Agent THINKS: I need to check the pipeline status
  Agent CALLS: query_pipeline_status(pipeline_name="daily_sales", date="2026-08-20")

  YOUR CODE runs:
    def query_pipeline_status(pipeline_name, date):
        # Call ADF REST API or Databricks API
        status = adf_client.pipeline_runs.query(...)
        return {"status": "Failed", "run_id": "run-123", "error": "Timeout"}

  Agent OBSERVES: Pipeline failed with timeout error
  Agent THINKS: I should get more details about the error
  Agent CALLS: get_error_details(run_id="run-123")

  YOUR CODE runs:
    def get_error_details(run_id):
        details = adf_client.pipeline_runs.get(run_id)
        return {"error": "Connection timeout to SQL Server", "activity": "CopyData"}

  Agent RESPONDS: "The daily sales pipeline failed today with a connection
                   timeout to SQL Server during the CopyData activity.
                   This typically indicates a network issue or the SQL Server
                   is overloaded. Would you like me to create an incident ticket?"

Code Interpreter — Running Code on the Fly

Code Interpreter executes Python code in a sandboxed environment.
The agent writes code, runs it, and uses the results.

What it can do:
  - Read and analyze CSV, JSON, Excel files
  - Perform calculations (statistics, aggregations)
  - Create charts (matplotlib, pandas plotting)
  - Process data (clean, transform, pivot)
  - Generate files (CSV exports, reports)

Example:
  User: "Analyze the attached sales data and create a monthly revenue chart."

  Agent writes and executes Python code:
    import pandas as pd
    import matplotlib.pyplot as plt

    df = pd.read_csv('/uploaded/sales.csv')
    monthly = df.groupby('month')['revenue'].sum()
    monthly.plot(kind='bar', title='Monthly Revenue')
    plt.savefig('/output/chart.png')

  Agent returns: the chart image + a summary of findings

Limitations:
  - Sandboxed: no internet access, no external API calls
  - Limited libraries: pandas, numpy, matplotlib, etc. (not all pip packages)
  - Session-based: files are available only during the session
  - Timeout: long-running code may be terminated

For AI-103:
  Know that Code Interpreter runs Python in a sandbox
  Know it can read files, perform analysis, and create charts
  Know it cannot access the internet or call external APIs
  Know it is best for data analysis and file processing tasks

File Search — Searching Document Libraries

File Search is a built-in RAG capability for agents.
Upload documents → agent searches them automatically.

How it works:
  1. Upload files to a vector store (part of Agent Service)
  2. Files are automatically chunked and embedded
  3. When the user asks a question, the agent searches the files
  4. Relevant chunks are included in the agent's context
  5. Agent answers grounded in the uploaded documents

File Search vs Azure AI Search:
  File Search (built-in):
    - Quick setup: upload files, done
    - Limited scale: designed for smaller document sets
    - No custom indexing pipeline
    - Good for: agent-specific document libraries (10-100 files)

  Azure AI Search (external):
    - Full search engine: hybrid, vector, semantic ranking
    - Enterprise scale: millions of documents
    - Custom skillsets, indexers, integrated vectorization
    - Good for: enterprise RAG across the organization

  Use File Search for: small, agent-specific document sets
  Use AI Search for: large, shared enterprise knowledge bases

Agent Memory and Threads

Agents maintain memory through THREADS -- conversation histories
that persist across multiple interactions.

Thread = a conversation session with the agent
  - Contains all messages (user + agent + tool results)
  - Persists across API calls (agent remembers previous messages)
  - Can be resumed later (continue a conversation)
  - Managed by Agent Service (you don't manage the history)

How threads work:
  1. Create a thread (new conversation)
  2. Add a message (user's question)
  3. Create a run (agent processes the message)
  4. Agent responds (may call tools during the run)
  5. Add another message (follow-up question)
  6. Agent remembers the full conversation

Memory scope:
  Within a thread: agent remembers everything
  Across threads: agent has no memory (each thread is independent)
  To share context across threads: use persistent storage (database, file)

Thread management:
  - Threads have a token limit (context window)
  - Long conversations may exceed the limit
  - Agent Service handles truncation automatically
  - Critical information should be in the system instructions, not just memory

For AI-103:
  Know that threads provide session-based memory
  Know that cross-thread memory requires external storage
  Know the token limit implications for long conversations

Multi-Agent Orchestration Patterns

Multi-agent orchestration uses multiple specialized agents working together on complex tasks.

Analogy — A hospital with specialist departments. Instead of one doctor handling everything (one agent), you have a triage nurse (router agent) who directs patients to the right specialist: cardiologist (database agent), radiologist (document agent), or surgeon (action agent). Each specialist has deep expertise in their domain and the right tools. The triage nurse coordinates the overall care plan.

Five orchestration patterns:

  1. SEQUENTIAL
     Agent A → Agent B → Agent C (pipeline)
     Each agent processes output from the previous one
     Example: Extract Agent → Transform Agent → Load Agent

  2. CONCURRENT (PARALLEL)
     Agents A, B, C run simultaneously
     Results combined by an orchestrator
     Example: Three agents evaluate three different proposals in parallel

  3. HANDOFF
     Agent A handles a request until it needs to transfer to Agent B
     Like transferring a phone call to a specialist
     Example: General agent → transfers to billing agent when topic is billing

  4. GROUP CHAT
     Multiple agents discuss and collaborate in a shared conversation
     Each agent contributes from their expertise
     Example: Data architect + security analyst + cost optimizer review a design

  5. MAGENTIC-ONE
     Microsoft's advanced multi-agent pattern (stable release 2026)
     Orchestrator agent dynamically assigns tasks to specialist agents
     Agents can request help from other agents
     Most flexible but most complex pattern
# Multi-agent with Responses API (concurrent subagents)
response = client.responses.create(
    model="gpt-4o",
    input="""Evaluate three disaster-recovery proposals:
      Alpha: active-active, RTO < 5min, $42K/month
      Beta: warm standby, RTO 30min, $18K/month
      Gamma: backup-restore, RTO 8hr, $6K/month
      Our requirement: RTO ≤ 30min, RPO ≤ 5min, budget ≤ $20K/month.""",
    multi_agent={
        "enabled": True,
        "max_concurrent_subagents": 3
    }
)
# Three subagents evaluate Alpha, Beta, Gamma in parallel
# Root agent consolidates findings and recommends Beta

Semantic Kernel and AutoGen — Framework Integration

Azure AI Agent Service integrates with two open-source frameworks:

  SEMANTIC KERNEL (Microsoft):
    - C# and Python SDK for building AI applications
    - Plugin system for tools and functions
    - Planner for multi-step task execution
    - Memory connectors for persistent storage
    - Integrates with Azure AI Agent Service
    - Best for: enterprise applications with structured workflows

  AUTOGEN (Microsoft Research):
    - Python framework for multi-agent conversations
    - Agents communicate via messages
    - Supports human-in-the-loop
    - Flexible orchestration patterns
    - Best for: research, prototyping, complex multi-agent scenarios

  Both frameworks support:
    - Sequential, concurrent, handoff, group chat patterns
    - Custom tool definitions
    - Azure AI model integration
    - Application Insights tracing

For AI-103:
  Know that Semantic Kernel and AutoGen are the supported frameworks
  Know that they integrate with Azure AI Agent Service
  Know the basic difference (SK = enterprise, AutoGen = multi-agent research)
  You do NOT need to code in these frameworks for the exam

Evaluating Agent Quality

Agent evaluation is more complex than RAG evaluation because
agents take ACTIONS, not just generate text.

Evaluation dimensions:

  1. TASK COMPLETION
     Did the agent accomplish the user's goal?
     Metric: success rate across test scenarios

  2. TOOL USE ACCURACY
     Did the agent call the right tools with correct parameters?
     Metric: tool call precision (correct calls / total calls)

  3. GROUNDEDNESS
     Are the agent's responses grounded in tool results and documents?
     Metric: groundedness score (same as RAG evaluation)

  4. SAFETY
     Does the agent follow Content Safety rules?
     Does it refuse inappropriate requests?
     Metric: safety violation rate

  5. TASK ADHERENCE
     Does the agent's tool use align with the user's intent?
     Does it avoid unintended or premature tool calls?
     Metric: task adherence score (Content Safety feature)

  6. EFFICIENCY
     How many tool calls does the agent make to complete the task?
     Fewer calls = faster and cheaper
     Metric: average tool calls per task

Evaluation in Foundry:
  1. Create test scenarios: user questions + expected tool calls + expected answers
  2. Run evaluation flows: agent processes each scenario
  3. Score: task completion, groundedness, safety, efficiency
  4. Identify failures: which scenarios failed? Why?
  5. Iterate: improve instructions, tool descriptions, or orchestration

For AI-103:
  Know the evaluation dimensions (task completion, tool accuracy, groundedness, safety)
  Know that task adherence (Content Safety) monitors agent tool use
  Know how to set up evaluation in Foundry

Building a Data Engineering Agent — Practical Example

Scenario: a data engineering team agent that can:
  - Check pipeline status (ADF, Databricks)
  - Look up error documentation
  - Create incident tickets
  - Analyze data quality metrics

Agent configuration:
  Model: GPT-4o
  Instructions: "You are a data engineering assistant for the DataPlatform team.
                 You can check pipeline statuses, look up errors, and create tickets.
                 Always check the pipeline status before advising on errors.
                 If a pipeline has failed, look up the error in the knowledge base.
                 Only create a ticket if the user confirms."

  Tools:
    1. check_pipeline_status(pipeline_name, date) → status, run_id, duration
    2. get_error_details(run_id) → error message, failed activity, stack trace
    3. search_knowledge_base(query) → relevant troubleshooting articles
    4. create_incident_ticket(title, description, priority) → ticket_id
    5. get_data_quality_metrics(table_name) → row count, null %, duplicate %

  Memory: thread-based (remembers the conversation)
  Safety: Content Safety enabled, task adherence enabled

  Conversation flow:
    User: "Did the daily sales pipeline run successfully?"
    Agent: [calls check_pipeline_status("daily_sales", "2026-08-20")]
           "The daily sales pipeline failed at 6:23 AM after 12 minutes."

    User: "What happened?"
    Agent: [calls get_error_details("run-abc-123")]
           [calls search_knowledge_base("SQL Server connection timeout")]
           "The failure was caused by a connection timeout to the SQL Server
            during the CopyData activity. Our knowledge base suggests this
            is typically caused by network congestion during peak hours.
            Recommended fix: retry the pipeline or increase the timeout setting."

    User: "Create a ticket for this."
    Agent: [calls create_incident_ticket("Daily Sales Pipeline Failure",
            "Connection timeout to SQL Server during CopyData...", "High")]
           "Created incident ticket INC-4567 with High priority.
            The on-call team has been notified."

Common Mistakes

  1. Building agents when RAG would suffice. If the task is “answer questions from documents,” RAG is simpler, cheaper, and more reliable than an agent. Agents add complexity: tool orchestration, memory management, error handling for tool failures. Use agents when the task requires ACTIONS (API calls, database queries, ticket creation), not just answers.

  2. Defining vague tool descriptions. The agent decides which tool to call based on the description. “Useful function” tells the agent nothing. “Query the status of a data pipeline by name, returning the run status, run ID, duration, and error message if failed” tells the agent exactly when and how to use the tool.

  3. Not handling tool execution failures. Tools can fail: the API times out, the database is down, the function throws an exception. If your tool returns an error, the agent may try to work with the error message or call the same tool repeatedly. Implement graceful error handling: catch exceptions, return structured error messages, and include retry guidance in the agent instructions.

  4. Giving agents too many tools. An agent with 20 tools struggles to choose the right one. Each additional tool increases the chance of wrong tool selection. Start with 3-5 focused tools. Add more only when evaluation shows the agent needs additional capabilities.

  5. Not using task adherence monitoring. Without task adherence, an agent might call a tool that does not match the user’s intent (calling a delete function when the user asked a question). Enable task adherence in Content Safety to monitor and flag misaligned tool use.

  6. Ignoring token limits in long conversations. Agent threads accumulate tokens over a conversation. Long conversations with many tool calls can exceed the context window. Important context gets truncated. Put critical information in the system instructions (which are always included), not in early conversation turns that may be dropped.

  7. Using multi-agent orchestration for simple tasks. Multi-agent adds complexity: inter-agent communication, state management, debugging across agents. A single agent with multiple tools handles most use cases. Use multi-agent only when the task genuinely requires different expertise or parallel processing (evaluating multiple proposals, specialist handoff).

  8. Not evaluating agents with diverse test scenarios. An agent that works for “check pipeline status” may fail for “check pipeline status and create a ticket if it failed.” Test: simple questions, multi-step tasks, edge cases (no data found, ambiguous requests), adversarial inputs (prompt injection through tool results), and long conversations. Evaluate task completion, tool accuracy, groundedness, and safety.

Interview Questions

Q: What is an AI agent and how does it differ from RAG? A: An AI agent is a system that reasons about tasks, plans steps, uses tools to gather information or take actions, observes results, and adapts. It follows a think-act-observe loop. RAG retrieves documents and generates answers — it answers questions but cannot take actions. An agent can call APIs, query databases, create tickets, run code, and coordinate multi-step workflows. RAG is the answer engine. An agent is the action engine that may use RAG as one of its tools.

Q: What is the Responses API and how does it differ from Chat Completions? A: The Responses API is Azure OpenAI’s stateful API for building agents. It manages conversation threads automatically, supports built-in tools (file search, code interpreter, web search, functions), and enables native multi-agent orchestration. Chat Completions is stateless — each request is independent, you manage conversation history, and tool execution is manual. Use Chat Completions for simple Q&A and RAG. Use the Responses API for complex tasks requiring tools, memory, and multi-step reasoning.

Q: What built-in tools does Azure AI Agent Service provide? A: File Search (RAG over uploaded documents), Code Interpreter (sandboxed Python execution for data analysis and chart generation), Bing Grounding (web search for current information), Azure AI Search (enterprise RAG with custom indexes), and Function Calling (custom tool integration where you define the function schema and execute the code). The agent decides which tool to use based on the conversation and tool descriptions.

Q: How does function calling work in AI agents? A: You define function schemas with name, description, and parameter types. The agent receives a user message, reasons about whether a function is needed, and returns a function call request with the name and arguments. Your application code executes the function (calling an API, querying a database) and returns the result to the agent. The agent may call multiple functions in sequence or decide to call another function based on the first result. The key principle: the agent requests function calls, your code executes them.

Q: What are the multi-agent orchestration patterns supported by Azure? A: Five patterns: Sequential (agents process in order like a pipeline), Concurrent (agents work in parallel on subtasks), Handoff (agent transfers to a specialist when needed), Group Chat (agents collaborate in a shared conversation), and Magentic-One (orchestrator dynamically assigns tasks to specialist agents). These are supported through the Responses API multi-agent feature, Semantic Kernel, and AutoGen. Use sequential for simple pipelines, concurrent for parallel evaluation, handoff for domain routing, and group chat for collaborative analysis.

Q: How do you evaluate AI agent quality? A: Evaluate six dimensions: task completion (did the agent accomplish the goal?), tool use accuracy (right tools, correct parameters), groundedness (responses based on tool results, not hallucinations), safety (follows Content Safety rules), task adherence (tool use aligns with user intent), and efficiency (minimum tool calls per task). Create diverse test scenarios including simple questions, multi-step tasks, edge cases, and adversarial inputs. Use Foundry evaluation flows to score systematically and iterate based on low-scoring areas.

Q: When should you use an agent vs RAG vs simple prompt engineering? A: Use prompt engineering when the task is answerable from the model’s knowledge or provided context (simple Q&A, summarization, classification). Use RAG when the task requires specific factual knowledge from your documents (company policies, product information, technical documentation). Use an agent when the task requires actions beyond answering — calling APIs, querying databases, creating tickets, running code, or orchestrating multi-step workflows. Each level adds capability and complexity: prompt engineering is simplest, agents are most powerful but most complex.

Wrapping Up

AI agents represent the next evolution beyond RAG: from answering questions to taking actions. The Responses API provides the foundation with built-in tools, thread-based memory, and multi-agent support. Azure AI Agent Service adds managed hosting, security, and monitoring. Function calling connects agents to your existing systems. And multi-agent orchestration handles complex scenarios where multiple specialized agents collaborate.

For AI-103, the key concepts are: the agent loop (think-act-observe), the Responses API vs Chat Completions, built-in tools (file search, code interpreter, function calling), thread-based memory, multi-agent patterns (sequential, concurrent, handoff, group chat, Magentic-One), and evaluation dimensions (task completion, tool accuracy, groundedness, safety, task adherence).

In the next post, we move to Domain 3: Computer Vision on Azure — Image Analysis, Custom Vision, OCR, Face API, and how Azure sees and understands images and video.

Related posts: – RAG Pipelines & AI Search – Prompt Engineering – Microsoft Foundry Platform – AI-103 Study Guide – Security & Responsible AI

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top