Document Intelligence and Information Extraction on Azure for AI-103: Prebuilt Models, Custom Template and Neural Models, Composed Models, Custom Classifiers, AI Search Skillsets, Custom Skills, Integrated Vectorization, Content Understanding, and the Complete Document Processing Pipeline

Leave a Comment / AI/ML, Azure

Table of Contents

In the previous post, we covered text analysis and speech — understanding and generating natural language. This final post in the AI-103 series covers Domain 5: Information Extraction — turning unstructured documents into structured, searchable data using Document Intelligence, AI Search skillsets, and Content Understanding. For data engineers, this is where AI meets your daily work: processing invoices, receipts, contracts, forms, and documents at scale.

Analogy — A mailroom that reads, sorts, and files every document automatically. Imagine a corporate mailroom that receives thousands of documents daily: invoices from vendors, receipts from employees, ID documents from new hires, contracts from legal, and medical forms from HR. Today, a human opens each document, reads it, identifies the type, extracts the key information, and enters it into the right system. Document Intelligence is the automated mailroom: it reads every document (Read model), identifies its type (custom classifier), extracts specific fields (prebuilt or custom models), and delivers structured data to your pipeline. AI Search then makes all of it searchable. And Content Understanding handles the unusual documents that do not fit standard templates.

Information Extraction Through a Data Engineer’s Lens

Why data engineers care about document processing:

  1. ENTERPRISE DATA IS MOSTLY DOCUMENTS
     Invoices, purchase orders, contracts, reports, forms, emails
     These are NOT in databases -- they are in PDFs, Word docs, images
     Someone has to turn them into structured data for analytics
     That someone is increasingly your data pipeline, not a human

  2. THE DOCUMENT PROCESSING PIPELINE
     Source: Blob Storage (PDFs, scanned images, Word docs)
     Extract: Document Intelligence (fields, tables, key-value pairs)
     Transform: data validation, normalization, deduplication
     Load: SQL database, Lakehouse, search index

  3. SEARCH AND RAG ENRICHMENT
     AI Search indexes documents for search and RAG
     Skillsets add intelligence: OCR, entity extraction, embedding
     Vector search enables semantic retrieval
     This is the ingestion half of every RAG pipeline

Real-world examples data engineers build:
  Accounts payable: vendor invoices → DI invoice model → ERP system
  Expense management: receipt photos → DI receipt model → expense database
  HR onboarding: ID documents → DI ID model → employee records
  Insurance claims: claim forms → custom DI model → claims processing
  Legal: contracts → AI Search index → searchable contract library
  Healthcare: patient forms → DI + Health NLP → clinical database

Azure AI Document Intelligence — The Document Processing Platform

Azure AI Document Intelligence (formerly Form Recognizer) extracts
structured data from documents using machine learning.

Name history:
  Form Recognizer → renamed to Document Intelligence (2023)
  Now part of Foundry Tools in Microsoft Foundry (2025-2026)
  AI-103 uses "Document Intelligence" -- if you see "Form Recognizer"
  in older materials, it is the same service

What Document Intelligence provides:
  Read model: extract raw text (like OCR but optimized for documents)
  Layout model: extract text + tables + structure + selection marks
  Prebuilt models: extract known fields from standard document types
  Custom models: extract fields YOU define from YOUR document types
  Composed models: combine multiple custom models behind one endpoint
  Custom classifiers: identify document type before extraction

Supported input:
  PDF, JPEG, PNG, BMP, TIFF, HEIF, DOCX, XLSX, PPTX, HTML
  Up to 500 MB per file (paid tier), 2,000 pages per document
  Scanned, photographed, and digital documents

The Layout Model — Understanding Document Structure

The Layout model extracts the complete structure of a document: text, tables, selection marks (checkboxes), and reading order.

Analogy — A document X-ray machine. If a regular camera takes a photo of a document (what it looks like), the Layout model is an X-ray machine that reveals the internal structure: where the text blocks are, how the tables are organized, which checkboxes are checked, and the logical reading order from top-left to bottom-right.

Layout model extracts:

  TEXT:
    Pages → paragraphs → lines → words
    With bounding polygons (where each element is on the page)
    Reading order (logical sequence, not just left-to-right)

  TABLES:
    Row and column structure
    Cell content with row/column indices
    Spanning cells (merged cells across rows or columns)
    Table headers identified separately

  SELECTION MARKS:
    Checkboxes: checked or unchecked
    Radio buttons: selected or unselected
    With bounding box location

  DOCUMENT STRUCTURE:
    Sections, headings, footnotes, page headers/footers
    Figures with captions
    Formulas and barcodes (add-on features)

Real-world example -- Extracting tables from annual reports:
  1. Annual reports (100-page PDFs) uploaded to Blob Storage
  2. Layout model processes each report
  3. Extracts: all tables with row/column structure
  4. Pipeline identifies financial tables (revenue, expenses, balance sheet)
  5. Table data loaded to Lakehouse as structured Delta tables
  6. Analysts query financial data across 10 years of reports
  Without Layout: someone manually copies tables from PDFs (days of work)
  With Layout: pipeline extracts all tables in minutes
# Layout model extraction
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.identity import DefaultAzureCredential

client = DocumentIntelligenceClient(
    endpoint="https://your-di.cognitiveservices.azure.com",
    credential=DefaultAzureCredential()
)

# Analyze a document
with open("annual-report.pdf", "rb") as f:
    poller = client.begin_analyze_document("prebuilt-layout", body=f)
result = poller.result()

# Extract tables
for table in result.tables:
    print(f"Table: {table.row_count} rows x {table.column_count} columns")
    for cell in table.cells:
        print(f"  Row {cell.row_index}, Col {cell.column_index}: {cell.content}")

# Extract text with reading order
for page in result.pages:
    for line in page.lines:
        print(f"Line: {line.content}")

The Read Model — Extracting Raw Text

The Read model is the simplest Document Intelligence model.
It extracts text content from documents -- nothing more.

Read model vs Layout model:
  Read: text only (pages, lines, words, languages)
  Layout: text + tables + selection marks + structure

When to use Read:
  You need raw text from a document (no tables, no structure)
  Preprocessing for downstream NLP (feed text to sentiment analysis, NER)
  High-volume text extraction where tables are not needed
  Read is faster and cheaper than Layout for text-only scenarios

Read model vs Vision OCR (Read API):
  Document Intelligence Read: optimized for documents (better paragraph detection)
  Vision OCR: optimized for images (better for photos, signs, license plates)
  Use DI Read for: documents (PDFs, Word files, scanned forms)
  Use Vision OCR for: natural images (photos, screenshots, whiteboards)

Prebuilt Models — Ready-Made Document Extractors

Prebuilt models extract specific fields from common document types with zero training.

Analogy — A team of specialist clerks. Instead of training one person to read every document type, you have specialists: one clerk who only reads invoices (and knows exactly where to find the total, vendor name, and due date), another who only reads receipts, another who handles ID documents. Each specialist is pre-trained and ready to work immediately.

Prebuilt models available:

  INVOICE (prebuilt-invoice):
    Extracts: VendorName, VendorAddress, CustomerName, InvoiceID,
              InvoiceDate, DueDate, SubTotal, Tax, TotalAmount,
              LineItems (description, quantity, unit price, amount)
    Supports: structured, semi-structured, and unstructured invoices
    Languages: multiple (English, French, German, Spanish, and more)

  RECEIPT (prebuilt-receipt):
    Extracts: MerchantName, MerchantAddress, TransactionDate,
              TransactionTime, Items (name, quantity, price), Total, Tax, Tip
    Supports: printed, thermal, handwritten receipts, hotel receipts

  ID DOCUMENT (prebuilt-idDocument):
    Extracts: FirstName, LastName, DateOfBirth, DocumentNumber,
              ExpirationDate, Address, Country/Region
    Supports: US driver's licenses, international passports

  W-2 (prebuilt-tax.us.w2):
    Extracts: EmployeeName, EmployerName, SSN, Wages, FederalTax,
              StateTax, Box1 through Box20

  HEALTH INSURANCE CARD (prebuilt-healthInsuranceCard.us):
    Extracts: InsurancePlan, MemberName, MemberID, GroupNumber,
              Copay, Prescriptions

  BUSINESS CARD (prebuilt-businessCard):
    Extracts: Name, JobTitle, Company, Email, Phone, Address, Website

  MARRIAGE CERTIFICATE (prebuilt-marriage certificate):
    Extracts: Spouse1, Spouse2, DateOfMarriage, Location

  CREDIT CARD (prebuilt-creditCard):
    Extracts: CardNumber, CardHolderName, ExpirationDate, Bank

  MORTGAGE (prebuilt-mortgage):
    Extracts: Borrower, Lender, PropertyAddress, LoanAmount

Real-world example -- Accounts payable automation:
  1. Vendors email invoices (PDF attachments) to a shared inbox
  2. Logic App or Power Automate saves PDFs to Blob Storage
  3. Data pipeline triggers on new files
  4. Document Intelligence prebuilt-invoice extracts fields:
     VendorName: "Acme Supplies"
     InvoiceID: "INV-2026-0456"
     InvoiceDate: "2026-08-15"
     DueDate: "2026-09-15"
     SubTotal: $4,250.00
     Tax: $552.50
     TotalAmount: $4,802.50
     LineItems:
       "Widget A" | Qty: 100 | Unit: $25.00 | Amount: $2,500.00
       "Widget B" | Qty: 50  | Unit: $35.00 | Amount: $1,750.00
  5. Extracted data loaded to AP database
  6. Three-way match: PO + receipt + invoice → auto-approved or flagged
  7. Dashboard: invoice aging, vendor spend, approval bottlenecks

  Before DI: AP clerk manually enters 200 invoices/week (40 hours)
  After DI: pipeline processes 200 invoices in 30 minutes
  Human review: only flagged invoices (low confidence or mismatches)

Custom Models — Training on Your Documents

Custom models extract fields that prebuilt models do not cover — fields specific to YOUR organization’s documents.

Two types of custom models:

  1. TEMPLATE (fixed layout):
     Best for: forms with CONSISTENT layout (same positions on every page)
     Example: your company's internal purchase order, a standard insurance form
     Training: minimum 5 labeled documents (same layout)
     How it works: learns field POSITIONS (field X is always at coordinates Y)
     Fast training, high accuracy for fixed layouts
     Fails if layout varies document to document

  2. NEURAL (variable layout):
     Best for: documents with VARYING layouts (different vendors, different formats)
     Example: invoices from 50 different vendors (each has different layout)
     Training: minimum 5 labeled documents (recommended 20+)
     How it works: learns field MEANING using deep learning (not just position)
     Slower training, handles layout variation
     Better for semi-structured and unstructured documents

  Template vs Neural decision:
    Same layout every time (your internal forms) → Template
    Different layouts (vendor invoices, external documents) → Neural
    When in doubt → Neural (more flexible)

Training process:
  1. Document Intelligence Studio → Custom Models → Create
  2. Upload training documents (5-50, more = better)
  3. Label fields: draw bounding boxes, assign field names and types
     Example: draw box around "Project Code" → label as "project_code" (string)
     Example: draw box around "$4,802.50" → label as "total_amount" (currency)
  4. Train the model (minutes for template, longer for neural)
  5. Test with new documents in the studio
  6. Publish → get a model ID
  7. Call from your pipeline:
     result = client.begin_analyze_document("your-custom-model-id", body=document)

Real-world example -- Insurance claim forms:
  Your company has a custom claim form with fields no prebuilt model covers:
    claim_number, policy_number, incident_date, incident_description,
    damage_type, estimated_repair_cost, adjuster_assigned, claim_status

  Training: label 30 sample claim forms
  Model learns to extract these custom fields from new claim forms
  Pipeline: claim form scanned → DI extracts fields → claims database updated

Composed Models — Routing Multiple Document Types

A composed model combines multiple custom models behind a single endpoint. The service automatically routes each document to the correct model.

Analogy — A hospital intake desk. When a patient arrives with a stack of paperwork — ID, insurance card, medical history form, consent form — the intake clerk does not use one form-reading process for all documents. They route each document to the appropriate handler. Composed models work the same way: one endpoint receives all documents, identifies the type, and routes to the specialist model.

How composed models work:

  Step 1: Train individual custom models
    Model A: trained on internal purchase orders
    Model B: trained on vendor invoices
    Model C: trained on expense receipts

  Step 2: Create a composed model
    Combine Models A, B, and C into a single composed model
    Assign a single model ID

  Step 3: Send any document
    Send a purchase order → composed model routes to Model A → PO fields extracted
    Send an invoice → composed model routes to Model B → invoice fields extracted
    Send a receipt → composed model routes to Model C → receipt fields extracted

  The classification happens automatically based on the document content.
  You can assign up to 200 custom models to a single composed model.

Real-world example -- Loan application processing:
  A loan package contains: ID document + bank statement + pay stub + application form
  Each page type → different custom model → all extracted via one API call
  Composed model ID: "loan-package-model"
  Pipeline sends the entire PDF → DI splits, classifies, and extracts per page

Custom Classifiers — Sorting Before Extracting

Custom classifiers identify the document type BEFORE extraction. This is useful when you need explicit control over routing.

Classifier vs Composed Model:

  Composed Model: classifies AND extracts in one step
    Send document → auto-classified → extracted → fields returned

  Custom Classifier: classifies ONLY (no extraction)
    Send document → classified → YOUR code decides what to do next
    More control: you can route to different systems, not just DI models

Training a classifier:
  1. Collect sample documents for each class (minimum 5 per class)
  2. Label by class: "invoice", "receipt", "purchase_order", "contract"
  3. Train the classifier model
  4. Test: send a document → classifier returns: "invoice" (confidence: 0.96)
  5. Your pipeline routes based on the classification result

Real-world example -- Digital mailroom:
  1. All incoming documents scanned to Blob Storage
  2. Custom classifier identifies type:
     "invoice" → route to accounts payable pipeline
     "contract" → route to legal review pipeline
     "resume" → route to HR onboarding pipeline
     "purchase_order" → route to procurement pipeline
  3. Each pipeline uses the appropriate DI model for extraction
  4. Unrecognized documents (low confidence) → human review queue

Azure AI Search — Building Searchable Knowledge Bases

AI Search stores and retrieves documents for search and RAG. We covered it in detail in the RAG post. Here we focus on the indexing pipeline — how documents become searchable.

The indexing pipeline:

  DATA SOURCE → SKILLSET → INDEX
       ↑            ↑          ↑
    where docs     how to     where to
    come from      process    store results

  Data Source: Blob Storage, SQL Database, Cosmos DB, ADLS Gen2
  Skillset: chain of AI skills that process each document
  Index: searchable store with text, vectors, and metadata

  The Indexer is the orchestrator that ties it all together:
    Indexer connects to data source
    Indexer runs skillset on each document
    Indexer maps skill outputs to index fields
    Indexer runs on a schedule (change detection for incremental updates)

For AI-103 Domain 5:
  Know the Data Source → Skillset → Index pipeline
  Know which data sources are supported
  Know the difference between built-in and custom skills
  Know how indexers handle incremental updates (change detection)

Built-In Skillsets — AI Enrichment During Indexing

Skillsets are chains of AI processing steps that run during indexing. Each skill takes input, processes it, and produces output that the next skill can use.

Key built-in skills for Domain 5:

  OCR SKILL (#Microsoft.Skills.Vision.OcrSkill):
    Input: image embedded in a document
    Output: extracted text
    Use: process scanned PDFs, images in documents
    This enables text search over scanned/image-based documents

  ENTITY RECOGNITION (#Microsoft.Skills.Text.V3.EntityRecognitionSkill):
    Input: text content
    Output: entities (people, organizations, locations, dates)
    Use: enable filtering and faceting by entity

  KEY PHRASE EXTRACTION (#Microsoft.Skills.Text.KeyPhraseExtractionSkill):
    Input: text content
    Output: key phrases
    Use: topic tagging, search suggestions

  LANGUAGE DETECTION (#Microsoft.Skills.Text.LanguageDetectionSkill):
    Input: text content
    Output: language code (en, fr, ja, etc.)
    Use: route to language-specific analyzers

  TEXT SPLIT (#Microsoft.Skills.Text.SplitSkill):
    Input: long text
    Output: chunks (pages or sentences)
    Use: break documents into searchable chunks for RAG

  AZURE OPENAI EMBEDDING (#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill):
    Input: text chunk
    Output: vector embedding (float array)
    Use: enable vector search and hybrid search

  DOCUMENT LAYOUT (#Microsoft.Skills.Util.DocumentIntelligenceLayoutSkill):
    Input: document
    Output: structured layout (text, tables, sections)
    Use: AI-aware chunking that preserves document structure

Skillset pipeline example -- Processing a document library:
  Document (PDF) from Blob Storage
    → OCR skill: extract text from scanned pages
    → Language Detection skill: identify language (English)
    → Text Split skill: chunk into 500-token pieces
    → Key Phrase Extraction skill: identify topics per chunk
    → Entity Recognition skill: extract people, organizations per chunk
    → Azure OpenAI Embedding skill: generate vector per chunk
    → Index: store text + vector + entities + key phrases + language

Custom Skills — Your Own Processing Logic

Custom skills call YOUR API during the indexing pipeline. When built-in skills do not cover your needs, you write a custom skill as an Azure Function or any REST endpoint.

Analogy — Adding a specialist to the assembly line. The built-in skills are the standard workstations on the factory assembly line. A custom skill is a specialist workstation you add for your specific product: maybe you need a custom quality check, a proprietary calculation, or a lookup against your internal database that no standard workstation handles.

Custom skill implementation:

  1. Create an Azure Function (Python, C#, JavaScript)
  2. Implement the Web API custom skill contract:
     Input: {"values": [{"recordId": "1", "data": {"text": "..."}}]}
     Output: {"values": [{"recordId": "1", "data": {"result": "..."}}]}
  3. Deploy the function
  4. Add to your skillset definition with the function URL

Example custom skill -- Domain-specific classification:
  Built-in skills cannot classify documents into YOUR categories
  Custom skill calls your own classification model (or GPT-4o)
  Input: document text
  Output: category ("legal", "financial", "technical", "hr")
  Result stored in the index → users filter by document category

Example custom skill -- External database lookup:
  Document mentions "Account #12345"
  Custom skill looks up the account in your CRM database
  Returns: customer name, tier, account manager
  Result stored alongside the document in the index
  Users search and see enriched results with customer context

For AI-103:
  Know the custom skill input/output contract (JSON with recordId + data)
  Know that Azure Functions is the typical hosting option
  Know when custom skills are needed (domain logic, external data)
Vector search and integrated vectorization were covered in depth
in the RAG post. Here is the Domain 5 summary:

Vector Search:
  Store document embeddings alongside text in the search index
  Search by semantic MEANING, not just keywords
  Configure: HNSW algorithm, dimensions (match embedding model), distance metric

Integrated Vectorization:
  Text Split skill → AzureOpenAIEmbedding skill → Vector field in index
  All handled automatically by the indexer
  No custom code for the embedding pipeline
  Vectorizer component handles query-time embedding

The Domain 5 perspective:
  Domain 5 asks about INDEXING (ingestion pipeline, skillsets, indexers)
  Domain 2 (RAG post) asks about QUERYING (hybrid search, retrieval)
  Same service (AI Search), different exam focus

For AI-103 Domain 5 specifically:
  Know how to configure a skillset with Text Split + Embedding skills
  Know what integrated vectorization automates
  Know the indexer schedule and change detection mechanism
  Know how to configure vector search dimensions and algorithm

Content Understanding — The Generative AI Approach

Content Understanding is Azure’s newest extraction service. It uses generative AI to extract structured data from any content type using natural language schemas.

Analogy — A brilliant intern vs a trained specialist. Document Intelligence prebuilt models are like trained specialists: they know exactly how to read an invoice or receipt because they have been trained on thousands of examples. Content Understanding is like a brilliant intern: give them any document and describe what you need (“find the project name, budget, and deadline”), and they figure it out without prior training. The specialist is faster and more accurate on known document types. The intern is more flexible on new or unusual documents.

Content Understanding capabilities:

  NATURAL LANGUAGE SCHEMA:
    Define what to extract in plain English
    "Extract: vendor name, invoice total, line items with descriptions and amounts"
    No labeled training data needed -- generative AI interprets the schema

  MULTIMODAL:
    Processes: documents, images, audio, video
    Not limited to text-based documents
    Can extract from photos, diagrams, presentations, recordings

  ANALYZER TEMPLATES:
    Predefined extraction templates for common scenarios
    Customize by modifying the natural language schema
    Example: "Analyze this video and extract scene descriptions,
             speaker identifications, and key topics per segment"

Content Understanding vs Document Intelligence:

  Document Intelligence prebuilt models:
    Best for: known document types (invoice, receipt, ID)
    How: trained ML models with fixed field schemas
    Accuracy: very high on supported document types
    Speed: fast (milliseconds per document)
    Cost: lower per document

  Document Intelligence custom models:
    Best for: YOUR specific document layouts
    How: you label training data, model learns your fields
    Accuracy: high after sufficient training data
    Speed: fast after training

  Content Understanding:
    Best for: new or unusual documents where no model exists
    How: generative AI interprets natural language schemas
    Accuracy: good, improves with better schema descriptions
    Speed: slower (generative AI processing)
    Cost: higher per document (LLM inference)
    Flexibility: handles ANY document type without training

  Decision guide:
    Prebuilt model exists for your document type → Document Intelligence prebuilt
    No prebuilt model, consistent layout → Document Intelligence custom (template)
    No prebuilt model, variable layout → Document Intelligence custom (neural)
    Unusual/new document type, low volume → Content Understanding
    Multimodal content (video, audio, images) → Content Understanding

Building a Complete Document Processing Pipeline

End-to-end pipeline for processing vendor invoices:

  INGESTION:
    1. Vendor emails arrive in shared inbox
    2. Logic App saves PDF attachments to Blob Storage
       Container: "incoming-invoices/{vendor}/{date}/{filename}.pdf"

  CLASSIFICATION (if multiple document types):
    3. Custom classifier identifies document type
       "invoice" (0.96) → continue to invoice extraction
       "purchase_order" (0.93) → route to PO pipeline
       "unknown" (0.45) → route to human review queue

  EXTRACTION:
    4. Document Intelligence prebuilt-invoice processes the PDF
       Extracts: VendorName, InvoiceID, InvoiceDate, DueDate,
                 TotalAmount, Tax, LineItems

  VALIDATION:
    5. Pipeline validates extracted fields:
       - Required fields present? (VendorName, InvoiceID, Total)
       - Confidence above threshold? (> 0.85)
       - Total = Sum of line items? (arithmetic validation)
       - Vendor exists in master data? (database lookup)
       - Duplicate invoice? (check InvoiceID against existing records)

  ENRICHMENT:
    6. Look up vendor details from master data (payment terms, category)
    7. Match against purchase order (three-way match: PO + receipt + invoice)

  LOADING:
    8. High-confidence, validated invoices → auto-load to AP system
    9. Low-confidence or failed validation → human review queue
    10. All invoices indexed in AI Search for searchable archive

  MONITORING:
    11. Dashboard: invoices processed, auto-approved rate, review queue size
    12. Alert: confidence below threshold trending up (model degradation)
    13. Monthly: accuracy report comparing DI extraction vs human review

  Metrics:
    Before DI: 200 invoices/week, 40 hours manual processing
    After DI: 200 invoices/week, 30 minutes automated + 2 hours human review
    Accuracy: 95%+ auto-approved, 5% human review
    Cost savings: ~35 hours/week of manual data entry eliminated

Choosing the Right Service — Document Intelligence vs OCR vs Content Understanding

"I need to..." → Which service?

  Extract raw text from any image or document
    → Vision Read API (images, photos, signs)
    → Document Intelligence Read model (documents, PDFs)

  Extract tables and structure from documents
    → Document Intelligence Layout model

  Extract fields from invoices
    → Document Intelligence prebuilt-invoice

  Extract fields from receipts
    → Document Intelligence prebuilt-receipt

  Extract fields from ID documents
    → Document Intelligence prebuilt-idDocument

  Extract custom fields from MY company's forms
    → Document Intelligence custom model (template or neural)

  Process a mix of document types from one endpoint
    → Document Intelligence composed model

  Sort documents by type before extracting
    → Document Intelligence custom classifier

  Make documents searchable with AI enrichment
    → Azure AI Search with skillsets (OCR, NER, embedding)

  Extract custom structured data from any content (including video)
    → Content Understanding

  Extract text during AI Search indexing
    → OCR skill in AI Search skillset

Quick decision:
  Standard document type (invoice, receipt, ID)? → DI prebuilt
  Your specific forms with consistent layout? → DI custom template
  Your specific forms with varying layouts? → DI custom neural
  Any document, no training, natural language schema? → Content Understanding
  Just need text for search/RAG? → Read model or OCR skill

Common Mistakes

  1. Building a custom model when a prebuilt model already exists. Document Intelligence has prebuilt models for invoices, receipts, IDs, W-2s, health insurance cards, and more. Training a custom model for invoices wastes time and produces worse results than the prebuilt model (which was trained on millions of invoices). Check prebuilt models first. Only build custom when no prebuilt matches your document type.

  2. Using the template custom model for variable-layout documents. Template models learn field POSITIONS. If your invoices come from 50 different vendors with different layouts, the template model fails because fields are in different positions on each invoice. Use the neural custom model, which learns field MEANING regardless of position.

  3. Not validating extraction results before loading to the database. Document Intelligence returns confidence scores for every extracted field. A field with 0.45 confidence is likely wrong. Always validate: check confidence thresholds, verify arithmetic (total = sum of line items), and route low-confidence extractions to human review. Blindly loading unvalidated extractions creates dirty data.

  4. Using Content Understanding for everything instead of Document Intelligence. Content Understanding is flexible (any document, natural language schema) but slower and more expensive than Document Intelligence (trained ML models). For standard document types and high-volume processing, Document Intelligence prebuilt and custom models are faster, cheaper, and more accurate. Reserve Content Understanding for unusual documents, multimodal content, or low-volume scenarios.

  5. Not including an OCR skill in AI Search skillsets for scanned documents. If your document library contains scanned PDFs or images, AI Search cannot extract text from them without the OCR skill. Without OCR, these documents are invisible to search. Always include OCR in skillsets when processing document libraries that may contain scanned content.

  6. Training custom models with too few or unrepresentative documents. A custom model trained on 5 perfect examples of your form will fail on real-world documents with smudges, skewed scans, missing fields, or handwriting. Train with at least 20 documents that represent the full range of variation you expect in production: different scan qualities, filled and partially filled forms, handwritten and typed content.

  7. Confusing Document Intelligence Read with Vision OCR. Both extract text from documents, but they are optimized for different inputs. Document Intelligence Read is optimized for documents (PDFs, forms, structured content). Vision OCR is optimized for natural images (photos, signs, license plates). Using the wrong one produces worse results and higher costs.

  8. Not using composed models for mixed document batches. If your pipeline receives invoices, receipts, and purchase orders in the same batch, calling three separate models requires your code to classify each document first. A composed model handles classification and extraction in one step, simplifying your pipeline and reducing error-prone manual routing.

Interview Questions

Q: What is Azure AI Document Intelligence and what document types does it support? A: Document Intelligence (formerly Form Recognizer) extracts structured data from documents using machine learning. It supports prebuilt models for common document types (invoice, receipt, ID document, W-2, health insurance card, business card, marriage certificate, credit card, mortgage), a Layout model for tables and structure extraction, a Read model for raw text, custom models (template and neural) for organization-specific documents, composed models for multi-type routing, and custom classifiers for document sorting. It processes PDFs, images, and Office documents up to 500 MB and 2,000 pages.

Q: What is the difference between template and neural custom models? A: Template custom models learn field positions on the page, so they work best for documents with a fixed, consistent layout (your company’s internal forms). Neural custom models use deep learning to understand field meaning regardless of position, handling varying layouts (invoices from different vendors). Template models train faster and are more accurate on fixed layouts. Neural models are more flexible and handle variation. Use template when every document has the same layout. Use neural when layouts vary.

Q: What is a composed model and when would you use it? A: A composed model combines up to 200 custom models behind a single endpoint. When a document is submitted, the composed model automatically classifies it and routes it to the correct custom model for extraction. Use it when your pipeline receives mixed document types (invoices, receipts, purchase orders) in a single batch. It eliminates the need for your code to classify documents before extraction, simplifying the pipeline and reducing routing errors.

Q: How do built-in skillsets enrich documents during AI Search indexing? A: Skillsets are chains of AI processing steps in the indexer pipeline. Built-in skills include OCR (extract text from images), Entity Recognition (extract people, organizations, locations), Key Phrase Extraction (identify topics), Language Detection (identify language), Text Split (chunk documents for RAG), Azure OpenAI Embedding (generate vectors for semantic search), and Document Layout (AI-aware structure extraction). Each skill adds metadata to the document, making it richer and more searchable. Multiple skills chain together: OCR → Text Split → Embedding creates a RAG-ready index from scanned PDFs.

Q: When should you use custom skills in AI Search? A: Use custom skills when built-in skills do not cover your processing needs. Implement as an Azure Function or REST endpoint following the Web API custom skill contract (JSON with recordId and data). Examples: domain-specific classification (categorize documents into your taxonomy), external database lookup (enrich documents with CRM or ERP data), proprietary calculations, or calling your own ML model during indexing. Custom skills extend the indexing pipeline with arbitrary processing logic.

Q: What is Content Understanding and how does it differ from Document Intelligence? A: Content Understanding uses generative AI to extract structured data from any content type using natural language schemas. You describe what to extract in plain language, and it interprets the document without prior training. Document Intelligence uses trained ML models (prebuilt or custom) with fixed field schemas. Content Understanding is more flexible (handles any document, including video and audio) but slower and more expensive. Document Intelligence is faster, cheaper, and more accurate for known document types. Use Content Understanding for unusual documents or multimodal content where no trained model exists.

Q: Describe a complete document processing pipeline from ingestion to analytics. A: Documents arrive in Blob Storage (via email, upload, or scan). A custom classifier identifies the document type. Document Intelligence extracts structured fields using the appropriate model (prebuilt for standard types, custom for organization-specific forms, composed for mixed batches). The pipeline validates extractions: checks confidence scores against thresholds, verifies arithmetic consistency, validates against master data, and checks for duplicates. High-confidence results load automatically to the database. Low-confidence results route to a human review queue. All documents are indexed in AI Search with skillsets (OCR, entities, embeddings) for searchable archive and RAG. Monitoring tracks extraction accuracy, auto-approval rate, and model confidence trends.

Wrapping Up

Document Intelligence and AI Search together turn unstructured documents into structured, searchable, retrievable data. Document Intelligence handles the extraction: prebuilt models for standard documents (invoices, receipts, IDs), custom models for your specific forms, composed models for mixed batches, and classifiers for routing. AI Search handles the indexing: skillsets enrich documents with OCR, entities, key phrases, and vector embeddings. Content Understanding adds generative AI flexibility for anything that does not fit standard models.

For AI-103, the key decisions are: prebuilt vs custom (does a prebuilt model exist?), template vs neural (fixed or variable layout?), composed vs classifier (auto-route or explicit control?), and built-in skills vs custom skills (standard enrichment or domain logic?). Every scenario question in Domain 5 comes down to matching the document type and requirement to the right combination of these services.

This completes the AI-103 series. From the Foundry platform to security, prompt engineering, RAG, agents, computer vision, text analysis, and document intelligence — you now have the complete map of every exam domain. Review the Study Guide for the 6-week plan, build hands-on in Foundry, and take practice exams until you consistently score 80%+.

Related posts: – AI-103 Study Guide – RAG Pipelines & AI Search – Computer Vision – Text Analysis & Speech – Microsoft Foundry Platform

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top