Azure Data Factory, Synapse, pipelines
Complete Materialized Lake Views guide. What MLVs are and how they differ from regular views and tables. When to use MLVs vs notebooks. Creating MLVs with aggregations and joins. Automatic refresh with Change Data Feed (CDF). Building Bronze-Silver-Gold layers with MLVs. Data quality monitoring MLV. Scheduling and debugging MLVs. MLV limitations.
Deep dive into Fabric Real-Time Analytics. Five window types with KQL examples: tumbling, hopping, sliding, session, snapshot. Accelerated vs standard shortcuts in KQL databases. Advanced KQL functions for dates, strings, aggregations, and JSON parsing. Materialized views for pre-computed aggregations. Eventhouse optimization: retention policies, caching policies, partitioning. RTI error resolution table.
Advanced Fabric Warehouse capabilities. COPY INTO for bulk loading from CSV and Parquet. CTAS for materialized summaries and snapshots. Dynamic Management Views for active queries and sessions. Query Insight views for performance analysis. Visual Query editor for no-code analysis. SSMS and Azure Data Studio connectivity with GRANT/DENY. Integrating T-SQL with Spark notebooks via cross-database queries. Warehouse optimization with statistics, result set caching.
Complete Spark Structured Streaming guide. Streaming vs batch same API, reading from Event Hubs, Delta Lake, and files. Output modes (Append, Complete, Update). Checkpoint location for crash recovery. Stateless vs stateful transformations. Windowing: tumbling, sliding/hopping, session windows with watermarks for late data. Writing to Delta tables and foreachBatch MERGE pattern. Trigger modes including availableNow for pipeline scheduling. Three real-world scenarios.
Complete Fabric REST APIs guide. Authentication three ways (Azure AD app, Azure CLI, mssparkutils). Core endpoints: workspace CRUD, item management, pipeline execution and monitoring, lakehouse table operations, capacity management. Six real-world Python scripts: create workspace with items, trigger and monitor pipeline from external system, audit all items across workspaces (CSV export), bulk assign roles, monitor pipeline failures, export table list. GitHub Actions CI/CD integration. Rate limits and best practices.
Master Data Activator (Reflex) in Fabric. Why dashboards are not enough (security camera vs motion sensor analogy). Core concepts (objects, properties, triggers, actions). Setting up from Power BI, Eventstream, and Fabric items. Trigger types: threshold, percentage change, absence, complex AND/OR conditions. Actions: email, Teams, Power Automate, start pipeline, webhook. Six real-world scenarios (inventory monitoring, revenue drop, IoT temperature, pipeline failure watchdog, churn early warning, data quality). Alert fatigue prevention. Comparison with Pipeline alerts and Power BI alerts.
Understand the foundation of Fabric. OneLake vs ADLS Gen2 vs S3 comparison, the hierarchy (tenant, workspace, item, tables, files), ADLS Gen2 API compatibility (same API, different endpoint), accessing from Azure Storage Explorer, AzCopy, Python SDK, and external Databricks. OneLake File Explorer for Windows. Multi-cloud shortcuts with caching. Data Hub for discovery. Storage billing details, soft delete, VACUUM for optimization. Three real-world patterns (centralized, hub-and-spoke, multi-cloud unified).
Open the Spark black box. Driver and executors explained, starter pools vs custom environments, key configurations (shuffle partitions with pizza analogy, AQE autopilot, auto-optimize, auto-compact, V-Order), broadcast joins, memory management, reading the Spark UI to identify bottlenecks (skew, small files, shuffles), four performance tuning patterns (small data, large data, join, write optimization), Spark Job Definitions for production, and high concurrency mode.
Master Fabric cost management. Capacity Units explained, F-SKU sizing guide (F2 to F512 with prices), CU consumption by workload type, throttling mechanics (10-min, 60-min, 24-hr windows), smoothing and burst model, monitoring with Capacity Metrics app, pause and resume automation, five cost optimization strategies (right-size, optimize Spark, stagger pipelines, pause dev, optimize Delta), and real-world sizing scenarios.
Every Fabric security layer explained. Seven layers from workspace roles to sensitivity labels. Workspace roles matrix (Admin, Member, Contributor, Viewer), item permissions, OneLake data access roles for table-level control, RLS in Warehouse and Semantic Models, CLS with GRANT on specific columns, dynamic data masking, sensitivity labels that flow downstream, Purview integration for lineage and catalog, endorsement (Promoted, Certified), and complete real-world security architecture.
