Data Engineering - DriveDataScience

Fabric Connections and Gateways: Connection Types, On-Premises Data Gateway, VNet Gateway, Managing Connections, and Accessing Data Behind Firewalls

Leave a Comment / Azure, Data Engineering

Complete Fabric connections and gateways guide. Connection types for Azure, on-premises, cross-cloud (AWS S3, GCS), and SaaS sources with authentication options. Connection vs ADF Linked Service migration. On-premises data gateway installation, architecture (outbound HTTPS only, no inbound ports), and high-availability clustering. VNet data gateway for private Azure resources. Creating, sharing, and reusing connections. Security best practices (Service Principal, Key Vault). Troubleshooting connection errors. Three real-world scenarios.

Fabric Connections and Gateways: Connection Types, On-Premises Data Gateway, VNet Gateway, Managing Connections, and Accessing Data Behind Firewalls Read More »

Delta Lake Table Properties: Every TBLPROPERTIES Setting, Retention, Change Data Feed, Column Mapping, Auto-Optimize, and Managing Delta Tables Like a Production Engineer

Leave a Comment / Azure, Data Engineering

The complete Delta Lake Table Properties reference. Every TBLPROPERTIES setting explained: deletedFileRetentionDuration and logRetentionDuration with VACUUM interaction, Change Data Feed (CDF) for MLVs and streaming, column mapping for rename and drop columns, autoOptimize.optimizeWrite and autoCompact, targetFileSize, protocol versions, schema.autoMerge, and data skipping. Five real-world configurations (Gold SCD dimension, Bronze staging, Silver standard, streaming target, large fact table). Fabric vs Databricks property comparison.

Delta Lake Table Properties: Every TBLPROPERTIES Setting, Retention, Change Data Feed, Column Mapping, Auto-Optimize, and Managing Delta Tables Like a Production Engineer Read More »

Fabric Optimization Guide: Lakehouse, Pipelines, Warehouse, Spark, Eventstream, and Query Performance Tuning Across All Workloads

Leave a Comment / Azure, Data Engineering

The consolidated Fabric optimization reference. Lakehouse: OPTIMIZE, VACUUM, Z-ORDER, V-Order, file sizing, partitioning. Pipelines: parallel execution, scheduling strategy, Copy Activity tuning. Warehouse: statistics, result set caching, query patterns. Spark: shuffle partitions, AQE, broadcast joins, caching. Eventstream: retention, caching policies, materialized views. Complete optimization checklist across all workloads.

Fabric Optimization Guide: Lakehouse, Pipelines, Warehouse, Spark, Eventstream, and Query Performance Tuning Across All Workloads Read More »

Fabric Triggers, Scheduling, and Orchestration: Schedule Triggers, Event-Based Triggers, Tumbling Window Triggers, Notebook Scheduling, and Advanced Orchestration Patterns

Leave a Comment / Azure, Data Engineering

Deep dive into Fabric scheduling and orchestration. Schedule triggers with cron syntax and time zones. Event-based triggers for file arrival and table changes. Tumbling window triggers for historical backfill. Notebook scheduling directly vs via pipeline. Five advanced orchestration patterns: master-child, conditional execution, retry with backoff, fan-out fan-in, cross-pipeline dependency chains. Dynamic scheduling expressions.

Fabric Triggers, Scheduling, and Orchestration: Schedule Triggers, Event-Based Triggers, Tumbling Window Triggers, Notebook Scheduling, and Advanced Orchestration Patterns Read More »

Fabric Monitoring and Troubleshooting: Monitoring Hub, Audit Logs, Error Resolution for Pipelines, Notebooks, Dataflows, Eventstreams, Shortcuts, and Deployment Errors

Leave a Comment / Azure, Data Engineering

Complete Fabric monitoring and troubleshooting manual. Monitoring Hub for all item types. Pipeline, notebook, Dataflow Gen2, semantic model, and Eventstream monitoring. Fabric audit logs for compliance. Error resolution tables for every item type: pipelines, notebooks, Dataflow Gen2, Eventstream, Eventhouse, OneLake shortcuts, T-SQL. Deployment pipeline errors with what can and cannot be deployed. Setting up proactive monitoring.

Fabric Monitoring and Troubleshooting: Monitoring Hub, Audit Logs, Error Resolution for Pipelines, Notebooks, Dataflows, Eventstreams, Shortcuts, and Deployment Errors Read More »

Materialized Lake Views in Fabric: What, When, Why, Bronze-Silver-Gold with MLVs, Automatic Refresh, Data Quality Checks, and Limitations

Leave a Comment / Azure, Data Engineering

Complete Materialized Lake Views guide. What MLVs are and how they differ from regular views and tables. When to use MLVs vs notebooks. Creating MLVs with aggregations and joins. Automatic refresh with Change Data Feed (CDF). Building Bronze-Silver-Gold layers with MLVs. Data quality monitoring MLV. Scheduling and debugging MLVs. MLV limitations.

Materialized Lake Views in Fabric: What, When, Why, Bronze-Silver-Gold with MLVs, Automatic Refresh, Data Quality Checks, and Limitations Read More »

Real-Time Analytics Deep Dive: Window Types, Accelerated Shortcuts, KQL Functions, Materialized Views, and Eventhouse Optimization

Leave a Comment / Azure, Data Engineering

Deep dive into Fabric Real-Time Analytics. Five window types with KQL examples: tumbling, hopping, sliding, session, snapshot. Accelerated vs standard shortcuts in KQL databases. Advanced KQL functions for dates, strings, aggregations, and JSON parsing. Materialized views for pre-computed aggregations. Eventhouse optimization: retention policies, caching policies, partitioning. RTI error resolution table.

Real-Time Analytics Deep Dive: Window Types, Accelerated Shortcuts, KQL Functions, Materialized Views, and Eventhouse Optimization Read More »

Fabric Warehouse Advanced: COPY INTO, CTAS, Dynamic Management Views, Query Insights, Visual Query Editor, SSMS Connectivity, and T-SQL with Notebooks

Leave a Comment / Azure, Data Engineering

Advanced Fabric Warehouse capabilities. COPY INTO for bulk loading from CSV and Parquet. CTAS for materialized summaries and snapshots. Dynamic Management Views for active queries and sessions. Query Insight views for performance analysis. Visual Query editor for no-code analysis. SSMS and Azure Data Studio connectivity with GRANT/DENY. Integrating T-SQL with Spark notebooks via cross-database queries. Warehouse optimization with statistics, result set caching.

Fabric Warehouse Advanced: COPY INTO, CTAS, Dynamic Management Views, Query Insights, Visual Query Editor, SSMS Connectivity, and T-SQL with Notebooks Read More »

Spark Structured Streaming in Fabric: Stateless vs Stateful Transformations, Checkpoints, Output Modes, Windowing, and Processing Real-Time Data with Delta Lake

Leave a Comment / Azure, Data Engineering

Complete Spark Structured Streaming guide. Streaming vs batch same API, reading from Event Hubs, Delta Lake, and files. Output modes (Append, Complete, Update). Checkpoint location for crash recovery. Stateless vs stateful transformations. Windowing: tumbling, sliding/hopping, session windows with watermarks for late data. Writing to Delta tables and foreachBatch MERGE pattern. Trigger modes including availableNow for pipeline scheduling. Three real-world scenarios.

Spark Structured Streaming in Fabric: Stateless vs Stateful Transformations, Checkpoints, Output Modes, Windowing, and Processing Real-Time Data with Delta Lake Read More »

Fabric REST APIs: Programmatic Management, Automating Workspace Setup, Triggering Pipelines, and Building Admin Scripts in Python

Leave a Comment / Azure, Data Engineering

Complete Fabric REST APIs guide. Authentication three ways (Azure AD app, Azure CLI, mssparkutils). Core endpoints: workspace CRUD, item management, pipeline execution and monitoring, lakehouse table operations, capacity management. Six real-world Python scripts: create workspace with items, trigger and monitor pipeline from external system, audit all items across workspaces (CSV export), bulk assign roles, monitor pipeline failures, export table list. GitHub Actions CI/CD integration. Rate limits and best practices.

Fabric REST APIs: Programmatic Management, Automating Workspace Setup, Triggering Pipelines, and Building Admin Scripts in Python Read More »