Data Engineering - DriveDataScience

Data Activator in Microsoft Fabric: Set Alerts on Your Data, Trigger Actions Automatically, and Monitor Without Building Dashboards

Leave a Comment / Azure, Data Engineering

Master Data Activator (Reflex) in Fabric. Why dashboards are not enough (security camera vs motion sensor analogy). Core concepts (objects, properties, triggers, actions). Setting up from Power BI, Eventstream, and Fabric items. Trigger types: threshold, percentage change, absence, complex AND/OR conditions. Actions: email, Teams, Power Automate, start pipeline, webhook. Six real-world scenarios (inventory monitoring, revenue drop, IoT temperature, pipeline failure watchdog, churn early warning, data quality). Alert fatigue prevention. Comparison with Pipeline alerts and Power BI alerts.

Data Activator in Microsoft Fabric: Set Alerts on Your Data, Trigger Actions Automatically, and Monitor Without Building Dashboards Read More »

OneLake Deep Dive: Architecture, ADLS Gen2 Compatibility, OneLake File Explorer, Multi-Cloud Shortcuts, Storage Billing, and the Foundation of Microsoft Fabric

Leave a Comment / Azure, Data Engineering

Understand the foundation of Fabric. OneLake vs ADLS Gen2 vs S3 comparison, the hierarchy (tenant, workspace, item, tables, files), ADLS Gen2 API compatibility (same API, different endpoint), accessing from Azure Storage Explorer, AzCopy, Python SDK, and external Databricks. OneLake File Explorer for Windows. Multi-cloud shortcuts with caching. Data Hub for discovery. Storage billing details, soft delete, VACUUM for optimization. Three real-world patterns (centralized, hub-and-spoke, multi-cloud unified).

OneLake Deep Dive: Architecture, ADLS Gen2 Compatibility, OneLake File Explorer, Multi-Cloud Shortcuts, Storage Billing, and the Foundation of Microsoft Fabric Read More »

Apache Spark in Fabric: Runtime Configurations, Starter Pools, Custom Environments, V-Order, Adaptive Query Execution, and Performance Tuning

Leave a Comment / Azure, Data Engineering

Open the Spark black box. Driver and executors explained, starter pools vs custom environments, key configurations (shuffle partitions with pizza analogy, AQE autopilot, auto-optimize, auto-compact, V-Order), broadcast joins, memory management, reading the Spark UI to identify bottlenecks (skew, small files, shuffles), four performance tuning patterns (small data, large data, join, write optimization), Spark Job Definitions for production, and high concurrency mode.

Apache Spark in Fabric: Runtime Configurations, Starter Pools, Custom Environments, V-Order, Adaptive Query Execution, and Performance Tuning Read More »

Fabric Administration and Cost Management: Capacity Units, Throttling, Smoothing, Monitoring, Pause and Resume, and Optimizing Your Fabric Spend

Leave a Comment / Azure, Data Engineering

Master Fabric cost management. Capacity Units explained, F-SKU sizing guide (F2 to F512 with prices), CU consumption by workload type, throttling mechanics (10-min, 60-min, 24-hr windows), smoothing and burst model, monitoring with Capacity Metrics app, pause and resume automation, five cost optimization strategies (right-size, optimize Spark, stagger pipelines, pause dev, optimize Delta), and real-world sizing scenarios.

Fabric Administration and Cost Management: Capacity Units, Throttling, Smoothing, Monitoring, Pause and Resume, and Optimizing Your Fabric Spend Read More »

Fabric Security and Governance: Workspace Roles, OneLake Data Access, Item Permissions, Sensitivity Labels, Purview Integration, and Data Lineage

Leave a Comment / Azure, Data Engineering

Every Fabric security layer explained. Seven layers from workspace roles to sensitivity labels. Workspace roles matrix (Admin, Member, Contributor, Viewer), item permissions, OneLake data access roles for table-level control, RLS in Warehouse and Semantic Models, CLS with GRANT on specific columns, dynamic data masking, sensitivity labels that flow downstream, Purview integration for lineage and catalog, endorsement (Promoted, Certified), and complete real-world security architecture.

Fabric Security and Governance: Workspace Roles, OneLake Data Access, Item Permissions, Sensitivity Labels, Purview Integration, and Data Lineage Read More »

Power BI in Fabric: Direct Lake, Semantic Models, Import vs DirectQuery vs Direct Lake, and Connecting Your Data to Reports

Leave a Comment / Azure, Data Engineering

Master Direct Lake in Fabric. Three connection modes compared (Import vs DirectQuery vs Direct Lake), how Direct Lake reads Delta files directly from OneLake, when it falls back to DirectQuery with guardrail thresholds, semantic models (auto-generated vs custom), building relationships and DAX measures, connecting Power BI Desktop, RLS with Direct Lake, V-Order optimization, and end-to-end pipeline-to-dashboard scenario.

Power BI in Fabric: Direct Lake, Semantic Models, Import vs DirectQuery vs Direct Lake, and Connecting Your Data to Reports Read More »

Real-Time Intelligence in Microsoft Fabric: Eventstream, Eventhouse, KQL Database, Real-Time Dashboards, and Processing Millions of Events Per Second

Leave a Comment / Azure, Data Engineering

Master Fabric Real-Time Intelligence. Eventstream for no-code streaming ingestion with 12 supported sources and in-flight transformations. Eventhouse for petabyte-scale time-series storage. KQL vs SQL comparison with essential queries. Real-Time Dashboards with auto-refresh. Data Activator for alerts. Four real-world scenarios (IoT factory, clickstream, financial transactions, app logs). The dual-path batch plus real-time architecture.

Real-Time Intelligence in Microsoft Fabric: Eventstream, Eventhouse, KQL Database, Real-Time Dashboards, and Processing Millions of Events Per Second Read More »

Fabric Notebooks: The Complete Guide to Spark Environments, Library Management, mssparkutils, Multi-Language Cells, and Production Notebook Patterns

Leave a Comment / Azure, Data Engineering

The complete Fabric Notebooks guide. Four languages in one notebook (PySpark, SQL, Scala, R) with temp view bridging. Spark Environments for persistent library management. Installing PyPI, wheel, and jar packages. mssparkutils deep dive: fs (file operations), notebook (run, runMultiple, exit), credentials (Key Vault, tokens), env (workspace info). Notebook chaining: percent-run vs mssparkutils.notebook.run vs runMultiple for parallel. The Config-Functions-Main production pattern. Parameters from pipelines with widgets. Session management, Spark configuration for performance, and error handling patterns.

Fabric Notebooks: The Complete Guide to Spark Environments, Library Management, mssparkutils, Multi-Language Cells, and Production Notebook Patterns Read More »

Fabric Git Integration and Deployment Pipelines: Version Control, CI/CD, and Promoting Changes from Dev to UAT to Production

Leave a Comment / Azure, Data Engineering

Complete Fabric CI/CD guide. Git Integration for source control: connecting to Azure DevOps and GitHub, commit and sync workflow, branching strategies with feature branch workflow, conflict resolution. Deployment Pipelines for release: creating stages (Dev, Test, Prod), assigning workspaces, one-click deployment, deployment rules for environment-specific connections, selective deployment. Three real-world scenarios (new pipeline development, hotfix, team of 5 collaboration). API automation, recommended production setup, and the complete flow from feature branch to production.

Fabric Git Integration and Deployment Pipelines: Version Control, CI/CD, and Promoting Changes from Dev to UAT to Production Read More »

Mirrored Databases in Microsoft Fabric: Real-Time Replication from SQL Server, Cosmos DB, Snowflake, and PostgreSQL Without Building a Single Pipeline

Leave a Comment / Azure, Data Engineering

Master Fabric Mirrored Databases. What mirroring is and how it eliminates ingestion pipelines, how it works under the hood (transaction log, initial snapshot, continuous CDC). Setup guides for all sources: Azure SQL Database, SQL Server 2016-2025, Cosmos DB (JSON to Delta conversion), Snowflake, and PostgreSQL. Querying mirrored data via SQL endpoint, Spark notebooks, and Power BI Direct Lake. Mirroring plus shortcuts for multi-cloud joins. Four real-world scenarios (e-commerce, banking, IoT, multi-cloud). Mirroring vs pipelines comparison and when to use both. Free compute, limitations, and security considerations.

Mirrored Databases in Microsoft Fabric: Real-Time Replication from SQL Server, Cosmos DB, Snowflake, and PostgreSQL Without Building a Single Pipeline Read More »