Azure - DriveDataScience

KQL Window Functions: serialize, prev, next, row_number, row_cumsum, row_rank_dense, row_rank_min, row_window_session, scan Operator, and Every Pattern for Fabric Real-Time Analytics

Leave a Comment / Azure, Data Engineering, Microsoft Fabric

The complete KQL window functions reference. The serialize operator and why window functions require it. prev() and next() for accessing adjacent rows. row_number() for sequential numbering. row_cumsum() for running totals. row_rank_dense() and row_rank_min() for ranking (dense vs gaps). row_window_session() for automatic session detection. The scan operator for stateful row processing. Partitioned windows with the restart parameter. KQL vs SQL window function comparison table. Eight real-world patterns, common mistakes, and interview Q&As.

KQL Window Functions: serialize, prev, next, row_number, row_cumsum, row_rank_dense, row_rank_min, row_window_session, scan Operator, and Every Pattern for Fabric Real-Time Analytics Read More »

DP-700 Certification Study Guide: Every Exam Objective Mapped to DriveDataScience Posts, Study Plan, and Tips to Pass the Microsoft Fabric Data Engineer Associate Exam

Leave a Comment / Azure, Data Engineering

The complete DP-700 study guide with every exam objective mapped to 34 DriveDataScience Fabric posts. Three domains: Implement and Manage, Ingest and Transform, Monitor and Optimize. 8-week study plan, quick reference cards, exam-day tips, and 5 practice questions.

DP-700 Certification Study Guide: Every Exam Objective Mapped to DriveDataScience Posts, Study Plan, and Tips to Pass the Microsoft Fabric Data Engineer Associate Exam Read More »

Fabric Data Factory Expression Language: Dynamic Pipelines with @pipeline(), @activity(), @formatDateTime(), Conditional Logic, and Every Expression You Need

Leave a Comment / Azure, Data Engineering

Complete Fabric Data Factory expression language guide. Where expressions work (Copy, Notebook, If Condition, ForEach, Web activities). Pipeline functions (@pipeline, @activity, @variables, @item). String functions (concat, replace, split, trim). Date functions (utcNow, formatDateTime, addDays, startOfMonth). Logical functions (if, equals, coalesce, and, or). Eight real-world patterns: yesterday incremental load, dynamic file paths with date partitioning, conditional full vs incremental, dynamic SQL, ForEach table list, notebook parameters, error handling, dynamic email subjects. ADF vs Fabric comparison (identical syntax).

Fabric Data Factory Expression Language: Dynamic Pipelines with @pipeline(), @activity(), @formatDateTime(), Conditional Logic, and Every Expression You Need Read More »

KQL (Kusto Query Language) Complete Guide: Syntax, Operators, Functions, Joins, Time Series, Anomaly Detection, and Real-World Query Patterns for Fabric Real-Time Intelligence

Leave a Comment / Azure, Data Engineering

The complete KQL reference guide. Pipe-based syntax explained with SQL comparison. Filtering with where (comparison, string operators, has vs contains performance). Selecting with project and extend. Aggregation with summarize (count, dcount, percentile, arg_max, make_list). Time-based analysis with bin and ago. All join types including lookup optimization. Complete function reference: string, date/time, numeric, dynamic/JSON. Let statements for variables and subqueries. Render for inline visualization. Advanced: make-series for time series, series_decompose_anomalies for anomaly detection, materialized views, stored functions. Eight real-world query patterns (latest per device, sessions, error rate, top N, funnel, spike detection, week-over-week, distributed tracing). Full KQL vs SQL comparison table.

KQL (Kusto Query Language) Complete Guide: Syntax, Operators, Functions, Joins, Time Series, Anomaly Detection, and Real-World Query Patterns for Fabric Real-Time Intelligence Read More »

M Language (Power Query) Complete Guide: Every Function You Need, Text, Date, Number, Table Operations, Error Handling, Custom Functions, and Real-World Patterns

Leave a Comment / Azure, Data Engineering

The complete M language reference for Dataflow Gen2. The let-in structure explained. Every essential function: text (20+ functions including split, combine, replace, extract), numbers (round, math, conversion), dates (30+ functions including arithmetic, extraction, start/end of periods, formatting), logical (if-then-else, null coalescing), lists (aggregate, filter, transform), and tables (filter rows, add columns, join with all 6 join kinds, group by, pivot, unpivot, buffer). Error handling with try-otherwise. Custom functions. Query folding explained. Five real-world patterns.

M Language (Power Query) Complete Guide: Every Function You Need, Text, Date, Number, Table Operations, Error Handling, Custom Functions, and Real-World Patterns Read More »

Fabric Connections and Gateways: Connection Types, On-Premises Data Gateway, VNet Gateway, Managing Connections, and Accessing Data Behind Firewalls

Leave a Comment / Azure, Data Engineering

Complete Fabric connections and gateways guide. Connection types for Azure, on-premises, cross-cloud (AWS S3, GCS), and SaaS sources with authentication options. Connection vs ADF Linked Service migration. On-premises data gateway installation, architecture (outbound HTTPS only, no inbound ports), and high-availability clustering. VNet data gateway for private Azure resources. Creating, sharing, and reusing connections. Security best practices (Service Principal, Key Vault). Troubleshooting connection errors. Three real-world scenarios.

Fabric Connections and Gateways: Connection Types, On-Premises Data Gateway, VNet Gateway, Managing Connections, and Accessing Data Behind Firewalls Read More »

Delta Lake Table Properties: Every TBLPROPERTIES Setting, Retention, Change Data Feed, Column Mapping, Auto-Optimize, and Managing Delta Tables Like a Production Engineer

Leave a Comment / Azure, Data Engineering

The complete Delta Lake Table Properties reference. Every TBLPROPERTIES setting explained: deletedFileRetentionDuration and logRetentionDuration with VACUUM interaction, Change Data Feed (CDF) for MLVs and streaming, column mapping for rename and drop columns, autoOptimize.optimizeWrite and autoCompact, targetFileSize, protocol versions, schema.autoMerge, and data skipping. Five real-world configurations (Gold SCD dimension, Bronze staging, Silver standard, streaming target, large fact table). Fabric vs Databricks property comparison.

Delta Lake Table Properties: Every TBLPROPERTIES Setting, Retention, Change Data Feed, Column Mapping, Auto-Optimize, and Managing Delta Tables Like a Production Engineer Read More »

Fabric Optimization Guide: Lakehouse, Pipelines, Warehouse, Spark, Eventstream, and Query Performance Tuning Across All Workloads

Leave a Comment / Azure, Data Engineering

The consolidated Fabric optimization reference. Lakehouse: OPTIMIZE, VACUUM, Z-ORDER, V-Order, file sizing, partitioning. Pipelines: parallel execution, scheduling strategy, Copy Activity tuning. Warehouse: statistics, result set caching, query patterns. Spark: shuffle partitions, AQE, broadcast joins, caching. Eventstream: retention, caching policies, materialized views. Complete optimization checklist across all workloads.

Fabric Optimization Guide: Lakehouse, Pipelines, Warehouse, Spark, Eventstream, and Query Performance Tuning Across All Workloads Read More »

Fabric Triggers, Scheduling, and Orchestration: Schedule Triggers, Event-Based Triggers, Tumbling Window Triggers, Notebook Scheduling, and Advanced Orchestration Patterns

Leave a Comment / Azure, Data Engineering

Deep dive into Fabric scheduling and orchestration. Schedule triggers with cron syntax and time zones. Event-based triggers for file arrival and table changes. Tumbling window triggers for historical backfill. Notebook scheduling directly vs via pipeline. Five advanced orchestration patterns: master-child, conditional execution, retry with backoff, fan-out fan-in, cross-pipeline dependency chains. Dynamic scheduling expressions.

Fabric Triggers, Scheduling, and Orchestration: Schedule Triggers, Event-Based Triggers, Tumbling Window Triggers, Notebook Scheduling, and Advanced Orchestration Patterns Read More »

Fabric Monitoring and Troubleshooting: Monitoring Hub, Audit Logs, Error Resolution for Pipelines, Notebooks, Dataflows, Eventstreams, Shortcuts, and Deployment Errors

Leave a Comment / Azure, Data Engineering

Complete Fabric monitoring and troubleshooting manual. Monitoring Hub for all item types. Pipeline, notebook, Dataflow Gen2, semantic model, and Eventstream monitoring. Fabric audit logs for compliance. Error resolution tables for every item type: pipelines, notebooks, Dataflow Gen2, Eventstream, Eventhouse, OneLake shortcuts, T-SQL. Deployment pipeline errors with what can and cannot be deployed. Setting up proactive monitoring.

Fabric Monitoring and Troubleshooting: Monitoring Hub, Audit Logs, Error Resolution for Pipelines, Notebooks, Dataflows, Eventstreams, Shortcuts, and Deployment Errors Read More »