DP-750: Implementing Data Engineering Solutions Using Azure DatabricksDemo

Overview

Company Information

Contoso, Inc. is a renewable energy provider operating solar and wind farms across North America.

Existing Environment

Azure Environment

Contoso has a single Azure Databricks workspace named Workspace1 in the West US Azure region. Workspace1 is enabled for Unity Catalog.

Workspace1 contains all-purpose clusters for both development and production workloads.

The company’s Azure environment includes:

• In the West US, Central US, and East US Azure regions, Azure event hubs that stream telemetry data and an Azure Data Lake Storage Gen2 account in each region for each hub
• A single Azure SQL database in the West US region that hosts enterprise resource planning (ERP) data
• An Azure Database for PostgreSQL server in the West US region that stores operational maintenance data

Data Environment

Contoso ingests the following operational and business data:

• Telemetry data: More than 40,000 IoT sensors across 28 sites emit JSON telemetry events every few seconds. Each site sends the events to the nearest event hub, which writes the data into the corresponding Data Lake Storage Gen2 account. These files often experience schema drift.
• Maintenance logs: Maintenance systems produce historical repair logs, daily incremental updates, technician notes, and unstructured attachments that are stored in the Data Lake Storage Gen2 accounts.
• Operational maintenance data: Structured operational maintenance data is stored on the Azure Database for PostgreSQL server.
• External weather data: Hourly weather forecasts are retrieved from a REST API and written to the Data Lake Storage Gen2 accounts.
• ERP data: Daily CSV extracts of 50 to 100 GB contain equipment metadata, work orders, and purchase order information.

Problem Statements

The company’s current analytics environment has several issues:

Ingestion

• Telemetry pipelines fall behind during peak loads.
• Telemetry ingestion fails when schema drift occurs.
• Streaming pipelines reprocess events after a pipeline restarts.

Compute

• Production and development workloads run on the same all-purpose clusters.
• Production and development workloads do NOT support autoscaling or workload isolation.

Governance

• The ERP data is duplicated across systems and development teams.
• Naming conventions are inconsistent across development teams, regions, and products.
• Ownership of the IoT sensors changes over time, and analysts must track the full history of the ownership.
• Occasionally, equipment manufacturers must correct data-entry mistakes in equipment names. Historical values are NOT required.

Pipeline operations

• Pipelines lack resiliency, alerting, and centralized scheduling.

Requirements

Planned Changes

Contoso plans to implement the following changes:

• Implement scalable data pipeline orchestration.
• Create a managed analytics catalog in Unity Catalog.
• Implement a consistent approach to creating curated datasets.
• Establish a centralized governance model across ingestion, cleansed, and curated layers.
• Grant data engineers access to the ERP tables by using minimal development effort.
• Adopt a compute strategy that isolates production workloads and supports autoscaling.
• Adopt a slowly changing dimension (SCD) approach to address current data modeling issues.

Technical Requirements

Contoso identifies the following environment and compute requirements:

• Ensure that production ingestion workloads run on compute clusters that can scale automatically during telemetry spikes.
• Provide fast and consistent performance for business intelligence (BI) workloads.
• Prevent development activity from affecting production pipelines.
• Production ingestion workloads must run as scheduled, non-interactive pipelines rather than on shared interactive development clusters.

Contoso identifies the following data ingestion and processing requirements:

• Auto-scale ingestion pipelines to handle bursty workloads.
• Handle schema drift for the maintenance and telemetry data.
• Ingest file-based telemetry data by using minimal operational effort.
• Store all ingested data in a format that supports incremental processing.
• Support the continuous ingestion of telemetry data from the event hubs by using exactly-once semantics.
• Support the ingestion of the structured maintenance data from the Azure Database for PostgreSQL server.
• Build a new telemetry pipeline that ingests raw events from the event hubs, cleanses the data, and publishes curated tables to Unity Catalog.
• Ensure that the Apache Spark Structured Streaming pipelines reading from the event hubs write the data into a managed Delta table named telemetry.raw_events. The pipelines must support schema drift and resume processing after failures without reprocessing the data.

Contoso identifies the following data modeling and optimization requirements:

• Build curated tables that standardize business logic.
• Overwrite equipment metadata attributes, such as name, manufacturer, model, and commissioning date, when the attributes change. Historical values are NOT required.

Contoso identifies the following pipeline deployment and operation requirements:

• Orchestrate multi-step ingestion and transformation workflows.
• Define a clear execution order and dependencies.
• Automatically retry failed steps and notify operators.
• Schedule ingestion and transformation workloads consistently.

Governance Requirements

Contoso identifies the following governance requirements:

• Centralize the metadata catalog.
• Provide isolated development areas that follow standard naming conventions.
• Establish a consistent structure for organizing raw, cleansed, and curated data.
• Provide a read-only mechanism to reference the ERP data through a foreign catalog.

Business Requirements

Contoso identifies the following business requirements:

• Improve ingestion reliability and reduce operational effort.
• Standardize data definitions across development teams.