DP-750: Implementing Data Engineering Solutions Using Azure DatabricksDemo

By Microsoft · Question Mode

QuestionQ45

Deploy and maintain data pipelines and workloads

You have an Azure Databricks workspace with Unity Catalog enabled.

You need to recommend a pipeline that ingests files from cloud storage, performs cleansing and enrichment transformations, and writes curated Delta tables for analytics. The solution must minimize development effort while providing built-in monitoring and automatic retries.

What should be included in the recommendation?

  • A a Databricks notebook triggered by a scheduled job
  • B an Apache Spark Structured Streaming job
  • C a Lakeflow Spark Declarative Pipelines (SDP) pipeline
  • D an Azure Data Factory pipeline that uses data flows
Explanation

Lakeflow Spark Declarative Pipelines provides a declarative framework for cloud-storage ingestion and batch or streaming transformations that produce curated Delta-backed datasets. It automates dependency orchestration and retries transient failures, while its monitoring UI and event log provide built-in operational visibility and alerts.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!