DP-750: Implementing Data Engineering Solutions Using Azure DatabricksDemo

By Microsoft · Question Mode

QuestionQ34

Deploy and maintain data pipelines and workloads

You have an Azure Databricks workspace.

You have an Apache Spark Structured Streaming job named Job1 that continuously processes data and periodically fails because of transient errors.

You need to ensure that Job1 meets the following requirements:

  • Resumes processing from the point at which Job1 failed
  • Minimizes the time required to restart Job1
  • Minimizes the cost of restarting Job1

What should you do?

  • A Decrease the retry interval.
  • B Increase the minimum number of nodes in the cluster.
  • C Add an alert and manually restart Job1.
  • D Implement checkpointing.
Explanation

Checkpointing stores processed source offsets, commit records, and streaming state in fault-tolerant storage. After a failure, Structured Streaming restores that checkpoint so processing continues from the last successful point rather than starting over, which reduces the work, time, and compute cost required for recovery.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!