Certified Data Engineer ProfessionalDemo

A junior data engineer must build a streaming data pipeline using DataFrame df that performs a grouped aggregation. The pipeline must calculate average humidity and average temperature for every non-overlapping five-minute interval. Incremental state data must be retained for 10 minutes to accommodate late-arriving data.

The streaming DataFrame df has this schema:

device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT

Code block:

Choose the response that correctly fills the blank in the code block to finish this task.

A small United States-based company has recently engaged a consulting firm in India to implement several new data-engineering pipelines that will power artificial-intelligence applications. All company data is stored in regional cloud storage in the United States.

The company’s workspace administrator is unsure where the Databricks workspace used by the contractors should be deployed.

Assuming all data-governance considerations have been addressed, which statement correctly guides this decision?

The data architect has required that every table in the Lakehouse be configured as an external (also called “unmanaged”) Delta Lake table.

Which approach ensures this requirement is satisfied?

The security team is investigating whether the Databricks secrets module can be used to connect to an external database.

After testing the code with all Python variables defined as strings, they upload the password to the secrets module and set the correct permissions for the currently active user. They then update their code as follows, leaving every other variable unchanged.

Which statement explains what will occur when the preceding code runs?

A Delta Lake table was created by using the query below:

After noticing that the original query contained a typographical error, the following code was run:

ALTER TABLE prod.sales_by_stor RENAME TO prod.sales_by_store

What result occurs after the second command runs?

A production cluster contains 3 executor nodes and uses the same virtual machine type for both the driver and executors.

When evaluating the Ganglia Metrics for this cluster, which indicator would signal a bottleneck caused by code running on the driver?

How do the operational characteristics of Lakeflow Spark Declarative Pipelines differ from those of Spark Structured Streaming?

A Databricks user who is new to the platform is attempting to troubleshoot lengthy execution times for pipeline logic they are developing. Currently, the user executes code one cell at a time and uses display() calls to verify that, as new transformations are added to an operation, the code produces logically correct results. To measure average execution time, the user runs every cell interactively several times.

Which adjustment will provide a more accurate measurement of how the code is likely to perform in production?

All records from an Apache Kafka producer are ingested into one Delta Lake table with this schema:

key BINARY, value BINARY, topic STRING, partition LONG, offset LONG, timestamp LONG

Five unique topics are being ingested. Only the registration topic contains Personally Identifiable Information (PII). The company wants to restrict access to PII. It also wants to retain records containing PII in this table for only 14 days after initial ingestion. However, it wants to retain non-PII records indefinitely.

Which of the following solutions meets these requirements?

When scheduling Structured Streaming jobs in production, which configuration automatically recovers from query failures while keeping costs low?

Which statement describes a primary technical challenge of ensuring consistent PII masking across every node in large-scale, distributed Databricks batch and streaming pipelines?

A nightly job ingests data into a Delta Lake table using the following code:

The next pipeline step requires a function that returns an object that can be used to manipulate new records that have not yet been processed into the next table in the pipeline.

Which code snippet completes this function definition?

def new_records():  

The data architect has required that every table in the Lakehouse be configured as an external (also called “unmanaged”) Delta Lake table.

Which approach will ensure this requirement is satisfied?

A junior data-engineering team member is investigating language interoperability in Databricks notebooks. The desired result of the code below is to register a view containing all sales made in countries from the continent of Africa that are present in the geo_lookup table.

Before the code is run, SHOW TABLES in the current database shows that it contains only two tables: geo_lookup and sales.

Which statement accurately describes the result of running these command cells sequentially in an interactive notebook?

A data engineer is ingesting JSON files from cloud object storage by using Databricks Auto Loader. The source folder may occasionally receive large data files, risking an overwhelmed stream. To ensure predictable micro-batch sizes, the team wants to throttle ingestion at 1 GB based on the volume of data scanned, irrespective of the number of files.

Which Auto Loader configuration should the data engineer use to accomplish this?

A data engineering team needs to implement a tagging system for its tables as part of an automated ETL process and must apply tags to Unity Catalog tables programmatically.

Which SQL command programmatically adds tags to a table?

Which statement describes the general programming model used by Spark Structured Streaming?

A junior data engineer is implementing logic for a Lakehouse table named silver_device_recordings. The source data has 100 unique fields within a highly nested JSON structure.

The silver_device_recordings table will be used downstream to support several production monitoring dashboards and a production model. Currently, 45 of the 100 fields are used by at least one of these applications.

The data engineer is determining the best approach to schema declaration given the data’s highly nested structure and numerous fields.

Which of the following correctly states information about Delta Lake and Databricks that could affect their decision-making process?

The data engineering team has configured a job to handle customer requests to be forgotten (to have their data deleted). All user data requiring deletion is stored in Delta Lake tables with the default table settings.

The team has chosen to process all deletions from the prior week as a batch job at 1am every Sunday. This job runs for less than one hour in total. Each Monday at 3am, another batch job runs a series of VACUUM commands on all Delta Lake tables across the organization.

The compliance officer has recently become aware of Delta Lake's time-travel functionality and is concerned it could permit ongoing access to deleted data.

Assuming that all deletion logic is implemented correctly, which statement correctly addresses this concern?

A platform-team lead is responsible for automating attribution of SQL Warehouse usage to individual teams. The requirement is to identify SQL warehouse usage at the individual-user level and produce a daily report to share with an executive team that includes leaders from every business unit.

How should the platform lead create an automated report that can be shared daily?

A Delta Lake table named customer_churn_params in the Lakehouse is used by the machine learning team for churn prediction. The table holds customer information derived from several upstream sources. At present, the data engineering team populates this table nightly by overwriting it with the currently valid values derived from upstream data sources.

Immediately after every successful update, the data engineering team wants to determine the difference between the new table version and its preceding version.

Given the current implementation, which method can be used?

The data engineering team has been assigned to configure connections to an external database for which Databricks has no supported native connector. The external database already has data security configured through group membership. These groups directly map to user groups already created in Databricks that represent different teams within the company.

A new login credential has been created for every group in the external database. The Databricks Utilities Secrets module will be used to make these credentials available to Databricks users.

Assuming all credentials are configured correctly on the external database and group membership is properly configured in Databricks, which statement describes how teams can be granted the minimum necessary access to use these credentials?

Given the following PySpark code in a Databricks notebook:

filtered_df=spark.read.format(“delta”).load(“/mnt/daya/large_table”) \  
.filter (“event_date> ‘2024-01-01’”)  
filtered_df.count ()  

The data engineer sees in the Query Profile that the scan operator for filtered_df reads nearly all files despite the filter being applied.

What is the likely reason for ineffective data skipping?

Which statement characterizes Delta Lake Auto Compaction?