Certified Machine Learning ProfessionalDemo

Question

A Data Scientist must run inference on a continuously updated Delta table named sales_data by using an MLflow-registered Spark ML pipeline model (catalog.prod.sales_forecaster). Predictions must be written to the Delta table forecast_results, which must be updated with low latency by leveraging a cluster with three executors. They want to maximize efficient cluster usage while doing so.

Which approach meets these needs?

A
B
C
D

Explanation

mlflow.pyfunc.spark_udf exposes a registered MLflow model as a Spark UDF, allowing inference to run over partitions of a streaming Spark DataFrame and therefore use the cluster executors. Combining that UDF with readStream and a checkpointed Delta writeStream continuously appends low-latency predictions to forecast_results. MLflow documents Spark UDF scoring for Spark clusters, including real-time Spark Streaming jobs, and its Spark UDF API supports calling a model with a struct of feature columns.

Learn more

Certified Machine Learning ProfessionalDemo

AstroTutor

Notes

Readiness

QuestionQ62

Community Discussion