Certified Machine Learning ProfessionalDemo

By Databricks · Question Mode

QuestionQ61

Scaling and Tuning

A Data Scientist needs fine-grained forecasts of clients’ energy consumption by type (that is, residential, industry, commercial, and others) and location at the district level. The data scientist already has a scikit-learn model that was tested on a small data subset and wants to apply it for this fine-grained forecasting. They have a cluster with eight executors available to perform the work.

The existing function is named forecast_consumption, and the Spark DataFrame that contains historic data is named history_df; client_type is the client-type column and district_id is the district-ID column.

Which code block would correctly complete this task with the least effort while using all available resources?

  • A
  • B
  • C
  • D
Explanation

A grouped-map pandas UDF applies a pandas DataFrame function independently to each district_id and client_type group, allowing the existing pandas/scikit-learn forecasting logic to run in Spark-distributed tasks. Declaring the output schema and using groupBy(...).apply(...) is the supported grouped pandas-UDF pattern; it avoids collecting the full dataset or group keys to the driver and avoids rewriting the model in Spark MLlib.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!