Certified Machine Learning ProfessionalDemo

Question

A Data Scientist trained a regression model with Spark’s MLlib to predict housing prices. The model must be evaluated to establish how accurately it predicts on a testing dataset. The Data Scientist wants to use a standard regression metric for this evaluation and also minimize the required processing overhead.

Which approach meets these needs?

A Convert the testing dataset to a pandas DataFrame and use scikit-learn’s root_mean_squared_error function to calculate the root mean squared error on the testing dataset.
B Use the MLlib’s RegressionEvaluator with the metricName set to rmse to calculate root mean squared error on the testing dataset.
C Implement a Spark UDF to calculate the root mean squared error for each row in the testing dataset and aggregate these values to obtain the overall RMSE score.
D Use the MLlib’s RankingEvaluator with the metricName set to rmse to calculate root mean squared error on the testing dataset.

Explanation

Spark MLlib’s RegressionEvaluator evaluates regression output from Spark datasets and supports rmse as the root mean squared error metric. Using this built-in distributed evaluator avoids the data transfer and driver-memory overhead of converting to pandas, as well as the extra computation and serialization overhead of a custom UDF.

Learn more

RegressionEvaluator — PySpark documentation

Certified Machine Learning ProfessionalDemo

AstroTutor

Notes

Readiness

QuestionQ139

Community Discussion