QuestionQ10

VerifiedData Operations and Support

A company runs an Apache Spark application nightly on an Amazon EMR cluster. The company uses Amazon EC2 instances to provide compute capacity for the EMR cluster. The company has deployed the Spark application in cluster mode.

An error occurs in the Spark application, and an error log is stored in the application's Spark driver standard error logs. A data engineer must investigate the error.

Where can the data engineer locate this error log?

Explanation

For an Amazon EMR Spark application deployed in YARN cluster mode, the Spark driver runs in the ApplicationMaster, which is the first YARN container. The driver’s log output therefore appears in the first YARN container log. Amazon EMR persistent application UIs provide access to YARN container logs.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!