QuestionQ43

VerifiedData Ingestion and Transformation

A data engineer needs to orchestrate a group of extract, transform, and load (ETL) jobs running on AWS. The ETL jobs include tasks that must run Apache Spark jobs on Amazon EMR, make API calls to Salesforce, and load data into Amazon Redshift.

The ETL jobs must automatically handle failures and retries. The data engineer must use Python to orchestrate the jobs.

Which service meets these requirements?

Explanation

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) runs Apache Airflow workflows defined in Python. Airflow DAGs support task dependencies, failure handling, and configurable retries. Amazon MWAA supports integrations with Amazon EMR and Amazon Redshift, and Airflow connections/operators can be used for third-party API calls such as Salesforce.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!