QuestionQ36
VerifiedData Pipeline OrchestrationYou are designing a pipeline to process data files that arrive in Cloud Storage by 3:00 am daily. Processing occurs in stages, with each stage’s output serving as input to the following stage. Every stage requires a long time to run. Occasionally, a stage fails and the issue must be resolved. You need to ensure the final output is produced as quickly as possible. What should you do?
Community Discussion