QuestionQ83
VerifiedMaintaining and automating data workloadsA financial services company has a critical daily reconciliation process with several distinct steps: retrieving data from an external SFTP server, decrypting the files, loading them into Cloud Storage, and finally executing a series of BigQuery SQL transformations. Each step has strict dependencies, and the entire process must notify you if it is not complete by 7:00 AM. Manual intervention for failures is expensive and delays compliance reporting. You need a highly observable, robust solution that allows individual steps to be easily re-run when errors occur. What should you do?
Community Discussion