QuestionQ118
VerifiedMaintaining and automating data workloadsYou need to schedule multiple sequential loading and transformation jobs. An upstream process will add data files to a Cloud Storage bucket, but there is no set schedule for when new data arrives. Next, a Dataproc job is triggered to perform transformations and write the data to BigQuery. You must then run additional transformation jobs in BigQuery. The transformation jobs differ for every table. These jobs can take hours to finish. You need to identify the most efficient and maintainable workflow for processing hundreds of tables while delivering the freshest data to end users. What should you do?
Community Discussion