QuestionQ80

VerifiedIngesting and processing the data

You are creating a report-only data warehouse in which data is streamed into BigQuery through the streaming API. In accordance with Google's best practices, you maintain both a staging table and a production table for the data. How should you design data loading so that there is only one master dataset, without impacting the performance of either ingestion or reporting?

Explanation

Recently streamed BigQuery rows can take up to 90 minutes to become available for copy operations. Consolidating staged rows into the production table and clearing the staging table every three hours leaves production as the only master dataset and batches production writes at a frequency that avoids unnecessary ingestion and reporting overhead. BigQuery also recommends fewer, larger writes to final production tables.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!