QuestionQ81

VerifiedIngesting and processing the data

You use a Dataflow streaming job to read messages from a message bus that does not support exactly-once delivery. The job applies transformations and loads the output into BigQuery. You want to ensure the data is streamed into BigQuery with exactly-once delivery semantics. You expect BigQuery ingestion throughput of approximately 1.5 GB per second. What should you do?

Explanation

Dataflow's STORAGE_WRITE_API write mode guarantees exactly-once semantics. BigQuery Storage Write API throughput is 3 GB/s in the US and EU multi-regions, while regional locations have a 300 MB/s quota; therefore, 1.5 GB/s requires a multiregional target table.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!