QuestionQ108

VerifiedIngesting and processing the data

An aerospace company stores its flight data in a proprietary data format. You need to connect this new data source to BigQuery and stream the data into BigQuery. You want to import the data into BigQuery efficiently while using as few resources as possible. What should you do?

Explanation

A custom Apache Beam I/O connector lets a Dataflow streaming pipeline read a proprietary source directly and transform its records for BigQuery. Avro is an efficient BigQuery load format; Google Cloud documentation notes that Avro typically outperforms JSON for file loads.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!