QuestionQ78

VerifiedIngesting and processing the data

Your infrastructure team has established an interconnect link between Google Cloud and the on-premises network. You are designing a high-throughput streaming pipeline to ingest streaming data from an Apache Kafka cluster hosted on-premises. You want to store the data in BigQuery with the lowest possible latency. What should you do?

Explanation

Dataflow supports a streaming Apache Kafka-to-BigQuery pipeline, including self-managed or external Kafka clusters, provided the Kafka broker is reachable from Dataflow worker machines. Reading Kafka directly and writing to BigQuery avoids the additional Kafka Connect and Pub/Sub stages, reducing end-to-end latency.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!