QuestionQ103

VerifiedIngesting and processing the data

You currently operate a single on-premises Kafka cluster in a data center in the us-east region that ingests messages from IoT devices worldwide.

Because large parts of the globe have unreliable internet connectivity, messages can accumulate at the edge, arrive simultaneously, and create a load spike on your Kafka cluster. This is becoming hard to manage and prohibitively costly. What is the Google-recommended cloud-native architecture for this scenario?

Explanation

An IoT gateway can aggregate data from remote devices and connect directly to Cloud Pub/Sub. Pub/Sub is a highly scalable asynchronous messaging service that decouples publishers from consumers, allowing ingestion bursts to be buffered independently of downstream processing. Dataflow can then read and process the Pub/Sub stream using managed streaming infrastructure.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!