QuestionQ79

VerifiedDesign scalable and highly available cloud database solutions

Your IoT monitoring platform receives sensor data from 100,000 devices, producing 2 million writes per second during peak hours and 500,000 writes per second during off-peak hours. Each record is 1 KB and uses the row-key format deviceId#timestamp.

Your existing 12-node Bigtable cluster has hotspotting in particular device ranges: three nodes handle most traffic while the remaining nodes are underutilized. Read queries generally retrieve the last 24 hours of data for individual devices. Storage cost is also becoming a concern because two years of historical data, totaling 500 TB, is retained.

You need to optimize performance, remove hotspotting, and lower storage costs while keeping read latency below 10 ms. What should you do?

Explanation

A deterministic hash prefix distributes device key ranges more evenly across Bigtable tablets, eliminating the concentration caused by lexicographically clustered device IDs while still allowing a known prefix and timestamp range for a device’s recent-data read. Autoscaling can adjust node capacity to CPU and storage targets as workload volume changes between peak and off-peak periods. Exporting and retaining data older than six months in Cloud Storage reduces costly online Bigtable storage while keeping the recent, latency-sensitive data in Bigtable.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!