Professional Data Engineer

By Google · Question Mode

QuestionQ50

Designing data processing systems

You are designing a data-processing pipeline. The pipeline must automatically scale as load grows. Messages must be processed at least once and be ordered within 1-hour windows. How should you design the solution?

Explanation

Cloud Pub/Sub provides at-least-once message delivery and can preserve order for related messages by using ordering keys. Cloud Dataflow is managed streaming processing that autoscales workers and supports windowing unbounded data into timestamp-based windows, including fixed one-hour windows.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!