Professional Data Engineer

Question

A web server publishes click events as messages to a Pub/Sub topic. The server includes an eventTimestamp attribute in each message that records when the click occurred. You have a Dataflow streaming job that reads this Pub/Sub topic through a subscription, performs some transformations, and writes the results to another Pub/Sub topic for the advertising department.

The advertising department must receive every message within 30 seconds of its corresponding click, but reports that messages arrive late. Your Dataflow job has system lag of about 5 seconds and data freshness of about 40 seconds. Inspection of several messages shows no more than 1 second of lag between eventTimestamp and publishTime. What is the issue, and what should you do?

Explanation

Data freshness measures the difference between an element’s event time and processing time; a value of about 40 seconds shows that some elements are being processed too late to meet a 30-second requirement. The approximately 1-second delay before Pub/Sub publication does not account for the lateness, so the pipeline requires performance optimization or additional worker capacity.

Learn more

Professional Data Engineer

AstroTutor

Notes

Readiness

QuestionQ2

Community Discussion