You are monitoring your organization’s data lake, which is hosted on BigQuery. The ingestion pipelines read data from Pub/Sub and write it into BigQuery tables. After deploying a new version of the ingestion pipelines, the daily stored data grew by 50%. Pub/Sub data volumes remained unchanged, and only some tables had their daily partition data size double. You need to investigate and resolve the cause of the data increase. What should you do?
A
Check for duplicate rows in the BigQuery tables that have the daily partition data size doubled.2. Schedule daily SQL jobs to deduplicate the affected tables.3. Share the deduplication script with the other operational teams to reuse if this occurs to other tables.
B
Check for code errors in the deployed pipelines.2. Check for multiple writing to pipeline BigQuery sink.3. Check for errors in Cloud Logging during the day of the release of the new pipelines.4. If no errors, restore the BigQuery tables to their content before the last release by using time travel.
C
Check for duplicate rows in the BigQuery tables that have the daily partition data size doubled.2. Check the BigQuery Audit logs to find job IDs.3. Use Cloud Monitoring to determine when the identified Dataflow jobs started and the pipeline code version.4. When more than one pipeline ingests data into a table, stop all versions except the latest one.
D
Roll back the last deployment.2. Restore the BigQuery tables to their content before the last release by using time travel.3. Restart the Dataflow jobs and replay the messages by seeking the subscription to the timestamp of the release.
Show Answer Answer Explanation Duplicate data in only a subset of tables after a pipeline deployment indicates that more than one pipeline version may be ingesting the same Pub/Sub data into those tables. Duplicate rows confirm the symptom; BigQuery audit logs identify the jobs associated with table writes, and Dataflow monitoring correlates those jobs with their start times and pipeline versions. Stopping all but the latest version removes the concurrent writer and prevents further duplicate ingestion. BigQuery audit logs record table and job activity, while the Dataflow monitoring interface provides job status and timing information.
Learn more
Community Discussion