QuestionQ102

VerifiedIngesting and processing the data

An external customer sends you a daily database data dump. The data lands in Google Cloud Storage (GCS) as comma-separated values (CSV) files. You want to analyze the data in Google BigQuery, but some rows might be incorrectly formatted or corrupted. How should you build this pipeline?

Explanation

A batch Dataflow pipeline can ingest CSV files from Cloud Storage into BigQuery while separating rejected CSV records into a bad-records/dead-letter table for later analysis. The Google-provided Cloud Storage CSV-to-BigQuery template supports a badRecordsOutputTable specifically for rejected data, allowing valid records to be loaded without losing visibility into malformed or corrupted rows.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!