QuestionQ82

VerifiedData Pipeline Orchestration

You work for a home-insurance company and are often asked to create and save risk reports, including charts, for particular areas by using a publicly available storm-event dataset. You want to be able to create and rerun those risk reports quickly as new data becomes available. What should you do?

Explanation

BigQuery public datasets are stored by Google in a public project, and you query them in place by their fully qualified table names (for example, bigquery-public-data.dataset.table), paying only for the queries you run. A query against the hosted tables therefore reads whatever the tables contain at run time. Copying the dataset into your own project creates a separate point-in-time copy that does not pick up later updates and adds storage cost. A Colab Enterprise notebook can hold the SQL that queries the referenced dataset, Markdown text that documents the report, and Matplotlib code that draws the charts, all saved together. The notebook can be rerun on demand, or scheduled to run once or on a recurring schedule, so a risk report can be regenerated quickly whenever new data is available. Exporting results to CSV or Google Sheets and building charts from cell data has to be repeated by hand for every refresh.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!