QuestionQ167

VerifiedPlanning and implementing a cloud solution

You have a large 5-TB AVRO file stored in a Cloud Storage bucket. Your analysts know only SQL and need to access the data in this file. You want the most cost-effective way to fulfill their request as quickly as possible. What should you do?

Explanation

BigQuery lets you create external (federated) tables that reference data files—including Avro—residing directly in Cloud Storage, without needing to load or copy the data into BigQuery storage. Because Avro is a self-describing format, BigQuery can automatically infer the schema, so analysts can immediately run standard SQL queries against the external table. This approach avoids the time needed to load 5 TB into a native table (and later delete it) and the duplicate BigQuery storage cost, and requires no infrastructure provisioning as would be needed with a Hadoop/Hive cluster, making it the fastest and most cost-effective option that also matches the analysts' SQL-only skill set.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!