BigQuery lets you create external (federated) tables that reference data files—including Avro—residing directly in Cloud Storage, without needing to load or copy the data into BigQuery storage. Because Avro is a self-describing format, BigQuery can automatically infer the schema, so analysts can immediately run standard SQL queries against the external table. This approach avoids the time needed to load 5 TB into a native table (and later delete it) and the duplicate BigQuery storage cost, and requires no infrastructure provisioning as would be needed with a Hadoop/Hive cluster, making it the fastest and most cost-effective option that also matches the analysts' SQL-only skill set.
Community Discussion