QuestionQ75

VerifiedData Operations and Support

A company wants to create a dimension table in an Amazon S3 bucket. The bucket holds historical data containing 10 million records. The historical data is 1 TB in size.

A data engineer needs a solution that updates changes to up to 10,000 records in the base table each day.

Which solution meets this requirement with the LOWEST runtime?

Explanation

Apache Hudi supports record-level inserts, updates, upserts, and deletes for datasets in Amazon S3. Its table metadata and incremental update capabilities allow a Spark job on Amazon EMR to apply the small daily change set without repeatedly processing the entire 1 TB historical dataset, yielding lower runtime than DataFrame-based full-table update approaches.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!