QuestionQ37

Data Storage and Processing

A retail company operates more than 3000 stores, all using the same Point Of Sale (POS) system. It wants to provide category managers with near-real-time sales results. The stores span multiple time zones and have a varying number of transactions per minute, with some stores generating higher sales volumes than others.

Sales results are delivered uniformly through data-engineered fields calculated in a complex data pipeline. The calculations include exceptions, aggregations, and scoring through external functions connected to scoring algorithms. The source data used for aggregations contains more than 100M rows.

Each minute, the POS sends files containing all sales transactions to a cloud-storage location. A naming convention includes store numbers and timestamps to identify the transaction set in each file. The files are generally smaller than 10MB.

How can near-real-time results be delivered to the category managers?

Choose two
Explanation

Snowpipe with AUTO_INGEST = TRUE uses cloud-storage event notifications to continuously load newly arrived files into a target table. A table stream captures the inserted change records and exposes stream metadata, while a task can run on the required schedule to consume those changes and perform the downstream transformations and scoring. This supports automated file ingestion plus near-real-time incremental processing without external polling or concatenating files.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!