What should a Data Engineer create to build a pipeline that runs a stored procedure to parse and load unstructured data files as they arrive in a stage?
A directory table catalogs files in a stage, and a stream on that directory table tracks changes such as newly added files. A task can consume those stream changes and invoke the stored procedure, providing an automated file-processing pipeline for unstructured data. Snowpipe is intended for continuous execution of COPY INTO loading rather than calling an arbitrary stored procedure.
COPY INTO
Community Discussion