QuestionQ85

VerifiedData Pipeline Orchestration

You have an existing weekly Storage Transfer Service job that transfers data from Amazon S3 to a Nearline Cloud Storage bucket in Google Cloud. Each week, the job moves a large number of relatively small files. As the number of files transferred each week has increased over time, you risk not completing the transfer within the allocated time frame. You need to reduce the overall transfer time by replacing the process. Your solution should minimize costs where possible. What should you do?

Explanation

Storage Transfer Service imposes a maximum QPS per transfer job. Transfers involving many relatively small files can therefore be QPS-limited. Splitting the workload into multiple parallel jobs, each scoped with include and exclude prefixes, works around that limit while retaining the managed, agentless S3-to-Cloud Storage transfer service.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!