Professional Data Engineer

By Google · Question Mode

QuestionQ9

Maintaining and automating data workloads

You are moving a large number of files from a public HTTPS endpoint to Cloud Storage. The files are safeguarded from unauthorized access by using signed URLs. You created a TSV file containing the object-URL list and started a transfer job using Storage Transfer Service. You observe that the job ran for a long time and ultimately failed. The transfer-job logs show that it was running successfully until a certain point, after which it failed with HTTP 403 errors for the remaining files. You confirmed that the source system had not changed. You need to resolve the issue so that you can resume the migration process. What should you do?

Explanation

Signed URLs grant access only until their configured expiration time. A long-running transfer can therefore receive HTTP 403 responses for objects whose signed URLs expire before Storage Transfer Service fetches them. Regenerating signed URLs with a longer validity period for the remaining objects restores authorization, and splitting the work into smaller parallel jobs shortens the time before each URL is used.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!