QuestionQ78

VerifiedServing and scaling models

You work for a large bank that serves customers through an application hosted in Google Cloud and running in the US and Singapore. You developed a PyTorch model that classifies transactions as potentially fraudulent or not. The model is a three-layer perceptron that takes both numerical and categorical features as input, with hashing performed within the model.

You deployed the model in the us-central1 region on nl-highcpu-16 machines, with predictions served in real time. The model’s current median response latency is 40 ms. You want to lower latency, particularly in Singapore, where some customers experience the longest delays. What should you do?

Explanation

Regional deployment places online inference close to the application workload, reducing cross-region network latency. Vertex AI endpoints are identified by project, region, and endpoint; private inference clients must be in the same region as the deployed model. Deploying private endpoints in both us-central1 and asia-southeast1 and routing requests to the appropriate regional endpoint minimizes the distance between each application deployment and its inference service.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!