Certified Generative AI Engineer AssociateDemo

Question

A Generative Al Engineer is building an LLM-based application. The documents used by its retriever have been chunked to a maximum of 512 tokens each. The Generative Al Engineer knows that cost and latency matter more than quality for this application. Several context-length levels are available.

Which option will meet their need?

A context length 514; smallest model is 0.44GB and embedding dimension 768
B context length 2048: smallest model is 11GB and embedding dimension 2560
C context length 32768: smallest model is 14GB and embedding dimension 4096
D context length 512: smallest model is 0.13GB and embedding dimension 384

Explanation

Because cost and latency are prioritized over quality, the engineer should choose the smallest, cheapest, lowest-latency embedding model whose context window still fits the data. The chunks are a maximum of 512 tokens, so a context length of exactly 512 (option D) accommodates them fully. Option D also has by far the smallest model size (0.13GB) and the smallest embedding dimension (384), which minimizes storage, compute, and inference latency. Option A (context 514, 0.44GB, dim 768) offers only trivially more context but a much larger, costlier model, while B and C are far larger models built for much longer contexts that are unnecessary for 512-token chunks.

Learn more

Use a custom embedding model (Databricks Vector Search)

Certified Generative AI Engineer AssociateDemo

AstroTutor

Notes

Readiness

QuestionQ41

Community Discussion