QuestionQ3

VerifiedIntroduction to Generative AI for Software Testing

In the context of Large Language Models (LLMs), which of the following statements (i–v) about tokenization and the context window are correct?

  1. Tokenization is the process of splitting textual input into smaller units called tokens.
  2. The context window determines the maximum number of tokens an LLM can consider at one time to maintain coherence.
  3. Increasing an LLM’s context-window size generally reduces computational complexity and processing time.
  4. Tokenization primarily consists of converting tokens into high-dimensional vectors to capture their semantic relationships.
  5. A larger context window enables an LLM to maintain coherence across longer passages, such as when analyzing large test logs.
Explanation

Tokenization breaks text into tokens, while the context window limits how many tokens the model can process together. Expanding that window allows more of a long passage or log to remain available for coherent analysis, but it generally increases computational cost. Converting tokens into high-dimensional vector representations is the embedding step, not tokenization.

Community Discussion

No comments yet. Be the first to start the discussion!