KV cache
A key-value (KV) cache stores the attention keys and values computed for earlier tokens in a sequence.
During text generation, a Transformer reuses these cached keys and values to produce the next token instead of recomputing them 1.
The cache is working state derived from the sequence, distinct from the model's learned weights. Its values depend on the tokens and their positions, and its memory use can grow with the prompt and generated output 1. The vLLM serving system can share cached prefixes across requests 1.
Related
Used in
Sources
- Tier A. W. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the 29th Symposium on Operating Systems Principles (SOSP 2023). Source recordSupports: cached attention keys and values; per-request state distinct from static model weights; dependence on tokens and positions; memory use and shared prefixes · §1; §2.2; §3; §4.4 'Shared prefix'