KV Cache

Memory usage in transformers

Untitled

Memory usage: $2 * precision * n_{layers} * d_{model} * seqlen * batch$

2 = 2 matrices for K and V

precision = # of bytes per parameter (e.g. 4 for fp32)

n_layers = layers in the model

d_model = dimension of embeddings

seqlen = length of context in tokens

batch = batch size

FlashAttention-2