
Memory usage: $2 * precision * n_{layers} * d_{model} * seqlen * batch$
2 = 2 matrices for K and V
precision = # of bytes per parameter (e.g. 4 for fp32)
n_layers = layers in the model
d_model = dimension of embeddings
seqlen = length of context in tokens
batch = batch size