For a distinguished site i∗ = ⌊\sigma0 N ⌋, with \sigma0 \in (0, 1), we ask whether the last token \thetaN (t) at time t decodes to the initial token at position i∗ .
Statement
For a distinguished site i∗ = ⌊\sigma0 N ⌋, with \sigma0 \in (0, 1), we ask whether the last token \thetaN (t) at time t decodes to the initial token at position i∗ . Large language models based on causal transformers are routinely deployed in regimes where the available context is much longer than the part of the prompt that actually matters for the final prediction. A striking empirical observation is the so-called lost-in-the-middle effect: retrieval is typically best when the rel- evant information is placed near the beginning or the end of the context, and substantially worse when it is placed in the middle [6]. The aim of this work is to give a rigorous mathematical explanation of this phenomenon in a minimal continuous-time model for causal self-attention with recency bias. Our starting point is the interacting particle system viewpoint on transformers developed in [3, 4, 5]. We consider a decoder-type dynamics on the circle T = R/2\piZ, where the jth token is represented by an angle \thetaj (t) and only attends to its predecessors. After reducing the original spherical model to dimension 2, and replacing the random softmax denominator by a deterministic normalization, we arrive at j X \lambda e− N (j−k) \thetȧj (t) = Pj \lambda W\beta′ (\thetaj (t) − \thetak (t)), W\beta (x) = e\beta cos x . e −N (j−ℓ) k=1 ℓ=1
Record
Comments
Nothing has been published against this problem yet, and nothing has been said about it.
Sign in with an institutional address to take part in the discussion. Reading every thread stays open to everyone.
Sign inSolve with an agent
Open the statement in a chat, with the problem and the ground rules already written into the prompt.