ProbXiv
Explorersign in
Problem archiveProblem record

Statement

For a distinguished site i∗ = ⌊\sigma0 N ⌋, with \sigma0 \in (0, 1), we ask whether the last token \thetaN (t) at time t decodes to the initial token at position i∗ . Large language models based on causal transformers are routinely deployed in regimes where the available context is much longer than the part of the prompt that actually matters for the final prediction. A striking empirical observation is the so-called lost-in-the-middle effect: retrieval is typically best when the rel- evant information is placed near the beginning or the end of the context, and substantially worse when it is placed in the middle [6]. The aim of this work is to give a rigorous mathematical explanation of this phenomenon in a minimal continuous-time model for causal self-attention with recency bias. Our starting point is the interacting particle system viewpoint on transformers developed in [3, 4, 5]. We consider a decoder-type dynamics on the circle T = R/2\piZ, where the jth token is represented by an angle \thetaj (t) and only attends to its predecessors. After reducing the original spherical model to dimension 2, and replacing the random softmax denominator by a deterministic normalization, we arrive at j X \lambda e− N (j−k) \thetȧj (t) = Pj \lambda W\beta′ (\thetaj (t) − \thetak (t)), W\beta (x) = e\beta cos x . e −N (j−ℓ) k=1 ℓ=1

Record

Comments

Nothing has been published against this problem yet, and nothing has been said about it.

Sign in with an institutional address to take part in the discussion. Reading every thread stays open to everyone.

Sign in

Solve with an agent

Open the statement in a chat, with the problem and the ground rules already written into the prompt.

This opens a third-party site. Nothing is posted back to ProbXiv and nothing you write there is recorded here — what a model gives you is an attempt, which a person still has to check.