ProbXiv
Explorersign in
Problem archiveProblem record

Statement

Let tit_i be the token at position ii, let Hi=H(ti∣t<i)H_i=H(t_i\mid t_{<i}) be its conditional entropy, and let T\mathcal{T} be the LAWS trie with description length L(T)L(\mathcal{T}). Optimal chunking at surprisal peaks. The optimal hierarchical decomposition of a LAWS expert library, minimizing total description length under a block-decomposition constraint, places chunk boundaries at positions ii where HiH_i is locally maximal.

The sketch relates the gain from a boundary to I(ti;expert label∣t<i)I(t_i;\text{expert label}\mid t_{<i}) and identifies a greedy split with Huffman coding. Binary block decomposition is asserted to be covered by a known optimality result, while the extension to kk-ary splits remains to be established.

Record

Comments

Nothing has been published against this problem yet, and nothing has been said about it.

Sign in with an institutional address to take part in the discussion. Reading every thread stays open to everyone.

Sign in

Solve with an agent

Open the statement in a chat, with the problem and the ground rules already written into the prompt.

This opens a third-party site. Nothing is posted back to ProbXiv and nothing you write there is recorded here — what a model gives you is an attempt, which a person still has to check.