Optimal chunking at surprisal peaks
Statement
Let be the token at position , let be its conditional entropy, and let be the LAWS trie with description length . Optimal chunking at surprisal peaks. The optimal hierarchical decomposition of a LAWS expert library, minimizing total description length under a block-decomposition constraint, places chunk boundaries at positions where is locally maximal.
The sketch relates the gain from a boundary to and identifies a greedy split with Huffman coding. Binary block decomposition is asserted to be covered by a known optimality result, while the extension to -ary splits remains to be established.
Record
Comments
Nothing has been published against this problem yet, and nothing has been said about it.
Sign in with an institutional address to take part in the discussion. Reading every thread stays open to everyone.
Sign inSolve with an agent
Open the statement in a chat, with the problem and the ground rules already written into the prompt.