Information Theory Foundations

When to Apply

Apply information-theory when:

• Compressing prompts, retrieval contexts, logs, or feature sets
• Drift detection — distribution shift from baseline (KL, JS divergence)
• Feature selection by mutual information with target
• Retrieval re-ranking, MMR, or diversity-aware candidate selection
• Prompt-quality diagnosis via output-conditional entropy / Fano bound
• Hallucination / abstention gating via semantic entropy over meaning-clustered samples (#1)
• RL post-training diagnostics — policy-entropy collapse is the dominant failure mode in RLVR (#1)
• Agent-to-agent message budgets and KV-cache handoff sizing, framed as a bottleneck/rate problem (#6, #8)

Skip and use simpler alternatives when:

• Question is about causation, not information — use foundations-causal-inference
• Single-feature linear correlation is sufficient — Pearson r is suitable for linear dependence; rank correlation may suit monotonic dependence, while MI targets broader dependence
• Streaming data with hard latency budget — full MI/KL is too slow; use sketches or sampled approximations
• N samples too small for stable entropy estimate (rule of thumb n > 5 × #bins per variable)
• Problem is system-stability or feedback control — use foundations-control-theory
• Bits/nats unit doesn't map to a business decision — risk of treating it as decoration, not signal

11 applied information-theory primitives for quantifying uncertainty, signal, and compression, backed by a formal theory map. Each primitive solves a specific measurement problem. Primitives are domain-agnostic: the same entropy calculation that budgets a context window also bounds a lossless compressor; mutual information that scores retrieval also measures feature relevance in ML.

Contents

• Quick Reference
• Primitive Index
• Formal Supporting Theory
• Anti-Patterns
• Misuse Boundaries
• Decision Checklist
• Composition Recipes
• Workflow
• ASCII Flow
• Practitioner Judgment
• Navigation
• Related Skills
• Fact-Checking

Quick Reference

Primitive Index

Each primitive is summarized here, expanded in references/primitives-overview.md, and covered by standalone playbooks under assets/templates/information-theory/. Use references/formal-theory-map.md when the task needs theorem assumptions or derivation boundaries.

Formal Supporting Theory

Anti-Patterns

Misuse Boundaries

Decision Checklist

• Uncertainty measurement: Need to quantify how many bits a distribution contains? → Shannon entropy (#1)
• Relevance scoring: Need to measure how much knowing X reduces uncertainty about Y? → mutual information (#2)
• Distribution comparison (asymmetric): Comparing a learned distribution to a reference where direction matters (e.g., RLHF KL penalty)? → KL divergence (#3)
• Distribution comparison (symmetric): Need a symmetric divergence? → JS divergence. Need a proper metric? → square-root JS distance (#3)

Before computing entropy or mutual information on product data, write the random variables and sampling distribution explicitly. A proxy such as token variety, label cardinality, embedding spread, or model confidence is not Shannon entropy or MI unless probabilities and outcomes are defined. Label proxy scores as heuristics and validate them against the decision outcome they are meant to predict.

• Training objective / model evaluation: Computing a loss between predicted and true distribution? → cross-entropy (#4)
• Model comparison across tokenizers: Need tokenizer-neutral perplexity? → bits-per-byte normalization (#4)
• Throughput ceiling: Need the theoretical limit on reliable transmission over a noisy channel? → channel capacity (#5)
• Compression with acceptable loss: Need to find the minimum bitrate for a target distortion? → rate-distortion (#6)
• Model selection / Occam's razor: Choosing between models of different complexity? → MDL (#7)
• Feature / representation compression: Building a compressed representation that retains task-relevant information? → information bottleneck (#8)
• Error lower bound: Need the minimum achievable classification error given residual uncertainty? → Fano's inequality (#9)
• Block code length planning: Determining how many samples are needed for near-optimal source coding? → AEP / typical sets (#10)
• Compression efficiency audit: Measuring how much redundancy remains in a source relative to its entropy? → redundancy / compression (#11)

Composition Recipes

Context-Window Budget

Problem: A retrieval or summarization pipeline fills a context window but needs to prioritize content under a token budget.

Use this foundation only when a distribution-based compression question is defined, or when interpreting a measured quality/budget tradeoff. A single segment has no Shannon entropy or MI without a random-variable model and sampling frame.

Inputs: Representative held-out tasks, candidate segments, actual token budget, task loss/success measure, and essential instructions, permissions, negations and constraints that must survive compression. Rules: Preserve essential constraints first. Benchmark relevance, diversity and surprisal proxies against task outcomes at several budgets, including an unpruned baseline. Unrelated random text can have high entropy while being useless; a predictable negation can be essential. If a joint distribution is supplied, marginal information is I(task;candidate|selected), not a bit value assigned to one document. Density ranking is a heuristic for indivisible budgets and conditional redundancy, not a global optimizer. MDL requires an explicit code/model and compatible bit units; do not compare raw token length with information gain in bits. Outputs: Measured task-loss/budget curve, chosen feasible context, retained constraints, proxy definitions, uncertainty and failure cases. KV-cache pruning and gist compression require their own task/latency benchmarks; high surprisal alone does not justify retaining or evicting a token.

Retrieval Reranking

Define task/query and candidate random variables over a representative sampling frame. Exact marginal information is I(task;candidate|selected); MI relevance minus pairwise MI is only a heuristic and needs validation. High conditional entropy indicates novelty rather than redundancy. For practical retrieval, evaluate diversity/relevance proxies on held-out task success instead of assigning invented bit values to single documents. Drift thresholds and feature-inclusion thresholds must be calibrated to uncertainty, sampling frequency, baseline and decision cost; no universal .05-nat or 10%-of-target-entropy gate is supplied.

Hypothetical feature example: H(churn)=.469 bits and a validated joint estimate H(churn|tickets)=.31 implies MI=.159 bits. This measures statistical dependence, not causation, and does not alone approve feature inclusion or establish a drift alert. KL=.69 nats is not a universal twofold odds shift; direction and full distributions determine divergence.

Prompt Complexity Diagnosis

Problem: A prompt produces high-variance outputs; unclear whether the source is prompt ambiguity, model uncertainty, or stochastic decoding.

Inputs: Fixed prompt, sampled outputs, named sampling law q (including temperature/top-p), evaluated probability law p, sample size, units (bits/nats per sequence or token), and a grounded finite target label Y with available evidence E when classification bounds are requested. Rules: The sample mean −log p(output|prompt) estimates cross-entropy of q relative to p; it estimates sampling entropy only when q=p and the evaluation covers the same events. Report estimation uncertainty. Arbitrary text entropy or confidence is not H(Y|E). For m=|Y|>1, a conservative Fano lower bound is max(0,(H(Y|E)−1)/log₂ m) using bits and the same label/evidence distribution; without that distribution, return “bound unavailable.” Controlled prompt edits, fixed/changed samplers, and model comparisons can test ambiguity, decoding and model hypotheses. Output variability alone cannot identify their causes or rank causally responsible prompt spans. Ground comparisons in held-out task labels/loss and account for confounding changes. Outputs: Sampling/cross-entropy measurements and uncertainty, controlled comparisons, supported hypotheses and unresolved alternatives, task-quality changes, and a Fano bound only where prerequisites hold. Input-span relevance and IB interpretations remain benchmarked proxies unless a joint distribution is defined.

Workflow

1. Identify the measurement problem (uncertainty quantification, distribution comparison, compression bound, model selection, representation learning).
2. Use the Decision Checklist to map the problem to a primitive.
3. Open references/primitives-overview.md for definitions, failure modes, and source anchors.
4. For multi-step pipelines, use the Composition Recipes to stack primitives.
5. Check references/patterns-scenarios-traps.md before applying to production — KL asymmetry, MI estimation bias, tokenizer effects, and finite-blocklength gaps are common failure modes.

ASCII Flow

Uncertainty, channel, or representation question
  -> Define random variables, distributions, code, channel, or embedding
  -> Select measure: entropy, KL, MI, capacity, MDL, bottleneck, or coding bound
  -> Verify estimator and finite-sample assumptions
     +-- estimator biased or data thin -> bound uncertainty or collect more data
     +-- assumptions acceptable -> compute metric
  -> Interpret directionality, units, and production limits

Practitioner Judgment

The formulas are the easy part. What separates a top-tier application of this skill from a decorative one is knowing when the framing is load-bearing, when an MI number can be trusted, and where the metaphor stops.

When Information-Theoretic Framing Helps vs. When It Decorates

Helps — the framing changes a concrete decision:

• A hard numeric budget exists (token limit, storage quota, wire bandwidth) and the question is "how many bits does this actually need" — entropy/rate-distortion bound the answer before you guess at a truncation heuristic.
• Two things must be compared on a common footing that raw scores hide — bits-per-byte across tokenizers, forward vs. reverse KL direction in an RLHF penalty, NCD instead of a hand-built similarity feature.
• A claim of "the model/feature/prompt has enough signal" can be falsified before spending compute — Fano's bound on a fixed feature set, or MI(feature; target) before adding a feature to a pipeline.
• Non-linear dependence matters and correlation would systematically miss it (MI as a first-pass relevance or leakage check, followed by a causal study if intervention claims are needed).

Decorates — the vocabulary is doing rhetorical work instead of analytical work:

• "Entropy" or "information content" is asserted with no distribution, alphabet, or estimator named. If you cannot write down p(x) and X, you have an analogy, not a measurement — say so explicitly rather than borrowing Shannon's authority.
• A result already reached by simpler means gets relabeled ("we picked the shorter prompt" becomes "we minimized MDL") without an actual two-part code or comparison — this adds jargon, not falsifiable content.
• Channel capacity, entropy, or IB is invoked as a one-line justification for a business or design decision with no channel model, noise process, or β sweep behind it.
• Rule of thumb: if the number would not change if you used a different but equally defensible estimator, it was never really an information-theoretic claim.

MI Estimation Is the Load-Bearing Risk — Treat the Estimator as Part of the Result

Every MI number in production is estimator(data, hyperparameters), not I(X;Y). The single most common way this skill gets misused is quoting an MI value as if it were estimator-free. Before trusting one:

• State which estimator produced it (plug-in/discrete, KSG k-NN, MINE/NWJ, InfoNCE, f-DIME) — each has a different bias/variance profile and, for InfoNCE-family estimators, a hard upper bound of log(K) that silently caps the reported value regardless of true MI.
• Report a confidence interval or a permutation-null baseline, not a point estimate — the Abdelaleem-Martini-Nemenman protocol (arXiv:2506.00330) is the current (2026) reference for when a neural MI estimate can be trusted at all: mainly when the true dependence lives in a low-dimensional latent subspace and N is large relative to that subspace's complexity, not the ambient dimension.
• Treat any MI-driven go/no-go decision (drop this feature, prune this KV cache entry, gate this token) as provisional until the estimator has been sanity-checked on a shuffled-label or synthetic-independence baseline that should read ≈0.
• A high-dimensional MI estimate that looks "too clean" (smooth curve, no variance across seeds) is a warning sign, not a reassurance — bias in finite-sample estimators is directional (usually inflates dependence), so noise-free-looking output often means the estimator is confidently wrong rather than precisely right.

Entropy Intuitions for Logging, Observability, and Feature Design

• A log field with H(field) ≈ 0 under normal operation (it almost always takes the same value) is not carrying information about normal operation — its entire value is in the rare cases where it deviates. Prioritize instrumenting fields with high conditional entropy given an incident (status codes, error classes, latency buckets) over fields that are merely present ("function entered" trace lines contribute ceremony, not bits).
• Cardinality blowups in metrics backends are usually an entropy/aggregation mismatch, not a tooling bug: a near-maximum-entropy field (user ID, request ID, raw timestamp) is being used as a group-by dimension meant for low-entropy categorical fields. Recognizing this as "someone is aggregating on a high-H(X) field" reframes a recurring on-call complaint as a design fix (bucket or hash the field) rather than a scaling problem to throw hardware at.
• In feature design, high entropy is necessary but nowhere near sufficient for value: a raw unique ID has maximum H(X) = log(N) and typically MI(X; target) ≈ 0 absent leakage. Always evaluate MI(feature; target), not H(feature) alone — a feature engineer who reports "this field has high entropy" without also reporting its MI with the label has described the feature's cost, not its worth.

Channel-Capacity Thinking for Org and Team Communication (Explicit Analogy, Not a Measurement)

This is a structural metaphor for reasoning about communication design, not a literal application of the noisy-channel coding theorem — there is no rigorously defined p(x,y) for a Slack channel, and computing a fake numeric "capacity" for a team is exactly the decorative misuse this section warns against above. Used as a checklist, the structure still transfers usefully:

• A communication chain's effective throughput is bounded by its noisiest, lowest-bandwidth hop (the one skim-read doc, the meeting half the team missed) — matching capacity-as-a-bound-set-by-the-worst-conditioned-link, not a sum of nominal channel widths.
• Redundancy (the same decision restated in a doc, a Slack post, and a meeting) is not waste; it is the same design tradeoff error-correcting codes make — accepting a lower effective rate in exchange for surviving dropped attention, partial reads, and turnover.
• Compressing a message below what the receiver's shared context can decode does not degrade understanding proportionally — it produces a cliff, the same shape as attempting a rate above capacity: below some density threshold a terse Slack message or cryptic PR title is not "a bit less clear," it is misread entirely.
• Use this framing to generate questions ("what's the noisiest hop in this rollout communication, is there redundancy built in, is this message denser than the reader's available context can decode right now") — not to produce a number. If a number is demanded, that is the signal to say the analogy has been pushed past where it is honest.

Navigation

• scripts/discrete_information.py — deterministic support artifact.

• scripts/test_discrete_information.py — deterministic support artifact.

• Practical completion contract and known-answer controls: references/practical-contract.md.

• Formal theory map: references/formal-theory-map.md

• Patterns, scenarios, and traps: references/patterns-scenarios-traps.md

• Primitives overview: references/primitives-overview.md

• Per-primitive playbooks: assets/templates/information-theory/README.md

• Sources: data/sources.json

Related Skills

(No cross-links at this time. Consumer skills — ai-prompt-engineering, ai-context-layer, dev-context-engineering, data-analytics-engineering, marketing-content-strategy, qa-observability — will receive applied-recipe files in a later wave.)

Fact-Checking

• All formulas and theorems are sourced to Cover & Thomas (2006) 2nd ed. and MacKay (2003) as primary references; verify equation numbers and chapter numbers before using in citations — do not assume a plausible-sounding chapter attribution is correct. A 2026-07-11 audit found and corrected two real instances of this failure mode in this skill's own files: primitives-overview.md had misattributed MacKay Ch.28 ("Model Comparison and Occam's Razor") to information bottleneck when it actually grounds MDL (#7), and 09-fano-inequality.md cited MacKay Ch.8 for Fano's inequality when MacKay's book never derives Fano's inequality at all. Treat every textbook chapter citation in this skill (and any you add) as needing independent verification, not just author/year/title.
• Numeric results (compression ratios, capacity values) are task- and channel-specific; do not transfer benchmarks across domains without re-deriving.
• Semantic entropy (Farquhar et al., Nature 630:625–630, 2024) is peer-reviewed and reproducible, but its scope is narrower than "hallucination detection" implies: it flags confabulations — answers that vary arbitrarily across samples — and is blind to errors the model states consistently. Do not present it as a general factuality check.
• The RLVR entropy law R = −a·e^H + b (Cui et al., arXiv:2505.22617) is an empirical fit across the model families tested, not a theorem. The qualitative claim (performance is traded from entropy; collapse caps gains) replicates widely; the fitted constants a and b do not transfer across setups — refit rather than reusing published values.
• The information bottleneck claims (IB = DNN compression) remain contested as of August 2026; see Saxe et al. (2018) rebuttal before asserting IB explains deep learning generalization, and treat the 2025/2026 reconciliation attempts (NIBT, CIBR, GIB) as partial and estimator-dependent, not a settled resolution — GIB in particular is an unreviewed preprint.
• If web access is unavailable, mark runtime-specific MI estimation results as unverified.
• Source links and verified dates in each per-primitive file are the canonical evidence tier.

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.