Surprise is measurable
H = βΞ£ p logβ p. Uniform sources maximize H; skewed sources carry fewer bits per symbol.
The single idea
Information removes uncertainty
Predictable outcomes carry little information; surprises carry more. In systems: flat metrics are dashboard noise; lossy summaries drop distribution tails.
Transmission distorts
Each bit flips with probability p. Without coding BER β p; redundancy enables recovery.
Speed limit
For a binary symmetric channel C β 1 β H(p). You cannot beat that error-free.
Lossy vs lossless
LLM summaries and Grafana rollups are lossy. What did you drop? Does it matter for decisions?
Interactive
Channel Lab β noise and redundancy
Generate a 16-bit message, run it through a channel with flip probability p, compare BER before and after majority decoding.
Channel quality
β
Decoder recovers the message β redundancy pays off.
What you should see
Information in engineering
Same laws show up in monitoring, logs, ML pipelines, and token limits.
Metrics & alerts
Ten KPIs correlated at 0.99 β low mutual information. One signal, noise elsewhere.
Logs & traces
Sampling and rollups are lossy compression. Drop rare tails β drop the incident.
Token budget
Context window = bandwidth. RAG routes signal; prompt injection injects noise.
Embeddings
Vectors compress meaning with loss. Similarity keeps signal; rare facts may vanish.
Cross-domain transfer
Same law β different context
After the lab, map the model to your stack.
ECC & channel
Memory bus BER.
SNR & ADC
Quantization and frontend noise.
CRC / protocol
Corrupted frame = lost symbol.
Metrics & logs
Aggregation = lossy compression.
Self-check
Did I get it?
I can explain why a predictable source has low entropy.
I connect noise p, BER, and capacity C β 1 β H(p).
I see why redundancy/error coding beats Β«just send it againΒ» blindly.
I can name a lossy compression example in AI/observability and what gets lost.