Name: Channel Capacity
Author: parcadei

When to Use

Use this skill when working on channel-capacity problems in information theory.

Decision Tree

Mutual Information
- I(X;Y) = H(X) + H(Y) - H(X,Y)
- I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)
- Symmetric: I(X;Y) = I(Y;X)
- scipy.stats.entropy(p) + scipy.stats.entropy(q) - joint_entropy
Channel Model
- Input X, output Y, channel P(Y|X)
- Channel matrix: rows = inputs, columns = outputs
- Element (i,j) = P(Y=j | X=i)
Channel Capacity
- C = max_{p(x)} I(X;Y)
- Maximize over input distribution
- Achieved by capacity-achieving distribution
Common Channels

Channel Capacity
Binary Symmetric (BSC)