Partial information decomposition

Partial Information Decomposition is an extension of information theory, that aims to generalize the pairwise relations described by information theory to the interaction of multiple variables.^[1]

Information theory can quantify the amount of information a single source variable ${\displaystyle X_{1}}$ has about a target variable ${\displaystyle Y}$ via the mutual information ${\displaystyle I(X_{1};Y)}$. If we now consider a second source variable ${\displaystyle X_{2}}$, classical information theory can only describe the mutual information of the joint variable ${\displaystyle \{X_{1},X_{2}\}}$ with ${\displaystyle Y}$, given by ${\displaystyle I(X_{1},X_{2};Y)}$. In general however, it would be interesting to know how exactly the individual variables ${\displaystyle X_{1}}$ and ${\displaystyle X_{2}}$ and their interactions relate to ${\displaystyle Y}$.

Consider that we are given two source variables ${\displaystyle X_{1},X_{2}\in \{0,1\}}$ and a target variable ${\displaystyle Y=XOR(X_{1},X_{2})}$. In this case the total mutual information ${\displaystyle I(X_{1},X_{2};Y)=1}$, while the individual mutual information ${\displaystyle I(X_{1};Y)=I(X_{2};Y)=0}$. That is, there is synergistic information arising from the interaction of ${\displaystyle X_{1},X_{2}}$ about ${\displaystyle Y}$, which cannot be easily captured with classical information theoretic quantities.

Partial information decomposition further decomposes the mutual information between the source variables ${\displaystyle \{X_{1},X_{2}\}}$ with the target variable ${\displaystyle Y}$ as

${\displaystyle I(X_{1},X_{2};Y)={\text{Unq}}(X_{1};Y\setminus X_{2})+{\text{Unq}}(X_{2};Y\setminus X_{1})+{\text{Syn}}(X_{1},X_{2};Y)+{\text{Red}}(X_{1},X_{2};Y)}$

Here the individual information atoms are defined as

• ${\displaystyle {\text{Unq}}(X_{1};Y\setminus X_{2})}$ is the unique information that ${\displaystyle X_{1}}$ has about ${\displaystyle Y}$, which is not in ${\displaystyle X_{2}}$
• ${\displaystyle {\text{Syn}}(X_{1},X_{2};Y)}$ is the synergistic information that is in the interaction of ${\displaystyle X_{1}}$ and ${\displaystyle X_{2}}$ about ${\displaystyle Y}$
• ${\displaystyle {\text{Red}}(X_{1},X_{2};Y)}$ is the redundant information that is in both ${\displaystyle X_{1}}$ or ${\displaystyle X_{2}}$ about ${\displaystyle Y}$

There is, thus far, no universal agreement on how these terms should be defined, with different approaches that decompose information into redundant, unique, and synergistic components appearing in the literature.^[1][2][3][4]

Despite the lack of universal agreement, partial information decomposition has been applied to diverse fields, including climatology,^[5] neuroscience^[6][7][8] sociology,^[9] and machine learning^[10] Partial information decomposition has also been proposed as a possible foundation on which to build a mathematically robust definition of emergence in complex systems^[11] and may be relevant to formal theories of consciousness.^[12]

• Mutual information
• Total correlation
• Dual total correlation
• Interaction information