Vtyjkyuk

The information inequality is a theorem that shows that the Kullback-Leibler divergence between two probability distributions is always non negative. This can be proved easily using the Jensen's inequality with the $-log$ function, but the proofs I have read always have to distinguish if the probability distributions are defined by continuous or discrete random variables.

I was trying to see if it is possible to not distinguish cases, and I thinked about using measure theory, with the Jensen's inequality provided in Rudin's book:

Let $mu$ be a probability measure on a $sigma$-algebra $mathcalM$ in a set $Omega$. If $f$ is a real integrable function with $a < f(x) < b$ for all $x in Omega$, and if $varphi$ is convex on $]a,b[$, then
$$varphileft( int_Omegaf dmuright) le int_Omega(varphi circ f) dmu. $$

So using this I have
$$KL(p|q) = intlogfracp(x)q(x)dp = int - log fracq(x)p(x) dp ge - log intfracq(x)p(x) dp, $$
but now the only way I find to continue is distinguishing if the distributions are discrete or continuous. I haven't studied measure theory and I only have some basic notions, so I'm not sure how to continue. Also, if my notations are wrong, please tell me. Any help will be appreciated.

A rigorous definition of information theory quantities can be found in M.S. Pinsker's book "Information and information stability of random variables and processes".

Consider two probability measures $mu$ and $nu$ defined on the measurable space $(Omega,mathcal M)$. Let $E_i$ be a partition of $Omega$. Then KL-divergence can be defined as
$$
D(mu|nu)=supsum_i mu(E_i)logfracmu(E_i)nu(E_i)
$$
where the supremum is taken over all partitions of $Omega$. Since this definition utilizes a sum like discrete KL-divergence, the non-negative property follows accordingly:
$$
D(mu|nu)geq 0.
$$

There is a theorem by Gelfand-Yaglom-Perez stating that if $D(mu|nu)$ is finite, then $mu$ is absolutely continuous with respect to $nu$ and
$$
D(mu|nu)=int_Omega logfracdmudnumathrmdmu,
$$
where $fracdmudnu$ is the Radon-Nikodym derivation. If you want, you can define the KL-divergence directly using the above equation.