Vtyjkyuk

I am reading some notes on Conditional Expectations as Orthogonal Projections (Page 60 - Section 3.5.2.3). There, they have an example of two biased coin, which I paraphrase.

Let $X$ denote a random variable that is $1$ if the first coin is heads (out of two unbiased coin tosses). I.e., the sample space is $langle TT, TH, HT, HHrangle$, and $X$ can be seen as a vector $langle 0, 0, 1, 1rangle$ in $mathbbR^4$ corresponding to these outcomes. Now, they say $E[X]$ is a projection of $X$ onto subspace of constants, that is, $E[X] = E[langle0, 0, 1, 1rangle] = langlefrac12, frac12, frac12, frac12rangle$. We can see that this is correct because $X - E[X]$ is orthogonal to the subspace of constants: $(X - E[X]) cdot langle 1, 1, 1, 1rangle = 0$.

I don't understand what the subspace of constants is, and how they arrive at the conclusion that $E[X] = langle frac12, frac12, frac12, frac12 rangle$. For example, when I don't understand how to generalize this example for two biased coin tosses with $Pr(textHeads) = p$.

The key to extending the ideas in that section of the notes to other probability spaces is hinted at in the last paragraph, where the authors write “… for example, $mathbf E[XY]$ acts like an inner product for random variables…” In fact, you have to use this inner product to compute the orthogonal projections. Relative to the basis that consists of the indicator variables for the elementary events and treating the random variables as vectors now, this inner product can be computed as the matrix product $Y^TQX$, where $Q$ is a diagonal matrix with the probabilities of the elementary events along its main diagonal. If we represent these probabilities as $p_i$, this expression can be written as $sum_i p_ix_iy_i$.

The authors are a bit cavalier about the inner product used in their examples: they use the dot product of the vectors to define orthogonality, which can be misleading. Per the previous paragraph, the matrix of the inner product for this space is actually $Q=operatornamediagleft(frac14,frac14,frac14,frac14right)$, yielding the inner product $frac14(Xcdot Y)$. This is zero iff $Xcdot Y$ is zero, so the factor of $frac14$ can be ignored when checking orthogonality. (In fact, this factor cancels when computing the angle between two vectors.)

For your unfair coin with $Pr(textHeads)=p$, the matrix of the inner product is instead $$Q=beginbmatrix(1-p)^2 &0&0&0 \ 0& p(1-p) &0&0 \ 0&0& p(1-p) &0 \ 0&0&0& p^2 endbmatrix.$$ You can then compute the corresponding orthogonal projections using well-known formulas: $mathbf E[X]=langle p,p,p,p rangle$, as we’d expect, while $mathbf E[X+Y mid X] = langle p,p,p+1,p+1 rangle$, which jibes with the computation $mathbf E[X+Ymid X] = X+mathbf E[Y]$ from a previous section. Using this inner product, $X-mathbf E[X]$ is indeed orthogonal to $mathbf 1 = langle1,1,1,1rangle$.

You could think of a "constant" as a degenerated random variable, with values equaling to a constant number $a$ for all scenarios $omega$, and that's why you get $(a,a,a,a)=a(1,1,1,1)$.

And in general, you could think a random variable $X$ as a function/mapping, and those random variables could conform a linear space (as the book says, it is linear in terms of addition and scalar multiplication). Then you define inner product on this linear space by $X cdot Y = mathbb E[XY]$. Let constant $c=1$ for all $omega$, and you have $(X-mathbb EX) cdot c = mathbb E[(X-mathbb E X)cdot 1]=0$