1.5 Averaging 1.5.6 Averages in involving multiple variables 1.6 Gaussians are quite normal

1.5.7 Variance: just how fat is your distribution?

The average of a probability distribution tells you a lot about the real world. What’s the average height of the lawn? What’s the average amount of money do you expect to lose playing video poker in Las Vegas? What is the life-expectancy of a chipmunk?

But aside from knowing this, you’d like to know how close to that average you expect to be. That is, when Bob looks at some random blade of grass, he might on average get $6cm$ , but he might find that in one instance it’s $5.5cm$ , another is $6.9cm$ . But does he expect to find $10cm$ ? The complete distribution tells you that, but you want just want one number to tell how close, for example $6\pm 1cm$ . This is a measure of how sharply peaked, or how wide is the distribution.

There are lots of different definitions you could come up with, but we’ll use the same one that everyone else does, the variance.

Var(x)=\langle(x-\langle x\rangle)^{2}\rangle

(1.38)

This means we take the average difference between and outcome and the mean, square it, and then average.

In terms of discrete probabilities this is:

Var(x)=\sum_{x}(x-\langle x\rangle)^{2})P(x)

(1.39)

and for a continuous distribution:

Var(x)=\int(x-\langle x\rangle)^{2})P(x)dx

(1.40)

This has the units of $x^{2}$ and is often referred to as $\sigma^{2}$ . The standard deviation is then just $\sigma=\sqrt{Var(x)}$ .

Note for those that remember their classical mechanics, that the variance corresponds to the moment of inertia of an object of mass 1, about the center of mass.

Let’s do some examples. What’s the variance for the distribution in fig. 1.1? That’s the case where you flip a coin and you get 1 for heads and -1 for tails. In that case the mean was zero, so applying eqn. 1.39 you have:

\sigma^{2}=(-1-0)^{2}{1\over 2}+(1-0)^{2}{1\over 2}=1

(1.41)

So the variance in this case is $1$ . For this trial you expect to make $0\pm 1$ .

Lastly, there’s an interesting identity for the variance that sometimes simplifies calculations:

	$\displaystyle Var(x)=\langle(x-\langle x\rangle)^{2}\rangle=\langle x^{2}-2% \langle x\rangle x+\langle x\rangle^{2}\rangle$		(1.42)
	$\displaystyle=\langle x^{2}\rangle-2\langle x\rangle\langle x\rangle+\langle% \langle x\rangle^{2}\rangle=\langle x^{2}\rangle-2\langle x\rangle^{2}+\langle x% \rangle^{2}=\langle x^{2}\rangle-\langle x\rangle^{2}$		(1.43)

Now we’ll move on to a more complex example.

*Variance of binomial distro

OK, what’s that ”*” doing up there? Well that means that if you look down the page and you start to feel like you might lose your lunch, just skip to eqn. 1.55 at the end of this section to see the final result.

Now let’s do a simple variation on this. as in problem 1.5.5, you get 0 for tails, and 1 for heads. So the mean for 1 trial is ${1\over 2}$ . What’s the variance?

\sigma^{2}=(0-{1\over 2})^{2}{1\over 2}+(1-{1\over 2})^{2}{1\over 2}={1\over 4}

(1.44)

This is a standard deviation $\sigma={1\over 2}$ . This makes sense, because it’s the same as the previous example except the peaks are now separated by half the distance, so you expect half the width.

Now let’s consider what you get with 10 tosses. What happens is now $\sigma^{2}\rightarrow 10\sigma^{2}$ . For $N$ tosses $\sigma^{2}=N{1\over 4}$ . The mean we found out was $N/2$ . So they are both proportional to $N$ .

How do I know this is the answer? What, you think I looked it up? You can do this following the methods we used in section 1.5.6.

We can write down the variance directly from the binomial distribution, just like we did for the mean (eqn. 1.36).

	$\displaystyle\langle(X-\langle X\rangle)^{2}\rangle=\sum_{X=0}^{n}(X-\langle X% \rangle)^{2}P(X)=\sum_{X=0}^{n}(X-\langle X\rangle)^{2}\left(\!\!\!\begin{% array}[]{c}n\\ X\end{array}\!\!\!\right)(p)^{X}(1-p)^{n-X}=$		(1.47)
	$\displaystyle\sum_{X=0}^{n}(X-np)^{2}{n!\over(n-X)!X!}(p)^{X}(1-p)^{n-X}$

As with the mean, you can do a bad-ass differentiation trick to get the answer, it’s a bit messy, but it works. However I’ll follow the same kind of reasoning we talked about for the mean to get the answer.

So you use the same variable as we did before $X=\sum_{i=1}^{n}x_{i}$ . Now we want to calculate the variance, so let’s see how far we can get:

\sigma^{2}=\langle(X-\langle X\rangle)^{2}\rangle=\langle(\sum_{i=1}^{n}x_{i}-% \langle\sum_{i=1}^{n}x_{i}\rangle)^{2}\rangle=\langle(\sum_{i=1}^{n}[x_{i}-% \langle x_{i}\rangle])^{2}\rangle

(1.48)

Well we’ve got pretty far, but not quite far enough. Now we have the the square of a sum. How do you handle that? Think of the case $n=2$ , $(x_{1}+x_{2})^{2}=x_{1}x_{1}+x_{1}x_{2}+x_{2}x_{1}+x_{2}x_{2}$ . This looks like a double summation of two indices. So generalizing this, we see that that summation in the above equation can be written as a double sum:

\sigma^{2}=\langle\sum_{i=1}^{n}\sum_{j=1}^{n}[x_{i}-\langle x_{i}\rangle][x_{% j}-\langle x_{j}\rangle]\rangle

(1.49)

But the average of a sum is the sum of the average (even for a double sum) so

\sigma^{2}=\sum_{i=1}^{n}\sum_{j=1}^{n}\langle(x_{i}-\langle x_{i}\rangle)(x_{% j}-\langle x_{j}\rangle)\rangle

(1.50)

Now there are two kinds of terms in this double sum; ones with $i=j$ and with $i\neq j$ . If $i\neq j$ , for example $\langle[x_{1}-\langle x_{1}\rangle][x_{2}-\langle x_{2}\rangle]\rangle$ then we can use independence to figure out the answer. In that case

\langle(x_{1}-\langle x_{1}\rangle)(\langle x_{2}-\langle x_{2}\rangle)\rangle% =\langle x_{1}-\langle x_{1}\rangle\rangle\langle x_{2}-\langle x_{2}\rangle\rangle

(1.51)

Because as we saw in section 1.5.6 for independent variables, the average of the product is the product of the average. But $\langle x_{1}\rangle$ is just a constant so averaging that gives doesn’t change it. So

\langle x_{1}-\langle x_{1}\rangle\rangle=\langle x_{1}\rangle-\langle x_{1}% \rangle=0

(1.52)

So all terms with $i\neq j$ vanish. We’re only left with the terms $i=j$ in other words a single sum:

\sigma^{2}=\sum_{i=1}^{n}\langle(x_{i}-\langle x_{i}\rangle)^{2}\rangle

(1.53)

Well each term in the sum is just the variance for a single trial. Let’s compute that, say for $x_{1}$ :

	$\displaystyle\langle(x_{1}-\langle x_{1}\rangle)^{2}\rangle=\langle(x_{1}-p)^{% 2}\rangle=(0-p)^{2}P(0)+(1-p)^{2}P(1)=$		(1.54)
	$\displaystyle p^{2}(1-p)+(1-p)^{2}p=p(1-p)(p+(1-p)=p(1-p)$

Good, you’re still awake! Must have had some pretty strong coffee. Well we’re almost there. Now we sum over all $n$ terms and obtain

\sigma^{2}=np(1-p)

(1.55)

That’s a pretty simple answer. I could’ve just written it down and left it that. But that would take away all the fun.

Problems

1. Suppose you have an unbiased coin, that is, it lands heads and tails with equal probability. You toss it 16 times and count the total number of heads you get. What’s the standard deviation of the number of total number of heads. Now toss it 256 times. What’s the standard deviation now?

2. Suppose you have the same setup as the last problem, with an unbiased coin. You give this coin to Bob so he can perform experiments on it. You want Bob to estimate the probability that a coin will land heads. Bob tosses it 16 times. He defines the probability of it landing heads to be $p=$ (total number of heads/total number of trials).

: (a) On average, what result will Bob obtain?
: (b) You want to estimate the standard deviation in Bob’s result. Using the formula for the standard deviation of the binomial distribution, calculate the standard deviation of $p$ that Bob will ascertain.
: (c) What happens to the standard deviation when Bob repeats the calculation 256 times instead?