Joint and conditional probability

In this section, we will study relationships between random variables in more detail. So far, we have often assumed different random variables to be independent from each other. But the more interesting case is when one random variable can provide information about the other. As an example, denote the data that you write on a flash drive as $X$ and let $Y$ be the data that you read from it. If $X$ and $Y$ are independent from each other, then the flash drive is completely useless. Ideally, you want $Y=X$, but in reality this is seldom the case, because noise is everywhere. So $X$ and $Y$ are neither independent nor can they be assumed to be equal. But $Y$ does provide information about $X$. We can study such relationships through joint and conditional probabilities.

Joint probability distributions

We have seen that a distribution can fully describe a single random variable. We have also seen that for two independent random variables, $X$ and $Y$, we have $P(X=i,Y=j)=P(X=i)P(Y=j)$. What about random variables that are not independent?

Suppose a coin is flipped twice. Let $X$ denote the result of the first flip ($X=1$ represents heads and $X=0$ represents tails). Also, let $Y$ be the total number of heads. Clearly, $Y$ and $X$ are not independent. For example, if we know $Y=2$, then necessarily $X=1$. How can we fully describe this relationship? We can do so using their joint probability distribution, which is essentially the set of values $P(X=i,Y=j)$ for all possible $i,j$. We can show it using a table. Let’s first find the probability values:

Now we can represent the distribution as a table (or a 3D plot):

If we have the joint probability distribution, how can we find the distribution of each random variable? To find $P(Y=i)$ say, we can sum up the corresponding row or column. In our example,

We can add this information to the table to present all the information at once:

Conditional probability

Suppose we have collected the following data over a year about traffic collisions and weather on a given road:

Let’s assume that this data is representative and we can use it to estimate probabilities.

• Let $B$ be the event that the weather is bad. Then $P(B)=(60+5)/365=18\mathrm{\%}$.
• Let $G$ be the event that the weather is good. Then $P(G)=(300)/(365)=72\mathrm{\%}$.
• Let $C$ be the event that there is a car clash. Then $P(C)=(5+5)/365=2.7\mathrm{\%}$.
• The probability of bad weather and car crash is $P(B\cap C)=5/365=1.4\mathrm{\%}$.

Now, we’d like to consider the effect of weather on the probability of car crash. To find the probability of a car crash when the weather is bad, we limit our attention to the 65 days in which we have bad weather. Among these, in 5 days there were car crashes. So we can estimate this probability as

This is a conditional probability. We assume some event to hold and then find the probability of another event under that assumption. We write the conditional probability of an event $A$ assuming $B$ as $P(A|B)$. So above we have found

Similarly,

which is smaller as can be expected.

The key point is that when computing the conditional probability of $A$ given $B$, in the denominator, we only consider those cases (days in our example) in which $B$ has occurred, and in the numerator, we consider cases in which both $A$ and $B$ occurred. While above we were dealing with cases (days), we can apply the same logic to probabilities (and get the same result):

If two events $A$ and $B$ are independent, then

So whether or not $B$ has occurred does not affect the probability of $A$.

The law of total probability

Conditional probability can be helpful for finding the (unconditional) probability of events too. Note that

This approach can be extended further. For example, suppose $B_1,B_2,B_3$ are events that that are pairwise disjoint (the intersection of any two of them is empty) and $B_1\cup B_2\cup B_3=\mathrm{\Omega }$. Then,

$$\begin{array}{}\text{(1)}& P(A)=P(B_1)P(A|B_1)+P(B_2)P(A|B_2)+P(B_3)P(A|B_3).\end{array}$$

Let’s see an example. Suppose a random 2-digit number (between 10 and 99, inclusive) is selected by choosing each digit at random. What is the probability of the event $A$ that it is larger than 63? It is easy to see that all choices are equally likely so we can easily find the desired probability by finding the number of choices that are larger than 63 and dividing by the total number of cases,

Let us also find it using conditional probability. The chosen number is larger than 63 if ‘the first digit (from the left) is larger than 6’ or ‘the first digit is equal to 6 and the second digit is larger than 3’. Let $B_1,B_2,B_3$ be the events that the first digit is less than 6, equal to 6, and larger than 6, respectively. Using $\text{(1)}$

The Monty Hall problem

In the Monty Hall game, a prize is hidden behind one of three doors. After you pick a door, the host opens another one without the prize and gives you the option to switch to the unopened door. There are two strategies. One is sticking with your original choice and the other is switching to the unopened door. Does the strategy you pick affect your chance of winning?

You can play the game below. A car and two goats are hidden behind the doors and your goal is to win the car (although I have heard goats are pretty good pets!). Play a few rounds of this game. After each round, clicking any of the door or prize resets the game.

Reset Record

Conditional distributions

Conditional probability also naturally applies to random variables. As an example, consider a coin being flipped twice. Let $X$ be the number of heads in the first flip (so either 0 or 1) and $Y$ be the total number of heads. Let us find $P(X=i|Y=j)$ for $i\in \{0,1\}$ and $j\in \{0,1,2\}$. Let us repeat the joint distribution table found above

Then,

The function $P(X=i|B)$ for an event $B$ and all possible values of $i$, is called the conditional distribution of $X$ given $B$.