Lecture 2 - Probability, Conditional Probability, and Independence

So far we have introduced probability through examples, counting arguments, and relative frequencies. These examples suggest how probabilities should behave, but they do not yet provide a general mathematical definition.

We now introduce the axiomatic definition of probability. This formulation does not require the elementary outcomes to be equally likely and does not depend on a particular interpretation of probability. Once the axioms are established, the usual rules of probability follow from them.

Axiomatic definition of probability¶

Let $\Omega$ be the sample space of a random experiment, and let $\mathcal{F}$ denote the collection of events to which we assign probabilities. A probability measure is a function

\mathbb{P}:\mathcal{F}\longrightarrow[0,1]

satisfying the following three axioms.

The first two axioms state that probabilities are non-negative and that the probability of an event that is certain to occur is one. The third axiom expresses the idea that probabilities add when events cannot occur simultaneously.

All the elementary rules of probability can be derived from these three properties.

Basic consequences of the axioms¶

We first determine the probability of the impossible event. Since

\Omega=\Omega\cup\varnothing,

and the two events are mutually exclusive, countable additivity gives

\mathbb{P}(\Omega) = \mathbb{P}(\Omega)+\mathbb{P}(\varnothing).

Since $\mathbb{P}(\Omega)=1$ , it follows that

\mathbb{P}(\varnothing)=0.

Next, let $A^c$ denote the complementary event of $A$ . Since $A$ and $A^c$ are mutually exclusive and

A\cup A^c=\Omega,

we have

\mathbb{P}(A)+\mathbb{P}(A^c)=1,

and therefore

\mathbb{P}(A^c)=1-\mathbb{P}(A).

The axioms also imply that probability is monotone.

If $A\subseteq B$ , then $B$ can be decomposed into two mutually exclusive events,

B=A\cup(B\setminus A).

Hence,

\mathbb{P}(B) = \mathbb{P}(A)+\mathbb{P}(B\setminus A) \geq \mathbb{P}(A).

Thus,

A\subseteq B \quad\Longrightarrow\quad \mathbb{P}(A)\leq\mathbb{P}(B).

In particular, since every event is contained in $\Omega$ ,

0\leq\mathbb{P}(A)\leq1.

Finally, for two arbitrary events $A$ and $B$ , we can write

A\cup B = A\cup(B\setminus A),

where the two events on the right are mutually exclusive. Since

B=(A\cap B)\cup(B\setminus A),

we obtain the addition rule

\mathbb{P}(A\cup B) = \mathbb{P}(A)+\mathbb{P}(B)-\mathbb{P}(A\cap B).

The subtraction of $\mathbb{P}(A\cap B)$ is necessary because the intersection is counted twice in $\mathbb{P}(A)+\mathbb{P}(B)$ .

Conditional probability¶

The probability of an event describes its likelihood before additional information about the outcome of the experiment is available. In many applications, however, we obtain information about the outcome and want to update the probability accordingly.

Suppose that we know that an event $B$ has occurred. We are then interested in the probability that another event $A$ occurs under the condition that $B$ has occurred.

Provided that $\mathbb{P}(B)>0$ , the conditional probability of $A$ given $B$ is defined by

\mathbb{P}(A\mid B) = \frac{\mathbb{P}(A\cap B)} {\mathbb{P}(B)}.

(1)

The event $A\cap B$ represents the outcomes for which both $A$ and $B$ occur. Thus, when we condition on $B$ , we restrict our attention to the part of the sample space in which $B$ occurs.

Equiprobable interpretation¶

The meaning of conditional probability is particularly transparent for a finite experiment with equally likely outcomes. Let $N$ be the total number of elementary outcomes, $N(B)$ the number for which $B$ occurs, and $N(A\cap B)$ the number for which both $A$ and $B$ occur. Then

\mathbb{P}(B)=\frac{N(B)}{N}, \qquad \mathbb{P}(A\cap B)=\frac{N(A\cap B)}{N}.

Consequently,

\mathbb{P}(A\mid B) = \frac{\mathbb{P}(A\cap B)} {\mathbb{P}(B)} = \frac{N(A\cap B)} {N(B)}.

Thus, after learning that $B$ has occurred, we effectively restrict attention to the outcomes satisfying $B$ . Among these outcomes, the conditional probability is the proportion for which $A$ also occurs.

This interpretation will be useful when we later discuss statistical independence.

Properties of conditional probability¶

Conditional probability satisfies the same basic probability rules as ordinary probability.

1 (Proof)

The first property follows from

0\leq\mathbb{P}(A\cap B)\leq\mathbb{P}(B),

after division by $\mathbb{P}(B)>0$ .

If $A\cap B=\varnothing$ , then $\mathbb{P}(A\cap B)=0$ , which gives the second property.

If $B\subseteq A$ , then $A\cap B=B$ , and therefore

\mathbb{P}(A\mid B) = \frac{\mathbb{P}(B)}{\mathbb{P}(B)} = 1.

Finally, if the events $A_1,A_2,\ldots$ are mutually exclusive, then

\left(\bigcup_{k=1}^{\infty}A_k\right)\cap B = \bigcup_{k=1}^{\infty}(A_k\cap B),

and the events $A_k\cap B$ are also mutually exclusive. Hence, by countable additivity,

\mathbb{P}\left( \left(\bigcup_{k=1}^{\infty}A_k\right)\cap B \right) = \sum_{k=1}^{\infty} \mathbb{P}(A_k\cap B).

Dividing by $\mathbb{P}(B)$ gives the result.

The definition of conditional probability can also be written as

\mathbb{P}(A\cap B) = \mathbb{P}(A\mid B)\mathbb{P}(B).

(2)

This is often called the multiplication rule. It expresses the probability that both $A$ and $B$ occur in terms of the probability of $B$ and the probability of $A$ once $B$ is known.

By interchanging $A$ and $B$ , we also have, whenever $\mathbb{P}(A)>0$ ,

\mathbb{P}(A\cap B) = \mathbb{P}(B\mid A)\mathbb{P}(A).

(3)

Equating the two expressions gives

\mathbb{P}(A\mid B)\mathbb{P}(B) = \mathbb{P}(B\mid A)\mathbb{P}(A).

This relation is the basis of Bayes’ formula.

Bayes’ formula¶

Suppose that $\mathbb{P}(A)>0$ and $\mathbb{P}(B)>0$ . From (2) and (3),

\mathbb{P}(A\mid B)\mathbb{P}(B) = \mathbb{P}(B\mid A)\mathbb{P}(A).

Therefore,

\mathbb{P}(A\mid B) = \frac{ \mathbb{P}(B\mid A)\mathbb{P}(A) }{ \mathbb{P}(B) }.

(4)

This is Bayes’ formula. It allows us to reverse the direction of a conditional probability: information about $\mathbb{P}(B\mid A)$ can be used to determine $\mathbb{P}(A\mid B)$ , provided that the required probabilities are known.

The denominator can often be computed by partitioning the possible causes of $B$ .

Law of total probability¶

Suppose that $B_1,B_2,\ldots$ are mutually exclusive events that form a partition of the sample space:

B_i\cap B_j=\varnothing, \qquad i\neq j,

and

\bigcup_{k=1}^{\infty}B_k=\Omega.

Assume also that $\mathbb{P}(B_k)>0$ for every $k$ . Then every event $A$ can be decomposed according to which of the events $B_k$ occurs:

A = \bigcup_{k=1}^{\infty}(A\cap B_k).

The events in this union are mutually exclusive. Therefore,

\mathbb{P}(A) = \sum_{k=1}^{\infty} \mathbb{P}(A\cap B_k).

Using the multiplication rule (2), we obtain the law of total probability.

Combining this result with Bayes’ formula gives

\mathbb{P}(B_j\mid A) = \frac{ \mathbb{P}(A\mid B_j)\mathbb{P}(B_j) }{ \displaystyle \sum_{k=1}^{\infty} \mathbb{P}(A\mid B_k)\mathbb{P}(B_k) }.

(6)

This form is particularly useful when the events $B_k$ represent different possible causes of an observed event $A$ .

Quality control and defective components

Suppose that a manufactured component can come from either of two production lines, $B_1$ and $B_2$ . Assume that

\mathbb{P}(B_1)=0.6, \qquad \mathbb{P}(B_2)=0.4,

and that the probabilities of producing a defective component are

\mathbb{P}(D\mid B_1)=0.01, \qquad \mathbb{P}(D\mid B_2)=0.03.

The total probability of obtaining a defective component is

\mathbb{P}(D) = \mathbb{P}(D\mid B_1)\mathbb{P}(B_1) + \mathbb{P}(D\mid B_2)\mathbb{P}(B_2).

Thus,

\mathbb{P}(D) = 0.01(0.6)+0.03(0.4) = 0.018.

If a component is found to be defective, Bayes’ formula gives the probability that it came from the second production line:

\mathbb{P}(B_2\mid D) = \frac{ \mathbb{P}(D\mid B_2)\mathbb{P}(B_2) }{ \mathbb{P}(D) } = \frac{0.03(0.4)}{0.018} = \frac{2}{3}.

Notice that $\mathbb{P}(B_2)=0.4$ before observing the component, whereas $\mathbb{P}(B_2\mid D)=2/3$ after learning that it is defective. The observation has therefore changed the probability assigned to the source of the component.

Statistical independence¶

Conditional probability provides a precise way of describing how information about one event changes the probability of another.

Suppose that $\mathbb{P}(B)>0$ . If

\mathbb{P}(A\mid B)=\mathbb{P}(A),

then learning that $B$ has occurred does not change the probability of $A$ . This motivates the definition of independence.

The definition is symmetric in $A$ and $B$ . Thus, independence is a mutual property: if $A$ and $B$ are independent, then $B$ and $A$ are also independent.

The connection with conditional probability is immediate. If $\mathbb{P}(B)>0$ , then

\mathbb{P}(A\mid B) = \frac{\mathbb{P}(A\cap B)} {\mathbb{P}(B)}.

If $A$ and $B$ are independent, then

\mathbb{P}(A\mid B) = \frac{\mathbb{P}(A)\mathbb{P}(B)} {\mathbb{P}(B)} = \mathbb{P}(A).

Conversely, if $\mathbb{P}(B)>0$ and

\mathbb{P}(A\mid B)=\mathbb{P}(A),

then multiplying by $\mathbb{P}(B)$ gives

\mathbb{P}(A\cap B) = \mathbb{P}(A)\mathbb{P}(B).

Therefore,

A\text{ and }B\text{ are independent} \quad\Longleftrightarrow\quad \mathbb{P}(A\mid B)=\mathbb{P}(A),

(8)

provided that $\mathbb{P}(B)>0$ .

Similarly, if $\mathbb{P}(A)>0$ ,

A\text{ and }B\text{ are independent} \quad\Longleftrightarrow\quad \mathbb{P}(B\mid A)=\mathbb{P}(B).

Thus, independence means precisely that learning that one event has occurred provides no information, in the probabilistic sense, about the other event.

Independence and complements¶

Independence is preserved when events are replaced by their complements. For example, if $A$ and $B$ are independent, then

\mathbb{P}(A^c\cap B) = \mathbb{P}(B)-\mathbb{P}(A\cap B).

Using (7),

\begin{aligned} \mathbb{P}(A^c\cap B) &= \mathbb{P}(B) - \mathbb{P}(A)\mathbb{P}(B)\\ &= (1-\mathbb{P}(A))\mathbb{P}(B)\\ &= \mathbb{P}(A^c)\mathbb{P}(B). \end{aligned}

Hence $A^c$ and $B$ are independent. The same argument applies to $A$ and $B^c$ , and therefore also to $A^c$ and $B^c$ .

Independence of several events¶

The definition of independence extends naturally to more than two events. For a collection of events, it is not sufficient to require only that every pair be independent.

For a sequence of events $A_1,A_2,\ldots$ , the same condition is required for every finite collection of distinct events.

This distinction will become important later when we introduce independent random variables. There, independence will mean that the events generated by the different random variables factorize in the corresponding way.

Conditional probability and relative frequency¶

The definition of conditional probability also gives a useful interpretation in terms of repeated experiments.

Suppose that an experiment is repeated $n$ times. Let $n(A)$ denote the number of trials in which $A$ occurs, and let $n(A\cap B)$ denote the number in which both $A$ and $B$ occur. Among the trials for which $B$ occurs, the relative frequency of $A$ is

\frac{n(A\cap B)}{n(B)},

provided that $n(B)>0$ .

The definition

\mathbb{P}(A\mid B) = \frac{\mathbb{P}(A\cap B)} {\mathbb{P}(B)}

therefore corresponds, under the relative-frequency interpretation, to the limiting proportion of occurrences of $A$ among those trials in which $B$ occurs.

Similarly, independence means that restricting attention to the trials in which $B$ occurs does not change the long-run frequency of $A$ :

\mathbb{P}(A\mid B)=\mathbb{P}(A).

Thus, the axiomatic definition, conditional probability, and the relative-frequency interpretation are consistent with the same underlying probabilistic structure.

Failures in series and parallel systems¶

Many physical and logical systems can be modeled as assemblies of components arranged in mechanical or logical configurations. Let $Y_i$ be an indicator variable where $Y_i = 1$ if component $i$ fails and $Y_i = 0$ if it functions properly, with individual failure probability $P_i = \mathbb{P}(Y_i = 1)$ .

Series Systems

A series system functions properly only if all of its components function properly, meaning it fails as soon as at least one component fails (e.g., chains, sequential workflows).

The general probability of failure is:

\mathbb{P}(\text{system failure}) = 1 - \mathbb{P}\big((Y_1 = 0) \cap (Y_2 = 0) \cap \dots \cap (Y_n = 0)\big).

If the $n$ components fail independently of one another:

\mathbb{P}(\text{system failure}) = 1 - \prod_{i=1}^n (1 - P_i).

If, in addition, all components have equal failure probabilities ( $P_i = P$ for all $i$ ):

\mathbb{P}(\text{system failure}) = 1 - (1 - P)^n.

Parallel Systems

A parallel system fails only if all of its components fail. It continues to operate as long as at least one component is in good working condition (e.g., redundant copy machines in an office).

The general probability of failure is:

\mathbb{P}(\text{system failure}) = \mathbb{P}\big((Y_1 = 1) \cap (Y_2 = 1) \cap \dots \cap (Y_n = 1)\big).

If the $n$ components fail independently of one another:

\mathbb{P}(\text{system failure}) = \prod_{i=1}^n P_i.

If, in addition, all components have equal failure probabilities ( $P_i = P$ for all $i$ ):

\mathbb{P}(\text{system failure}) = P^n.

Combined System Reliability (Heater, Pumps, and Turbines)¶

To illustrate how series, parallel, and $m$ -out-of- $n$ logic combine when calculating probabilities, consider a power generation facility consisting of three main sub-systems connected in series:

Heater Sub-system: A single heater ( $R_1$ ).
Pump Sub-system: Two pumps ( $R_2$ and $R_3$ ) operating in parallel.
Turbine Sub-system: Five turbines ( $R_4, R_5, R_6, R_7, R_8$ ) operating as a 3-out-of-5 system (requires at least 3 functioning turbines for the sub-system to work).

Combined System Reliability (Heater, Pumps, and Turbines)

Between scheduled maintenances, components fail independently. For simplicity, assume that all turbines have an identical failure probability of 0.15:

Component	$R_1$ (Heater)	$R_2$ (Pump 1)	$R_3$ (Pump 2)	$R_4$ through $R_8$ (Turbines)
$\mathbb{P}(\text{Failure})$	0.05	0.10	0.08	0.15 each

Let $W_i$ denote the event that component $i$ works properly.

Step 1: Heater Sub-system Reliability

The heater sub-system consists of a single component $R_1$ . Its reliability is simply the probability that $R_1$ does not fail:

\mathbb{P}(\text{Heater Works}) = \mathbb{P}(W_1) = 1 - 0.05 = 0.95.

Step 2: Pump Sub-system Reliability

The pump sub-system operates in parallel, meaning it survives as long as at least one pump works. It fails only if both pumps fail simultaneously:

\mathbb{P}(\text{Pumps Work}) = 1 - \mathbb{P}(W_2^c \cap W_3^c).

Since $R_2$ and $R_3$ fail independently:

\mathbb{P}(\text{Pumps Work}) = 1 - \mathbb{P}(W_2^c)\mathbb{P}(W_3^c) = 1 - (0.10)(0.08) = 1 - 0.008 = 0.992.

Step 3: Turbine Sub-system Reliability ( $m$ -out-of- $n$ System)

The turbine sub-system requires at least 3 out of 5 turbines to function. Because all 5 turbines have identical probabilities of success ( $1 - 0.15 = 0.85$ ) and failure (0.15), any specific configuration with $k$ working turbines and $5-k$ failed turbines has probability $(0.85)^k (0.15)^{5-k}$ .

Since the different working states are mutually exclusive, we compute the total reliability by counting the number of valid outcome configurations for 5, 4, or 3 working turbines and summing their probabilities:

Case 1: All 5 turbines work ( $\binom{5}{5} = 1$ state)
$1 \times (0.85)^5 = 0.4437.$
Case 2: Exactly 4 turbines work ( $\binom{5}{4} = 5$ states)
$5 \times (0.85)^4(0.15)^1 = 5 \times (0.5220)(0.15) = 0.3915.$
Case 3: Exactly 3 turbines work ( $\binom{5}{3} = 10$ states)
$10 \times (0.85)^3(0.15)^2 = 10 \times (0.6141)(0.0225) = 0.1382.$

Summing the probabilities across all 16 mutually exclusive functional states yields:

\mathbb{P}(\text{Turbines Work}) = 0.4437 + 0.3915 + 0.1382 = 0.9734.

Step 4: Overall System Reliability

Because the heater, pump, and turbine sub-systems are connected in series, the overall plant functions if and only if all three sub-systems work. Assuming independence among sub-systems:

\mathbb{P}(\text{System Works}) = \mathbb{P}(\text{Heater Works}) \times \mathbb{P}(\text{Pumps Work}) \times \mathbb{P}(\text{Turbines Work})

\mathbb{P}(\text{System Works}) = (0.95) \times (0.992) \times (0.9734) \approx 0.9173.

Looking ahead¶

We have now established the basic rules needed to describe how events interact:

\text{probability} \longrightarrow \text{conditional probability} \longrightarrow \text{independence}.

In the next lecture, we move from events to random variables. A random variable assigns a numerical value to the outcome of a random experiment, allowing us to describe random phenomena quantitatively. We will introduce probability mass functions, probability densities, and distribution functions, which provide a systematic way of describing the distribution of a random variable.