Lecture 7 - Joint Distributions and Joint Moments - Matematica Computazione per L'Ingegneria

Joint Distributions and Joint Moments¶

So far, we have studied random variables one at a time. We have introduced their probability distributions, cumulative distribution functions, densities, expectations, moments, and variances.

In many applications, however, several random quantities are observed simultaneously.

Examples include:

the position and velocity of a particle;
the temperature and pressure of a physical system;
the errors of two measuring instruments;
the dimensions of two components produced by the same manufacturing process;
the number of events observed in two different regions.

In such situations, knowing the distribution of each random variable separately is generally not enough. We also need to describe their joint behavior.

This leads to the concepts of:

joint distributions;
marginal distributions;
conditional distributions;
independence of random variables;
joint moments;
covariance and correlation.

The case of two random variables¶

Let

\xi,\eta:\Omega\to\mathbb{R}

be two random variables defined on the same probability space.

The pair

(\xi,\eta)

can itself be regarded as a random variable taking values in $\mathbb{R}^2$ .

For example, if $\xi$ represents the temperature of a system and $\eta$ represents its pressure, then one observation of the experiment produces a point

(\xi(\omega),\eta(\omega)) \in\mathbb{R}^2.

The distribution of this random point describes how the two quantities behave together.

Joint distribution function¶

The joint cumulative distribution function of $\xi$ and $\eta$ is defined by

F_{\xi,\eta}(x,y) = \mathbb{P} \left( \{\xi\leq x,\eta\leq y\} \right).

(1)

This function gives the probability that the random point $(\xi,\eta)$ lies in the lower-left rectangle

(-\infty,x]\times(-\infty,y].

The joint CDF contains the complete information about the joint distribution of the pair $(\xi,\eta)$ .

In particular, it can be used to compute the probability of a rectangle:

\mathbb{P} \left( \{a<\xi\leq b,\ c<\eta\leq d\} \right)

by combining four values of the joint CDF:

\begin{aligned} &\mathbb{P} \left( \{a<\xi\leq b,\ c<\eta\leq d\} \right)\\ &\qquad= F_{\xi,\eta}(b,d) -F_{\xi,\eta}(a,d) -F_{\xi,\eta}(b,c) +F_{\xi,\eta}(a,c). \end{aligned}

(2)

This is the two-dimensional analogue of the relation between a one-dimensional CDF and interval probabilities.

Joint distribution of discrete random variables¶

Suppose that $\xi$ and $\eta$ are discrete random variables.

Their joint probability mass function is

P_{\xi,\eta}(x,y) = \mathbb{P} \left( \{\xi=x,\eta=y\} \right).

(3)

The probabilities must satisfy

P_{\xi,\eta}(x,y)\geq0

and

\sum_x\sum_y P_{\xi,\eta}(x,y) = 1.

(4)

The joint mass function gives the probability of every possible pair of values.

For example, suppose that two components are tested and

\xi= \begin{cases} 1,&\text{first component is defective},\\ 0,&\text{otherwise}, \end{cases}

and similarly for $\eta$ .

The joint distribution distinguishes between the probabilities

\begin{split} \mathbb{P}(\{\xi=1,\eta=1\}), \qquad \mathbb{P}(\{\xi=1,\eta=0\}), \\ \mathbb{P}(\{\xi=0,\eta=1\}), \qquad \mathbb{P}(\{\xi=0,\eta=0\}). \end{split}

These four probabilities describe how the two defect indicators behave together.

Marginal distributions¶

Suppose that $\xi$ and $\eta$ are discrete random variables with joint mass function $P_{\xi,\eta}$ .

The distribution of $\xi$ alone is obtained by summing over all possible values of $\eta$ :

P_\xi(x) = \sum_y P_{\xi,\eta}(x,y).

(5)

Similarly,

P_\eta(y) = \sum_x P_{\xi,\eta}(x,y).

(6)

These are called the marginal distributions.

The terminology reflects the fact that a joint probability table can be displayed with the marginal probabilities in its margins.

Continuous joint distributions¶

Suppose that $(\xi,\eta)$ is a pair of continuous random variables.

A joint probability density function is a nonnegative function

p_{\xi,\eta}(x,y)

satisfying

\int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} p_{\xi,\eta}(x,y)\,dx\,dy = 1.

(7)

The probability that $(\xi,\eta)$ belongs to a region $D\subseteq\mathbb{R}^2$ is

\mathbb{P} \left( \{(\xi,\eta)\in D\} \right) = \iint_D p_{\xi,\eta}(x,y)\,dx\,dy.

(8)

For a rectangle,

\begin{aligned} &\mathbb{P} \left( \{a\leq\xi\leq b,\ c\leq\eta\leq d\} \right)\\ &\qquad= \int_a^b\int_c^d p_{\xi,\eta}(x,y)\,dy\,dx. \end{aligned}

(9)

Thus, just as in one dimension, probabilities correspond to areas under a density. In two dimensions, the density is a surface and probabilities correspond to volumes under that surface.

Marginal densities¶

The marginal density of $\xi$ is obtained by integrating out $\eta$ :

p_\xi(x) = \int_{-\infty}^{+\infty} p_{\xi,\eta}(x,y)\,dy.

(10)

Similarly,

p_\eta(y) = \int_{-\infty}^{+\infty} p_{\xi,\eta}(x,y)\,dx.

(11)

This is the continuous analogue of summing the rows and columns of a discrete joint probability table.

The terminology “marginal” is therefore used in both cases:

\boxed{ \begin{aligned} P_\xi(x) &= \sum_y P_{\xi,\eta}(x,y), & p_\xi(x) &= \int p_{\xi,\eta}(x,y)\,dy, \\[6pt] P_\eta(y) &= \sum_x P_{\xi,\eta}(x,y), & p_\eta(y) &= \int p_{\xi,\eta}(x,y)\,dx. \end{aligned}}

Example

Let $\xi$ and $\eta$ be continuous random variables that are not independent.

Suppose $\xi$ is uniformly distributed over the interval $[0, 10]$ :

\xi \sim \mathrm{Uniform}(0, 10), \quad p_\xi(x) = \begin{cases} \frac{1}{10}, & 0 \le x \le 10, \\ 0, & \text{otherwise.} \end{cases}

Instead of $\eta$ being purely independent, let its conditional distribution given $\xi = x$ depend directly on $x$ . Specifically, let the mean of $\eta$ increase linearly with $x$ :

\eta \mid \xi = x \;\sim\; \mathcal{N}(\mu(x), \sigma^2), \quad \text{where } \mu(x) = x \text{ and } \sigma = 1.

The conditional probability density function is given by:

p_{\eta \mid \xi}(y \mid x) = \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{(y - x)^2}{2}\right).

By the product rule for joint densities, $p_{\xi,\eta}(x,y) = p_\xi(x) \cdot p_{\eta \mid \xi}(y \mid x)$ . Thus, the joint PDF is:

p_{\xi,\eta}(x,y) = \begin{cases} \frac{1}{10 \sqrt{2\pi}} \exp\left(-\frac{(y - x)^2}{2}\right), & 0 \le x \le 10, \, y \in \mathbb{R}, \\[8pt] 0, & \text{otherwise.} \end{cases}

Marginal Density $p_\xi(x)$ : Integrating the joint PDF over all $y \in \mathbb{R}$ for $x \in [0, 10]$ :
$\begin{split} p_\xi(x) = & \int_{-\infty}^{+\infty} p_{\xi,\eta}(x,y) \, dy = \\ = & \frac{1}{10} \int_{-\infty}^{+\infty} \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{(y - x)^2}{2}\right) dy = \\ = & \frac{1}{10} \cdot 1 = \frac{1}{10}. \end{split}$
Marginal Density $p_\eta(y)$ :** Integrating the joint PDF over all $x \in [0, 10]$ :
$\begin{split} p_\eta(y) = & \int_{-\infty}^{+\infty} p_{\xi,\eta}(x,y) \, dx = \\ = & \frac{1}{10} \int_{0}^{10} \frac{1}{\sqrt{2\pi}} \exp\left(-\frac{(y - x)^2}{2}\right) dx = \\ = & \frac{1}{10} \left[ \Phi(y) - \Phi(y - 10) \right], \end{split}$

where $\Phi(\cdot)$ is the cumulative distribution function (CDF) of the standard normal distribution. Because $p_{\xi,\eta}(x,y) \neq p_\xi(x) p_\eta(y)$ , the random variables $\xi$ and $\eta$ are dependent.

import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import simpson
from scipy.stats import norm, uniform

# 1. Define distribution parameters
a, b = 0.0, 10.0  # Uniform bounds for xi
sigma = 1.0  # Standard deviation for conditional normal eta | xi = x

# 2. Create spatial grid
x = np.linspace(a - 2, b + 2, 300)
y = np.linspace(a - 4 * sigma, b + 4 * sigma, 300)
X, Y = np.meshgrid(x, y)

# 3. Compute joint PDF values p(x, y) = p_xi(x) * p_(eta|xi)(y|x)
p_xi_grid = uniform.pdf(X, loc=a, scale=b - a)
p_eta_given_xi_grid = norm.pdf(Y, loc=X, scale=sigma)
Z = p_xi_grid * p_eta_given_xi_grid

# 4. Compute marginal densities by numerical integration
# p_xi(x) = integral over y of p(x, y)
p_xi_marginal = simpson(Z, y, axis=0)

# p_eta(y) = integral over x of p(x, y)
p_eta_marginal = simpson(Z, x, axis=1)

# Analytical expressions for theoretical comparison
p_xi_analytical = uniform.pdf(x, loc=a, scale=b - a)
p_eta_analytical = (norm.cdf(y, loc=0, scale=1) - norm.cdf(y - 10, loc=0, scale=1)) / 10.0

# 5. Create 2x2 multi-panel plot
fig = plt.figure(figsize=(14, 10))

# --- ROW 1: JOINT DISTRIBUTION PLOTS ---

# Panel (1,1): 3D Surface Plot of Dependent Joint Density
ax1 = fig.add_subplot(2, 2, 1, projection='3d')
surf = ax1.plot_surface(
    X, Y, Z, cmap='viridis', edgecolor='none', alpha=0.85
)
ax1.set_title(r'Joint Density Surface $p_{\xi,\eta}(x,y)$', fontsize=12, pad=12)
ax1.set_xlabel(r'$\xi$ (Uniform)')
ax1.set_ylabel(r'$\eta$ (Conditionally Normal)')
ax1.set_zlabel(r'Density $p(x,y)$')
ax1.view_init(elev=28, azim=-55)

# Panel (1,2): 2D Contour Plot
ax2 = fig.add_subplot(2, 2, 2)
contour = ax2.contourf(X, Y, Z, levels=15, cmap='viridis')
fig.colorbar(contour, ax=ax2, label=r'Density $p_{\xi,\eta}(x,y)$')
x_line = np.linspace(a, b, 100)
ax2.plot(
    x_line,
    x_line,
    color='red',
    linestyle='--',
    linewidth=2,
    label=r'$\mathbb{E}[\eta \mid \xi = x] = x$',
)
ax2.set_title(r'Joint Contours with Mean Trend $\mu(x) = x$', fontsize=12)
ax2.set_xlabel(r'$x$')
ax2.set_ylabel(r'$y$')
ax2.grid(True, linestyle='--', alpha=0.5)
ax2.legend(loc='upper left')

# --- ROW 2: MARGINAL DISTRIBUTION PLOTS ---

# Panel (2,1): Marginal Density p_xi(x)
ax3 = fig.add_subplot(2, 2, 3)
ax3.plot(
    x,
    p_xi_marginal,
    color='navy',
    lw=2,
    label=r'Integrated $\int p(x,y)dy$',
)
ax3.plot(
    x,
    p_xi_analytical,
    color='deepskyblue',
    linestyle='--',
    lw=2,
    label=r'Theoretical $\mathrm{Uniform}(0,10)$',
)
ax3.fill_between(x, p_xi_marginal, color='skyblue', alpha=0.3)
ax3.set_title(r'Marginal Density $p_\xi(x)$', fontsize=12)
ax3.set_xlabel(r'$x$')
ax3.set_ylabel(r'Density $p_\xi(x)$')
ax3.grid(True, linestyle='--', alpha=0.5)
ax3.legend()

# Panel (2,2): Marginal Density p_eta(y)
ax4 = fig.add_subplot(2, 2, 4)
ax4.plot(
    y,
    p_eta_marginal,
    color='crimson',
    lw=2,
    label=r'Integrated $\int p(x,y)dx$',
)
ax4.plot(
    y,
    p_eta_analytical,
    color='orange',
    linestyle='--',
    lw=2,
    label=r'Theoretical $\frac{1}{10}[\Phi(y)-\Phi(y-10)]$',
)
ax4.fill_between(y, p_eta_marginal, color='lightcoral', alpha=0.3)
ax4.set_title(r'Marginal Density $p_\eta(y)$', fontsize=12)
ax4.set_xlabel(r'$y$')
ax4.set_ylabel(r'Density $p_\eta(y)$')
ax4.grid(True, linestyle='--', alpha=0.5)
ax4.legend()

plt.tight_layout()
plt.show()

Conditional Distributions¶

The joint distribution describes how two random variables behave together, while the marginal distributions describe each variable separately. A third useful description is obtained when the value of one random variable is known.

Suppose that $\xi$ and $\eta$ are random variables. If we know that $\eta=y$ , we may ask how the distribution of $\xi$ changes. This leads to the notion of a conditional distribution.

Conditional distribution in the discrete case¶

Suppose that $\xi$ and $\eta$ are discrete random variables. For a value $y$ such that

P_\eta(y)>0,

the conditional probability that $\xi=x$ given $\eta=y$ is

P_{\xi\mid\eta}(x\mid y) = \mathbb{P}(\{\xi=x\}\mid\{\eta=y\}) = \frac{P_{\xi,\eta}(x,y)}{P_\eta(y)}.

(12)

For every fixed $y$ with $P_\eta(y)>0$ , this is a probability distribution in $x$ . In particular,

P_{\xi\mid\eta}(x\mid y)\geq 0,

and

\sum_x P_{\xi\mid\eta}(x\mid y)=1.

(13)

The joint distribution can therefore be reconstructed from the marginal and conditional distributions:

P_{\xi,\eta}(x,y) = P_{\xi\mid\eta}(x\mid y)P_\eta(y).

(14)

This is the random-variable version of the multiplication rule introduced for events in Lecture 2.

Conditional distribution of a defective component

Suppose that a production line produces components classified according to their type and whether they are defective. Let

\xi= \begin{cases} 1,&\text{the component is defective},\\ 0,&\text{otherwise}, \end{cases}

and let $\eta$ denote the production line, with $\eta\in\{1,2\}$ .

Suppose that the joint distribution is

\begin{array}{c|cc} & \eta=1 & \eta=2\\ \hline \xi=0 & 0.45 & 0.40\\ \xi=1 & 0.05 & 0.10 \end{array}

Then

P_\eta(1)=0.50, \qquad P_\eta(2)=0.50.

If we are told that the component came from line 1, its conditional probability of being defective is

P_{\xi\mid\eta}(1\mid 1) = \frac{0.05}{0.50} = 0.10.

Thus, conditioning on the production line changes the distribution of $\xi$ .

Conditional density in the continuous case¶

Suppose now that the pair $(\xi,\eta)$ admits a joint density $p_{\xi,\eta}$ , and let $p_\eta$ be the marginal density of $\eta$ .

For values $y$ such that $p_\eta(y)>0$ , the conditional density of $\xi$ given $\eta=y$ is defined by

p_{\xi\mid\eta}(x\mid y) = \frac{p_{\xi,\eta}(x,y)}{p_\eta(y)}.

(15)

For each fixed $y$ for which the conditional density is defined,

\int_{-\infty}^{+\infty} p_{\xi\mid\eta}(x\mid y)\,dx = 1.

The corresponding conditional distribution function is

F_{\xi\mid\eta}(x\mid y) = \int_{-\infty}^{x} p_{\xi\mid\eta}(t\mid y)\,dt.

(16)

As in the discrete case, the joint density can be written as

p_{\xi,\eta}(x,y) = p_{\xi\mid\eta}(x\mid y)p_\eta(y).

(17)

Independence of Random Variables¶

For events, independence means that knowing whether one event occurred does not change the probability of another event. The same idea extends naturally to random variables.

Two random variables $\xi$ and $\eta$ are said to be independent if, for every pair of Borel sets $A,B\subseteq\mathbb{R}$ ,

\mathbb{P}(\{\xi\in A\}\cap\{\eta\in B\}) = \mathbb{P}(\{\xi\in A\}) \mathbb{P}(\{\eta\in B\}).

(18)

In words, observing the value of $\xi$ provides no probabilistic information about the value of $\eta$ , and vice versa.

Independence in the discrete case¶

For discrete random variables, independence is equivalent to the factorization

P_{\xi,\eta}(x,y) = P_\xi(x)P_\eta(y)

(19)

for every pair $(x,y)$ .

For example, if $\xi$ and $\eta$ are independent Bernoulli random variables with parameters $p$ and $q$ , respectively, then

\begin{aligned} \mathbb{P}(\{\xi=1,\eta=1\}) &= pq,\\ \mathbb{P}(\{\xi=1,\eta=0\}) &= p(1-q),\\ \mathbb{P}(\{\xi=0,\eta=1\}) &= (1-p)q,\\ \mathbb{P}(\{\xi=0,\eta=0\}) &= (1-p)(1-q). \end{aligned}

The joint distribution is completely determined by the two marginal distributions.

Independence in the continuous case¶

If $(\xi,\eta)$ admits a joint density, independence is equivalent to

p_{\xi,\eta}(x,y) = p_\xi(x)p_\eta(y)

(20)

for almost every $(x,y)\in\mathbb{R}^2$ .

Equivalently, the joint distribution function factorizes as

F_{\xi,\eta}(x,y) = F_\xi(x)F_\eta(y).

(21)

Thus, independence is a special situation in which the joint distribution contains no additional information beyond the two marginal distributions.

Independence and conditional distributions¶

The connection with conditional distributions is particularly useful.

If $\xi$ and $\eta$ are independent, then, whenever the conditional distribution is defined,

P_{\xi\mid\eta}(x\mid y) = P_\xi(x)

(22)

in the discrete case, and

p_{\xi\mid\eta}(x\mid y) = p_\xi(x)

(23)

in the continuous case.

Thus, conditioning on $\eta$ does not change the distribution of $\xi$ .

The converse also holds under the usual conditions: if the conditional distribution of $\xi$ does not depend on $y$ , then $\xi$ and $\eta$ are independent.

Functions of Two Random Variables¶

Once a joint distribution is available, we can compute expectations not only of $\xi$ and $\eta$ separately, but also of functions involving both variables.

Let

g:\mathbb{R}^2\to\mathbb{R}

be a suitable function.

If $\xi$ and $\eta$ are discrete, then

\mathbb{E}[g(\xi,\eta)] = \sum_x\sum_y g(x,y)P_{\xi,\eta}(x,y).

(24)

If $(\xi,\eta)$ admits a joint density, then

\mathbb{E}[g(\xi,\eta)] = \int_{-\infty}^{+\infty} \int_{-\infty}^{+\infty} g(x,y)p_{\xi,\eta}(x,y)\,dx\,dy.

(25)

These formulas are direct extensions of the one-dimensional formulas introduced in Lecture 6.

For example, choosing

g(x,y)=xy

gives the mixed first moment

\mathbb{E}[\xi\eta].

(26)

More generally, the quantities

\mathbb{E}[\xi^r\eta^s], \qquad r,s\in\mathbb{N},

(27)

are called joint moments of $\xi$ and $\eta$ .

The exponents determine the order of the moment. For example,

\mathbb{E}[\xi\eta]

is a mixed moment of total order two, while

\mathbb{E}[\xi^2\eta]

has total order three.

Products of independent random variables¶

Independence leads to an important simplification.

If $\xi$ and $\eta$ are independent and the expectations exist, then

\mathbb{E}[\xi\eta] = \mathbb{E}[\xi]\mathbb{E}[\eta].

(28)

More generally, if the relevant expectations exist,

\mathbb{E}[g(\xi)h(\eta)] = \mathbb{E}[g(\xi)]\mathbb{E}[h(\eta)].

(29)

For the discrete case, this follows immediately from the factorization of the joint pmf:

\begin{aligned} \mathbb{E}[g(\xi)h(\eta)] &= \sum_x\sum_y g(x)h(y)P_{\xi,\eta}(x,y)\\ &= \sum_x\sum_y g(x)h(y)P_\xi(x)P_\eta(y)\\ &= \left(\sum_x g(x)P_\xi(x)\right) \left(\sum_y h(y)P_\eta(y)\right). \end{aligned}

The continuous case follows in the same way by using the factorization of the joint density.

Covariance¶

The product $\mathbb{E}[\xi\eta]$ alone does not tell us whether large values of $\xi$ tend to occur together with large values of $\eta$ . To measure this type of joint variation, we introduce the covariance.

Assume that $\xi$ and $\eta$ have finite second moments. Their covariance is defined by

\operatorname{Cov}(\xi,\eta) = \mathbb{E} \left[ (\xi-\mathbb{E}[\xi]) (\eta-\mathbb{E}[\eta]) \right].

(30)

The covariance measures whether the two variables tend to deviate from their means in the same direction.

Expanding the product gives

\operatorname{Cov}(\xi,\eta) = \mathbb{E}[\xi\eta] - \mathbb{E}[\xi]\mathbb{E}[\eta].

(31)

Thus, covariance compares the mixed moment $\mathbb{E}[\xi\eta]$ with the product of the means.

If large values of $\xi$ tend to be associated with large values of $\eta$ , the covariance is typically positive. If large values of one variable tend to be associated with small values of the other, it is typically negative.

The covariance is symmetric:

\operatorname{Cov}(\xi,\eta) = \operatorname{Cov}(\eta,\xi).

(32)

It is also linear in each argument. For constants $a,b,c,d$ ,

\operatorname{Cov}(a\xi+b,c\eta+d) = ac\,\operatorname{Cov}(\xi,\eta).

(33)

In particular,

\operatorname{Cov}(\xi,c)=0

for every constant $c$ .

Most importantly, variance is a special case of covariance:

\operatorname{Cov}(\xi,\xi) = \operatorname{Var}(\xi).

(34)

Variance of a sum¶

Using covariance, the variance of a sum can be written as

\operatorname{Var}(\xi+\eta) = \operatorname{Var}(\xi) + \operatorname{Var}(\eta) + 2\operatorname{Cov}(\xi,\eta).

(35)

More generally,

\operatorname{Var} \left( \sum_{k=1}^n \xi_k \right) = \sum_{k=1}^n\operatorname{Var}(\xi_k) + 2\sum_{1\leq i<j\leq n} \operatorname{Cov}(\xi_i,\xi_j).

(36)

If the random variables are pairwise uncorrelated, all covariance terms vanish and therefore

\operatorname{Var} \left( \sum_{k=1}^n \xi_k \right) = \sum_{k=1}^n\operatorname{Var}(\xi_k).

(37)

In particular, independent random variables are uncorrelated whenever their second moments exist. Hence the variance formula for sums of independent random variables introduced in Lecture 6 is a consequence of the more general covariance formula.

Correlation¶

The numerical value of covariance depends on the units in which the variables are measured. For example, changing a measurement from metres to centimetres multiplies the covariance by a factor of 10².

To obtain a dimensionless measure, we normalize the covariance by the standard deviations.

Suppose that

0<\operatorname{Var}(\xi)<\infty, \qquad 0<\operatorname{Var}(\eta)<\infty.

The correlation coefficient of $\xi$ and $\eta$ is

\rho_{\xi,\eta} = \frac{\operatorname{Cov}(\xi,\eta)} {\sqrt{\operatorname{Var}(\xi)} \sqrt{\operatorname{Var}(\eta)}}.

(38)

Using the standard deviations $\sigma_\xi$ and $\sigma_\eta$ , this can be written as

\rho_{\xi,\eta} = \frac{\operatorname{Cov}(\xi,\eta)} {\sigma_\xi\sigma_\eta}.

(39)

The Cauchy--Schwarz inequality implies that

-1\leq \rho_{\xi,\eta}\leq 1.

(40)

The value of $\rho_{\xi,\eta}$ describes the strength and direction of a linear association between the two variables:

$\rho_{\xi,\eta}>0$ indicates positive linear association;
$\rho_{\xi,\eta}<0$ indicates negative linear association;
$\rho_{\xi,\eta}=0$ means that the variables are uncorrelated;
values close to 1 or -1 indicate a strong linear relationship.

It is important to distinguish correlation from independence.

\xi\text{ and }\eta\text{ independent} \quad\Longrightarrow\quad \rho_{\xi,\eta}=0,

(41)

provided the variances are finite and positive.

The converse is generally false.

Zero correlation does not imply independence¶

Consider a random variable

\xi\sim U(-1,1)

and define

\eta=\xi^2.

Clearly, $\eta$ is completely determined by $\xi$ , so the two variables cannot be independent.

However, by symmetry,

\mathbb{E}[\xi]=0

and

\mathbb{E}[\xi^3]=0.

Consequently,

\operatorname{Cov}(\xi,\eta) = \operatorname{Cov}(\xi,\xi^2) = \mathbb{E}[\xi^3] - \mathbb{E}[\xi]\mathbb{E}[\xi^2] = 0.

Thus $\xi$ and $\eta$ are uncorrelated but not independent.

An Interpretation Through Linear Prediction¶

The correlation coefficient also has a useful interpretation in terms of linear prediction.

Suppose we want to approximate $\xi$ by an affine function of $\eta$ ,

\widehat{\xi}=c_1+c_2\eta.

We choose $c_1$ and $c_2$ to minimize the mean squared error

\mathbb{E} \left[ (\xi-c_1-c_2\eta)^2 \right].

(42)

Assume that $\xi$ and $\eta$ have finite, positive variances. The optimal coefficients are

c_2 = \frac{\operatorname{Cov}(\xi,\eta)} {\operatorname{Var}(\eta)}

(43)

and

c_1 = \mathbb{E}[\xi] - c_2\mathbb{E}[\eta].

(44)

The corresponding minimum mean squared error is

\min_{c_1,c_2} \mathbb{E} \left[ (\xi-c_1-c_2\eta)^2 \right] = \operatorname{Var}(\xi)(1-\rho_{\xi,\eta}^2).

(45)

Thus $\rho_{\xi,\eta}^2$ measures the fraction of the variance of $\xi$ that can be explained by the best affine prediction based on $\eta$ .

This interpretation also clarifies why zero correlation does not necessarily mean independence. If $\rho_{\xi,\eta}=0$ , the best affine predictor of $\xi$ based on $\eta$ is simply its mean. Nevertheless, nonlinear relationships between the variables may still exist, as in the example $\eta=\xi^2$ .

Summary¶

The joint distribution provides a complete probabilistic description of two random variables and allows us to study their dependence.

For discrete random variables, the joint distribution is described by

P_{\xi,\eta}(x,y) = \mathbb{P}(\{\xi=x,\eta=y\}),

while for a pair admitting a joint density it is described by $p_{\xi,\eta}(x,y)$ .

From the joint distribution we can obtain:

Object	Discrete case	Continuous case
Joint distribution	$P_{\xi,\eta}(x,y)$	$p_{\xi,\eta}(x,y)$
Marginal of $\xi$	$\displaystyle P_\xi(x)=\sum_yP_{\xi,\eta}(x,y)$	$\displaystyle p_\xi(x)=\int p_{\xi,\eta}(x,y)\,dy$
Conditional distribution	$\displaystyle P_{\xi\mid\eta}(x\mid y)=\frac{P_{\xi,\eta}(x,y)}{P_\eta(y)}$	$\displaystyle p_{\xi\mid\eta}(x\mid y)=\frac{p_{\xi,\eta}(x,y)}{p_\eta(y)}$
Independence	$P_{\xi,\eta}=P_\xi P_\eta$	$p_{\xi,\eta}=p_\xi p_\eta$
Joint moment	$\displaystyle \sum_x\sum_y x^ry^sP_{\xi,\eta}(x,y)$	$\displaystyle \iint x^ry^s p_{\xi,\eta}(x,y)\,dx\,dy$
Covariance	$\operatorname{Cov}(\xi,\eta)=\mathbb{E}[\xi\eta]-\mathbb{E}[\xi]\mathbb{E}[\eta]$	same
Correlation	$\displaystyle \rho_{\xi,\eta}=\frac{\operatorname{Cov}(\xi,\eta)}{\sigma_\xi\sigma_\eta}$	same

The main relationships to remember are

\text{joint distribution} \quad\longrightarrow\quad \begin{cases} \text{marginal distributions},\\ \text{conditional distributions},\\ \text{joint moments}. \end{cases}

Independence is characterized by factorization of the joint distribution:

P_{\xi,\eta}(x,y)=P_\xi(x)P_\eta(y)

in the discrete case, or

p_{\xi,\eta}(x,y)=p_\xi(x)p_\eta(y)

in the continuous case.

Finally,

\text{independence} \quad\Longrightarrow\quad \operatorname{Cov}(\xi,\eta)=0,

but zero covariance does not generally imply independence.

From Two Random Variables to Many Observations¶

The concepts introduced in this lecture provide the language needed to study collections of random variables.

In particular, we can now distinguish between:

the distribution of each individual observation;
the joint distribution of several observations;
independence between observations;
the moments of individual observations;
the covariance between different observations.

These ideas become especially important when considering a sequence of random variables

\xi_1,\xi_2,\ldots,\xi_n,\ldots

and quantities such as their sum

S_n=\sum_{k=1}^n\xi_k

or their average

\overline{\xi}_n = \frac{1}{n}\sum_{k=1}^n\xi_k.

The variance calculation from Lecture 6 shows that, for independent identically distributed random variables with variance $\sigma^2$ ,

\operatorname{Var}(\overline{\xi}_n) = \frac{\sigma^2}{n}.

Thus the average becomes increasingly concentrated around its mean as the number of observations grows.

In the next lecture we will study this phenomenon more systematically. The Law of Large Numbers describes the convergence of averages to their expected value, while the Central Limit Theorem describes the fluctuations around that limiting value and explains the fundamental role of the normal distribution in statistics and scientific computation.