Review of Probability

Random Experiments, Sample Spaces, and Events¶

• Random experiment: a well-defined procedure whose outcome cannot be predicted with certainty in advance. Example: tossing a coin twice.

• Sample space $S$: the set of all possible outcomes. Example: $S = \{(H,H),\, (H,T),\, (T,H),\, (T,T)\}$.

• Event $A$: any subset of the sample space, $A \subseteq S$. Example: “at least one head” $= A = \{(H,H),(H,T),(T,H)\} \subseteq S$.

Probability of Events¶

A probability function $P(\cdot)$ over events in $S$ satisfies:

1. Non-negativity: $P(A) \geq 0$

2. Normalization: $P(S) = 1$

3. Additivity:

   • If $A \cap B = \emptyset$: $P(A \cup B) = P(A) + P(B)$

   • In general: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

Random Variables¶

A random variable is a function mapping outcomes to real values:

Example (two coin tosses): Let $X$ = number of tails.

The state space (support) of $X$ is $\mathcal{X} = \{0, 1, 2\}$.

Two types:

• Discrete random variables — countable state space.

• Continuous random variables — uncountable state space.

Probability Mass Function (PMF) — Discrete¶

For a discrete random variable $X$ with state space $\mathcal{X}$, the PMF is $P(X = x)$ for $x \in \mathcal{X}$.

Properties:

• Non-negativity: $P(X = x) \geq 0$ for all $x \in \mathcal{X}$

• Normalization: $\displaystyle\sum_{x \in \mathcal{X}} P(X = x) = 1$

Joint, Marginal, and Conditional Probabilities¶

For two discrete random variables $X$ and $Y$:

• Joint probability: $P(X = x,\, Y = y)$

• Marginalization:

  $$P(X = x) = \sum_{y \in \mathcal{Y}} P(X = x,\, Y = y), \qquad P(Y = y) = \sum_{x \in \mathcal{X}} P(X = x,\, Y = y)$$

  (3)

• Conditional probability:

  $$P(X = x \mid Y = y) = \frac{P(X = x,\, Y = y)}{P(Y = y)}$$

  (4)

Bayes’ Rule¶

Independent Random Variables¶

$X$ and $Y$ are independent if:

Expectation and Variance¶

For a discrete random variable $X$:

Common Distributions¶

Bernoulli Distribution¶

Models a single binary trial (e.g., one coin flip).

where $\mu = P(X = 1)$. Mean and variance:

Binomial Distribution¶

Models the number of successes in $N$ independent Bernoulli trials.

Mean and variance:

Continuous Random Variables¶

For a continuous random variable $X$, $P(X = x) = 0$ for any single point. Probabilities are described by the probability density function (PDF) $p_X(x)$:

The cumulative distribution function (CDF) is:

Bivariate Continuous Distributions¶

For joint PDF $p_{X,Y}(x,y)$ satisfying $\int_{-\infty}^\infty \int_{-\infty}^\infty p_{X,Y}(x,y)\, dx\, dy = 1$:

• Marginals:

  $$p_X(x) = \int_{-\infty}^\infty p_{X,Y}(x,y)\, dy, \qquad p_Y(y) = \int_{-\infty}^\infty p_{X,Y}(x,y)\, dx$$

  (16)

• Conditional:

  $$p_{X \mid Y}(x \mid y) = \frac{p_{X,Y}(x,y)}{p_Y(y)}$$

  (17)

• Independence: $X \perp Y \iff p_{X,Y}(x,y) = p_X(x)\, p_Y(y)$

Gaussian (Normal) Distribution¶

The Gaussian distribution is one of the most important models in machine learning.

Univariate:

where $\mu$ is the mean and $\sigma^2$ is the variance.

Multivariate ($D$-dimensional vector $\mathbf{x}$):

where $\boldsymbol{\mu}$ is the mean vector and $\boldsymbol{\Sigma}$ is the covariance matrix.