Probabilistic Methods

Let $X_1, X_2, \ldots$ be i.i.d. Bernoulli(p) random variables (1 = heads, 0 = tails). The running relative frequency after n flips is the sample mean.

The Weak LLN states that for any $\varepsilon > 0$, the probability that the sample mean deviates from the true expected value by more than $\varepsilon$ goes to zero as $n \to \infty$.

For a fair coin, $\mu = E[X] = p = 0.5$. The simulator above draws from Bernoulli(0.5) and plots $\bar{X}_n$ vs $n$. Watch the running frequency bounce wildly at first, then stabilize near the red reference line at 0.5.

The LLN guarantees convergence in probability, not almost-sure equality for any finite n. Early fluctuations are large; they shrink as $1/\sqrt{n}$. The LLN is the theoretical foundation for using observed frequencies to estimate true probabilities.

Increasing n reduces the standard error of $\hat{p}$. The histogram of $\hat{p}$ values becomes narrower and more concentrated around the true p. This reflects real statistical precision: you are collecting more data per estimate.

Increasing R does NOT change the spread of the histogram; it only makes the histogram smoother and less ragged. You are simply drawing more replications to better approximate the theoretical sampling distribution. R is a simulation parameter, not a data parameter.

n governs statistical precision (real-world quantity). R governs simulation quality (computational quantity). Confusing the two is a common mistake: running more simulations does not make your estimator more accurate in practice.

Because $E[X_i] = p$ for each i, linearity of expectation gives the mean of the sampling distribution.

Since the $X_i$ are independent with $\mathrm{Var}(X_i) = p(1-p)$, the variance of the sum scales by $1/n^2$.

By the Central Limit Theorem, for large n the standardized $\hat{p}$ is approximately standard normal.

With p = 0.4, n = 100, the theoretical SE = $\sqrt{0.4 \cdot 0.6 / 100} = 0.049$. Run the simulator and check that the histogram spread matches this value and that the bell shape is visible.

The normal approximation is reliable when $np \geq 10$ and $n(1-p) \geq 10$. Here $100 \times 0.4 = 40$ and $100 \times 0.6 = 60$, both well above 10.

No matter the shape of the population distribution (provided it has finite mean and variance), the standardized sample mean converges in distribution to the standard normal as $n \to \infty$.

The sample mean $\bar{X}$ is approximately normal for large n, regardless of the shape of the population.

An Exponential($\lambda=1$) distribution is right-skewed with $\mu = 1$ and $\sigma = 1$. Individual values are nowhere near normal. But the simulator draws n = 30 values, computes their mean, and repeats R = 600 times.

Even with n = 30, the histogram of $\bar{X}$ is approximately bell-shaped and centered at $\mu = 1$. The spread is $\sigma/\sqrt{n} = 1/\sqrt{30} \approx 0.183$.

The CLT is why so many statistical procedures (z-tests, t-tests, confidence intervals) work even when the data are not normally distributed. The normality is in the estimator, not the raw data.

Using the normal approximation and the fact that the 97.5th percentile of N(0,1) is 1.96, the two-sided 95% CI is:

The total width of the 95% CI depends on $\hat{p}$ and n. The maximum width (over p) occurs at $p = 0.5$.

Run the confidence simulator: for each of many simulations, a fresh $\hat{p}$ is drawn and its 95% CI plotted. Green intervals contain the true p = 0.5; red ones miss it. About 95% should be green.

A 95% CI does NOT mean "there is a 95% probability that p is in this interval." The true p is fixed. The interval is random. The correct statement: if we repeated this procedure many times, 95% of the resulting intervals would contain the true p.

Increasing n from 100 to 400 cuts the interval width roughly in half (since width $\propto 1/\sqrt{n}$). You need 4 times as much data to halve the margin of error.

If the population standard deviation $\sigma$ is known (rare in practice), use the z-based interval.

In practice $\sigma$ is unknown and estimated by s. Replacing $\sigma$ with s introduces extra variability, which the t-distribution accounts for. With degrees of freedom $\nu = n-1$:

The t-distribution with $\nu$ degrees of freedom is wider than N(0,1). For small n, $t_{n-1, 0.025}$ can be much larger than 1.96, leading to a wider (more honest) interval. As $n \to \infty$, $t_{\nu} \to N(0,1)$.

For large n the CLT makes the t-interval valid regardless of the data distribution. For small n, the t-interval requires approximately normal data. The interval is more sensitive to skewness when n is small.

The same coverage idea from Problem 8.5 applies: run the confidence simulator to see that about 95% of t-intervals contain the true mean when the data are normal.

The null hypothesis is that the coin is fair; the alternative is two-sided (coin could be biased either way).

The observed proportion is $\hat{p} = 61/100 = 0.61$. Under $H_0$, the standard error is computed using $p_0 = 0.5$.

Use the simulator to generate 800 samples of size 100 from Bernoulli(0.5) (the null distribution). The p-value is the fraction of simulated $\hat{p}$ values at least as extreme as 0.61.

Standardize the observed $\hat{p}$ under $H_0$ to get a z-score, then find the two-tailed p-value from the standard normal.

The two-tailed p-value is $2\Phi(-|z|)$ where $\Phi$ is the standard normal CDF.

Since $0.0278 < 0.05$ we reject $H_0$ at the 5% level. There is statistically significant evidence that the coin is not fair. However, statistical significance does not imply practical significance: 61% vs 50% may or may not be meaningful depending on context.

Under the null, each face has probability 1/6; the expected count for each face is $E_i = 120/6 = 20$.

Compute the chi-square test statistic by summing the squared deviations of observed from expected, divided by expected.

With observed counts [18, 22, 17, 25, 19, 19] and expected 20 for each:

Summing the terms: $4/20 + 4/20 + 9/20 + 25/20 + 1/20 + 1/20 = 44/20 = 2.2$.

Under $H_0$, D follows approximately a chi-square distribution with $k - 1 = 5$ degrees of freedom. The critical value at $\alpha = 0.05$ is $\chi^2_{5,\,0.05} = 11.07$.

Since $D = 2.2 < 11.07$, we fail to reject $H_0$. The data are consistent with a fair die. Use the die simulator to see what typical D values look like from a fair die.

You can also simulate D under $H_0$ by rolling a fair die 120 times, computing D, and repeating thousands of times. The resulting histogram should match $\chi^2(5)$, confirming the theoretical result.

Under the null hypothesis $H_0: \mu_A = \mu_B$, all group labels are exchangeable. We can shuffle the 40 observations between the two groups, recompute the mean difference, and repeat many times to build the null distribution.

The observed test statistic is the difference in sample means.

Pool all 40 values. For each of B = 5000 permutations: randomly assign 20 to "group A" and 20 to "group B", compute $T^* = \bar{X}^*_A - \bar{X}^*_B$. The collection $\{T^*_1, \ldots, T^*_{5000}\}$ approximates the null distribution of the test statistic.

Count the fraction of permuted differences at least as extreme as the observed value (two-tailed).

Assuming approximately normal data (or invoking the CLT), a 95% CI uses the Welch t-approximation with pooled standard error.

The permutation test makes no distributional assumptions beyond exchangeability under $H_0$. It is especially valuable for small samples or non-normal data. The t-based CI is complementary: it quantifies the magnitude and uncertainty of the difference.

If the permutation p-value is below $\alpha = 0.05$, the observed difference is unlikely under exchangeability — evidence that the groups differ. The CI tells us the plausible range of the true difference $\mu_A - \mu_B$.

Clearly describe the data-generating process. Identify the population, the sampling mechanism, and the outcome variable. Example: "We recruit 80 visitors to a website and record whether each one clicks a call-to-action button (1) or not (0)." State what makes the outcomes random and whether independence is plausible.

Separate the fixed unknowns (parameters) from the random quantities (statistics). The true click probability p is a fixed but unknown parameter. The sample proportion $\hat{p}$ is a random variable that varies across repetitions of the experiment.

Obtain the data. For a simulation, draw n = 80 Bernoulli(p) values. For a real study, collect observations from the population. Record $n$, the individual outcomes, and any relevant covariates. Check for obvious data quality issues (missing values, implausible entries).

Calculate point estimates of the parameters of interest. Verify their properties: unbiasedness ($E[\hat{p}] = p$), standard error, and consistency (SE $\to 0$ as $n \to \infty$).

Construct a 95% CI for the parameter using the appropriate method (Wald z-interval for proportions; t-interval for means). State it in plain language: "We are 95% confident the true click rate is between a% and b%."

State the null and alternative hypotheses, choose $\alpha$ (usually 0.05), compute the test statistic and p-value, and state the decision. Example: test $H_0: p = 0.5$ vs $H_1: p \neq 0.5$.

Draw conclusions that are consistent with both the CI and the hypothesis test. Address four key questions: (1) Is the result statistically significant? (2) Is it practically meaningful? (3) What are the limitations (sample size, non-randomness, confounding)? (4) What would you do next?

The LLN guarantees your estimate is close to the truth for large n. The CLT justifies the normal-based CI and z-test. The CI tells you precision; the test tells you whether chance alone could explain the result. Neither replaces subject-matter judgment.

Do not say "the probability that p is in the CI is 95%" (p is not random). Do not equate statistical significance with practical importance. Do not ignore multiple testing. Do not extrapolate beyond the sampled population. Report effect sizes and CIs, not just p-values.