HumachLearnLearn · explore · apply
Start learning
Statistics & Probability · Lesson 18

Hypothesis Testing

Hypothesis Testing is a probability/statistics concept used to describe uncertainty.

ConceptWorked examplePracticeKnowledge check
Textbook walkthrough

What Hypothesis Testing actually means

Hypothesis Testing is a probability/statistics concept used to describe uncertainty. Probability models connect random outcomes to numerical summaries such as expectation, variance and conditional probability; inference uses the distribution of statistics across hypothetical samples to reason from a sample to a population.

Hypothesis Testing matters because probability and statistics describe variation rather than just calculate point summaries. They provide the basis for uncertainty, sampling reasoning, hypothesis tests and probabilistic model interpretation.

Deeper walkthrough

Read Hypothesis Testing as a mechanism, not a recipe

Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Define the random variable/event and probability model. Stage 2: Compute or estimate probabilities/statistics under the model. Stage 3: Distinguish a population parameter from a sample statistic. Final checkpoint: Quantify sampling uncertainty with an interval, posterior or other appropriate method.

Mechanism

Follow the transformation

Define the random variable/event and probability model.

Compute or estimate probabilities/statistics under the model.

Distinguish a population parameter from a sample statistic.

Evidence

Know what would convince you

  • Compute a tiny example by hand or simulate a simple case where the expected behaviour is known.
  • Check units, sample size, denominator and assumptions before interpreting the statistic.
Useful distinctionRandom variable: Maps outcomes to numerical values.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1

Define the random variable/event and probability…

Define the random variable/event and probability model. This is an input-preparation stage for Hypothesis Testing. Verify the relevant type, shape, units, keys, missingness or assumptions before later steps depend on them.

Input focus: confirm the data/object, units, type, shape and assumptions before the next operation depends on them.
How it works

Trace the mechanism step by step

  1. Define the random variable/event and probability model.
  2. Compute or estimate probabilities/statistics under the model.
  3. Distinguish a population parameter from a sample statistic.
  4. Use conditional probability when information changes what outcomes are plausible.
  5. Quantify sampling uncertainty with an interval, posterior or other appropriate method.
Worked demonstration

Make the concept concrete

Demonstration

Python example

# Step 1 — Compute the right-hand expression and store its result in `observed_diff` for the next step.
observed_diff = 4.2
# Step 2 — Compute the right-hand expression and store its result in `null_se` for the next step.
null_se = 1.5
# Step 3 — Compute the right-hand expression and store its result in `z` for the next step.
z = observed_diff/null_se
# Step 4 — Display the current value explicitly so the result/state can be inspected during execution.
print("z≈", round(z,2))
Expected / illustrative result
z≈2.8
A test asks how extreme the observed statistic is under a null model; practical importance still requires an effect-size interpretation.
Interpret the result.

For Hypothesis Testing, connect the displayed result to the specific input and mechanism above; independently verify one value/state change rather than treating successful execution as proof.

Distinctions & related ideas

Know what this is — and what it is not

Random variableMaps outcomes to numerical values.
ExpectationProbability-weighted long-run average.
VarianceExpected squared deviation from the mean.
Sampling distributionDistribution a statistic would have across repeated samples.
Bayes ruleUpdates a conditional probability by combining prior plausibility and likelihood of evidence.
Use deliberately

When it is appropriate

Use Hypothesis Testing when the statistical quantity or inferential idea matches the variable type, sampling process and question being asked.

Boundary conditions

When to stop or reconsider

Do not interpret the result beyond the assumptions and design that support it; distinguish descriptive evidence, uncertainty and causal claims.

Common mistakes

Failure modes to recognise

  • Using a summary or test that does not match the variable scale, dependence structure or sampling design.
  • Treating a point estimate or p-value as a complete statement without effect size, uncertainty or context.
  • Confusing association with causation or sample behaviour with a guaranteed population truth.
Verification

How to check the result

  • Compute a tiny example by hand or simulate a simple case where the expected behaviour is known.
  • Check units, sample size, denominator and assumptions before interpreting the statistic.
  • Change one observation/assumption and predict how the estimate or uncertainty should respond.
Connection to feature importance

Feature significance in ML

A feature-level p-value can be used as one screening signal, but it should be paired with effect size, multiple-testing awareness and predictive validation. Compare it with univariate AUC and RFE rather than treating p < 0.05 as automatic importance.

Open Explainable AI Playground →Feature-selection workflow
Hands-on practice

Demonstrate understanding

Try this:

Build a tiny, inspectable example of Hypothesis Testing. First define the random variable/event and probability model. Then compute or estimate probabilities/statistics under the model. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.

Use a very small numeric/categorical example and calculate one quantity manually. Separate what is observed in the sample from what is inferred about a wider process.
Knowledge check

Check reasoning, not memorisation

Before trusting a result from Hypothesis Testing, which check provides the strongest evidence that you understand and applied it correctly?

Quick reference

Keep the important distinctions visible

Step 1Define the random variable/event and probability model.
Step 2Compute or estimate probabilities/statistics under the model.
Step 3Distinguish a population parameter from a sample statistic.
Step 4Use conditional probability when information changes what outcomes are plausible.
Lesson summary

What to remember

  • Hypothesis Testing is a probability/statistics concept used to describe uncertainty. Probability models connect random outcomes to numerical summaries such as expectation, variance and conditional probability; inference uses the distribution of statistics across hypothetical samples to reason from a sample to a population.
  • Define the random variable/event and probability model.
  • Using a summary or test that does not match the variable scale, dependence structure or sampling design.
  • Compute a tiny example by hand or simulate a simple case where the expected behaviour is known.