Follow the transformation
Construct the design matrix X and target y.
Estimate coefficients β that minimise squared residuals.
Generate predictions ŷ = Xβ (plus intercept).
Linear Models · Lesson 21
Linear Regression
Linear regression models a continuous target as a weighted sum of input features plus an intercept.
What Linear Regression actually means
Linear regression models a continuous target as a weighted sum of input features plus an intercept. Ordinary least squares chooses coefficients that minimise the sum of squared residuals between observed and predicted targets.
Linear Regression matters because linear models provide an interpretable baseline and expose central ideas such as coefficient estimation, probability links and regularisation. They also show clearly how preprocessing and penalty choices affect fitted parameters.
Deeper walkthrough
Read Linear Regression as a mechanism, not a recipe
Treat this as a sequence of observable decisions rather than one opaque command. Stage 1: Construct the design matrix X and target y. Stage 2: Estimate coefficients β that minimise squared residuals. Stage 3: Generate predictions ŷ = Xβ (plus intercept). Final checkpoint: Use validation to assess predictive generalisation; regularise when complexity/collinearity warrants it.
Know what would convince you
- Fit a tiny or baseline case first and confirm prediction shape/range and a few outputs.
- Evaluate with the same held-out folds/metric as competing models and inspect variability, not just the mean.
Useful distinctionLinear regression: Continuous target; linear in chosen features.
Click a stage to inspect what happens, what changes, and what should be checked before moving on.
Stage 1
Construct the design matrix X and…
Construct the design matrix X and target y. This is an input-preparation stage for Linear Regression. Verify the relevant type, shape, units, keys, missingness or assumptions before later steps depend on them.
Input focus: confirm the data/object, units, type, shape and assumptions before the next operation depends on them.
Mathematical / formal view
Prediction: ŷ = β0 + β1 x1 + ... + βp xp. OLS minimises Σ_i (y_i - ŷ_i)^2.
Trace the mechanism step by step
- Construct the design matrix X and target y.
- Estimate coefficients β that minimise squared residuals.
- Generate predictions ŷ = Xβ (plus intercept).
- Inspect residuals for nonlinearity, heteroscedasticity, outliers and dependence.
- Use validation to assess predictive generalisation; regularise when complexity/collinearity warrants it.
Make the concept concrete
Demonstration
Python / scikit-learn example
# Step 1 — Import the module so its functions/classes are available to the rest of this example.
import numpy as np
# Step 2 — Import only the named objects needed by the following steps, keeping dependencies explicit.
from sklearn.linear_model import LinearRegression
# Step 3 — Construct `X` as an array so vectorised numerical operations can be applied consistently.
X=np.array([[1],[2],[3],[4]],float); y=np.array([2,4.2,5.8,8.1])
# Step 4 — Fit the model or transformer, learning its parameters from the supplied training data.
m=LinearRegression().fit(X,y)
# Step 5 — Display the current value explicitly so the result/state can be inspected during execution.
print(round(m.coef_[0],2), round(m.intercept_,2))
# Step 6 — Display the current value explicitly so the result/state can be inspected during execution.
print(round(m.predict([[5]])[0],2))Expected / illustrative result
A slope near 2 and prediction near 10 illustrate fitting a linear relationship by squared-error minimisation.
Interpret the result.
For Linear Regression, connect the reported result to the exact training/validation/prediction step that produced it and check one prediction, fold or metric component independently.
Know what this is — and what it is not
Linear regressionContinuous target; linear in chosen features.RidgeLinear regression with L2 shrinkage.LassoLinear regression with L1 shrinkage and possible sparsity.Polynomial featuresStill a linear model in transformed features, but nonlinear in the original input.When it is appropriate
Use Linear Regression when its inductive assumptions fit the feature/target structure and it can be compared fairly with a simpler baseline on unseen data.
When to stop or reconsider
Prefer a simpler or different model when the sample size, representation, computational budget, interpretability requirement or data geometry conflicts with this method.
Failure modes to recognise
- Judging the model only by training fit instead of generalisation on held-out data.
- Comparing models with inconsistent preprocessing, folds or evaluation metrics.
- Tuning complexity without checking a simple baseline, error patterns and variance across splits.
How to check the result
- Fit a tiny or baseline case first and confirm prediction shape/range and a few outputs.
- Evaluate with the same held-out folds/metric as competing models and inspect variability, not just the mean.
- Inspect errors/residuals or decision boundaries and vary one key hyperparameter to verify expected behaviour.
Demonstrate understanding
Try this:
Build a tiny, inspectable example of Linear Regression. First construct the design matrix X and target y. Then estimate coefficients β that minimise squared residuals. Write the expected result before running it, and explain one condition that would make the result misleading or invalid.
Start with a small baseline and a fixed validation split/fold assignment. Predict what increasing or decreasing one complexity control should do before testing it.
Check reasoning, not memorisation
Before trusting a result from Linear Regression, which check provides the strongest evidence that you understand and applied it correctly?
Keep the important distinctions visible
Step 1Construct the design matrix X and target y.Step 2Estimate coefficients β that minimise squared residuals.Step 3Generate predictions ŷ = Xβ (plus intercept).Step 4Inspect residuals for nonlinearity, heteroscedasticity, outliers and dependence.What to remember
- Linear regression models a continuous target as a weighted sum of input features plus an intercept. Ordinary least squares chooses coefficients that minimise the sum of squared residuals between observed and predicted targets.
- Construct the design matrix X and target y.
- Judging the model only by training fit instead of generalisation on held-out data.
- Fit a tiny or baseline case first and confirm prediction shape/range and a few outputs.