Feature Engineering & Selection
Feature engineering creates informative representations; feature selection removes redundant, noisy or costly variables. Both should be evaluated inside the validation process. The topic is split into focused lessons so definitions, implementation decisions, diagnostics and common failure modes can be learned separately.
Learn the mechanism one decision at a time
Work through the lessons in order if the topic is new. If you already know the basics, open the specific leaf lesson that matches the operation, diagnostic or failure mode you need.
1Definition→
2Mechanism→
3Example→
4Diagnostic→
5Decision
Domain featuresRatios, interactions, lags, rolling statistics and physically meaningful transformations can expose structure that simple learners otherwise miss. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
02Filter methodsVariance filters, correlation, mutual information and univariate tests rank features without repeatedly fitting the final model. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
03Wrapper methodsRecursive Feature Elimination and sequential selection evaluate subsets using a predictive model, often at high computational cost. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
04Embedded methodsL1 regularisation, tree importance and sparsity-inducing learners perform selection during model fitting. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.
05Stability and leakageA selected feature should remain useful across folds and time. Selection performed before CV can leak target information. The important practical question is not only how the technique is defined, but what assumptions it introduces, which data are allowed to influence it, and how its effect should be validated on unseen evidence.