Text Feature Preparation
Text Feature Preparation groups the core ideas a learner needs at the 5 · data preprocessing & feature engineering stage. Work through the lessons in order when new to the area, or use them independently as a reference when implementing an analysis.
Learn the mechanism one decision at a time
Work through the lessons in order if the topic is new. If you already know the basics, open the specific leaf lesson that matches the operation, diagnostic or failure mode you need.
1Definition→
2Mechanism→
3Example→
4Diagnostic→
5Decision
Tokenisation intuitionTokenisation intuition is a practical concept within Text Feature Preparation. It helps turn the broader workflow stage “5 · Data Preprocessing & Feature Engineering” into an explicit analytical decision that can be explained, implemented and checked. The concept should be understood in terms of purpose, mechanism, assumptions, evidence and downstream consequences.
02Bag-of-wordsBag-of-words is a practical concept within Text Feature Preparation. It helps turn the broader workflow stage “5 · Data Preprocessing & Feature Engineering” into an explicit analytical decision that can be explained, implemented and checked. The concept should be understood in terms of purpose, mechanism, assumptions, evidence and downstream consequences.
03TF-IDFTF-IDF is a practical concept within Text Feature Preparation. It helps turn the broader workflow stage “5 · Data Preprocessing & Feature Engineering” into an explicit analytical decision that can be explained, implemented and checked. The concept should be understood in terms of purpose, mechanism, assumptions, evidence and downstream consequences.
04N-gramsN-grams is a practical concept within Text Feature Preparation. It helps turn the broader workflow stage “5 · Data Preprocessing & Feature Engineering” into an explicit analytical decision that can be explained, implemented and checked. The concept should be understood in terms of purpose, mechanism, assumptions, evidence and downstream consequences.
05Text normalizationText normalization is a practical concept within Text Feature Preparation. It helps turn the broader workflow stage “5 · Data Preprocessing & Feature Engineering” into an explicit analytical decision that can be explained, implemented and checked. The concept should be understood in terms of purpose, mechanism, assumptions, evidence and downstream consequences.
06EmbeddingsEmbeddings is a practical concept within Text Feature Preparation. It helps turn the broader workflow stage “5 · Data Preprocessing & Feature Engineering” into an explicit analytical decision that can be explained, implemented and checked. The concept should be understood in terms of purpose, mechanism, assumptions, evidence and downstream consequences.
07Train-test vocabulary leakageTrain-test vocabulary leakage is a practical concept within Text Feature Preparation. It helps turn the broader workflow stage “5 · Data Preprocessing & Feature Engineering” into an explicit analytical decision that can be explained, implemented and checked. The concept should be understood in terms of purpose, mechanism, assumptions, evidence and downstream consequences.