Key Points

Choosing Your First Conventional ML Baseline

Choose the task type before choosing the algorithm.
A good starter model basket includes both simple baselines and one or two stronger comparison options.
Conventional models are usually the right first step for structured or limited data.
Stronger models should be added for a reason, not because they sound more advanced.

Training in the bootcamp means following a short, reproducible process, not just running one line of code.
A train/test split is the standard first check for generalisation.
Default model settings may not work well; tuning hyperparameters with validation or cross-validation can help.
Random and stratified splits are common defaults for ordinary classification, but time series and grouped subject data need different validation strategies.
Any further development should be justified by the data, the task, and the current state of the model and analysis.

Choose the primary metric before training, not after seeing the result.
Use residuals, confusion matrices, or cluster interpretation to look beyond a single headline score.
Responsible evaluation means documenting who may be missing, what errors matter, and what claims the evidence can support.

Better features often improve a model more than switching to a more complex algorithm.
Different task types need different representations.
Text, images, and time series usually need task-specific feature extraction before a conventional baseline makes sense.
Spectral data and time-series data are often similar from a feature engineering perspective because both are ordered signals.
TF-IDF is useful because it reduces the influence of very common words and highlights terms that are more informative for a particular document.
Genomics and bioinformatics data often need sequence or marker-based features such as k-mers, motifs, SNP markers, or pathway summaries.
Feature engineering and feature learning sit on the same continuum of representation choices.

Feature learning means the model learns useful internal representations rather than relying only on hand-crafted features.
Transfer learning is usually more realistic than training from scratch for workshop-scale projects.
Transfer learning is about reusing a representation that has already been learned elsewhere.
The most practical question is often not “should I use deep learning?” but “is there a useful pre-trained representation for my data type?”
Conventional baselines still matter because they tell you whether the extra complexity is justified.

Neural networks are useful when patterns are too complex for hand-crafted features
A neuron performs a weighted sum + bias → activation function
Without activation functions, networks behave like linear models
A single perceptron can only solve linearly separable problems
Stacking layers allows networks to learn complex nonlinear relationships

Self-attention: each token evaluates relationships with all others
Enables contextual understanding (e.g., resolving pronouns)
Multi-head attention: multiple attention mechanisms run in parallel
Residual connections + normalization: prevent information loss, improve training stability (Pre-Norm common in modern models)
Vision Transformers (ViTs): treat image patches as tokens use attention to model global relationships
Transformers are highly flexible and used across: NLP, computer vision, time series

LLMs are built on Transformer architectures and trained at massive scale
Core idea: predict the next token given previous tokens
Language modelling is based on probability of sequences
Tokenization breaks text into manageable units: word-level or subword-level (BPE, WordPiece, SentencePiece)
Most LLMs use decoder-only transformers: embeddings → self-attention → output probabilities
Self-attention enables context-aware predictions across tokens