Supervised Learning

Concept Definition: Learning with the Answer Key

Supervised learning is the most intuitive of the three modeling paradigms: given the model a batch of "questions + answer keys" (pairs of (x, y)), it learns the mapping from x to y, so it can make reasonable predictions for unseen x's. The "supervision" comes from the fact that training data comes with labels — like a teacher grading assignments, each sample has the correct answer, and the model keeps correcting itself against it.

Formally: assume there exists an unknown function f such that y = f(x). The training set D = {(x₁,y₁),…,(xₙ,yₙ)} is a sample of f, and the learning algorithm searches for a hypothesis ĥ in the hypothesis space H that minimizes error on the training set while also performing well on unseen data (generalization). See the four-element framework in What is Machine Learning.

Three Task Types

Task	Type of Output y	Typical Problem	Evaluation Metrics
Classification	Discrete categories	Spam detection, image classification, disease diagnosis	Accuracy, Precision/Recall, F1, AUC
Regression	Continuous values	House price prediction, sales forecasting, temperature prediction	MSE, MAE, R²
Ranking	Order relationships	Search ranking, recommendation ranking	NDCG, MAP, MRR

These three often convert into one another: regression outputs can be thresholded to become classification; probabilities output by classifiers can themselves be used for ranking. Which task definition you choose depends on what the business needs — predicting "whether this user will churn" (classification) and "how high this user's churn risk is" (regression + threshold) look similar on the surface, but their decision approaches are completely different.

Training Objectives: Loss Functions

The "answer key" of supervised learning is converted into an optimizable objective through loss functions. Choosing a loss function is answering the question "what kind of mistake is most expensive?":

Loss Function	Use Case	Characteristics
Mean Squared Error (MSE)	Regression	Heavy penalty for large errors (squared), sensitive to outliers
Mean Absolute Error (MAE)	Regression	Robust to outliers, but gradient is discontinuous
Huber loss	Regression	A compromise between MSE and MAE, commonly used in industry
Cross-entropy (log loss)	Classification	Naturally matches probabilistic outputs, well-behaved gradients, the de facto standard for classification
Hinge loss	Classification	Margin loss for SVMs
Pairwise loss	Ranking	Compares relative order of sample pairs

Why use cross-entropy instead of MSE for classification?

In classification, y is a one-hot category. Computing the gradient of MSE on probabilistic outputs causes vanishing gradients (the derivatives of sigmoid approach zero at both ends), leading to extremely slow convergence. The gradient of cross-entropy is proportional to "predicted probability minus true label" — the bigger the error, the harder the push. This is a classic example of "loss functions should match the output distribution."

Classic Algorithm Families

1. Linear Models: the Simplest, Most Reliable Baseline

Linear regression: assumes ĥ(x) = w·x + b, solved via least squares or gradient descent. Extremely interpretable (coefficients w directly represent "contribution per unit of feature to the output"), making it the default baseline for all regression tasks. Logistic regression: wraps a sigmoid around the linear output to produce probabilities, paired with cross-entropy loss. Despite the name containing "regression," it is the most important classification model.

Positioning of linear models: always run these first. They train fast, are interpretable, and are the fastest way to determine "whether the problem itself has signal" — if a linear model doesn't even beat the majority-class baseline, the problem likely lies in the features or data, not in insufficient model complexity. See Linear Models and Logistic Regression.

2. Tree Models: the Kings of Tabular Data

Decision trees: recursively split samples by feature values, forming if-then rule trees. Interpretable, no normalization needed, natively handles categorical features. Random forests: bagging ensembles of multiple trees, low variance and robust. Gradient boosting trees (GBDT / XGBoost / LightGBM): additive models that iteratively fit residuals, representing the long-standing best practical choice for tabular data — Kaggle competitions, risk control, and recommendation ranking feature engineering all rely on them.

Positioning of tree models: the primary weapon for tabular data. Their advantages include insensitivity to feature scales, ability to capture non-linear interactions, fast training, and available feature importance for interpretability. See Tree Models and Ensemble Learning.

3. Kernel Methods: SVM

Support Vector Machines (SVMs) map samples to a high-dimensional space via kernel functions to find the maximum-margin hyperplane. Before deep learning, they were the benchmark approach for text and images. Today's position: still valuable for small-sample, high-dimensional sparse features (e.g., text TF-IDF), but large-sample and unstructured data have been taken over by neural networks.

4. Neural Networks: the Champions of Unstructured Data

From multilayer perceptrons to CNNs and Transformers, neural networks dominate images, text, and speech with end-to-end representation learning. Positioning: the default choice for unstructured data; on tabular data, they may not beat tree models. See Deep Learning Fundamentals.

Algorithm Selection Decision Tree

What is the data format?
├── Tabular data → Small sample / need interpretability → Logistic regression or decision tree
│                  Medium sample → Random forest / XGBoost / LightGBM
│                  Large sample + strong features → Try neural networks (may not beat trees)
├── Images → CNN family (ResNet, EfficientNet, ViT)
├── Text → Transformer family (BERT, GPT) pre-training + fine-tuning
├── Sequences / time → Time feature engineering + tree models (tabular) or Transformer / LSTM
└── Mixed multimodal → Pre-trained multimodal models

First rule of selection: start with a baseline

Don't pick the "most advanced model" on intuition. The correct order is always: majority-class baseline → linear model → simple tree model → complex model. Each step answers "is more complex truly better?" Jumping straight into a Transformer, tuning for three months, and discovering XGBoost was stronger is the most classic tuition fee for Kaggle beginners.

Four Core Mechanisms of Supervised Learning

1. Overfitting: Learning Too Much

The model memorizes the noise in the training set, leading to poor generalization. Signal: training metrics are far better than validation metrics. Countermeasures: regularization, more data, early stopping, cross-validation. This is the number one topic in supervised learning; for the full discussion, see Overfitting and Regularization.

2. Bias-Variance Tradeoff

Prediction error = bias² + variance + irreducible noise. Simple models have high bias (underfitting), complex models have high variance (overfitting). Tuning model complexity is about finding the balance between bias and variance. For the full discussion, see Model Evaluation and Validation.

3. Class Imbalance

When positive and negative samples are highly skewed (e.g., fraud detection at 1:1000), accuracy is an invalid metric (predicting all negative gives 99.9% accuracy). Countermeasures: resampling (under-sampling / over-sampling / SMOTE), adjusting class weights, switching to PR-AUC / recall metrics, threshold tuning. First figure out "which type of mistake is more costly" before deciding on a strategy.

4. Data Leakage

Test set information leaks into training, causing inflated offline metrics and production failures. Common sources: global normalization using test set statistics, random splitting of time series, features containing future information. Before splitting data, think "what could I actually see at this moment during real prediction?" See Data and Data Engineering.

An End-to-End Example: Logistic Regression Training

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.preprocessing import StandardScaler

# 1. Data: features X (2D), labels y (0/1)
rng = np.random.default_rng(42)
X = rng.normal(size=(1000, 2))
y = (X[:, 0] + 2 * X[:, 1] + rng.normal(size=1000) > 0).astype(int)

# 2. Split: split first, then normalize, to avoid data leakage
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

# 3. Feature scaling (only fit on training set!)
scaler = StandardScaler().fit(X_train)
X_train_s, X_test_s = scaler.transform(X_train), scaler.transform(X_test)

# 4. Train
model = LogisticRegression()
model.fit(X_train_s, y_train)

# 5. Evaluate
y_pred = model.predict(X_test_s)
y_prob = model.predict_proba(X_test_s)[:, 1]
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"AUC:      {roc_auc_score(y_test, y_prob):.3f}")
print(f"Learned coefficients: {model.coef_[0]}, intercept: {model.intercept_[0]:.3f}")

Two important details: split first, then normalize (to avoid leakage), and use stratify for class imbalance (to ensure consistent category distribution across train/test). These details represent the watershed between "can call an API" and "understands modeling"; see Common Pitfalls and Anti-Patterns.

Tradeoffs

• Interpretability vs Predictive Power: linear/tree models are interpretable, deep models are black boxes. Risk control, healthcare, and legal domains require explanations; recommendation ranking only cares about performance. A compromise: use black-box models + post-hoc explanation via SHAP (see Interpretability and Fairness).
• Training Cost vs Performance: XGBoost trains in minutes, fine-tuning BERT takes GPU hours. Small teams start with "good enough and cheap."
• Offline Metrics vs Online Business Impact: an AUC increase of 0.01 doesn't necessarily translate to business improvement. Connecting offline evaluation with A/B testing is a hallmark of mature teams.
• Model Complexity vs Maintenance Cost: every additional model and feature pipeline adds long-term maintenance burden. If a simple model solves it, don't use a complex one.

Further Reading

• What is Machine Learning — supervised learning's position among the three paradigms
• Unsupervised Learning — what to do without labels
• Model Evaluation and Validation — metrics, cross-validation, bias-variance
• Overfitting and Regularization — the number one pitfall in supervised learning
• Feature Engineering — the model's ceiling is determined by features
• Linear Models and Logistic Regression — applied breakdown of the baseline models in this article
• Tree Models and Ensemble Learning — the kings of tabular data
• Deep Learning Fundamentals — how neural networks work
• How to Choose Frameworks and Tools — how to choose between sklearn, XGBoost, and PyTorch

References

• Tom Mitchell. Machine Learning (1997), Chapter 1–4 — classic textbook on supervised learning
• Bishop. Pattern Recognition and Machine Learning (2006) — authoritative textbook from a probabilistic perspective
• James, Witten, Hastie, Tibshirani. An Introduction to Statistical Learning (ISLR) — the bible of supervised learning, available free online
• scikit-learn User Guide: Supervised learning — complete algorithm family and API reference
• Pedregosa et al. Scikit-learn: Machine Learning in Python (JMLR, 2011)