Clustering and Dimensionality Reduction

One-line positioning: clustering and dimensionality reduction are two classes of unsupervised methods with "no standard answer" — clustering answers "how many groups does the data form, what is each group like," and dimensionality reduction answers "how to compress high-dimensional data into low dimensions without losing the main structure." They are the two most commonly used wrenches in the data exploration phase: clustering groups samples, dimensionality reduction compresses features, and the two are often used together (reduce first then cluster, or visualize clustering results after dimensionality reduction).

Compared to supervised learning, clustering intuition is universal — "put similar things together" is something anyone can say; but getting truly professional, you'll hit a wall: clustering has no objective "correct answer." Classification has accuracy, regression has MSE, but clustering can only rely on "does it look reasonable" and a series of flawed metrics. This article's goal is to make you thoroughly understand the math behind K-Means, choosing K, DBSCAN's density intuition, PCA's eigendecomposition, and t-SNE's random walk, and know what each can and can't do in real business. For a concept overview, see Unsupervised Learning; this article is its deep hands-on companion.

1. Clustering: Task Definition and Formalization

The task clustering solves is:

Given an unlabeled sample set X = {x₁, x₂, …, xₙ} (each xᵢ ∈ ℝᵈ is a d-dimensional feature vector), partition the samples into k mutually exclusive clusters C₁, C₂, …, Cₖ, such that intra-cluster similarity is maximized and inter-cluster similarity is minimized.

Formally, clustering solves for the mapping from a sample set to cluster labels:

Input and output of a clustering problem
┌─────────────────────────────────────────────────────┐
│ Input: X = {x₁, ..., xₙ}, each xᵢ ∈ ℝᵈ (unlabeled)    │
│ Output: cluster partition {C₁,...,Cₖ} and sample labels yᵢ ∈ {1..k} │
│ Objective: intra-cluster compactness + inter-cluster separation │
└─────────────────────────────────────────────────────┘

Three "degrees of freedom" here determine clustering's dilemmas:

1. How to measure "similarity"? Euclidean distance, cosine similarity, Mahalanobis distance, correlation coefficient... different measures yield completely different clusters. This is the first seemingly trivial but actually decisive choice in clustering.
2. How to quantify "compactness and separation"? Different objective functions (the inertia of K-Means mentioned below, the linkage criteria of hierarchical clustering) will select different partitions, even if they all "make sense."
3. What is k? Many algorithms (K-Means, GMM) require you to specify the number of clusters upfront, and this number almost never comes from the data itself in real business.

So the professional perspective on clustering is: an engineering tradeoff between objective functions and domain knowledge, not "automatically discovering the truth." We'll revisit "how to evaluate without a ground truth" in the Evaluation section later.

2. K-Means: The Most Popular, Also the Most Worth Explaining

K-Means is the most widely used clustering algorithm: a few dozen lines of code, O(n·k·d) per iteration, almost no hyperparameters to tune (except K). Precisely because it's simple, it's often treated as "blindly running one line KMeans(n_clusters=3).fit(X)," but its principles hide two core ideas of all unsupervised learning: coordinate descent and EM iteration.

1. Objective function and algorithm flow

K-Means minimizes the within-cluster sum of squares (WCSS, called inertia in sklearn):

$$ J = \sum_{i=1}^{n} \min_{j} | x_i - \mu_j |^2 $$

where μ₁, …, μₖ are k centroids. An equivalent formulation assigns each sample to the nearest centroid:

$$ J = \sum_{j=1}^{k} \sum_{x_i \in C_j} | x_i - \mu_j |^2 $$

The algorithm is the classic two-phase alternating iteration (coordinate descent):

① Initialize: randomly (or heuristically) pick k centroids μ₁, ..., μₖ
② Loop until centroids no longer change (or max iterations reached):
    E-step (assign): assign each sample xᵢ to the nearest centroid → update cluster membership
    M-step (update): μⱼ = mean of all samples in Cⱼ → update centroids
③ Output: final cluster membership and centroids

# 50-line hand-written K-Means, stripping all wrappers
import numpy as np

def kmeans(X, k, max_iter=100, tol=1e-4, seed=0):
    rng = np.random.default_rng(seed)
    # ① Randomly pick k initial centroids from samples
    centers = X[rng.choice(len(X), k, replace=False)]
    for _ in range(max_iter):
        # ② E-step: distance matrix argmin to assign clusters
        dists = np.linalg.norm(X[:, None, :] - centers[None, :, :], axis=2)
        labels = dists.argmin(axis=1)
        # ③ M-step: new centroids = cluster mean
        new_centers = np.array([X[labels == j].mean(axis=0) for j in range(k)])
        if np.abs(new_centers - centers).max() < tol:   # Convergence check
            break
        centers = new_centers
    return centers, labels

Note the name "centroid takes the mean" in the M-step — that's why it's K-Means. Euclidean distance and the mean are a natural pair: in Euclidean distance, the cluster representative that minimizes within-cluster sum of squares is the mean. This is a direct conclusion from calculus ("take the derivative of μ and set to zero"), and is why "squared distance + mean" can't be separated.

2. EM perspective: K-Means is a "hard" Gaussian Mixture Model

Describing K-Means as "iterative assignment + update" is only surface-level. A deeper understanding comes from the EM (Expectation-Maximization) algorithm: K-Means is a special case of the Gaussian Mixture Model (GMM) in the limit where "each sample belongs to exactly one cluster, and cluster variance goes to uniform and infinitely small."

• GMM assumes data is generated by a mixture of k Gaussians, and estimates parameters (mean, covariance, mixing weights) using EM iteration;
• K-Means's E-step "assign samples to the nearest centroid" is equivalent to GMM's E-step "compute posterior probability" in the limit where variance → 0 — the posterior becomes a 0/1 hard assignment;
• K-Means's M-step "take the mean" is equivalent to GMM's M-step "update Gaussian means."

This perspective has two practical values. First, it explains a common confusion: why K-Means only "excels at" spherical clusters — because in the GMM family it forces the simplest shape of "isotropic Gaussian"; when data is elongated, ring-shaped, or nested, K-Means's Euclidean distance will cut them apart rather than surround them. Second, it gives you an upgrade path: when you need "soft clustering" (probability of a sample belonging to multiple clusters) and clusters are elliptical, simply swap K-Means for GMM — the math is completely isomorphic.

3. Initialization: why n_init defaults to 10

K-Means's objective function is non-convex; iteration only guarantees convergence to a local optimum. Poor initial centroids may converge to a very poor local solution (e.g., two centroids land in the same true cluster). Common countermeasures:

Initialization method	Idea	Notes
Random initialization	Randomly pick k points from samples	Most straightforward; paired with multiple runs to keep the best
K-Means++	Select centroids sequentially, each new centroid more likely to be far from existing centroids	sklearn default; significantly improves solution quality for complex data
Multiple runs	n_init different initializations, keep the one with minimum inertia	n_init=10 in sklearn means this

K-Means++ (Arthur & Vassilvitskii, 2007) guarantees that the obtained solution is within a factor of 2(ln k+1) of the optimal value with high probability, a typical engineering wisdom of "trading randomness for quality." So in practice, keep the default n_init=10 or increase it; don't set it to 1 to save a few dozen milliseconds — clustering result instability is one of the most common rookie pitfalls (more pitfalls in Common Pitfalls).

4. Choosing K: the elbow method and silhouette score

K doesn't come from the algorithm; it must be given by a person. Two classic auxiliary approaches, each with limitations:

Elbow Method — plot the inertia curve vs. K, find the "inflection point where the decline flattens":

inertia
  │
  │               ← inflection point = "elbow"
  │            ·
  │         ·
  │      ·
  │   ·
  │·
  └─────────────────────── K
        1   2   3   4   5   6

The logic: as K increases by 1, clusters get finer, and inertia necessarily drops, but "the marginal benefit of splitting into one more cluster" shrinks quickly; the inflection point is where the cost-performance ratio is highest. Fatal weakness: real data often has no obvious inflection point, and the curve flattens gradually — "the elbow" depends on the observer's hand gesture. This is a heuristic, not a method.

Silhouette Score — for each sample, define:

$$ s_i = \frac{b_i - a_i}{\max(a_i, b_i)} $$

where aᵢ is the average distance between sample i and other samples in the same cluster (intra-cluster cohesion), and bᵢ is the average distance between sample i and samples in the nearest other cluster (inter-cluster separation). sᵢ ∈ [−1, 1]: close to 1 means "close and far" (close to same cluster, far from other); close to −1 means the sample was assigned to the wrong cluster. Overall silhouette score = the mean of all sᵢ; take the K that peaks the curve:

from sklearn.metrics import silhouette_score

Ks = range(2, 9)
scores = {k: silhouette_score(X, KMeans(n_clusters=k, random_state=0).fit_predict(X))
          for k in Ks}
best_k = max(scores, key=scores.get)   # K with the highest silhouette score

Silhouette score computation is O(n²), very slow for large samples (>100K), so sample first. It's a bit more "objective" than the elbow method, but still only geometric goodness, not guaranteed business meaning.

The real K comes from business

Print out the customer segmentation results for K=3 and K=5, and show them to the business stakeholder: "what are the profiles of these three groups? is there a group of 'high-value dormant customers'?" The K that business can explain is the correct K. The elbow method and silhouette score only help narrow the range from 3~8; the final call rests with domain knowledge. This echoes the core of Interpretability: model results must be explainable in human language.

3. DBSCAN and Hierarchical Clustering: Two Other Worldviews Beyond K-Means

K-Means has an implicit assumption: clusters are "spherical, uniformly dense, and roughly the same size." Real data often violates these assumptions. DBSCAN and hierarchical clustering circumvent them from two different angles.

1. DBSCAN: density-based clustering

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) defines clusters by density rather than distance. It has only two hyperparameters:

• eps (ε): neighborhood radius;
• min_samples (minPts): the minimum number of neighbors required to be a "core point."

Three types of points make up its worldview:

● = core point (≥ minPts points in neighborhood)   ○ = border point   × = noise point
         eps
     ○ ┌────┐
    ●──│ ●  │──○       Algorithm: from any core point, extend clusters along "density-connected"
    ●  │    │          relationships; core points connected to each other belong to one cluster;
     ○ └────┘          border points attach to nearby core points' clusters;
     ×  ×   × ×        isolated points labeled as noise (-1).

No need to specify K, can discover arbitrarily shaped clusters, automatically identifies noise points — these three features make it the first choice for anomaly detection and arbitrary-shape clustering. In sklearn:

from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.5, min_samples=5)
labels = db.fit_predict(X)   # noise points get label -1

Drawbacks: eps is sensitive to density variation (with a single eps, dense regions and sparse regions can't be simultaneously handled when densities vary); not applicable when cluster densities differ greatly; the "nearest neighbor" concept fails in high dimensions (see the motivation for dimensionality reduction below); parameters are unintuitive. Algorithm and parameter details can be found in the DBSCAN entry in the Glossary.

2. Hierarchical clustering: dendrograms and linkage criteria

Hierarchical clustering (agglomerative) works bottom-up: each sample starts as its own cluster, then repeatedly merge the "closest clusters" until only one remains. The record of the merging process is the dendrogram — a horizontal cut gives clustering at any granularity.

"Distance between clusters" can be defined in multiple ways (i.e., linkage criteria):

Linkage criterion	Inter-cluster distance	Tendency	Classic reference
Single linkage	Distance between the two closest points in each cluster	Prone to long "chain" clusters	Often produces unreasonable chains
Complete linkage	Distance between the two farthest points	Prefers spherical; sensitive to noise
Average linkage	Average distance between all pairs of points	More robust; commonly used in practice
Ward	Increment of within-cluster sum of squares after merging	Consistent with K-Means objective; clusters tend spherical	Ward (1963)

Hierarchical clustering's unique value is in exploration: the dendrogram lets you see the full spectrum from "1 cluster to n clusters," and choosing K becomes "at which level of the tree to cut." The cost is O(n²)~O(n³) computation; 100K samples become unmanageable.

3. Comparison of the three

Dimension	K-Means	DBSCAN	Hierarchical (agglomerative)
Cluster shape	Approximately spherical only	Arbitrary shapes	Depends on linkage criterion (Ward tends spherical)
Must specify K?	Yes	No	No (but need to decide "where to cut")
Noise handling	No concept; noise forced into a cluster	Natively identifies noise	No concept
Uneven cluster density	Not applicable	Not applicable (single eps)	Can handle
Time complexity	O(n·k·d·iter)	O(n²) (can be accelerated with indexing)	O(n²)~O(n³)
Hyperparameters	K	eps, min_samples	Linkage criterion, distance measure
Typical use	Large-scale customer segmentation	Anomaly detection, geo/spatial clustering	Exploratory analysis, gene clustering

One-line selection guide: certainly spherical and need speed → K-Means; shape unknown and need to catch outliers → DBSCAN; want to see the complete spectrum of clustering "from small to large" → hierarchical clustering.

4. The Clustering Evaluation Dilemma: Internal vs. External Metrics

Supervised learning evaluation has a "ground truth" (y); clustering doesn't. Thus evaluation naturally splits into two camps, each with hard flaws.

1. External metrics: using "ground truth" as a ruler

When data happens to have real labels (e.g., you know certain customers truly belong to three known market segments), you can use external metrics to measure how well the clustering results match the true groupings:

Metric	Idea	Value range
ARI (Adjusted Rand Index)	Statistically count the proportion of "sample pairs" that agree across two partitions, subtracting the expected agreement by chance	0 ≈ random, 1 = perfect match
NMI (Normalized Mutual Information)	The amount of information shared between two partitions	0 ≈ independent, 1 = perfect match

But here's a fatal logical trap, also pointed out in the concept page Unsupervised Learning: if you have real labels in hand, why do unsupervised learning? Directly using supervised learning to learn those labels will almost certainly perform better. External metrics are only valuable in two places: ① using labeled benchmark data to compare algorithm implementations (not solving business problems); ② checking whether "unsupervised-discovered clusters" match "known domain groupings" (validation, not replacement).

2. Internal metrics: without labels, but biased

Internal metrics only look at geometric structure:

• inertia / WCSS: the lower the better, but monotonically decreases with K, and favors spherical clusters;
• Silhouette score: measures "cohesion + separation"; friendly to convex clusters, gives low scores to irregular clusters — not because the clustering is bad, but because the metric itself is biased;
• DB index (Davies–Bouldin): ratio of intra-cluster scatter to inter-cluster distance; the lower, the better; fast to compute.

3. The ultimate judge: business interpretability

Laying three approaches together, the real engineering conclusion is:

External metrics  → Can only use when labels exist; when you have labels, you likely don't need clustering
Internal metrics  → Usable without labels; but only reflect geometric assumptions, contain no business meaning
Business interpretability  → Always usable; the true acceptance criterion for customer segmentation and market subdivision

The standard practice action: internal metrics select candidate K → manually inspect each cluster's profile (means, distributions, typical samples) → business stakeholder confirms "this group is indeed a market segment." Metrics narrow the search scope; humans make the final call. Don't use "highest silhouette score" as a delivery reason — that's not the same as "highest accuracy." For the full theory behind evaluation metrics, see Model Evaluation and Validation.

5. PCA: Principal Component Analysis

Now switching topics to dimensionality reduction. Dimensionality reduction solves two problems caused by high dimensions: ① the curse of dimensionality (data becomes sparse in high-dimensional space, distance metrics fail); ② the limits of visualization (humans can only see 2D/3D). PCA (Principal Component Analysis) is the most classic, most interpretable, and most widely applied linear dimensionality reduction method.

1. From covariance matrix to eigendecomposition

PCA's mathematical core has only three steps: centering → covariance matrix → eigendecomposition.

Let X be the n×d sample matrix (already column-centered, i.e., each column has mean 0). The covariance matrix is

$$ C = \frac{1}{n} X^\top X \quad \text{(d×d, symmetric positive semi-definite)} $$

Eigendecompose C:

$$ C v_j = \lambda_j v_j, \quad \lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d \ge 0 $$

Principal components are the eigenvectors vⱼ of the covariance matrix, and the corresponding eigenvalues λⱼ are the variance in that direction. The projection of data onto the subspace spanned by the first k eigenvectors is PCA's dimensionality reduction:

$$ Z = X V_k, \qquad Z \in \mathbb{R}^{n \times k}, \quad V_k = [v_1, \dots, v_k] $$

It can be proved (Rayleigh quotient / variational theorem) that: in d-dimensional space, the j-th principal component is the direction "orthogonal to the previous j−1 components and with maximum projected variance." In other words: PCA finds a set of mutually orthogonal coordinate axes that successively capture maximum variance. The linear algebra background needed for the mathematical derivation is in Math Primer.

Numerically, scikit-learn's PCA uses truncated SVD (singular value decomposition) by default, rather than direct eigendecomposition of the covariance matrix — because SVD is numerically more stable, doesn't require computing the covariance matrix first (squaring the covariance matrix amplifies numerical errors), and can directly get the truncated version. The two are mathematically equivalent: for centered data, X's right singular vectors = C's eigenvectors, and singular values squared / n = eigenvalues.

2. Explained variance ratio: how many dimensions to keep?

Each principal component's importance = its variance share:

$$ \text{explained variance ratio}j = \frac{\lambda_j}{\sum^{d} \lambda_i} $$

Read directly from explained_variance_ratio_ in sklearn. A classic heuristic is cumulative explained variance ratio ≥ 80% (or 95%): keep the minimum number of principal components such that the cumulative variance ratio reaches the threshold. For example, a 20-dimensional dataset where the first 3 principal components explain 85% of the variance — you can safely compress the data to 3 dimensions — that remaining 15% of discarded variance is mostly noise.

from sklearn.decomposition import PCA

pca = PCA(n_components=0.95)      # Pass the threshold directly; automatically selects the number of principal components
Z = pca.fit_transform(X_scaled)
print(pca.n_components_, pca.explained_variance_ratio_.cumsum()[-1])

3. Standardization: PCA without standardization is a slave to units

This is PCA's most overlooked pitfall. PCA maximizes variance, and the magnitude of variance is directly tied to the feature's units:

Feature unit           Variance magnitude      Consequence
Annual spending (yuan)  10⁸ level              Dominates the first principal component
Purchase frequency (count)  10¹ level         Ignored by PCA

If you don't standardize first, PCA's "most important direction" will almost certainly point to the feature with the largest unit — this isn't discovering data structure; it's reproducing the unit of measurement. So you almost always want to StandardScaler before PCA (scaling each column to mean 0, standard deviation 1). This is consistent with the "unit-independence" principle in feature engineering, see Feature Engineering and Data Engineering. Note: when all features themselves have the same unit and scale (e.g., pixels from the same sensor, questions from the same scale), standardization can be skipped — this is one of the few exceptions.

4. Properties and limitations of PCA

• Strengths: linear, reversible (can reconstruct approximate original data), globally optimal (under the "maximum variance" objective), principal components are mutually orthogonal with no redundancy, has a clearly interpretable quantity (explained variance ratio);
• Limitations: can only capture linear structures — PCA can't flatten data's nonlinear manifolds (e.g., spirals, rings); principal components are linear combinations of original features, so semantics are often hard to interpret ("0.31× spending + 0.52× frequency − 0.44× time gap" = what?); sensitive to outliers (variance will be dominated by extreme values).

6. t-SNE and UMAP: Nonlinear Visualization Dimensionality Reduction

PCA can't handle nonlinear structures; manifold learning methods take over. The two most famous are t-SNE (t-distributed Stochastic Neighbor Embedding) and UMAP (Uniform Manifold Approximation and Projection).

1. The principle of t-SNE

t-SNE (van der Maaten & Hinton, 2008)'s core idea is very elegant: keep "neighbor relationships in high-dimensional space" as unchanged as possible in low-dimensional space.

1. High-dim side: convert Euclidean distances between samples to conditional probabilities pⱼ|ᵢ — "the probability that xⱼ is xᵢ's neighbor" — determined by a Gaussian distribution centered at xᵢ. Perplexity controls the width of this Gaussian distribution (effective number of neighbors, default 30 in sklearn);
2. Low-dim side: model the similarity of low-dimensional points yᵢ, yⱼ with a Student's t-distribution (degree of freedom 1, i.e., Cauchy distribution), denoted qᵢⱼ;
3. Align both sides: measure the difference between the two probability distributions with KL divergence, minimize it with gradient descent.

The key design is that the low-dim side uses a heavy-tailed t-distribution instead of Gaussian: points that are "far apart in high dimensions" get pushed farther in low dimensions — this alleviates the "crowding problem," letting clusters in the low-dim plot truly separate. This is where the "t" in the name comes from.

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, perplexity=30, random_state=0)
Z_tsne = tsne.fit_transform(X_scaled)   # Get 2D coordinates, can be directly plotted as a scatter

2. Essential differences from PCA

Dimension	PCA	t-SNE / UMAP
Transformation type	Linear (projection)	Nonlinear (manifold learning)
Globally optimal?	Yes (maximum variance)	No (random initialization + gradient descent)
Distance meaning	Preserves global distances (approximately)	Only preserves neighbor relationships; global distances are meaningless
Can reconstruct original data?	Yes (inverse transform)	No
Interpretability	Yes (explained variance ratio)	Almost none
Determinism?	Deterministic	Results differ each run
Typical use	Feature compression, preprocessing	Visual exploration, confirming cluster structure

3. Three pitfalls of t-SNE

Visualization ≠ clustering conclusions

"Close together" in a t-SNE plot only represents local neighbor relationships; distances between clusters, cluster sizes, and cluster shapes have no real meaning. Seeing two blobs close in the plot doesn't imply "they're also close in high dimensions." Worse, t-SNE can "manufacture" fake cluster structures — uniformly distributed random noise may also present cluster-like illusions in t-SNE plots. Conclusions must be based on clustering algorithms + business validation; t-SNE is just a display tool.

Three must-know pitfalls:

1. Perplexity sensitive: perplexity too small (<5) breaks the plot into fragmented small blobs; too large (>50) degenerates into a uniform cloud; in practice, try a range from 5~50 to check stability;
2. Randomness: different random_state gives different plots; to judge "whether a structure is real," fix the seed and run multiple times, checking whether cluster structures appear repeatedly;
3. Complexity O(n²): tens of thousands of samples become painfully slow. The standard approach is: first reduce to 30~50 dims with PCA, then run t-SNE (denoise + speed up), or sample for very large data. This echoes "dimensionality reduction alleviates the curse of dimensionality" from the opening of this section.

4. UMAP: t-SNE's contemporary successor

UMAP (McInnes et al., 2018) takes the route of "first build a fuzzy topological graph using the manifold hypothesis, then optimize the low-dimensional layout." Its mathematical roots lie in Riemannian geometry (with claimed theoretical guarantees), but practically it's stronger than t-SNE in three aspects: ① much faster (supports approximate nearest neighbor, can run on hundreds of thousands of points); ② preserves more global structure (its objective is cross-entropy, not KL divergence, balancing local and global); ③ can transform unknown new data (has a transform method, while t-SNE can only handle "already-seen samples"). When the visualization target reaches the hundred-thousands scale, UMAP basically replaces t-SNE. However, the two share the same pitfalls: distances and cluster sizes on the plot are uninterpretable, randomness, and parameter sensitivity.

7. Practice: Complete KMeans + PCA Workflow in sklearn

Below we tie together all the knowledge into a runnable complete case: customer segmentation for a retail platform. The data is synthetic RFM-style features (or use any table you have), walking through the standard pipeline of "standardize → PCA visualization → choose K → KMeans → profile interpretation."

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import matplotlib.pyplot as plt

# ── ① Simulated data: 1000 customers × 4 RFM-style features ─────
rng = np.random.default_rng(42)
n = 1000
# Three customer segments: high-value(0) / mid-tier(1) / dormant(2)
seg = rng.integers(0, 3, n)
# Class means per column [high-value, mid-tier, dormant]
means = np.array([
    [8000, 40,  20,  2.0],   # High-value: high spending, high frequency, short interval, high avg order value
    [3000, 15,  60,  1.5],   # Mid-tier
    [ 800,  3, 180,  1.0],   # Dormant: low spending, low frequency, long time since last purchase
])
scales = np.array([500, 5, 12, 0.3])            # Noise std per column
X = means[seg] + rng.normal(0, scales, size=(n, 4))

# ── ② Standardize: must do before PCA (different units!) ────────────
X_scaled = StandardScaler().fit_transform(X)

# ── ③ PCA: first check explained variance ratio, confirm how many dims ──
pca = PCA().fit(X_scaled)
cum = np.cumsum(pca.explained_variance_ratio_)
print("Cumulative explained variance ratio:", np.round(cum, 3))          # First 2 dims usually >0.8
X_2d = pca.transform(X_scaled)[:, :2]

# ── ④ Choose K: elbow method + silhouette score double-confirmed ─────────
inertias, sils = [], []
Ks = range(2, 8)
for k in Ks:
    km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(X_scaled)
    inertias.append(km.inertia_)
    sils.append(silhouette_score(X_scaled, km.labels_))

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].plot(Ks, inertias, "o-"); axes[0].set_title("Elbow method (inertia)")
axes[1].plot(Ks, sils, "s-");  axes[1].set_title("Silhouette score")
plt.tight_layout(); plt.show()

# ── ⑤ Take K with best silhouette (in real business, also manually review) ──
best_k = Ks[np.argmax(sils)]
print("Best K by silhouette =", best_k)

# ── ⑥ Final modeling + PCA plane visualization ───────────────────
km = KMeans(n_clusters=best_k, n_init=10, random_state=0).fit(X_scaled)
plt.scatter(X_2d[:, 0], X_2d[:, 1], c=km.labels_, cmap="tab10", s=8, alpha=0.7)
plt.scatter(pca.transform(km.cluster_centers_)[:, 0],
            pca.transform(km.cluster_centers_)[:, 1],
            marker="X", s=200, c="black")
plt.title(f"PCA projection + KMeans segmentation (K={best_k})"); plt.show()

# ── ⑦ Profile interpretation: translate clusters back to original business units ──
df = pd.DataFrame(X, columns=["annual spending", "annual frequency", "days since last purchase", "avg order value"])
df["segment"] = km.labels_
print(df.groupby("segment").mean().round(1))

This code demonstrates all key actions: standardization first, PCA-assisted visualization, K double-metric confirmation, centroids projected to the PCA plane to show cluster centers, and cluster profiles printed in original units for the final step (⑦) — the three rows of means from groupby are precisely the basis for you to present to the business stakeholder as "three groups: high-value active / mid-tier / dormant-to-reawaken."

Note that the centroids in the plot are pca.transform(km.cluster_centers_) projected, not km.cluster_centers_ directly plotted — because the model works in the standardized space, and the plot is in the PCA plane; coordinates must be consistent. This kind of "confusing training space with display space" is a common bug; more engineering pitfalls are in Common Pitfalls.

8. Real Business Applications

1. Customer segmentation (market subdivision)

The most classic application. RFM (Recency, Frequency, Monetary) clustering → identify groups like "high-value loyal / dormant churning / price-sensitive / new customers" → differentiated operations: send reactivation coupons to the dormant group, exclusive services to high-value. It complements collaborative filtering in recommender systems: clustering segmentation solves "what overall strategy for whom," while collaborative filtering solves "which single product to push to this person." The key discipline: segmentation is not a one-time thing — customer behavior drifts; clustering must be re-run periodically and monitored for group migration rates (e.g., "12% of last quarter's high-value customers slipped into the dormant group").

2. Anomaly detection

DBSCAN's noise points are natively "outliers"; points "far from any centroid" in K-Means are also outlier candidates. In financial anti-fraud, industrial inspection, and network intrusion detection, anomalies are often more valuable than normal — fraud orders, faulty equipment, and attack traffic are extremely small in the data, precisely the objects clustering easily misses or misclassifies. Two deployment approaches:

• Unsupervised anomaly detection: DBSCAN (labels == -1) or "distance to nearest centroid > threshold";
• Supervised anomaly detection: once historical labels exist, switch to Isolation Forest or one-class SVM.

The boundary between the two must be clear: clustering-type anomaly detection assumes "normal data is dense, anomalies are sparse and isolated"; when the data-generating mechanism doesn't fit this assumption, switching to tree models approaches (isolation forest is essentially a tree model family) is often more stable. Anomaly detection is also commonly combined with time series, see time-series cleaning in Data Engineering.

3. Feature compression and denoising

• Compression: compress 1000-dimensional sparse text features (TF-IDF) to 100 dimensions with PCA / SVD, serving as input to downstream models, with better speed and stability;
• De-correlation: PCA's output principal components are mutually orthogonal, more friendly to models sensitive to collinearity (linear regression, logistic regression) than the original highly-correlated features;
• Visualization: t-SNE/UMAP lets humans "see" data — visualizing before model training often surfaces data leakage, duplicate samples, and dirty partitions early. This is the starting point for data exploration (the process is in the project skeleton from What is Machine Learning).

Two red lines for using PCA in feature engineering

① Only fit on the training set first, then use the same pca to transform the test set — fitting on all data simultaneously is a classic data leakage that artificially inflates test evaluation; ② On linear models, PCA compression is generally lossless or beneficial; but for tree models (XGBoost, LightGBM), which use quantile splits on features, PCA swapping the original features sometimes reduces both interpretability and performance, and principal components can't do feature importance attribution — in this case, feature selection is preferred over dimensionality reduction.

4. Other high-value scenarios at a glance

Scenario	Method	Business value
Image/vector retrieval	PCA/UMAP for dimensionality reduction before building index	Order-of-magnitude retrieval speedup
Document topic exploration	Topic models / LSA (essentially SVD)	Auto-summarize corpus topics
Gene expression profiles	Hierarchical clustering	Discover co-expressed gene modules
Anomaly-based anti-fraud	DBSCAN noise points	Identify unknown fraud patterns

9. Tradeoffs and Decision Points

Putting clustering and dimensionality reduction into the project's global context, there are several tradeoffs that must be faced:

• "Automatically discovering structure" vs. "structure is your prior": clustering has no objective correct answer; the shape of clusters, the distance measure, the value of K — all carry your subjective choices. Acknowledging this is far more professional than pretending "the data spoke for itself." Cluster profiles must be explainable in human language (interpretability); otherwise, don't deploy.
• Interpretability vs. powerful distance: K-Means is simple but only eats spherical clusters; GMM can model ellipses; t-SNE plots look great but are irreversible and uninterpretable. Which to choose depends on the downstream use: if the cluster profile needs to be presented to business → choose interpretable (K-Means with profile tables); if pure visual exploration → then t-SNE/UMAP.
• Computational savings from dimensionality reduction vs. lost semantics: compressing PCA to 5 dims may explain 90% of variance, but each principal component is a linear combination of 50 features, and you can't explain to business "what this dimension is." When interpretability is needed, use feature selection over dimensionality reduction.
• Local fidelity vs. global fidelity: t-SNE preserves local structure at the expense of global distances; PCA preserves global variance but can't handle nonlinearities. No free lunch — depends on whether your analysis purpose is "finding neighbors" or "measuring the global picture."
• Fast vs. slow: K-Means is second-level, DBSCAN O(n²), hierarchical clustering O(n³), t-SNE O(n²) and non-incremental. On million-scale data, algorithm choice is basically halved by complexity alone.
• Unsupervised vs. going straight to supervised: when labels are available and the business goal is clear (e.g., predicting churn), don't cluster just "to use clustering" — supervised models learning the target directly are often better. Clustering's most suitable stage is: no labels, unclear objectives, and you need to first understand what the data looks like.

A mature data science team treats the tools in this article as exploration-phase standard toolkit: when faced with new data, first standardize + PCA visualization + clustering profiles, forming intuition about the data, then decide whether to go supervised or continue unsupervised. This workflow, together with the rigorous evaluation of model evaluation and pipeline construction of data engineering, forms a complete methodological closed loop.

Further Reading

• Unsupervised Learning — The concept overview of this article, including GMM, association rules, and four-task framework
• What is Machine Learning — Three modeling paradigms and the overall "data generation rule" perspective
• Feature Engineering — Standardization, feature selection, and PCA's engineering connection
• Model Evaluation and Validation — The rigorous framework of supervised evaluation, mirroring clustering's "no-ground-truth evaluation"
• Data Engineering — Cleaning, missing values, and time-series processing, clustering's pre-requisite
• Interpretability and Fairness — How cluster profiles "speak in human language," and fairness risks (group biases amplified by clustering)
• Recommender Systems — The division of labor between collaborative filtering and segmentation in personalization scenarios
• Tree Models and Ensemble Learning — Complementary scenarios between tree models and clustering/dimensionality reduction
• Glossary — Entries for K-Means, inertia, silhouette score, perplexity, etc.
• Math Primer — Eigendecomposition, SVD, KL divergence, and other math used in this article
• Common Pitfalls — Data leakage, random seeds, unit traps, and other engineering pitfalls

References

• MacQueen. Some methods for classification and analysis of multivariate observations (Proc. 5th Berkeley Symposium, 1967) — The original paper coining "k-means" and the algorithm
• Lloyd. Least Squares Quantization in PCM (IEEE Trans. Information Theory, 1982) — Lloyd proposed K-Means in 1957, formally published in 1982
• Arthur & Vassilvitskii. k-means++: The Advantages of Careful Seeding (SODA 2007) — K-Means++ initialization
• Ester, Kriegel, Sander & Xu. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise (KDD 1996) — Original DBSCAN paper
• Ward. Hierarchical Grouping to Optimize an Objective Function (JASA, 1963) — Hierarchical clustering Ward linkage criterion
• Rousseeuw. Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis (J. Computational and Applied Mathematics, 1987) — Original silhouette score paper
• van der Maaten & Hinton. Visualizing Data using t-SNE (JMLR 9, 2008) — Original t-SNE paper
• McInnes, Healy & Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction (arXiv:1802.03426, 2018) — Original UMAP paper
• scikit-learn: Clustering official documentation — sklearn implementations and parameter details for K-Means, DBSCAN, and hierarchical clustering
• scikit-learn: PCA and Decomposition official documentation — sklearn implementations of PCA, SVD, and explained variance ratio
• scikit-learn: t-SNE official documentation — Perplexity, randomness, and usage recommendations
• James, Witten, Hastie & Tibshirani. An Introduction to Statistical Learning (Springer) — Chapter 12: Unsupervised learning (PCA and clustering), free online version
• Hotelling. Analysis of a Complex of Statistical Variables into Principal Components (J. Educational Psychology, 1933) — The classic foundational paper for PCA