Deep Learning Handbook

Theme

Interpretability and Fairness

Quick overview The stronger the model, the more we need to answer "why did it decide that, and is it fair to whom?" This article surveys post-hoc explanations (Saliency/Grad-CAM/LIME/SHAP), mechanistic interpretability (probes/circuits), the limitations of explanations and "explanation hallucinations," and expands into fairness bias, evaluation, and alignment safety.

Interpretability and Fairness ​

One-sentence definition: Interpretability is the ability to answer "why did the model produce this result"; fairness is ensuring the model doesn't systematically disadvantage any group of people. Together, they address deep learning's most pointed question: when models intervene in healthcare, finance, law, and education, what gives us the right to trust them — and to let them decide people's fates?

1. Why Explain? ​

Three genuine motivations:

  1. Compliance and accountability: the EU's GDPR (2018) grants users a "right to explanation"; financial lending's Equal Credit Opportunity Act requires explainable, reviewable reasons for loan denials. Regulatory pressure is turning "explainability" from a nice-to-have into a prerequisite.
  2. Trust and adoption: doctors, judges, and engineers won't accept a "black-box judgment" — even if it's statistically more accurate. Trust requires "something that can be inspected and challenged."
  3. Debugging and improvement: explanation is a diagnostic tool. Visualizing attributions reveals what the model is "looking at" — is it the lesion or the scanner's metal artifact? This is far more efficient than blind hyperparameter tuning and pairs naturally with Debugging and Diagnostics.

2. Post-Hoc Explanations: Saliency, Grad-CAM, Attention Weights ​

Post-hoc explanations don't modify the model; they only "ask questions" of an already-trained model. Three commonly used methods:

  • Saliency maps (2014): compute the gradient of the loss/prediction w.r.t. the input ∂ŷ/∂x, and draw heatmaps showing "which input pixels contributed most to the prediction." Intuitively clear and simple to implement, but gradients are noisy and unstable — the same model with different implementations may produce different heatmaps.
  • Grad-CAM (2017): class activation maps for CNNs — localize where the model is "looking" by "weighting feature map channels using gradients." More stable than Saliency, with clearer semantics — the de facto standard in CV. Limitation: it can only localize regions, not explain "why this class."
  • Attention weights: attention heatmaps from Transformers (see Attention). The most intuitive NLP explanation, but as we've emphasized — "high attention" ≠ "causally important" (Jain & Wallace 2019 showed that removing high-attention heads often leaves predictions unchanged). Attention weights can only serve as clues, not rigorous explanations.

3. Feature Attribution: LIME, SHAP, Integrated Gradients ​

Feature attribution answers "which input features drove this prediction." Three mainstream methods:

  • LIME (2016): perturb the input around a single sample, train an interpretable local surrogate model (linear regression/decision tree) to approximate the black box. Strength: model-agnostic (works with any model); weakness: results depend on perturbation method and local kernel width.
  • SHAP (2017): based on Shapley values from game theory — fairly distribute the "total prediction" across all features. Best theoretical properties (satisfies axioms like additivity, symmetry), but exact computation is exponential; relies on approximations. SHAP's global summary plots (feature importance rankings, dependency plots) are the standard weapon in team reviews.
  • Integrated Gradients (2017): integrate gradients along a straight path "from zero baseline to the input," solving the gradient noise and saturation problems of Saliency. Simple to implement, satisfies axioms — a common choice for deep models.

All three methods can only answer "local, per-sample" explanations — they tell you "why this patient was flagged positive," but not "how the model works overall." This isn't a flaw; it's something to know about the capability boundaries of each tool.

4. Mechanistic Interpretability: Probes, Activation Analysis, Circuits ​

One step beyond "per-sample explanation" is mechanistic interpretability (MI): opening up the model's internals to understand what "neurons/attention heads/circuits" are actually computing. Three techniques:

  • Probes: train lightweight linear classifiers to read intermediate-layer hidden vectors — check whether "this layer encodes some kind of information" (e.g., "layer 6 encodes parts of speech"). Note: probes can only prove "information exists in a linearly separable space," not that "the model actually uses it."
  • Activation analysis: find "neurons highly activated by specific concepts" (e.g., the "Golden Gate Bridge neuron"); decompose internal representations into readable features using dictionary learning (SAE, sparse autoencoders) — this has been the mainstream direction for LLM interpretability since 2023.
  • Circuits: trace the information flow "from input to output" (e.g., "the induction head circuit for coreference resolution"), using causal intervention (activation patching, ablation) to verify whether the flow actually performs the function.

MI's value lies in moving from "what" to "how": not just knowing what information the model uses, but how it uses it. Its methodological rigor far exceeds post-hoc heatmaps and also forms the technical foundation for alignment and safety. A reading path for related papers is in Classic Papers.

5. The Limitations of Explanations and "Explanation Hallucination" ​

Four limitations of explanations that must be honestly acknowledged:

  1. The "manipulability" of post-hoc explanations: results from Saliency/LIME/SHAP are sensitive to implementation details; different tools may give contradictory conclusions — the "explanations" themselves need validation.
  2. Explanations are approximations, not truths: surrogate models (LIME/SHAP) explain the "local approximation," not the original model; local explanations can be wildly wrong (e.g., "linear approximation" breaks down quickly in high-dimensional spaces).
  3. Explanation hallucination: models don't have a "thinking" process, yet we tell stories about their outputs — especially LLM "self-explanations," which are often unrelated to the true mechanism.
  4. The explanation ceiling for complex models: the more parameters, the more nonlinear the behavior — "global explanation" becomes impossible. The pragmatic goal is "local, task-relevant, verifiable" explanations, not "seeing through the entire model."

Don't be comforted by explanations

"Having a heatmap" ≠ "having safety." Explanations must serve a specific decision (compliance, debugging, trust); explanations divorced from purpose are decorations.

6. Fairness and Bias ​

Unfairness is rarely self-created by models — it's an amplifier of existing biases in data, objectives, and processes. Bias sources fall into three layers:

  • Data bias: insufficient samples for certain groups in training data, or systematically skewed labels (e.g., "minority samples are underrepresented in medical datasets") — the root source lives in Data and Data Engineering and the bias discussion in Representation Learning and Pretraining.
  • Algorithmic bias: the optimization objective itself is skewed (using "click-through rate" as the objective makes the model inherently favor young, highly active users); proxy features encode sensitive attributes (e.g., "zip code" proxies "race").
  • Process bias: systematic bias introduced by annotation, sampling, and deployment methods.

Evaluating fairness (at its core, the political nature of "how to define fairness"):

  • Demographic parity: equal acceptance rates across groups.
  • Equalized odds: equal "false positive/false negative rates" across groups.
  • Individual fairness: similar people should receive similar outcomes.

These three definitions often conflict — there is no "absolute fairness." In engineering practice: include protected attributes in audits, report confusion matrices and key metrics separately across groups (see Deep Learning Evaluation and Experiments), and use fairness metrics (e.g., Demographic Parity Difference) as gates.

7. Brief on Alignment and Safety ​

Alignment: making model behavior consistent with human intentions and values. Technical approaches:

  • RLHF (Reinforcement Learning from Human Feedback, see Deep Reinforcement Learning): use human preferences as rewards and fine-tune with PPO — the key step for ChatGPT's alignment.
  • DPO (Direct Preference Optimization): bypass the reward model and RL, directly optimize for preferences — simpler and more stable.
  • Red teaming: actively find the model's failure modes (harmful outputs, jailbreaks, hallucinations) and harden it with adversarial examples.

The relationship between alignment and interpretability: alignment answers "will the model do the wrong thing?" while interpretability answers "can we see it do the wrong thing?" — they complement each other. Safety requires "capability + intent + oversight" working together; cutting-edge progress is in Frontier Advances.

8. Trade-offs ​

Trade-offs

Interpretability vs. performance: simple models (linear/trees) are natively interpretable but less accurate; deep models are more accurate but hard to explain globally. The practical compromise: use a high-accuracy black box for the primary decision, and pair it with local explanations (SHAP/Grad-CAM) + manual review as a supervision layer — "black box + guardrails" is the current engineering mainstream.

Efficiency vs. rigor of post-hoc explanations: LIME/SHAP are cheap but may be unfaithful; mechanistic explanations (probes/circuits) are rigorous but expensive and only cover smaller models. Choose based on purpose: heatmaps suffice for debugging; compliance audits need stricter attribution.

The three fairness definitions are incompatible: demographic parity, equalized odds, and individual fairness cannot all be satisfied simultaneously across most distributions — you must first agree with stakeholders on what "fairness" means before talking about measurement.

Privacy vs. interpretability: explanation requires access to model internals/samples, creating tension with privacy protection (differential privacy); the boundary between transparency and safety needs to be weighed per scenario.

Interpretability and fairness are not add-ons outside the model — they are part of deep learning engineering capability — determining whether a model can be trusted, regulated, and maintained long-term. Looping back to the monitoring cycle in MLOps and Model Deployment, integrating fairness and drift into continuous auditing is the responsible way to deploy. For related terminology, see the Glossary.

Further Reading ​

References ​