Interpretability and Fairness

Concept Definition: A Model Must Not Only Be Right — It Must Be Trustworthy

The biggest barrier to deploying machine learning is often not accuracy, but trust. Doctors won't use a diagnostic model they can't explain; risk control must be able to explain reasons when rejecting a user (regulatory requirement); if a hiring model is accused of discrimination, it must be able to defend itself. Interpretability studies "why the model made this judgment"; fairness studies "whether the model treats all groups equally" — together they form "Trustworthy ML."

Why black boxes can't survive in the real world

• Regulation: regulations like GDPR require "interpretable algorithmic decisions" (you must give reasons for rejecting loans/insurance);
• Debugging: when a model makes a mistake, without interpretation you can't tell whether it's a data problem, a feature problem, or a model problem;
• Trust: business stakeholders/users don't accept decision systems that say "I don't know why either";
• Security: risks like adversarial attacks and hallucinations require auditable fallback.

Three Levels of Interpretability

Level	Questions Answered	Methods
Global interpretability (model-level)	What does this model depend on overall?	Feature importance, partial dependence plots (PDP), tree structure
Local interpretability (sample-level)	Why was this specific sample classified as X?	SHAP, LIME, counterfactual explanation
Transparent models (intrinsically interpretable)	The model itself can be read as rules	Linear models, decision trees, rule-based systems

Key insight: interpretability is not binary — white-box models (linear/tree) are natively readable but have limited expressiveness; black-box models (deep networks) are powerful but require post-hoc explanation to approximate interpretability. The tradeoff here is essentially the same as the "expressiveness vs transparency" tradeoff.

The Interpretability Methods Toolkit

1. Feature Importance

• Native to tree models: importance weighted by split gain (XGBoost/LightGBM can output with one line of code);
• Permutation importance: shuffle the values of a feature and observe how much the metric drops — the bigger the drop, the more important the feature. Model-agnostic, more trustworthy than native tree importance;
• Limitation: only answers "which features are important overall," not "for a specific sample, why."

2. SHAP: The Current De Facto Standard for Local Explanation

SHAP (SHapley Additive exPlanations) is based on the Shapley value from game theory: treat each feature as a "player" and calculate its marginal contribution to the prediction. Output the contribution value of each feature for each sample (positive contributions push the prediction, negative ones pull it down).

import shap

explainer = shap.TreeExplainer(model)          # Tree-model specific, fast
shap_values = explainer.shap_values(X_sample)
shap.force_plot(explainer.expected_value, shap_values[0], X_sample[0])

Advantages: strict theoretical foundation (Shapley values are the only method that satisfies "fair contribution allocation"), model-agnostic (with TreeExplainer/DeepExplainer/KernelExplainer variants), and can do both local + global (summary plots). Disadvantage: computationally expensive (exact Shapley is combinatorial explosion; approximations are used).

Classic uses of SHAP:

• force plot: single-sample explanation — "why was this loan rejected?";
• summary plot: global feature importance + direction (how high/low feature values affect the prediction);
• dependence plot: nonlinear relationship between a feature and the prediction.

3. LIME: Local Linear Approximation

LIME (Local Interpretable Model-agnostic Explanations) perturbs the input near the sample to be explained and fits a simple model (linear) to the black-box's local behavior, using the local weights as the explanation.

• Advantages: model-agnostic, works with any model;
• Disadvantages: approximation quality depends on the perturbation method; less stable than SHAP (random perturbation causes explanation jitter).

4. Counterfactual Explanation

Answers "what would need to change for the result to flip?" — "if this user's income were above 30k, the loan wouldn't have been rejected." Friendly to users and products; implementation-wise, it searches the input space for the minimum modification.

5. Attention Weights and Interpretability

Warning: attention weights ≠ causal explanation. Transformer attention scores are often used to "explain" the model, but there's substantial academic work showing that attention distributions have weak correlation with true attribution. LLM "interpretability" is currently an approximation at the level of "prompt engineering + chain-of-thought readability," not a rigorous mechanistic explanation. See Large Language Models (LLM).

Fairness: Where Bias Comes From, How to Assess It

Sources of Bias

Source	Example
Data bias (the primary source)	Historical resumes are mostly male → model learns "male preference"
Labeling bias	Subjective standard differences among annotators
Proxy variables	Using "zip code" as a proxy for "race," causing indirect discrimination
Algorithmic bias	The optimization goal itself favors the majority class (class imbalance)
Feedback loops	Model recommendations amplify existing bias (the Matthew effect in recommendation systems)

Core insight: bias mainly comes from data and social reality, not from the model "deciding on its own" — the model simply amplifies the patterns in the data (including bias) into decisions. Amazon's hiring system (2018) was shut down because it learned to discriminate against women — a landmark case of data bias.

Formal Definitions of Fairness

There is no single mathematical definition of "fair"; common metrics each have their own focus:

Definition	Requirement	Limitation
Demographic Parity	Acceptance rate is equal across groups P(ŷ=1|A=a)	May sacrifice accuracy
Equalized Odds	TPR/FPR are equal across groups	Finer-grained, more demanding
Calibration	Predicted probabilities match true probabilities across groups	Cannot be simultaneously satisfied with equalized odds

Key fact (the impossibility theorem): unless base rates are exactly the same, demographic parity and equalized odds cannot both hold. So "fairness" must be defined by business/law — engineering can transparently report per-group metrics and let decision-makers choose the definition.

Practical Assessment

• Report metrics grouped by sensitive attributes: compare accuracy/false positive rate/recall across gender, age, race;
• Test "proxy variables": check whether features strongly correlated with sensitive attributes are active in the model;
• Bias mitigation: data level (re-sampling, re-weighting, removing sensitive features), training level (fairness regularization constraints), post-processing level (threshold adjustment).

Trustworthy ML Engineering Practices

Putting interpretability and fairness into engineering practice, mature teams follow this playbook:

1. Development: sensitive attribute inventory (which attributes need fairness audits)
2. Training: record model version + training data composition (diversity metrics)
3. Evaluation: per-group metric report (grouped by sensitive attribute) → compare baselines
4. Deployment: explanation service (SHAP/rules) delivered alongside the model
5. Monitoring: continuously monitor per-group metrics and explanation drift

Minimum viable starting point (even small teams can do this):

1. Tree model / XGBoost projects: use built-in feature importance + SHAP directly, produce an explanation report;
2. Classification models: fix a "per-group metric table," have a human review before going live;
3. Deep models / LLMs: at minimum, keep decision logs (input, output, key feature snapshots) for traceability if something goes wrong.

The balance between interpretability and accuracy

Interpretability isn't free: post-hoc explanations like SHAP are only approximations, and transparent models may sacrifice accuracy. The engineering practice's tiered strategy is: high-risk decisions (credit, healthcare) prioritize interpretable models; low-risk high-throughput (recommendation ranking) use black boxes + post-hoc explanation. There's no silver bullet for "both high accuracy and fully interpretable" — tier by risk and make tradeoffs.

Tradeoffs

• White-box vs. black-box: high risk, strong regulation → white-box (linear/tree); performance-first, explanation can be added later → black box + SHAP;
• Local explanation cost: SHAP is accurate but expensive, LIME is cheap but jittery — consider pre-computation or rule-based approaches for high-frequency online explanation scenarios;
• Fairness definitions: different definitions are incompatible, so business/law must set the standard first, then pick metrics;
• De-biasing vs. accuracy: removing sensitive features is often not enough (proxy variables remain), but strong fairness constraints sacrifice accuracy — "reporting + monitoring" is more pragmatic than "blanket deletion."

Further Reading

• Model Evaluation and Validation — fairness metrics are an extension of the evaluation system
• Data and Data Engineering — the data roots of bias
• Tree Models and Ensemble Learning — the primary models with built-in interpretability
• Large Language Models (LLM) — interpretability challenges in large models
• MLOps and Model Deployment — landing interpretation and monitoring in production
• Common Pitfalls and Anti-Patterns — the anti-pattern of "treating accuracy as everything"

References

• Lundberg & Lee. A Unified Approach to Interpreting Model Predictions (SHAP, NeurIPS 2017)
• Ribeiro, Singh, Guestrin. "Why Should I Trust You?": Explaining the Predictions of Any Classifier (LIME, KDD 2016)
• Molnar. Interpretable Machine Learning (free book) — the most comprehensive free resource on interpretability
• Hardt, Price, Srebro. Equality of Opportunity in Supervised Learning (NeurIPS 2016) — equalized odds definition
• Barocas, Hardt, Narayanan. Fairness and Machine Learning (free textbook) — authoritative textbook on fairness
• Reuters. Amazon scraps secret AI recruiting tool (2018)
• Jain & Wallace. Attention is not Explanation (NAACL 2019) — the famous argument that attention ≠ explanation