Model explainability
Explanation methods attribute a model's prediction to its input features, which is a requirement for using models anywhere a decision must be defended.
- Operational use
- Explaining why a transaction was classified a certain way, why a forecast moved, or why a record was flagged, in terms a reviewer can evaluate.
- When not to use it
- If a decision requires a true causal account, feature attribution is the wrong tool; it describes the model, not the world.
- How it is validated
- Sanity checks on known cases; stability of explanations across similar records; review by someone who knows the domain.
- Limitations
- SHAP-style attributions describe feature contributions within the model. They do not prove causation, and they can mislead when features are correlated.
Gradient-boosted trees (XGBoost is the reference implementation) remain among the strongest learners for structured business data: tabular records, transactions, attributes. Their practical weakness is opacity, which SHAP addresses by assigning each feature a contribution to each individual prediction.
Eigenvalue’s position: a model that cannot explain itself does not belong in a financial workflow. Attribution is shown beside the prediction wherever a person reviews model output, phrased in the domain’s own vocabulary. And the limits are stated plainly: an attribution describes the model’s arithmetic, not the business’s causality.
Where it appears in our work
- Quantitative Forecasting and Scenario Engine Considered during system design
Primary sources
- Chen and Guestrin (2016). XGBoost: A Scalable Tree Boosting System
- Lundberg and Lee (2017). A Unified Approach to Interpreting Model Predictions
Last reviewed 2026-08-02. Sources link to primary or official pages; verify current versions before relying on them.