Production ML engineering
A model is a small component of a production system; data dependencies, monitoring, tests, and ownership determine whether the system keeps working after the demo.
- Operational use
- Any model or pipeline expected to run unattended in a business process, where drift, upstream changes, and silent failure are the real enemies.
- When not to use it
- If a problem is deterministic, a model adds fragility without adding capability. The high-interest debt starts accruing the day you deploy one you did not need.
- How it is validated
- The ML Test Score rubric shapes review; data contracts and control totals guard inputs; golden scenarios and release gates guard outputs; monitoring watches drift after deploy.
- Limitations
- A notebook or demo is not a production ML system, and Eigenvalue does not describe one as such. Offline accuracy says little about behavior under real, changing data.
Google’s technical-debt paper made the canonical argument: the model is the small box in the middle, and the system around it, data dependencies, configuration, serving, monitoring, feedback loops, is where production risk lives. The ML Test Score turned that argument into a checkable rubric.
This is the engineering culture Eigenvalue applies even where no neural network is present: versioned rules, tested calculations, release gates, golden scenarios, reconciliation against source truth, and monitoring that notices when inputs drift. The question is never only “was the model accurate,” but “what happens when the world changes and nobody is watching.”
Where it appears in our work
- Internal AI Assistant Considered during system design
- Eigen Conductor Considered during system design
- Finance Operations Software Relevant research reference
- Multi-Organization Operations Software Relevant research reference
Primary sources
- Sculley et al. (2014). Machine Learning: The High-Interest Credit Card of Technical Debt
- Breck et al. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction
Sources link to primary or official pages; verify current versions before relying on them.