MLOps is the set of practices — versioning, CI/CD, monitoring, and drift detection — that keeps a machine learning model reliable after launch, not just accurate on the day it shipped. Here's the checklist we run before any model goes to production.
MLOps (Machine Learning Operations) is the discipline of deploying, monitoring, and maintaining machine learning models in production — the operational layer that sits after a model is trained and before it's forgotten about. It borrows from DevOps (CI/CD, versioning, automated testing) but adds practices specific to ML: accuracy monitoring, data drift detection, and retraining triggers.
Without MLOps, a model that scored well in testing silently degrades in production as real-world data shifts away from what it was trained on — a failure mode with no error message, no crash, and no alert unless one is explicitly built.
This applies across every system covered in our cluster — LLM integrations, RAG and fine-tuned models, and AI agents all need a monitoring layer once they're live, not just at launch.
Grouped by category. Every item here is something we verify before a model ships — not something we add after a client asks why accuracy dropped.
Model version tracking
Every deployed model tied to a specific version, training data snapshot, and hyperparameters.
Rollback capability
Previous model version can be restored within minutes if a new deployment underperforms.
Reproducible training pipeline
Training run can be re-executed from stored config to produce the same model.
Dataset versioning
Training and evaluation datasets are snapshotted, not pulled live and untracked.
Automated evaluation gate
New model versions must clear an accuracy benchmark before deployment is allowed.
Staged rollout
New models go to a canary or shadow deployment before full traffic — not straight to 100%.
A/B testing framework
Ability to compare model versions on live traffic with statistical significance, not gut feel.
Accuracy tracking in production
Ongoing measurement against ground truth or human review, not just at launch.
Data drift detection
Alerts when incoming data distribution shifts meaningfully from the training distribution.
Latency & error rate monitoring
Standard infrastructure monitoring applied to the model-serving layer specifically.
Retraining triggers
Defined thresholds that automatically flag or initiate a retraining cycle when performance drops.
Per-request inference cost
Token or compute cost tracked per call, not discovered at month-end on the cloud bill.
Usage-based alerting
Thresholds set so an unexpected spike in calls or cost triggers a notification before it triggers a budget crisis.
Resource autoscaling policy
Serving infrastructure scales with load without manual intervention.
Output logging & auditability
Model inputs and outputs logged for review — required in most regulated deployments.
Access controls on the model endpoint
Not every internal service or user should have unrestricted access to the model API.
Data residency compliance
Inference and any stored logs respect the same regional requirements as the rest of the stack.
Prompt/input sanitization checks
Guardrails against injection or malformed input are part of deployment, not an afterthought.
MLOps is usually priced as part of a build, or as an ongoing monitoring retainer for systems already in production.
Scope
Setup cost
Ongoing cost driver
Scope
Basic monitoring
Setup cost
$3K–6K
Ongoing cost driver
Logging & alerting infra, minimal
Scope
Full MLOps pipeline
Setup cost
$12K–25K
Ongoing cost driver
CI/CD, drift detection, automated retraining triggers
Scope
Enterprise multi-model platform
Setup cost
$40K+
Ongoing cost driver
Cross-model dashboards, compliance logging at scale
Fraud scoring, clinical support, financial decisioning — drift here has direct cost or regulatory consequences. Full monitoring and retraining triggers are non-negotiable.
An internal knowledge assistant or content-generation helper can run on basic accuracy and error-rate monitoring without a full CI/CD-for-models setup.
High data drift risk — fast-moving markets, seasonal behavior shifts — needs automated drift detection, not a quarterly manual check.
Pure RAG systems don't drift the way trained models do — the priority shifts to index freshness and retrieval quality monitoring instead of accuracy drift.
