ML Model Deployment
Getting models out of notebooks and into production, with versioned rollouts, autoscaling inference, drift monitoring and a rollback that actually works.
The gap between a trained model and a served one
A model that scores well in a notebook is perhaps a third of the way to production. The remaining work is deployment engineering: packaging the artefact, serving it under concurrency, versioning it so you can roll back, and watching for the day the input distribution shifts and accuracy quietly decays.
That last one catches most teams. A model does not throw an exception when the world changes underneath it; it just becomes gradually wrong while every dashboard stays green. Drift monitoring is not optional infrastructure.
We deploy models the way we deploy any other service: containerised, autoscaled, observable, and reversible. The interesting parts are the model registry, the shadow deployment path and the monitoring, not the YAML.
Teams that start here often pair it with on-premise to cloud migration, monolith to serverless migration and AI product development.
Models do not fail loudly. They drift, and every dashboard stays green while accuracy quietly slides for three months.
Why models stall on the way to production
Four gaps between a good notebook and a served model.
The model was trained on a laptop with unpinned dependencies, so nobody can rebuild the exact artefact that is running.
Weights are copied to a server by hand, so there is no registry, no lineage and no reliable way to roll back to the previous model.
Inference containers load multi-gigabyte weights on start, so an autoscaling event produces a wave of timeouts.
Nothing watches the input distribution or the prediction mix, so accuracy decay is discovered by a customer complaint months later.
What an ML deployment engagement covers
Reproducible artefacts, safe rollouts and monitoring that catches decay.
Versioned artefacts with training lineage, metrics and dependencies recorded, so every deployed model can be traced and rebuilt.
Reproducible images with pinned dependencies, health checks and warm starts, running on your own cluster or managed endpoints.
Scaling on queue depth and latency rather than CPU alone, with pre-warmed replicas so scale-out does not produce timeouts.
New versions run against live traffic without serving results, so accuracy is compared on real data before any switch.
Input distribution, prediction mix and, where labels arrive later, realised accuracy tracked with alerting on decay.
One-command reversion to the previous version, tested during the engagement rather than discovered during an incident.
From notebook to served model through controlled rollout
Reproducibility first, then serving, then the safety net.
We review the training pipeline, dependencies and evaluation, and establish whether the artefact can be reproduced at all.
Training containerised with pinned dependencies and a registry entry carrying metrics and lineage.
Inference service with warm starts, batching where it helps, and load testing against realistic concurrency.
The model serves in shadow against live traffic while we compare predictions with the incumbent.
Staged traffic shift with monitoring and a tested rollback, then handover with runbooks.
The stack behind our deployments
Portable by default, managed services where they earn their scope.
Reproducible images with pinned dependencies, so the artefact that passed tests is the artefact that serves.
Inference servers chosen for the model shape rather than one framework for everything.
Kubernetes or managed endpoints, autoscaled on the signals that actually predict load.
Latency and errors plus the model-specific signals that ordinary APM never captures.
What this looks like in production
Fixing a forecast that had been wrong for a quarter
A retailer ran demand forecasting models copied onto an EC2 instance by hand. There was no registry, no lineage and no monitoring beyond whether the process was alive.
A supplier change had shifted the input distribution three months earlier. The model kept returning confident forecasts, the service kept reporting healthy, and inventory decisions had been quietly degrading the whole time.
We rebuilt the pipeline with a registry, shadow deployment and drift monitoring on both inputs and predictions. The next distribution shift raised an alert within two days rather than a quarter.
Deployment discipline for probabilistic systems
We treat a model as a versioned artefact with a rollback path and a monitor watching for decay, because the failure mode that matters is silent.
Every deployed model can be rebuilt from its registry entry, with pinned dependencies and recorded lineage.
A working demo and a written note every Friday covering what shipped, what slipped and what it means for the date. No status theatre.
Nothing goes live in one jump. We run in parallel or behind a flag until the numbers say it is safe to switch over.
