PART 5 · MODULE 2Getting models to users
Model Deployment & Serving
Deploying models to production
Loading module...
Deploying models to production
A model sitting in a Jupyter notebook is like a recipe that has never been cooked in a real kitchen. Deployment is the process of taking that model and making it available to actual users -- reliably, quickly, and at scale.
Not every model needs to respond in milliseconds. The right serving pattern depends on your use case.
Process data in bulk on a schedule. Think nightly product recommendations or weekly fraud reports. High throughput, but results are not instant.
"Every night, score all customers for churn risk"Respond to requests instantly. When a user types a message, the chatbot replies in seconds. Low latency is critical here.
"User asks a question, get an answer in 200ms"Run the model directly on the user's device -- phone, browser, or IoT sensor. No internet needed, maximum privacy, but limited compute power.
"Autocorrect runs entirely on your phone"Model serving architecture
Models are saved in efficient formats for loading and running. ONNX is a universal format that works across frameworks. TensorFlow SavedModel and PyTorch are framework-specific options.
Wrap your model in a web endpoint so any application can send it data and get predictions back. REST APIs are the most common, while gRPC is faster for high-throughput systems.
Package your model, its dependencies, and the serving code into a Docker container. This guarantees it runs the same way everywhere -- your laptop, a server, or the cloud.
Large language models and image generators need GPU power. But simpler models -- like a spam classifier -- run perfectly fine on CPUs, which are much cheaper.
When thousands of users hit your model at once, you need multiple copies running in parallel. Auto-scaling spins up new instances when demand spikes and shuts them down when it drops.
A model that cannot serve users reliably is just a science experiment. Deployment is where ML meets engineering -- latency, uptime, cost, and scalability matter just as much as accuracy. The best model in the world is useless if it takes 30 seconds to respond or crashes under load.