ML System Design: What Changes When a Model Produces the Answer

Four things change, and the rest of the design stays the same. Training and serving become two separate systems. Correctness turns into a measured score.

The other two are less obvious. The inputs to the model must match in training and in serving. Cost per request becomes a design constraint rather than an afterthought.

Everything else you already know still applies. You still need a load balancer, a cache, a database, and a queue.

The model does not replace that design. It is one more part inside it.

This article explains what an ML system is, then shows the request path with a model in it. It covers the four differences one at a time. It also covers what large language models add on top.

What counts as an ML system

An ML system is one where the answer comes from a model trained on data, not from rules a person wrote.

A model is a file of learned numbers plus the code to apply them. Inference is the act of running the model on one input to get an output. Training is the separate, much slower job that produces the model file.

A spam filter written as a list of banned words is an ordinary system. The same filter, trained on a million labeled messages, is an ML system. The product looks identical from outside. The design behind it does not.

The request path with a model in it

Start with the path a single request takes. This is where most designs go wrong, so draw it before anything else.

1. The request arrives and the service looks up the features, which are the numeric inputs the model expects.
2. Some features come from the request itself. Others are read from a store, because they were computed earlier.
3. The service calls the model and gets a score back.
4. The service applies a rule to that score, such as a cut-off, and returns an answer.
5. The request, the features, and the score are written to a log for later training.

Step 5 is the one candidates skip. Without it you have no data to train the next model, and no way to check whether the current one is still working.

Step 2 is where the latency usually goes. A model that runs in five milliseconds is not fast if collecting its inputs takes two hundred.

The four things that change

Training and serving are two systems, not one. Training is a batch job that runs for hours on a large machine. Serving answers one request in milliseconds. They have different hardware, different scaling limits, and different failure modes. Design them separately and say how the model file moves from one to the other.

Correct is replaced by a score. An ordinary system is right or wrong on each request. A model is right most of the time, and you pick the metric that measures it. Name the metric out loud, and say what number is good enough to ship. A design with no metric has no definition of working.

Features must match in both places. The value computed during training and the value computed during serving must be the same. When they differ, the model gets inputs it was never trained on, and quality drops for no visible reason. This is called training and serving skew. The usual repair is a feature store, which is a service that computes each feature once and serves it to both sides.

Cost per request is part of the design. A database read costs a fraction of a cent. Model inference can cost a hundred times more, especially on specialized hardware. So you cache scores that do not change often. You run cheap filters before expensive ones. You move work to a batch job when the answer can be a few hours old.

Grokking Modern AI Fundamentals covers these building blocks before the interview framing, which is the right order if the terms above are new.

Ordinary systems and ML systems side by side

The failure with no alarm

An ordinary system fails loudly. A request returns a 500, a graph drops, and somebody is paged.

An ML system usually fails quietly. The world changes, the data the model was trained on gets older, and predictions get worse week by week. Nothing throws an error. This is called drift.

The design has to catch it, so two things belong in every answer. Keep measuring the metric on live traffic, not only on the data used for training. Compare the current model against the previous one on a small share of traffic before switching over.

Naming this failure without being asked is the single strongest thing you can do in an ML design discussion.

What large language models add

A large language model, or LLM, is a model that produces text instead of a number. Three parts of the design change again.

The answer is text, so it needs checking. You add filters that block unsafe output, and rules for what the system does when a check fails.

Facts come from a retrieval step. The model is given documents to work from, found by matching meaning rather than exact words. This is called retrieval-augmented generation, and the search side of it needs its own store and its own latency budget.

Cost and latency are much higher. Answers stream back token by token, so the design has to hold a connection open. Caching, shorter inputs, and smaller models for easy requests are the usual repairs.

The handbook covers both layers in depth. AI and ML system design covers the training and serving path. Generative AI and LLM system design covers the parts above.

What does not change

The ordinary parts still carry most of the design, and most candidates under-serve them.

You still estimate traffic and storage. You still decide where data lives and how it is split.

You still put slow work in a queue and answer the user now. You still name what breaks first at ten times the traffic.

An answer that covers the model perfectly and forgets the caching and storage decisions is an incomplete answer.

How to practice these designs

Pick three questions and design them out loud with a timer. A recommendation feed, a fraud detector, and a search ranker cover most of the ground.

For each one, draw the request path first and the training path second. Then answer three questions before moving on. What is the metric? Where do the features come from? What does one request cost?

Do one design with an LLM in it after those three. The extra parts only make sense once the ordinary ML path is familiar.

Frequently asked questions

Is ML system design the same as AI system design? In interviews the two names are used for the same round. AI system design sometimes means the question involves a large language model. The framework is the same either way, and the extra LLM parts sit on top of it.

Do I need a machine learning background to answer these? No. You are being asked to design a system around a model, not to derive the model. Knowing what training and serving each need, and what inference costs, covers most of the round.

How much of the answer should be about the model itself? Less than candidates expect. Naming a reasonable model family and moving on is usually enough. The design questions are about data, serving, cost, and measurement.

What is the most common mistake? Skipping the logging step. If nothing records the inputs and the outcome, there is no way to train the next model and no way to detect drift. It is a small box on the diagram that changes the whole answer.

Which questions are most likely? A recommendation feed, a ranking system for search, a fraud or abuse detector, and a chatbot built on retrieval. Most other questions reuse decisions from one of those four.

Related reading

• AI and ML System Design, for the full training and serving reference
• Generative AI and LLM System Design, for retrieval, serving, and guardrails
• System Design Interview Questions, Organized by Difficulty, for choosing the level to practice at
• System Design Interview Questions by Company, for predicting which question you will get
• Data Intensive Applications, on the data layer these systems sit on

If the ordinary system design layer is the weaker half, Grokking the System Design Interview works through the common questions in full.