An AI chat app is a text chat interface where a user types a message, the app sends it to a service that can call an AI model, and the app shows the model’s reply back to the user.
In many products, the reply arrives as a stream, so the user sees tokens appear over time instead of waiting for the full answer.
To keep the picture simple, start with a small internal team using one web app, talking to one backend service, which calls one model provider.
With one server, the flow is a loop. The client sends the user’s message to a single API server, the API server loads the conversation so it has context, then it calls a model provider, and finally it returns the assistant reply to the client.
The conversation store matters even at this baseline because a page refresh should not erase the chat. The API server persists each user message and assistant message, then uses those stored messages to continue the same conversation on the next turn.
Here is the one chat turn we are building as the baseline.
The diagram has four jobs. The client collects input and renders the stream, the API server owns the request lifecycle, the conversation store keeps messages so the thread survives refresh, and the model API produces the assistant tokens the server relays back to the client.
The functional scope is plain text chat only. There are no tools, no images, and no file uploads, so the only work per turn is storing messages, assembling context, and calling the model.
Now add one number that makes a single server feel pressure. With 2,000 concurrent users at 2 turns per minute, the backend faces about user turns per second, which translates into the same order of parallel model calls.
That single number creates two non-functional signals the rest of the design will keep reacting to. Latency becomes visible because model calls dominate end to end time, and cost becomes sensitive because the system pays for each request and each generated token.
Even before scaling, the baseline breaks cleanly into reusable blocks that each own one responsibility.