Generate custom courses on any topic — with hands-on practice, AI guidance, and visuals built in.
Already have an account?
A risk team gets handed a new healthcare member support chatbot two days before launch. In a late test, the bot gives a member step by step dosing advice for a prescription drug, even though the product requirement says it must refuse medical advice and route the member to a nurse line. Nobody can answer the first question you need for a risk rating, which is where in the system this behavior could have been prevented, detected, or contained.
The rest of this lesson builds a threat model for harmful outputs by walking the incident chain end to end, then turning that chain into concrete control points you can measure and govern.
In this course, a threat model is a structured description of how a specific harm can happen in your LLM application, which components participate, what conditions must be true, and what evidence you can collect to confirm or refute the path. For harmful outputs, that means we treat the output as the failure state and trace backward through prompt, retrieval, generation, moderation, and delivery until we find the controllable breaks in the chain.
Simulated — incident narrative excerpt:
A member asks the chatbot whether they can double their dose to recover from a missed day. The chatbot retrieves a general medication FAQ and a community forum snippet that includes informal dosing discussion. The model replies with conditional dosing guidance and a warning, and the UI renders it without any additional gate. No refusal message is triggered, and no human review occurs.
The key move is to separate what went wrong into observable system events. A user request arrived, retrieval returned a mix of acceptable and unsafe context, the model produced regulated advice, the moderation step did not block or route it, and the delivery channel presented it as normal help content. We want a threat model because each hop has different telemetry and different control limits.
We can make that chain concrete in the system diagram. As you review it, focus on which component turned a latent risk into an actual delivered response. Now map the failure points on the architecture:
A control point is a place in the request to response path where you can apply a security or safety decision using an input, a policy, and an enforcement action. For harmful outputs, it helps to group control points into pre generation, during generation, and post generation, because each group offers different guarantees and different false positive costs.
Pre generation controls shape what the model sees. They include user input filtering, retrieval constraints, and prompt templates. They can reduce the chance of harmful outputs, but they cannot guarantee safety because safe looking inputs can yield unsafe completions, and retrieval can still surface borderline text that is contextually dangerous.
During generation controls shape what the model is allowed to produce. This is where you use system instructions, tool policies for agents, and model side safety features where available. They can narrow behaviors, but they still rely on the model following instructions under adversarial or ambiguous inputs, and they are hard to verify without systematic testing.
Post generation controls decide what gets delivered and what gets routed. This includes output moderation classifiers, allow and deny lists for high risk topics, and human review queues. They are often the only place you can guarantee that prohibited text does not reach a user, but they can add latency, create inconsistent user experience, and require disciplined logging to prove they worked when you need evidence.
Guarantees need enforcement
A policy statement is not a control. A control point only exists when there is an enforceable decision and an audit trail you can review.
A common misconception is that harmful output risk is only a model quality issue. In practice, the model is one component in a distributed system. The same base model can be acceptable in a nurse line triage workflow with strict post generation routing and unacceptable in a self serve chat UI with no gating, because the delivery path changes impact and detectability.
To make the control points operational, we attach evidence. For each point, we should be able to answer what signal is logged, where it is stored, and how it is reviewed. Walk through the intervention points and the logs you would expect to see: