Northbridge Health, a US health organization with 1,200 employees, is preparing CareGuide for its patient portal. CareGuide uses a large language model (LLM), a system that generates text from a user’s request. Before launch, the team faces a practical decision. Does useful-looking chat make CareGuide ready for patients when it might expose patient information or offer unsafe health guidance?
Safety evaluation gathers evidence about how an LLM application behaves in situations that could cause harm. It differs from quality evaluation, which asks whether answers are fluent, relevant, or helpful. Safety evidence supports launch, pause, redesign, and oversight decisions. It also gives the team a record of what was tested, what failed, and what remains uncertain. Let’s trace the patient information, possible harms, and deployment decision around CareGuide.
Three activities contribute different evidence. Safety testing checks whether CareGuide meets defined expectations for known risks. Red teaming deliberately searches for unexpected ways a person could cause unsafe behavior, such as bypassing a restriction. Benchmarking measures a system against a consistent public or internal test set, allowing comparison across versions or systems. None gives a complete answer alone.
For a patient chatbot, the same visible failure can need all three views. Testing checks the intended behavior, red teaming probes weaknesses, and benchmarking adds a repeatable comparison point. Public benchmark results inform questions, but they cannot prove CareGuide is safe in its patient portal context. Consider how the three activities frame these CareGuide failures.