Guardrails and evaluation

What it is for

Guardrails are filters and classifiers that sit around a model. They help, they also fail, and the failure rates on your traffic are rarely the ones on a product page. S3 chooses guardrails on evidence and keeps measuring them.

What we look at

• Which risks the guardrails are meant to reduce, against the 2025 OWASP entries LLM01 Prompt Injection, LLM05 Improper Output Handling and LLM06 Excessive Agency.^⁠[1]
• Open-source and commercial options, compared on your own test data. The choice is vendor-neutral and rests on the test results.
• False positives (legitimate users blocked) and false negatives (attacks let through), measured separately.
• Where guardrails cannot help, and an architectural control is needed instead; see AG4 Injection-resistant architecture review.

Why we report the method with every number

An accuracy figure without its method is a claim no one can check. Regulators treat it that way too: in 2024 the US Federal Trade Commission’s proposed order against a security-screening company barred it from misrepresenting its products’ accuracy, its false alarm rates and the results of any testing.^⁠[2] Every error rate we report states the test set, the method and the date.

What you receive

A guardrail recommendation with its tuning, an evaluation suite that runs in your CI, and a dated report of error rates.