50ms is not a reasonable overhead for a model call. A security check cannot add that much to the critical path when inference already takes time.
That concern makes sense if inline inspection behaves like a WAF: every request passes through rules or an LLM classifier that reasons before returning a verdict. At those latencies, it adds substantial overhead to a path where model inference already takes seconds. The math fails.
Our engineering challenge with Repellox was to make classification useful without adding observable latency during normal operation. This post explains the decisions that brought us there.
Setting the Inspection Latency Ceiling
For a production agent, latency usually means time to the first meaningful model token. It is typically 300ms to 1500ms, depending on context length, model, and infrastructure. Adding 50ms means 3-15% more latency on every request. Teams see that, and users sometimes do too.
An inspection taking under 5ms sits below network-jitter noise in most deployments. Under 2ms, it is effectively invisible within the full request lifecycle.
Working backward: what classifier can deliver a useful adversarial-content verdict in under 2ms at p99? Not a GPT-4 class model, a 7B parameter model, or anything compute-heavy. The answer is a purpose-built lightweight classifier, fast by design rather than by adding hardware.
Staged Classification, Early Termination
Our architecture uses several classification stages, from fastest to most expensive, with each stage able to exit early.
Stage 1 matches patterns with heuristic rules and lightweight regex for known injection templates, instruction markers, and high-confidence attack signatures. It runs in microseconds and catches many unsophisticated attacks. Obvious payloads, including ChatGPT jailbreak templates and crude instruction overrides, usually stop here.
Stage 2 is a fast semantic classifier, a small fine-tuned model scoring a few injection-relevant dimensions. Is the text mainly instructional? Does it mention system concepts such as prompts, roles, or instructions? Is it presented as an override or correction? It runs in single-digit milliseconds on modest hardware and catches inputs that look benign syntactically but are adversarial in meaning.
Stage 3 adds context. It uses the agent's behavioral profile and the content source to revise the Stage 2 verdict. A moderately scoring chunk from a user-writable source merits more risk than the same chunk from a curated internal corpus. Compute remains negligible because this stage mainly adjusts the Stage 2 score instead of running new model inference.
// Simplified inspection pipeline
async function inspect(input: InspectionRequest): Promise<Verdict> {
// Stage 1: fast pattern matching (microseconds)
const patternResult = await patternLayer.check(input.content);
if (patternResult.confidence > BLOCK_THRESHOLD) {
return { action: 'block', stage: 1, reason: patternResult.reason };
}
// Stage 2: semantic classifier (2-4ms)
const semanticScore = await semanticLayer.score(input.content);
if (semanticScore.injectionLikelihood > HIGH_THRESHOLD) {
return { action: 'block', stage: 2, score: semanticScore };
}
// Stage 3: context adjustment (sub-millisecond)
const adjusted = contextLayer.adjust(semanticScore, input.metadata);
if (adjusted.riskScore > ADJUSTED_THRESHOLD) {
return { action: 'flag', stage: 3, score: adjusted };
}
return { action: 'allow', score: adjusted };
}The important property is early exit in the common case. Most benign requests finish at Stage 1, with no adversarial pattern found, in microseconds. A small fraction reaches Stage 2 and adds a few milliseconds. Only ambiguous semantic results proceed to Stage 3.
Speed Tradeoffs We Accepted
The tradeoffs are real. A lightweight classifier is less accurate than a large reasoning model on ambiguous inputs. Some attacks that a GPT-4 class model would identify correctly receive a moderate Stage 2 score below the block threshold. We address this with sensitivity-oriented threshold calibration, accepting more false positives to reduce false negatives, plus behavioral anomaly detection at the tool-call layer.
The behavioral backstop is important. If an injection passes inspection and affects the model's output, the resulting tool call is checked against the agent's expected behavioral profile. Calls outside that profile can be flagged or blocked even when the upstream layer misses the payload. The layers are independent, so an attacker must defeat both.
Our classification is not perfect. It is calibrated to the attack distribution we currently observe, and that distribution changes as attackers adapt. We retrain and recalibrate regularly. No classifier offers 100% detection with zero false positives. The objective is useful risk reduction without degrading the user-facing application.
Choosing the Deployment Path
The lowest-latency option runs in process, with no network hop. In our SDK path, the classifier is a library beside the application code. No external service request is needed. The inspect() call is local and adds single-digit milliseconds before the model API call.
Teams that prefer an out-of-process design can route LLM requests through a proxy or sidecar. A local sidecar hop is typically 0.5-2ms on the same host or in the same data center. Inspection adds 2-4ms, producing 3-6ms of end-to-end overhead in general, acceptable for most production contexts.
Out-of-process deployment has one operational benefit: the inspection pipeline can be updated without redeploying application code. That matters when deployment procedures are strict.
Testing Inline Inspection in Your Stack
To assess fit, measure your application's current p99 model-call latency, including the API round-trip. Add 5ms. If the total remains inside your response-time budget, inline inspection fits your requirements. Most production agents we discuss have budgets where 5ms is not detectable in practice.
Inline inspection genuinely may not fit high-frequency, latency-critical agents requiring sub-100ms total latency. Those use cases are real, and the current overhead is not compatible with them. In other cases, latency is less of a barrier than commonly assumed.