Skip to main content
Before discussing the ticketing system, note an important design choice we made: we use a large language model (LLM) to audit policy compliance either instead of or in addition to a traditional static rule engine such as Open Policy Agent (OPA). This is a deliberate trade-off that introduces specific costs, operational considerations, and risk mitigations you must plan for.

​
Deterministic rule engines vs. LLM auditors

A static rule engine is deterministic: the same input always produces the same output, and it’s cheap to run. Static policies are ideal for “hard blockers” — rules that must never be violated (for example: “do not deploy images from untrusted registries” or “do not grant cluster-admin to a service account”). An LLM-based auditor is more flexible and can:
  • Reason about intent rather than only syntactic matches.
  • Catch edge cases a static rule might not anticipate.
  • Produce natural-language explanations that help operators, auditors, and incident responders.
However, LLMs come with trade-offs:
  • Cost: per-run costs and additional infrastructure to scale.
  • Latency and rate limits: hosted models add latency and can be throttled.
  • Non-determinism: identical inputs can yield different outputs.
  • Failure modes: hallucinations, misclassifications, prompt-injection risks, or context-window truncation.
  • Auditability: explanations can be incomplete or misleading unless outputs are structured and logged.
Use a hybrid approach: keep deterministic, low-cost rules for guaranteed-blocking checks, and use an LLM layer for ambiguous or intent-based judgments where human-like reasoning adds value.

​
Quick comparison

​
Practical pattern and mitigations

Follow this pattern to get the most value from a hybrid design while minimizing risk:
  • Use static rules for hard blockers. Gate critical decisions with a rule engine (e.g., OPA) so that safety invariants never rely solely on an LLM.
  • Put the LLM after the rules for “gray area” cases. Let the rule engine filter obvious violations and call the LLM only for cases that require judgment.
  • Enforce structured outputs from the LLM. Require JSON schemas or function-like outputs (for example: verdict, confidence, and explanation) and validate them with a strict schema validator.
  • Reduce nondeterminism where possible. Use lower sampling or deterministic decoding, canonical prompt templates, and representative examples. Still plan for occasional variance.
  • Add secondary checks. Re-run high-risk decisions through a separate verifier or different model and compare results (ensemble or majority vote) before taking irreversible actions.
  • Human-in-the-loop for high-risk decisions. Escalate to an on-call engineer or compliance reviewer when confidence is low or the decision has significant consequences.
  • Instrument and log everything. Capture inputs, prompts, LLM outputs, confidence scores, and final decisions for auditability and postmortem analysis.
  • Cache and batch results. Reduce cost and latency for repeated checks by caching judgments for identical inputs and batching low-priority requests.
  • Monitor for drift and model issues. Track model performance, cost, and unexpected verdicts. Create alarms for sudden changes in rejection rates or explanation patterns.
  • Protect against prompt injection. Sanitize inputs, restrict what the model can access, and validate any executable outputs the model produces.
Do not rely solely on an LLM for safety-critical enforcement. LLMs should augment human judgment and deterministic rules, not replace them.

​
Deployment implications

When you deploy an LLM-based auditing layer, account for these operational realities:
  • Cost and scaling: estimate per-request costs and provision capacity (or throttling) to meet peak demand.
  • Latency and retries: design fallbacks, timeouts, and retry strategies for cases where the LLM is slow or unavailable.
  • Security and compliance: ensure sensitive inputs are handled according to policy; never send secrets or protected data to external models unless explicitly approved and logged.
  • Observability: include the LLM in traces, metrics, and audit logs so reviewers can reconstruct how decisions were made.
  • Access controls: separate who can change the rule engine vs. who can modify LLM prompts or policies; maintain change logs and approvals.

​
Quick operational checklist

  • Implement static hard blockers with OPA or equivalent.
  • Define clear criteria for when to invoke the LLM.
  • Require structured JSON outputs and validate against schema.
  • Cache repeated checks and batch low-priority requests.
  • Add human approval gates for high-impact decisions.
  • Log inputs, prompts, model outputs, and final decisions.
  • Monitor model behavior, cost, and request latency.
  • Harden inputs and model interfaces against prompt injection.

​
Conclusion

In practice, many organizations run both: static rules for hard blockers and an LLM layer for gray areas where judgment helps. This hybrid pattern maximizes safety and developer productivity, but it changes how you build, deploy, monitor, and audit policy agents. Design for the trade-offs up front: deterministic blocks for safety, LLMs for nuance, and operational controls for resilience and compliance.

Watch Video