SKY-BEN-004Reliability BenchmarksRev. 2026.10
Fifty thousand trajectories, two architectures, one model.
We ran the same planner, prompts and tools through an unconstrained ReAct loop and through Skyla's state-gated supervisor. This chapter reports what changed, what it cost, and what the numbers do not show.
Does structure help, or does it only restrict?
The common worry about guardrails is that they trade capability for safety. A model boxed in by policy, the argument goes, will finish fewer tasks. We wanted to measure that honestly, on work that looks like what our customers run, rather than on puzzles.
So we held everything constant except the execution architecture. Same model, same system prompts, same tool definitions, same seeds. The only difference was whether model output went straight to tools or through a gate, a sandbox and a receipt check first.
Five families of consequential work.
The corpus holds 50,000 trajectories, 10,000 in each of five families: treasury operations, clinical order entry, infrastructure change, data subject requests and procurement. Trajectories average 14.3 steps, for 715,000 proposed tool calls in total.
Every tool talks to a seeded simulation of the real system, so outcomes can be scored against a known correct end state. In 12% of trajectories we planted adversarial content inside tool responses, such as an email asking the agent to change a payee. Another 8% contain stale or contradictory records.
The two systems under test.
The baseline is a well-tuned ReAct loop with tool-call validation by schema only, the way most production agents are built today. It may retry on errors and has the same credentials Skyla would issue, but no state machine, quota or receipt.
The Skyla arm wraps the identical planner in a machine per family, with policies written by an engineer who had not seen the test set. Escalations were answered by a scripted approver that followed the written policy.
More tasks finished, and no unauthorised writes.
Gating raised the correct end-state rate in every family, from 71.4% to 88.9% overall. Most of the gain came from denials that turned into better second attempts: a malformed payment that the baseline sent and the simulated bank rejected became, under Skyla, a denial, a corrected proposal and a clean commit.
On safety, the baseline made 1,312 state changes its credentials allowed but its task did not, including 692 driven by injected instructions. The gated arm made none.
- ReAct baseline (unconstrained loop)
- Skyla state-gated supervisor
| Task family | ReAct | Skyla | Δ points |
|---|---|---|---|
| Treasury operations | 68.2% | 90.4% | +22.2 |
| Clinical order entry | 64.9% | 87.1% | +22.2 |
| Infrastructure change | 73.5% | 88.3% | +14.8 |
| Data subject requests | 76.8% | 91.0% | +14.2 |
| Procurement | 73.6% | 87.7% | +14.1 |
| All 50,000 | 71.4% | 88.9% | +17.5 |
| Measure | ReAct | Skyla |
|---|---|---|
| Unauthorised state mutations | 1,312 | 0 |
| Malformed payloads reaching a tool | 4,870 | 0 |
| Injected instructions acted on | 692 | 0 |
| Runs replayed to identical end state | 38.0% | 100.0% |
| Runs escalated to a human | n/a | 2.1% |
| Added latency per call, p50 / p99 | 0 / 0 ms | 11 / 38 ms |
What the gate costs.
Each gated call added a median of 11 ms and a 99th percentile of 38 ms, dominated by invariant lookups against live state. Against model latency measured in seconds, users did not notice. Token use rose by 6% because denials add a short feedback message and a retry.
The larger cost is human attention. 2.1% of runs escalated, and someone has to answer them. We think that is the right place to spend people, but it is a real cost and you should plan for it.
What these numbers do not prove.
Simulations are cleaner than production, and our policies were written with knowledge of the families even if not of the test cases. A different model may close some of the gap on its own. Zero unauthorised mutations means zero against the policy we wrote; a policy with holes would let the same mistakes through, legally.
Customers can rerun the full harness, seeds and oracles under NDA. Write to systems@skyla.ai and we will set it up against your own model and tools.