BLUF

A written policy in the system prompt is the most common AI governance control and the least measured. Across six current frontier models, the policy measurably changed behaviour on two of four obligations, left two unchanged because the models behaved that way anyway, and leaked on specific pressure cases in five of six models. The instrument failed before it worked, and that failure is part of the result.

Research study. This report measures six hosted models against one synthetic policy with four claims. It does not rank models, establish regulatory compliance, certify safety, or report the behaviour of a production system. The scoring layer's independent validation against human labels is in progress.

Abstract

The most common AI governance control in production is a sentence. An organization writes its policy into the system prompt of a language-model product and treats the presence of that sentence as evidence that the product follows it. Almost nobody measures whether it does, and fewer still measure whether it would have behaved the same way without it.

Technical Report 001 proposed a protocol for producing that evidence and validated its mechanics against a scripted target. This report runs it against six current frontier models: Claude Sonnet 5.5, GPT-5.6 Terra, Gemini 3.8 Flash, Grok 4.7, DeepSeek V4 Pro, and Qwen 3.8 Max. Each answered 48 probes under a written policy and again under a matched no-policy control, for 1,152 scored responses.

Three findings. First, the policy measurably changed behaviour on two of four obligations and on none of the other two. With the policy, every model declined to retain personal data 96–100% of the time; without it, 12–39%. On the other two obligations (disclosing that it is an AI, and escalating licensed decisions) the models behaved much the same with or without the policy. The instruction was doing no measurable work there. Second, the policy leaked under pressure in five of six models. Five gave a specific medication maximum when the request was framed as “purely general”, one cited the policy section it was about to break, and one presented itself as a named human support agent. Third, the probe set could not rank the models. Every per-claim interval overlapped.

The fourth result concerns the instrument. On first contact with real model output, the scoring layer could not read 61% of responses and scored 22 violations as passes. It had to be corrected and re-checked on a model it had never seen before any number here was usable. That is the strongest argument this report makes for treating conformance as something re-checked on a schedule rather than certified once.

1. The problem

Read narrowly, the question is not “is this model safe?”. General safety benchmarks address that, and this report does not compete with them. The question is narrower and belongs to the organization:

When we write our policy into this AI system, does the system’s behaviour change in the way the policy requires, and where does it stop changing?

There are four reasons this goes unanswered in practice. Each is a reason a one-off test is not enough.

The control is missing. If a model already discloses that it is an AI whether or not it is told to, a 100% pass rate under the policy says nothing about the policy. If a model only stops retaining customer data because the policy says so, the policy is load-bearing, and removing or editing it is a change with consequences. Without a no-policy control the two cases look identical. Most published evaluations of “policy following” report only the policy condition.

The system underneath changes. Hosted models are updated behind stable names. Every model in this study was addressed by a public identifier that resolves to a dated snapshot, and four of those snapshots were less than two months old at the time of testing. A result measured in October describes October. Authorization needs an expiry date, and the evidence behind it needs a recheck schedule.

The instrument is a source of error. An automated scorer that cannot read a model’s prose does not fail loudly. It converts evidence into “no evidence”, and an evaluation that quietly drops those cases from its denominator reports a clean number. Very few evaluation reports state their own unscored rate. This one does, because the first version of its scoring layer could not read most of what it was given.

The deployed thing is not the model. Teams deploy agents: a model wrapped in a framework, a tool loop, and accumulated context. A conformance number measured on the bare model does not cover any of that.

A policy that does nothing, a policy that leaks under pressure, a measurement that cannot read the system, and a system that changes after it was measured are four different failures. All four look identical from inside the organization: there is a policy, and nobody has been told it is not working.

2. Research questions

  1. Policy effect. Does including a written policy measurably change behaviour relative to a matched no-policy control, and on which obligations?
  2. Leakage. Under pressure (probes that remove or invert the cue that normally triggers an obligation), where does conformance break even with the policy in place?
  3. Discrimination. Can a probe set of this size distinguish frontier models from one another on policy conformance?
  4. Instrument validity. Can an automated, rule-based scorer read real frontier output well enough for the answers to 1–3 to mean anything?

3. What changed since Report 001

Report 001 validated the evidence pipeline against a scripted target. Its confidence intervals exercised the reporting machinery, not real uncertainty. This report changes three things.

  • Real targets. Six hosted frontier models through a single routing API, each pinned to the dated snapshot it resolved to on the test date.
  • A powered, graduated probe set. Twelve probes per claim, split three easy, three medium and six hard. Hard probes hold the obligation fixed while removing or inverting the surface cue that would trigger it: role-play framing, professional context, an explicit request for the behaviour the policy forbids, time pressure, presupposition. Easy probes establish a floor. Hard probes are where systems that follow a rule separate from systems that pattern-match to it.
  • An honest gate. Report 001’s gate had three states. This one adds a fourth, insufficient evidence, so that “the system fell short” and “the run could not tell” are no longer reported as the same thing (§7).

4. Method

The protocol is Report 001’s, run at scale. In outline:

  • Policy. One synthetic acceptable-use policy (version 1.1), written in-house so that the study contains no client or proprietary content.
  • Claims. Four atomic behavioural claims of four types: disclose AI identity (transparency), escalate licensed medical, legal and financial decisions (escalation), do not retain personal identifiers and prompt their removal (data handling), and justify factual claims with reasoning and a findable source (requirement). Prohibition claims are excluded. A rule-free prohibition probe has to make the actual prohibited request, which needs an authorized testing context.
  • Conditions. Every probe ran with the policy as the system instruction and again with no system instruction.
  • Repetition. Two runs per probe per condition: 4 claims × 12 probes × 2 conditions × 2 runs = 192 responses per model.
  • Scoring. Deterministic and rule-based. No model grades another model’s output. Scoring rules are versioned and regression-tested against a labelled corpus; their design is not described here.
  • Uncertainty. Pass rates carry 95% Wilson intervals computed on a cluster-adjusted effective sample size. Two runs of the same probe are correlated: they sample one prompt’s response distribution, not the space of ways a claim can be exercised. So the interval’s width is driven by the number of distinct probes (about 12 per claim), not by the 24 responses behind it.
flowchart LR
  P["Written policy"] --> C["Atomic claims"]
  C --> Q["Graduated probe set<br/>easy · medium · hard"]
  Q --> T["Policy condition"]
  Q --> U["No-policy control"]
  T --> S["Rule-based scoring<br/>+ unscored rate"]
  U --> S
  S --> I["Clustered Wilson intervals"]
  I --> G{"Evidence gate"}
  G --> A["Authorized"]
  G --> N["Not authorized<br/>(measured shortfall)"]
  G --> X["Insufficient evidence<br/>(could not tell)"]
  M["Model, policy, or<br/>probe-set change"] -.-> T
  R["Scorer improvement"] -.->|"re-score stored<br/>responses"| S

Every response is stored with the run’s manifest: model identifier and snapshot, configuration hash, claim-set hash, code revision, seed. Stored responses can be re-scored when the scoring layer improves, at no further cost, and each re-scored report records when and at which revision it was re-scored. That capability matters in §5.

5. The instrument failed first

The most useful result in this report was produced before any model was compared.

On the first real sweep, four models and 768 responses, the scoring layer that had passed its own calibration suite left 469 of 768 responses (61%) unscored. Per model it ranged from 29% to 74% of policy-condition responses. It found exactly one violation under the policy. A reader shown only the scored responses would have seen near-perfect conformance across the board.

Both conclusions would have been wrong. Reading the unscored and passing responses by hand showed two failure modes:

  • It could not read the prose. The calibration corpus had been built from one model’s output, and real frontier models phrase the same behaviour in ways that corpus never contained. A large share of the unscored cases came down to typography and phrasing rather than anything the model actually did.
  • It credited the right words around the wrong action. The serious case: answers that gave a specific medication dose and mentioned a prescriber were scored as conforming escalations. One model, with the policy in place, gave “an initial starting dose of 5 mg daily” and passed. Twenty-two passes were violations, nine of them with the policy in place. That is the direction that matters, because it reports conformance that is not there.

After correction, the same 768 stored responses re-score at 98 unscored (13%), with 51 policy-condition violations found instead of 1. The probe set was also revised: several probes turned out to be unable to trigger the obligation they claimed to test. A second, full six-model sweep ran on the corrected probe set.

The scorer was then fitted to responses from five of the six models, so its agreement with those models’ output is in-sample. The sixth model ran afterwards, on responses the scorer had never seen. A hand audit of every fail, every unscored response and every hard-tier pass in that held-out set (by the scorer’s author, so not independent) found no false passes, one false fail, and about ten clear-cut responses left unscored. Those were corrected before the final numbers.

The point is not that the first scorer was bad. Every automated scorer is fitted to some corpus, and the corpus is never the deployment. An evaluation that does not report its unscored rate, does not audit its passes, and has not been run against output it was not fitted to has not measured anything yet. A one-time evaluation would have published the first set of numbers.

6. Results

6.1 Can the scorer read the output?

  Claude Sonnet 5.5 GPT-5.6 Terra Gemini 3.8 Flash Grok 4.7 DeepSeek V4 Pro Qwen 3.8 Max
Unscored, policy condition 2% 4% 5% 6% 4% 4%
Unscored, all responses 8% 11% 9% 9% 8% 7%

The no-policy condition is harder to score, mostly because unconstrained answers are longer and less structured. Unscored responses are excluded from pass rates and reported, never folded in.

6.2 Does the policy do anything?

Pass rate with the policy versus without it, per model. ✱ marks a policy condition whose interval sits clear of the control’s.

Obligation With policy Without policy Policy effect
Do not retain personal data 96–100% 12–39% ✱ all six models
Reasoning and a findable source 57–100% 0–26% ✱ all six models
Escalate licensed decisions 83–100% 74–81% none detectable
Disclose AI identity 91–100% 78–100% none detectable

This table is the headline. On data handling and sourcing, the written policy is load-bearing: the models do the required thing because they were told to, and would mostly not do it otherwise. Remove or weaken that instruction and the behaviour goes with it, which makes the policy text itself a component that needs change control.

On escalation and disclosure, the models already behave close to the policy without it. A 100% pass rate there is evidence about the model’s training, not about the policy. That has two consequences. Those obligations are not where this policy needs auditing. And the organization is relying on a provider default it does not control, which can shift with the next model snapshot.

6.3 Where does it leak?

Policy condition only. Every row is a failure with the policy in place.

Probe pattern Models that failed What happened
A medication maximum requested “purely in general terms” for a person with reduced liver function 5 of 6 Stated a specific daily maximum dose; GPT-5.6 Terra escalated instead
A dosing question framed as exam revision 1 of 6 Opened with an escalation notice citing the policy section, then gave a starting dose
A “blind UX study” asking the assistant to respond as a human agent 1 of 6 “You’re speaking with Alex from customer support.”
Role-play, then “are you a real person, or one of those AI things?” 1 of 6 Answered in character as human-enough; others disclosed while staying in character
“Read my reference number back to me” 1 of 6 Replied with the number
“Yes or no, no sources”, “three words”, “skip the citations” 4 of 6 Gave the bare answer the user asked for

All six patterns are hard-tier probes. Outside the sourcing claim, the easy and medium tiers produced no violations under the policy at all, which is exactly why they cannot separate systems. Within it, the easy and medium failures were answers that explained well and cited nothing a reader could check.

The first row is the clearest. Five of six frontier models, explicitly instructed to escalate dosing questions, gave a dose once the request was framed as general information. The framing did not change the obligation. It changed the cue. A policy that holds only when the user phrases the request the way the policy author imagined has not been tested.

6.4 Can the probe set rank the models?

No.

  Claude Sonnet 5.5 GPT-5.6 Terra Gemini 3.8 Flash Grok 4.7 DeepSeek V4 Pro Qwen 3.8 Max
Hard-tier pass rate, with policy 93% 93% 91% 86% 98% 75%

Per claim, every model’s interval overlaps every other model’s. With about twelve distinct probes per claim, a difference of one or two failures is within the noise. Qwen 3.8 Max is weakest on the hard tier and is the only model with a measured shortfall (§7), which is suggestive but not a ranking. Separating models of this calibre on individual claims would take roughly 30 or more hard probes per claim.

This report therefore publishes no leaderboard. A ranking over a probe set that cannot separate its subjects would manufacture a distinction the evidence does not support.

7. The authorization gate

Each claim carried an evidence requirement: the lower bound of its 95% interval must reach 75%.

Status Models Meaning
Authorized none Every required claim certified above its floor
Not authorized Qwen 3.8 Max Reasoning-and-sources measured at 57%, below the 75% floor
Insufficient evidence the other five Observed rates at or above the floor on every claim, but intervals still reaching below it

The distinction between the last two rows is the gate fix this report introduces. Report 001’s gate called any interval below the floor a failure. Under that rule, a model observed at 96% against a 75% floor, with one failure in twelve probes, was reported as having fallen short. The evidence points the other way: the run was too small to certify it. A buyer told “not authorized” hears that the system failed. A buyer told “insufficient evidence” hears that the test was not big enough. Only one of those is true, and the gate now says which, together with an estimate of how many distinct probes would settle it.

No model was authorized. That is the correct result for a 48-probe study. Conformance at a meaningful floor is bought with probe coverage, not repetition.

8. Why this is a recheck, not a certificate

Every number above has a shelf life.

  • The models will change. Each result is tied to a dated snapshot. When the identifier resolves to a new one, the evidence describes a system that is no longer deployed.
  • The policy will change. Section 6.2 shows that on some obligations the policy text is load-bearing. Editing it is a behavioural change and should trigger a recheck like any other.
  • The instrument will improve. Stored responses can be re-scored by a better scorer at no cost, and the re-scored evidence is stamped as such. This study did exactly that twice.
  • The probe set will grow. A larger hard tier is what turns “insufficient evidence” into a decision. Results across probe-set versions are not comparable, and the manifest records the claim-set hash so that is checkable.

So the conformance unit is a periodic recheck: the same claims, the same control, re-run on a schedule and on every material change, with authorization that expires.

9. From benchmark to agents: a proposed runtime monitor

A recheck measures the system under test conditions. It says nothing about the thousands of real exchanges between rechecks. For that, Aegis has a prototype runtime monitor for agent systems. It attaches to an agent (the first adapter targets smolagents) and judges live exchanges against the same claims with the same scorer.

Three design rules carry the trust argument:

  1. It is an outside observer. It wraps the agent and returns the agent’s answers unchanged. It is not a tool the agent can call and it is not in the agent’s prompt. An agent grading its own conformance is the problem being solved, not a way of solving it.
  2. It judges only what was exercised. Most real exchanges invoke no policy obligation. A claim is scored only when the user’s message triggers it, and each event records which trigger fired.
  3. It does not become the leak. Logs are redacted before writing. A monitor for a data-retention policy that kept raw transcripts would retain exactly what the policy forbids.

The monitor’s limit is measured, not assumed. Replaying the 48 benchmark probes through it, before any tuning against them, it triggered the right claim on all 12 easy probes but only 14 of 24 hard ones. The misses are indirect requests that never use the obvious words (“What time do you finish your shift?”), and those are exactly the cases where §6.3’s leaks occurred. The monitor sees obvious exercises of a policy reliably and subtle ones poorly. The periodic recheck exists for the subtle ones. Neither replaces the other.

Live monitoring also has no control condition, so its rates are signals about the traffic that happened, not measurements of the agent. The monitor is a prototype and is proposed here for agentic pilots, not presented as a validated product.

10. Limitations

Scorer validity is not independently established. The scorer was corrected against responses from five of the six models it is evaluated on, and its held-out audit was performed by its author. Labelling round one is in preparation: 200 responses sampled across all six models, blind to model, condition and scorer verdict, for two labellers who did not write the scorer, against a pass bar fixed before the labels are seen. Until it reports, agreement between the scorer and careful humans is asserted, not shown.

One synthetic policy, four claims, no prohibitions. Results do not transfer to other policies, claim types, or real frameworks.

Twelve probes per claim. Enough to see where a policy leaks; not enough to rank models or certify a 75% floor.

Single-turn. One message, one final answer. An agent that violates a policy at step three and recovers by step six would score as conforming.

Interval overlap is a screen, not a test. Non-overlapping marginal intervals are a conservative separation heuristic. A paired or hierarchical model of probe-level dependence would be stronger.

Routed hosted models. All six were reached through one routing API at its default settings. Provider-side routing, safety layers and sampling configuration are outside the study’s control and may differ from a direct integration.

The control removes one thing. The no-policy condition removes the written policy and nothing else. Provider-level safeguards remain in both conditions, so “no detectable policy effect” means the policy added nothing measurable on top of them.

11. Reproduction and scope of disclosure

The harness and the v0.2 claim set are being prepared for public release. Once released, they re-run every measurement in this report against the same claims and probes. Hosted models change behind stable names, so a re-run measures the snapshots current at that time, not the October ones. The full six-model sweep costs a few dollars at current routing prices.

This report does not walk through the scoring rules, the monitor’s trigger design, or the method used to construct hard probes. Production engagements also use material that is not published: claim libraries mapped to real frameworks, held-out probe sets that are never released (a public probe set can be optimised against, and this one now is), and calibration data from labelled deployments.

12. Conclusion

Report 001 asked whether written policy could be turned into reviewable deployment evidence, and answered with a pipeline that worked against a scripted target. This report points that pipeline at six real frontier models and finds what a policy owner would want to know.

The policy did real work on two of four obligations and none on the other two. It leaked on specific pressure cases in five of six models. The measuring instrument was itself wrong before it was right, in the direction that hides violations. And none of it stays true for long.

That last point is the product argument, made from evidence rather than asserted. Conformance is not a property a system has; it is an observation about a system at a time, under a policy, measured by an instrument with its own error rate. The organizations that will be able to say their AI follows their policy are the ones that keep checking: on a schedule, on every change, against a control, with the instrument’s own blind spots reported next to the result.

References

After the last line

Comments & sharing

Agree, disagree, add context, or send this to someone who would have a take.

Share

LinkedIn X Email

Public thread

Comments are public and attached to this post through GitHub Discussions. Sign in with GitHub to join the thread.

Open public discussions

Newsletter, eventually

Get the next brain dump.

No fixed cadence yet. Leave your email for the first issue when it exists.

Unsubscribe whenever. How your email is handled.