# Retrieval audience pilot — investigation record

**I06 · Executed October 23, 2026 · Scripted pilot; no language model, no cloud service, no network.** Gives [Retrieval is not permission](https://rethinkingagents.com/blog/retrieval-is-not-permission) its first observation. It runs the paired-user cases from the [retrieval review packet](https://rethinkingagents.com/resources/retrieval-review.md) against two pieces of real third-party code, run locally: the Azure sample revision pinned in the [K09 source register](https://rethinkingagents.com/research/retrieval-boundary.md), and LangChain's response caches. Not a test of any Azure tenant, Azure AI Search, Entra ID, Cosmos DB, deployed copy of the sample or production cache.

## Question

The article argues that a correctly filtered search does not settle who receives an answer. Identity failures, citation routes, stored history and answer caches each need their own audience decision. For the packet's two fictional users, Amira (account team) and Ben (general support), what does real code do?

1. **Azure sample, application layer.** What does the pinned backend do when the caller's identity is absent, malformed, from another tenant or temporarily unverifiable (R10, R11)? Does the citation route repeat the access decision on its own (R09)? Does a valid token from the same tenant open a restricted file (R13)? Does reopening stored history recheck current access (R06)?
2. **Response cache.** With Amira's authorized answer cached, does Ben's identical question get it (R03)? Does the reverse happen (R04)? Does a stored answer survive a membership removal, a document permission change or a content change (R05, R07, R08)? Does the cache key contain any caller identity?

## Data and provenance

- **Azure sample:** `Azure-Samples/azure-search-openai-demo` commit `3f4a21f03ae3d565aca37cc300e3d38b0c7b582a` (committer date 2026-07-27, as in K09). Downloaded as `https://codeload.github.com/Azure-Samples/azure-search-openai-demo/tar.gz/3f4a21f03ae3d565aca37cc300e3d38b0c7b582a` (archive SHA-256 `b06974b45c9278edca4279c7ac9562c9e83d8572c5bc14df927c67faa743caa9` on 2026-10-23; GitHub does not guarantee stable archive bytes, so the file digests below are the pins). [`run.py`](https://rethinkingagents.com/research/retrieval-audience-pilot.py) refuses to run if any pinned file differs.

  | File at the pinned revision | SHA-256 | Role |
  |---|---|---|
  | `app/backend/core/authentication.py` | `f77e2dc9989265570ea6a685228ffbe50dca2c83b5b8c5118b12eab1264723dd` | imported and driven; copy in [`pinned/authentication.py`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/programme-2026-09/investigations/retrieval-audience-pilot/pinned/authentication.py) |
  | `app/backend/decorators.py` | `83e4224c6f3a9f86a5ad2f6c2bf882906182d29e100c9ebd123030038def3e9f` | imported and driven; copy in [`pinned/decorators.py`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/programme-2026-09/investigations/retrieval-audience-pilot/pinned/decorators.py) |
  | `app/backend/chat_history/cosmosdb.py` | `1c692178ff9dea796ea2a2edff5ef56225a48091c0453b99dc04346000bf6c94` | imported and driven; copy in [`pinned/cosmosdb.py`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/programme-2026-09/investigations/retrieval-audience-pilot/pinned/cosmosdb.py) |
  | `app/backend/approaches/approach.py` | `8fb4cb0c5310da12aed5aacfa6edcc8f144a7019714d7e8ab2902705a1cbee90` | imported; `Approach.search` driven |
  | `app/backend/approaches/chatreadretrieveread.py` | `b7a05012d89c727bf4c6ab0e9bc3a3464e7c299114e46784a7e5cfe770801494` | read only (lines 320 and 431) |
  | `app/backend/app.py` | `7b6397ab1187faccdaf52ea933516e30d479226a097501ae3446bf3e4360daf9` | read only (`content_file`, lines 142–186) |
  | `docs/login_and_acl.md` | `43f66e5e3712526d078f6332f46f535a5aad0c20f2640512efb429ab02125e05` | read only (configuration flags) |

  The sample is MIT-licensed ([`pinned/LICENSE`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/programme-2026-09/investigations/retrieval-audience-pilot/pinned/LICENSE)); the three copies are unmodified.
- **Azure sample environment:** CPython 3.10.20 (the version the sample's lockfile was compiled for), dependencies installed from the sample's own `app/backend/requirements.txt` at the pinned revision with uv 0.12.0. Freeze: [`requirements-azure.freeze.txt`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/programme-2026-09/investigations/retrieval-audience-pilot/requirements-azure.freeze.txt) (every lockfile pin, plus four transitive Python 3.10 backports uv added that the lockfile does not list: `async-timeout`, `exceptiongroup`, `taskgroup`, `tomli`). Key versions: `quart` 0.20.0, `pyjwt` 2.13.0, `msal` 1.37.0, `tenacity` 9.1.4, `azure-search-documents` 12.1.0b1.
- **Cache environment:** CPython 3.12.6, `langchain-core` 1.7.0 (wheel SHA-256 `e82b24d7ac51fc74874baa29d6b3c0b8234880d4f3d897f8ff57fb80dfd9b07d`) and `langchain-community` 0.4.2 (wheel SHA-256 `84dd8c5122532394d5b6849a5fc9995ef28e4f77227daeb09f24b3d942e9e466`), both from PyPI. Installed module digests: `langchain_core/caches.py` `ef38ab793ad7439cf412170b0b683ef0eb10c9ff3617a78194a6a814474d94b1`, `langchain_community/cache.py` `56c8be4aed235ca486e634ede2ada1f025cef1c9d20717c69ec98c19a630bd65`. Freeze: [`requirements-cache.freeze.txt`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/programme-2026-09/investigations/retrieval-audience-pilot/requirements-cache.freeze.txt).
- **Inputs:** the packet's fixture text, question and users. Tenant, application and user IDs are fictional all-zero GUIDs. Signing keys are generated per run and never written out. No credentials, no model calls, no cloud accounts. Sockets were blocked in both parts; `results.json` records zero attempts.
- **Committed here:** [`run.py`](https://rethinkingagents.com/research/retrieval-audience-pilot.py) (SHA-256 `b5fa33598e25e504a6a87af1206efab9633bd434635c82f1b35975f81406d7b3`) and [`results.json`](https://rethinkingagents.com/research/retrieval-audience-pilot.json) (SHA-256 `cc8f8ab56ec2051c2013e73b42bf84764b3482fee27cf75dff471e4bcd772516`).

## Method

### Part 1: the Azure sample's identity path, run locally

[`run.py`](https://rethinkingagents.com/research/retrieval-audience-pilot.py) imports the pinned backend modules unmodified. It replaces only their network dependencies with local recording stubs:

- **Entra signing keys:** the URL the helper requests is checked and answered locally with a generated RSA key, so the real `validate_access_token` (pinned lines 228–283: issuer, audience, signature and expiry checks) runs on real JWTs.
- **On-behalf-of exchange:** `ConfidentialClientApplication` is replaced by a stub that returns `{"access_token": "search-token:<oid>", "id_token_claims": {"oid": …}}`, an error dict, or raises.
- **Azure AI Search:** a stub `search` that records the `filter` and `query_source_authorization` it receives and returns fixture documents. The public guide is marked `["all"]`; the account plan is restricted to group `account-team`. A tokenless query returns only `["all"]` documents, as the pinned guide describes (lines 93 and 463). **This stub is ours and stands in for the service. No service-side decision in this record was observed.**
- **Cosmos DB:** a stub container scoped by partition key `[oid, session_id]`.

Requests go through Quart's test client to three routes. `/probe/chat` is wrapped in the pinned `@authenticated` decorator and calls the pinned `Approach.search` in both its plain and semantic branches. It reads the token with `auth_claims.get("access_token")`, as `chatreadretrieveread.py` does at lines 320 and 431. `/content/<path>` is wrapped in the pinned `@authenticated_path`; its body stands in for `app.py`'s `content_file`, so reaching it means the gate let the request through. The third route is the pinned chat-history blueprint, `GET /chat_history/sessions/<id>`.

Four configurations, set through the sample's three flags (`AZURE_USE_AUTHENTICATION`, `AZURE_ENFORCE_ACCESS_CONTROL`, `AZURE_ENABLE_UNAUTHENTICATED_ACCESS`): **login-off** (F, F, F), **login-no-enforce** (T, F, F), **enforce** (T, T, F) and **enforce-plus-unauthenticated** (T, T, T). In each, ten identities: valid Amira, valid Ben, no header, a `Basic` header, `Bearer` with no token, a non-JWT, an expired token, a wrong audience, another tenant's issuer, and a token signed by a different key. Four outage modes for valid Amira: signing-key endpoint unreachable, on-behalf-of error response, on-behalf-of exception, search unavailable. Plus two sequences: Amira loses `account-team` between two citation requests, and Amira reopens a stored session whose answer contained the private date, after losing `account-team`.

### Part 2: LangChain response caches

`GenericFakeChatModel` from `langchain-core` with a cache attached: `InMemoryCache` (`langchain-core`) and `SQLiteCache` (`langchain-community`, a fresh file per run). Our fixture retriever applies the packet's audience to the current membership and document state. Before each call the harness sets the fake model's next message to the answer that is currently authorized for that caller. A model call therefore always returns a correct answer. What is under test is whether the cache serves a stored answer instead of making that call. Model calls are counted outside the model, from the scripted message generator.

Three prompt arrangements an application might use: **question-only** (system text plus the question), **caller ID as invocation parameter** (the same prompt, `invoke(…, user=<oid>)`), and **retrieved passages in prompt** (the caller's currently visible passages, followed by the question). Five sequences from the packet: R03 Amira then Ben; R04 Ben then Amira; R05 Amira, remove her membership, Amira; R07 Amira, change the plan's group without changing its text, Amira; R08 Amira, change the plan's date without changing its permission, Amira. After each sequence the harness reads the cache store directly: `InMemoryCache._cache` keys, and the SQLite file through `sqlite3`.

### Static reading, labelled as such

`langchain-community` 0.4.2 `cache.py` was read, not run, for Redis and the semantic caches. No Redis, vector store or embedding model was started.

### Status conditions

The conditions are ours, derived from the packet's "Required observation" column. There is no upstream specification for them.

- **Cache cases.** PASS: the answer served at the last step meets the packet's requirement. R03: Ben gets the procedure and not the private date. R04: Amira gets the date. R05 and R07: Amira's second answer has the procedure and not the date. R08: the second answer has the new date and not the old one. FAIL otherwise. InMemoryCache and SQLiteCache must agree, or the run aborts.
- **Azure cases, application layer only.** PASS: the pinned code meets the requirement before any service decision is needed, for example by refusing the request, or by repeating a caller-scoped query on every request. FAIL: the pinned code alone contradicts the requirement, for example by serving a route with no check or returning a restricted stored answer with no check. UNKNOWN: the outcome turns on a service decision that only our stub made. NOT EXECUTED: no executable path in what we ran.

```sh
W=/tmp/ra-i06; D=azure-search-openai-demo-3f4a21f03ae3d565aca37cc300e3d38b0c7b582a
mkdir -p $W/archive && cd $W/archive
curl -sSfL -o demo.tar.gz https://codeload.github.com/Azure-Samples/azure-search-openai-demo/tar.gz/3f4a21f03ae3d565aca37cc300e3d38b0c7b582a
tar xzf demo.tar.gz
cd $W
uv venv --python 3.10 venv-azure
VIRTUAL_ENV=$W/venv-azure uv pip install -r archive/$D/app/backend/requirements.txt
uv venv --python 3.12 venv-cache
VIRTUAL_ENV=$W/venv-cache uv pip install langchain-core==1.7.0 langchain-community==0.4.2
python3 -I "$REPO/docs/research/programme-2026-09/investigations/retrieval-audience-pilot/run.py" --work $W > results.json
```

`$REPO` is this repository's root. The orchestrator uses only the standard library and runs each part with `-I` in its own environment. A run takes about 25 seconds; three runs produced byte-identical `results.json`. Signing keys and timestamps change per run but are not written out. $0.

## Results

| Case | Subject · configuration | Status | Observed |
|---|---|---|---|
| R01, R02 | No subject run | NOT EXECUTED | No real retrieval system or model. The first, empty-cache step of R03 and R04 exercises only our fixture retriever and scripted model. |
| R03 | Cache · question-only | **FAIL** | Ben's request was served Amira's stored answer, including “November 12”. No model call was made. |
| R03 | Cache · caller ID as parameter | PASS | Separate entries per user. The caller's OID appears in the stored `llm_string`. |
| R03 | Cache · passages in prompt | PASS | Different prompts, so a cache miss and a fresh public-only answer for Ben. |
| R04 | Cache · question-only | **FAIL** | Amira was served Ben's stored public-only answer and lost her authorized date. |
| R04 | Cache · caller ID / passages | PASS, PASS | Amira got a fresh answer with the date. |
| R05 | Cache · question-only / caller ID | **FAIL**, **FAIL** | After her membership was removed, Amira was still served her stored answer with the date. |
| R05 | Cache · passages in prompt | PASS | Her visible passages changed, so the prompt changed: a miss. The old restricted entry stays in the store. |
| R05 | Azure · enforce | NOT EXECUTED | The decision is the service's. Recorded: each request made a fresh on-behalf-of exchange and a fresh caller-token query (1 and 1 per request). The app holds no access decision between requests. |
| R06 | Azure · enforce | **FAIL** (against the packet's strict policy) | After Amira lost `account-team`, `GET /chat_history/sessions/s-amira-1` returned 200 with the stored private date and made 0 search calls. History is scoped by owner OID only (pinned `cosmosdb.py` lines 142–150). Ben requesting the same session ID got 200 with 0 answers, through our stub's partition scoping; a request with no identity got 403. The packet says owner-keyed retention can be a legitimate product policy. |
| R07 | Cache · question-only / caller ID | **FAIL**, **FAIL** | Plan regrouped, text and digest unchanged: the stored answer with the date was served. |
| R07 | Cache · passages in prompt | PASS | Miss, then a fresh answer without the date. |
| R07 | Azure · enforce | NOT EXECUTED | Same as R05: the decision is the service's. |
| R08 | Cache · question-only / caller ID | **FAIL**, **FAIL** | The plan's date changed to “December 3”; the stale “November 12” answer was served. |
| R08 | Cache · passages in prompt | PASS | Miss, then a fresh answer with the new date. |
| R08 | Azure | NOT EXECUTED | Content freshness is an index property; no index was run. |
| R09 | Azure · enforce | PASS (application layer) | Ben's direct `/content/account-plan-v1.md` made its own query: path filter, Ben's token, no earlier state. It returned 403, while Ben could still open the public guide (200). `check_path_auth`, pinned lines 180–207. |
| R09 | Azure · login-no-enforce | **FAIL** (documented) | `check_path_auth` returned `True` before any query (line 182), so Ben reached the restricted file. The route docstring documents this (app.py lines 148–149). |
| R10 | Azure · enforce, login-no-enforce | PASS | All eight absent or malformed identities got 403 on chat, public and restricted routes, with 0 search calls. |
| R10 | Azure · enforce-plus-unauthenticated | UNKNOWN | All eight became `{}` claims: no header, `Basic`, empty `Bearer`, non-JWT, expired, wrong audience, other tenant and wrong signature. Chat then ran a tokenless search (HTTP 200); whether that returns public-only results is the service's decision. The citation route raised `KeyError` at line 200 and returned 500 for **every** path, including the public guide. |
| R10 | Azure · login-off | **FAIL** (documented) | No identity exists; the citation gate passes every path, including the restricted file. |
| R11 | Azure · enforce | PASS | On-behalf-of error: 403. On-behalf-of exception, signing keys unreachable, search unavailable: 500. Nothing was served. |
| R11 | Azure · enforce-plus-unauthenticated | **FAIL** | When the on-behalf-of exchange errored or raised, or the signing-key endpoint was unreachable, signed-in Amira silently became anonymous (lines 169–178). Chat proceeded with HTTP 200 and a tokenless search, with no unavailable state. Whether results stay public-only is the service's decision. |
| R12 | No subject run | NOT EXECUTED | The pinned backend has no tool or export action to gate. |
| R13 | Azure · login-no-enforce | **FAIL** (documented) | A valid same-tenant token alone opened the restricted file through the citation route. |
| R13 | Azure · enforce | NOT EXECUTED | The group decision is made in Azure AI Search from the forwarded token. At this revision the app builds no group or OID filter (`build_filter` handles only categories, approach.py lines 277–285). Another tenant's token was refused (403). |
| R14 | No subject run | NOT EXECUTED | The pinned backend has no conversation download, export or share route. |
| R15 | No subject run | NOT EXECUTED | Only the 180 log records emitted by the executed paths were checked: none contained the private date or a token. Telemetry export was not run. |

**Cache keys.** For every run, the stored key was exactly (serialized prompt, `llm_string`). The `llm_string` was `[('_type', 'generic-fake-chat-model'), ('stop', None)]`, plus `('user', <oid>)` only when the application passed it. No caller identity appeared unless the application put it in the prompt or the invocation parameters. The SQLite table's columns are `prompt, llm, idx, response`. Its `response` column held Amira's answer, private date included, as plain text, with no audience field.

**Static reading of `langchain-community` 0.4.2 `cache.py`, not run.** Every cache implements `BaseCache.lookup(prompt, llm_string)`, which has no caller argument. `RedisCache` keys on `_hash(prompt + llm_string)` (line 392). `RedisSemanticCache` keeps one index per `llm_string` (line 641) and returns the nearest stored prompt within `score_threshold` 0.2 (lines 682–690), with no identity or source filter. We did not measure whether Ben's public-only prompt would fall within that distance of Amira's.

## What the pilot shows

1. **With enforcement on and unauthenticated access off, the pinned sample refused every bad identity before retrieval.** It also repeated the access check on the citation route with the caller's own token. Malformed, expired, wrong-audience, other-tenant and forged tokens all got 403 with no search call. Ben's direct request for the restricted file triggered its own path-scoped query and was refused, while the public guide stayed reachable. This is the application half of the boundary; the service half was not run.
2. **Adding `AZURE_ENABLE_UNAUTHENTICATED_ACCESS` turns identity failures into anonymous sessions.** Every malformed or forged token, and an outage of the identity service for a signed-in user, produced empty claims rather than an error. Chat then ran a tokenless query, and the user saw no signal. The design relies on the service returning only globally readable documents to tokenless queries, which the sample's guide states and we did not observe. The citation route in the same configuration fails with a 500 for every path, so anonymous users cannot open even the public guide.
3. **Stored history is scoped to its owner, not to current access.** Reopening a session after membership removal returned the restricted answer without any search. Under the packet's strict policy, this is a FAIL; under a retention policy, it is the intended behaviour. The packet asks the product owner to choose, and this observation shows what the pinned code chose.
4. **LangChain's cache interface has no audience.** The key is whatever the application put in the prompt and parameters. A question-only prompt gave Ben Amira's private date and gave Amira Ben's public-only answer. A per-user key stopped cross-user reuse but still served the restricted date after Amira's membership was removed. With each caller's retrieved passages in the prompt, all five sequences passed. That happened because the passages changed the key, not because anything checked access, and the restricted answer remained on disk.

## Limits

- **No service ran.** Every Azure AI Search, Entra ID and Cosmos DB decision was made by our stubs. The tokenless "public-only" behaviour is the sample guide's statement (lines 93, 463), not an observation. Group evaluation, permission propagation delay and ACL freshness were not tested.
- **Application layer, not application.** `app.py` was not imported; its `content_file` body and the chat route's error handling were replaced by probe routes around the pinned decorators. The query-rewrite, model and agentic knowledge-base paths (`run_agentic_retrieval_approach`) were not run. MSAL's own token cache was bypassed by the stub, so its effect on revocation was not observed.
- **Signing-key 5xx retries were not run.** They wait 15–60 s between up to five attempts (lines 233–237). By static reading, the final failure surfaces as a `tenacity.RetryError` rather than an `AuthError`. We did not execute that.
- **The configuration is unknown.** No reader's deployment was inspected. Four flag combinations were run; `AZURE_ENABLE_GLOBAL_DOCUMENT_ACCESS`, user upload, SharePoint and the agentic path were not.
- **The cache model is a scripted fake with our retriever.** The retriever and the "authorized answer" are ours, so R03–R08 test only the cache's reuse behaviour in three prompt arrangements an application might choose, not any product. Semantic and Redis caches were read, not run.
- **One run per configuration, deterministic.** Rerunning reproduces the output byte for byte, and no timing or performance figure is reported.

## What should follow

- Only if a reader needs the service half: an authorized synthetic Azure AI Search index with query-time ACLs and the packet's two users, to observe tokenless queries, a group removal and propagation delay. That needs a tenant and spend outside D48.
- A semantic-cache run (`RedisSemanticCache` against a local Redis container with a fixed local embedding) to measure whether Ben's and Amira's retrieval-augmented prompts fall within the default threshold.
- A trace of the full chat route (`app.py`, query rewriting stubbed) to confirm the R10/R11 observations end to end and to record what the response's thought payload exposes (R15).

## Change log

- 2026-10-23: executed; three identical runs; script, results, freezes and pinned copies committed.