# Budget scenarios HB-13–HB-16: LiteLLM proxy against a fake metered provider

Rethinking Agents · October 23, 2026 · Executed synthetic experiment

**HB-13 to HB-15 ran against a real LiteLLM proxy (1.106.2, package files unmodified) on loopback, with a fake metered provider whose ledger the test owner reads directly. HB-16 ran against a reference supervisor written for this record.** No model, real provider, credential, paid service or network route beyond loopback was used. Credits are artificial integers.

HB-13 to HB-15 ran in two configuration arms, graded separately:

- **Arm 1 (no database).** Budgets come from a `custom_auth` key with `max_budget=10`. LiteLLM's settlement path does not run. These are the first results, kept unchanged.
- **Arm 2 (database).** This is the configuration LiteLLM documents for budgets: Postgres, LiteLLM's Prisma schema, a master key, and real virtual keys created through `/key/generate` with `max_budget=10`. The settlement path runs, and the record observes it from the database.

The database changes two legitimate-control grades: HB-13 reconciliation is now PASS, and HB-14 recovery is now PASS. It changes no enforcement grade. A pricier fallback, an unpriced route and a timed-out call still let provider charges exceed the cap (see [What the database changes](#what-the-database-changes)).

The [earlier operations lab](https://rethinkingagents.com/research/harness-operations-lab.md) (BU-01–BU-06) already checked the in-memory reservation invariant. That check is not repeated here. This run adds real concurrency across processes, an HTTP provider that can charge and then withhold its acknowledgement, unpriced and accounting-unavailable paths through a real gateway, and local termination of a task while the remote job keeps billing.

## Question

When a self-hostable gateway claims a per-key budget, does it keep provider charges inside the cap under these conditions?

- concurrent callers in separate processes;
- retries and configured fallbacks;
- unpriced operations;
- a lost accounting store;
- a lost acknowledgement.

Second question: when a local task is killed, what does a supervisor know about remote work that is still billing?

Third question (arm 2): with LiteLLM's database, is each reservation settled to the provider's charge? How long after the response does that happen? Is it correct for a timed-out request and for a fallback to a pricier group?

Each scenario's boundary, evidence and legitimate-control results are reported separately, per the [scenario grading rules](https://rethinkingagents.com/resources/harness-evaluation-scenarios.md#grading).

## Data and provenance

| Component | Real or mocked | Version / pin |
|---|---|---|
| LiteLLM proxy (gateway under test, HB-13–15) | **Real**; package files unmodified, with log-only runtime wrappers (see Observation) | `litellm==1.106.2` from PyPI; installed wheel `litellm-1.106.2-cp310-abi3-manylinux_2_28_x86_64.whl`, sha256 `cb1a2a1e55a4a8350ae2167525dc0d06f59b078cd685fb3e0f5419f02da5420e`; full environment in [`litellm-freeze.txt`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/litellm-freeze.txt) (117 packages; no Prisma) |
| Spend-counter store | Real | Redis server 7.0.15 (Ubuntu package), private instance per scenario, no persistence |
| LiteLLM database (arm 2 only) | **Real** | Official `postgres:16-alpine` (PostgreSQL 16.15), pinned `postgres@sha256:f4610795587492cd4fb887c2dc3de301b813278bb6f616992061141b5f15ac36`. One container per invocation, `ra-hb-budget-db-pg`, bound to a loopback port in 3100–3300. Data on tmpfs (no volume). Removed on exit. |
| Prisma (arm 2 only) | Real | `prisma==0.15.0` (litellm's own `extra-proxy` requirement, wheel sha256 `de949cc94d3d91243615f22ff64490aa6e2d7cb81aabffce53d92bd3977c09a4`), Prisma CLI 5.17.0, engines commit `393aa359…`; client generated from litellm's `schema.prisma`. Full pins in [`litellm-db-freeze.txt`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/litellm-db-freeze.txt) |
| Accounting outage | Mocked | Test-owned TCP gate in front of Redis; a flag file resets existing connections and refuses new ones; Redis keeps its state |
| Metered provider | Mocked (fixture contract) | `run.py --role provider`: stdlib HTTP server and SQLite ledger, OpenAI-compatible `/v1/chat/completions`, operation lookup and async jobs |
| Callers | Scripted | One OS process per concurrent request, all released at one wall-clock instant |
| Supervisor (HB-16) | **Reference implementation** | `supervise()` in `run.py`; not LiteLLM, not any product |
| Host | — | Linux 6.8.0-147-generic, Python 3.12.6 (driver is standard-library only) |

**Fixture provider contract.** All values are fixed:

| Model group | Charge | Gateway price |
|---|---|---|
| `fake-priced` | 1 credit per completion token; completion tokens ≤ `max_tokens` (4) | Same rate |
| `fake-direct` | Same as `fake-priced` | Same rate; no fallback configured |
| `fake-fallback` | 2 credits per token | Same rate; configured as the router fallback for `fake-priced` |
| `fake-unpriced` | 4 credits per accepted request | None configured |

Other rules:

- Requests that are rejected (401) or fail (503) are not charged.
- A request carrying `x-hb-idempotency-key` commits and charges once per key. A replay returns the stored receipt without a new charge.
- Async jobs bill 1 credit per tick of a fixture clock until they are cancelled or reach their authorised `max_ticks` (20 for task-a's jobs, 10 for task-b's). The test owner advances the clock, so counts are deterministic.
- Every task key has a 10-credit cap.

**LiteLLM configuration.** The config is written by `run.py` (`LITELLM_CONFIG`):

- two uvicorn workers;
- `REDIS_HOST`/`REDIS_PORT` pointing at the gate, so both workers share counters;
- `fail_closed_budget_enforcement: true`;
- `num_retries: 0`, `request_timeout: 8`;
- router fallback `fake-priced → fake-fallback`;
- `forward_client_headers_to_llm_api: true`, so the client's `x-` headers reach the provider;
- `LITELLM_LOCAL_MODEL_COST_MAP=True`, so nothing is fetched from the network;
- an isolated temporary `HOME`.

**Configuration arm 1 (no database).** Virtual keys with budgets need LiteLLM's Postgres database, and none was available offline. Keys therefore come from a `custom_auth` function with `custom_auth_run_common_checks: true`. It returns a `UserAPIKeyAuth` with `max_budget=10` and `spend=0`. That puts every request through LiteLLM's normal common checks and its `reserve_budget_for_request` path.

**Configuration arm 2 (database).** The model list, Redis, gate, two workers, timeouts, fallback and `fail_closed_budget_enforcement: true` are the same as arm 1 (`LITELLM_DB_CONFIG` in `run.py`). The differences:

- No `custom_auth`. LiteLLM's own virtual-key auth runs against `DATABASE_URL` and `LITELLM_MASTER_KEY`.
- Keys are created through `POST /key/generate` with `{"key_alias": …, "max_budget": 10}`. LiteLLM stores them in `LiteLLM_VerificationToken`. The Redis counter is `spend:key:<sha256 of the key>`.
- LiteLLM pushes its own schema once into a template database (`litellm --use_prisma_db_push --skip_server_startup`). Each stack gets a fresh `CREATE DATABASE … TEMPLATE`, and the proxy starts with `--use_prisma_db_push`, a no-op on the clone.
- The log-only wrappers load as a callback module (`litellm_settings.callbacks: hb_hooks.handler`), because custom auth would replace the auth under test. A wrapper on `user_api_key_auth_builder` tags each request with its `x-hb-op` header.
- `--timeout_worker_healthcheck 120`. This is worker supervision only. On a host at load average around 60, uvicorn's default 5 s killed every worker while it was still importing (see Method, attempts).
- `HOME` is the isolated directory that holds the Prisma toolchain, not a per-stack directory.
- Defaults are left alone: `PROXY_BATCH_WRITE_AT` is 10 s (plus 0–5 s jitter per worker), and there is no spend-log collector sidecar.

**Observation.** Each source below is separate from what the client or gateway reports:

1. The provider's SQLite ledger (charges, attempts, operations, jobs), read directly by the test owner.
2. LiteLLM's spend counter (`spend:key:<key>`), read directly from its Redis.
3. LiteLLM's own reservation, estimate and settlement calls, recorded by log-only wrappers installed from the custom-auth module (`HB_AUTH` in `run.py`). In arm 2, they come from the callback module (`HB_HOOKS`), which also wraps the router's fallback budget check (`is_token_within_budget_for_model`). The wrappers call the original functions and return their results unchanged. The worker PID is recorded to show that requests reached more than one process.
4. Arm 2 only: `LiteLLM_VerificationToken.spend` and `LiteLLM_SpendLogs`, read with `psql` inside the Postgres container. After each step's last response, the driver samples the Redis counter and the DB spend every 0.1 s for 35 s. Each sample costs one `docker exec`, so real resolution is a few tenths of a second.

## Method

From the repository root, with `redis-server` on `PATH`:

```sh
uv venv /path/to/venv -p 3.12
VIRTUAL_ENV=/path/to/venv uv pip install 'litellm[proxy]==1.106.2'
python3 docs/research/harness-effects-2026-10/runs/budget/run.py --scenario all --litellm /path/to/venv/bin/litellm > results.json
# or one scenario: --scenario HB-13 | HB-14 | HB-15 | HB-16 (HB-16 needs neither LiteLLM nor Redis)
```

**Configuration arm 2.** This needs Docker. The Prisma client is generated once, with an isolated `HOME` that also receives the Prisma toolchain (about 240 MB of npm modules and engines):

```sh
VIRTUAL_ENV=/path/to/venv uv pip install -r docs/research/harness-effects-2026-10/runs/budget/litellm-freeze.txt 'prisma>=0.11.0,<1.0'
mkdir -p /tmp/ra-hb-budget-db/home
(cd /path/to/venv/lib/python3.12/site-packages/litellm/proxy && HOME=/tmp/ra-hb-budget-db/home /path/to/venv/bin/prisma generate --schema schema.prisma)
python3 -I docs/research/harness-effects-2026-10/runs/budget/run.py --arm db --scenario all --litellm /path/to/venv/bin/litellm --prisma-home /tmp/ra-hb-budget-db/home > arm2.json
```

`results.json` is the arm-1 output (the first run, unchanged except for an added `configuration` field) followed by arm-2 final run 3. Final run 4, a rerun of the same revision, is in [`evidence/arm2-rerun.json`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/arm2-rerun.json). Final runs 1 and 2 are in [`evidence/arm2-earlier-runs.json`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/arm2-earlier-runs.json). `--arm db` refuses to start if any Docker container, volume or network named like `ra-hb-budget-db-pg` exists. It removes its container in a `finally` block and drops each stack's database in `Stack.close()`. HB-16 has no gateway and so no arm 2.

Arm 2 changes three procedures. Everything else is identical:

- after each step's last response, a 35 s settlement watch (Observation 4);
- in HB-13-control, the third request is sent after that watch rather than immediately;
- in HB-14-control, the restore loop retries for up to 300 s (arm 1: 90 s), so the record shows when recovery happens, not only whether it beats 90 s.

Arm-2 attempts, all kept:

1. An exploratory HB-14 run failed at startup. Under host load around 55, the proxy's parent spent about 2.5 min on Prisma setup, the 120 s startup wait expired, and a stack that failed to start leaked its processes (they were then removed by process group). `Stack` now closes itself on a failed start, and arm 2 waits up to 600 s.
2. A full run with the 600 s wait was interrupted by SIGINT, which runs the cleanup paths. uvicorn's 5 s worker healthcheck was killing every worker during import at load around 60–80. Arm 2 now passes `--timeout_worker_healthcheck 120`.
3. A full exploratory run (06:42–07:10 UTC, load 36–87) completed with an earlier revision of `run.py`. That revision had no fallback-check wrapper or caller timing, and a 90 s restore window. It differed from the final runs in two rows, kept in [`evidence/arm2-exploratory-run.json`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/arm2-exploratory-run.json):
   - HB-13b: one fallback was refused, so only 8 credits were charged (PASS).
   - HB-14-control: 2 of 3 cycles had not recovered within 90 s.
4. A diagnostic run of two HB-14 cycles with preserved stack directories reproduced the slow recovery. Excerpts of LiteLLM's log are in [`evidence/arm2-hb14-breaker-reopen.txt`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/arm2-hb14-breaker-reopen.txt).
5. Final runs 1 and 2 (07:28–07:50 and 07:50–08:08 UTC, load about 6–28) used the same grading logic as the committed `run.py`. Afterwards, two `observed` templates were reworded: HB-13-control had called an already-settled counter "the reserved bounds", and HB-14-control printed "0 of 0 losing cycles".
6. Final runs 3 and 4 used the committed `run.py`: 08:11–08:29 and 08:30–08:49 UTC, load about 7–16.

After the arm-2 code was added, arm 1 was rerun with it (07:21–07:27 UTC; the arm-1 code path is as committed). Every field matched the committed arm-1 output except HB-14-control, which lost a prior reservation in 1 of 3 cycles (status FAIL). That is the variation the arm-1 record already documents.

Each scenario starts a fresh stack:

- provider;
- Redis;
- gate;
- LiteLLM on free ports in 3100–3300, all bound to `127.0.0.1`.

Each process runs in its own session. `Stack.close()` terminates each process group it started (SIGTERM, then SIGKILL), waits until every group is empty and deletes its temporary directory. If anything remains, the script fails with `Cleanup incomplete` rather than reporting success. Infrastructure errors exit non-zero; boundary FAILs are results and exit 0. Outcomes are deterministic except where the kernel decides which LiteLLM worker accepts a request. Two values can vary between runs: `distinct_workers` in HB-13a, and the number of HB-14 outage cycles that lose a prior reservation (see HB-14-control). Apart from those two values, two consecutive full runs produced identical JSON; their only difference was the per-cycle invalidation counts inside the HB-14-control text.

Procedures, in the order the code runs them:

- **HB-13a.** Three caller processes send to one key at the same instant. Each request has a 4-credit bound and a 1 s provider delay. Afterwards, the same key sends a retry and then a direct request to the fallback group.
- **HB-13b.** On a new key, the primary group is made to fail with an uncharged 503. Two concurrent requests are sent; the router falls back to the 2-credit-per-token group.
- **HB-13c.** A direct provider call is made without the provider credential. Then one unpriced request goes through the gateway.
- **HB-13-control.** Two concurrent requests each have a 4-credit bound and use 2 credits. A third request follows.
- **HB-14a.** Three unpriced requests are sent on one key, then one priced request.
- **HB-14b.** Three outage cycles, each on a fresh stack. In each, a request with a 4 s provider delay holds a reservation; meanwhile the store is made unreachable and two priced requests are sent.
- **HB-14-control.** In each cycle the store is restored. The script waits for the in-flight request, then retries a priced request once a second until it succeeds or 90 s pass. The 90 s window exceeds LiteLLM's default 60 s Redis circuit-breaker recovery time. Expected final counter: 8 (the prior reservation of 4 plus the new reservation of 4).
- **HB-15a.** On `fake-direct` (no fallback), the provider commits and charges, then holds the acknowledgement for 12 s, longer than LiteLLM's 8 s timeout. The script then reads the counter and the provider lookup, and sends two new requests.
- **HB-15b.** The same lost acknowledgement on `fake-priced`, which has a fallback, without an idempotency key.
- **HB-15-control.** A lost acknowledgement with an idempotency key, followed by a provider lookup and a client replay of the same key.
- **HB-16a–c.** A local task process submits a 20-tick job. The fixture clock advances 3 ticks; the supervisor sends SIGTERM to the task's process group; the clock advances 2 more ticks. The supervisor then queries status, requests cancellation and polls while the clock advances. The cancel-response mode is `honor`, `ignore` or `unavailable`.
- **HB-16-control.** A separately owned 10-tick job runs throughout.

## Results

### Configuration arm 1 (no database)

These are the first results, unchanged. Statuses use the scenario pack's definitions, applied to this configuration. "Gateway ledger" compares LiteLLM's counter with the provider's charges. It is UNKNOWN wherever the gateway cannot see a charge the provider made.

| ID | Status | Observed | Provider credits (cap 10) | Gateway ledger |
|---|---|---|---|---|
| HB-13a concurrent admission, retry | **PASS** | 2 of 3 simultaneous callers admitted, 1 refused (`budget_exceeded`, HTTP 422); both reservations written before dispatch. The retry and the direct fallback request were refused. | 8 | covers (8) |
| HB-13b configured fallback | **FAIL** | The admission estimate (4.0 each) covered only the requested group, so both requests were admitted. Both fell back to the 8-credit group. | **16** | UNKNOWN (8 held) |
| HB-13c route coverage | **FAIL** | The direct provider call got 401 and no charge. The unpriced group through the gateway had max-cost estimate `None`, so no reservation was written, and the request was admitted. | 4 | UNKNOWN (0 held) |
| HB-13-control | **NOT EXECUTED** (settlement needs LiteLLM's DB) | Useful work: two requests completed. No settlement to actual cost ran; the counter stayed at the reserved bounds (8). A third request that would fit after reconciliation was refused. | 4 | covers (8) |
| HB-14a unpriced operation | **FAIL** | Three unpriced requests were admitted with no reservation while fail-closed mode was on. A priced request was admitted afterwards. | **16** | UNKNOWN (4 held) |
| HB-14b accounting store unavailable | **PASS** | All 6 requests sent during three outages were refused with HTTP 503 ("Budget enforcement unavailable … fail_closed_budget_enforcement is enabled"). | 0 | — |
| HB-14-control | **FAIL (intermittent)**; `results.json` run: PASS, 0 of 3 cycles | Supported requests always succeeded after restoration, and refused retries were never charged. But the in-flight request's reservation was **lost in 3 of 13 outage cycles** overall: 3 of 7 earlier single-cycle runs of the same procedure, 0 of 6 cycles in the two final runs. Each loss ended with a counter of 4 instead of 8. Two losing outputs, with invalidation logs, are in [`evidence/`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/). | 4 per cycle | covers when kept; 4 short when lost |
| HB-15a timeout settlement | **FAIL** | LiteLLM returned 408 and released the 4-credit reservation to 0 while the provider held a committed 4-credit charge. Two new requests were then admitted. | **12** | UNKNOWN (0 held) |
| HB-15b timeout then fallback | **FAIL** | On timeout, LiteLLM re-dispatched the uncertain request to the fallback group. The client saw HTTP 200. The provider recorded two committed executions of one logical operation. | **12** | UNKNOWN (4 held) |
| HB-15-control | **PASS** | The receiver lookup reported `committed`, 4 credits. The client's replay of the same key returned the stored receipt; there was one charge. | 4 | covers (4) |
| HB-16a cancellation honoured | **PASS** | Cancellation was confirmed by receipt and status. 5 of 20 ticks were billed, 2 of them after the local process had exited. | 5 | supervisor: 5 |
| HB-16b cancellation acknowledged, ignored | **PASS** (boundary as specified) | The cancel request got 202 `cancel_accepted`, but the job kept running to its authorised bound. The supervisor saw this and reported `completed`, 20 billed, 17 of them after local exit. Cancellation axis: not honoured. | 20 | supervisor: 20 |
| HB-16c job service unavailable | **UNKNOWN** | Status and cancel both returned 503. The supervisor reported UNKNOWN for both, rather than "stopped". The observer saw the job bill all 20 ticks, 17 after local exit. | 20 | supervisor: UNKNOWN |
| HB-16-control | **PASS** | task-b's job finished (10 credits attributed to task-b). No cancellation was ever requested on it. | 10 | — |

Axes by scenario:

| Scenario | Boundary | Evidence quality | Legitimate control |
|---|---|---|---|
| HB-13 | FAIL: concurrency held (13a); fallback (13b) and route coverage (13c) did not | Gateway ledger UNKNOWN for 13b and 13c | NOT EXECUTED for reconciliation; useful work completed |
| HB-14 | FAIL on unpriced (14a); PASS on store loss (14b) | Gateway ledger UNKNOWN for 14a | FAIL (intermittent): prior reservation lost in 3 of 13 recoveries |
| HB-15 | FAIL: timeout released committed spend (15a); fallback duplicated the operation (15b) | Gateway ledger UNKNOWN for 15a and 15b | PASS, because of the receiver's idempotency and the client's replay, not the gateway |
| HB-16 (reference supervisor) | PASS in 16a/16b; UNKNOWN in 16c | PASS in all three: no summary inferred remote state from local exit | PASS |

What the run shows:

1. **Admission-time reservation across processes worked.** Concurrent callers on two workers sharing Redis never got more than two 4-credit operations admitted (HB-13a). With the store gone, fail-closed mode refused work instead of guessing (HB-14b).
2. **The reservation covers the requested model group, not the route that serves it.** `_get_request_models` in `budget_reservation.py` returns only the requested group, so a pricier fallback is never reserved. Concurrent fallbacks overspent the cap by 60% (HB-13b). This admission decision does not depend on the database.
3. **Unpriced routes skip reservation even in fail-closed mode.** LiteLLM's source says so directly: when `estimate_request_max_cost` returns `None`, "we fall back to read-time enforcement only". The fail-closed flag governs an unwritable store, not missing prices. Three unpriced calls pushed the provider total to 16 against a 10 cap (HB-14a). Nothing refused the configuration.
4. **A timeout counts as zero spend.** On a terminal failure LiteLLM releases the reservation to zero by design (`release_or_invalidate_budget_reservation`; a cancelled client request reconciles to the input-token cost). When the provider did charge, the cap was exceeded (HB-15a). On a group with a fallback, the same uncertainty became a second execution on another deployment while the client saw success (HB-15b). LiteLLM forwards a client's idempotency key, so a receiver that honours it can deduplicate (HB-15-control). The gateway itself never separates attempts from logical operations and has no way to ask the receiver what happened.
5. **Recovery can erase live reservations.** After the store came back, one worker's Redis circuit breaker could stay open for up to the default 60 s, and that worker refused each reservation. On every refusal, `_reserve_counters` calls `invalidate_spend_counter`, which deletes the *shared* counter key. In 3 of 13 recoveries a delete landed, and the in-flight request's 4-credit reservation vanished; the next admission then counted from the spend floor. Without a database the floor is 0. With one, the floor is settled spend, which also excludes in-flight reservations. Whether this happens depends on which worker the kernel hands the retries to.
6. **Local exit says nothing about remote billing.** In every HB-16 variant the remote job was still running after the local task had exited. Even with cancellation honoured, 2 ticks were billed between local exit and the cancel request. When the service ignored or could not answer cancellation, 17 of 20 ticks came after local exit. Only the authorised `max_ticks` bound limited exposure.

### Configuration arm 2 (database)

These are final run 3. Final runs 1, 2 and 4 gave the same status and the same credit counts in every row (see [Reruns](#reruns)). "DB ledger" compares `LiteLLM_VerificationToken.spend` at the end of the 35 s settlement watch with the provider's charges. Settlement times are measured from the step's last response.

| ID | Status | Observed | Provider credits (cap 10) | DB ledger |
|---|---|---|---|---|
| HB-13a concurrent admission, retry | **PASS** | 2 of 3 simultaneous callers admitted, 1 refused (HTTP 422 `budget_exceeded`), across 2 workers. The retry and the direct fallback request were refused. The Redis counter already held 8; DB spend reached 8 at +2.7 s. | 8 | matches (8) |
| HB-13b configured fallback | **FAIL** | Both requests were admitted on 4-credit reservations. LiteLLM's fallback budget check allowed both fallbacks, at +0.15 s and +0.75 s. The first fallback's settlement to 8 credits landed at +1.22 s, after both checks. Both requests were billed at the fallback rate. Counter and DB spend settled to 16. | **16** | matches (16) |
| HB-13c route coverage | **FAIL** | The direct provider call got 401 and no charge. The unpriced group was admitted with estimate `None` and no reservation, and recorded as a success row with spend 0. | 4 | UNKNOWN (0 recorded) |
| HB-13-control | **PASS** | Two requests used 2 credits each against 4-credit bounds. Each reservation was settled to 2. The Redis counter already read 4, not the reserved 8, when the driver read it after the last response. DB spend reached 4 at +9.5 s. A third request that fits only after reconciliation was admitted. Final counter and DB spend were 6. | 6 | matches (6) |
| HB-14a unpriced operation | **FAIL** | Three unpriced requests were admitted with no reservation, each recorded at spend 0. A priced request afterwards was admitted. | **16** | UNKNOWN (4 recorded) |
| HB-14b accounting store unavailable | **PASS** | All 6 requests sent during three Redis outages were refused with HTTP 503 ("the budget reservation could not be written to the spend counter backend"). Postgres stayed up throughout. | 0 | — |
| HB-14-control | **PASS** | All 3 cycles recovered and kept the prior reservation: the counter was 8 right after the successful retry. Two cycles took 120.6 and 117.8 s, because each worker's breaker reopened once on a failed recovery probe. They invalidated the counter 114 and 112 times, and each deleted counter was reseeded from DB spend, which already held the in-flight request's 4 credits. The third cycle recovered in 1.1 s with 3 invalidations. | 4 per cycle | matches (8 per cycle) |
| HB-15a timeout settlement | **FAIL** | LiteLLM returned 408 and released the reservation to 0. It wrote a failure row with spend 0 while the provider held a committed 4-credit charge. Two new requests were admitted. | **12** | UNKNOWN (8 recorded) |
| HB-15b timeout then fallback | **FAIL** | On timeout LiteLLM re-dispatched to the fallback group. The client saw HTTP 200, and there were two committed executions of one operation. LiteLLM recorded one success row for the fallback (8). | **12** | UNKNOWN (8 recorded) |
| HB-15-control | **PASS** | The receiver lookup reported `committed`, 4 credits. The replay returned the stored receipt, and there was one charge. LiteLLM recorded a failure row (0) and the replay (4). | 4 | matches (4) |

### Both arms, graded separately

| Scenario | Axis | Arm 1 (no database) | Arm 2 (database) |
|---|---|---|---|
| HB-13 | Boundary | FAIL: fallback (13b) and route coverage (13c) | FAIL: the same two cases. 13b charged 16 in all four final runs. |
| HB-13 | Evidence quality | Gateway ledger UNKNOWN for 13b and 13c | 13b settled: counter and DB match the provider. 13c still UNKNOWN (recorded at 0). |
| HB-13 | Legitimate control | NOT EXECUTED (no settlement path) | **PASS**: reservations reconciled to actual cost, and a request that fits only afterwards was admitted |
| HB-14 | Boundary | FAIL on unpriced (14a); PASS on store loss (14b) | Same |
| HB-14 | Evidence quality | Gateway ledger UNKNOWN for 14a | UNKNOWN for 14a: the DB records unpriced calls at 0 |
| HB-14 | Legitimate control | FAIL (intermittent): prior reservation lost in 3 of 13 recoveries, and in 1 of 3 on the recheck run | **PASS**: 0 of 12 recoveries across four final runs lost it. Recovery took 116.9–120.6 s in 10 of 12 cycles. |
| HB-15 | Boundary | FAIL: timeout released committed spend (15a); fallback duplicated (15b) | Same |
| HB-15 | Evidence quality | Gateway ledger UNKNOWN for 15a and 15b | UNKNOWN: the DB records 8 of 12 in both |
| HB-15 | Legitimate control | PASS, through the receiver's idempotency | Same |

### What the database changes

All citations are to the pinned wheel, relative to `site-packages/`, by line in litellm 1.106.2.

1. **Settlement runs only with a database (HB-13-control: NOT EXECUTED → PASS).**
   - `cost_tracking()` in `litellm/proxy/proxy_server.py:2957-2964` registers `ProxyDBLogger` only when `prisma_client is not None`.
   - Its success path starts at `_PROXY_track_cost_callback` (`litellm/proxy/hooks/proxy_track_cost_callback.py:426`). It calls `_update_database_and_spend_counters` (`:535`, defined at `:871`), which calls `_reconcile_budget_reservation_before_db_update` (`:897`).
   - That reconciles each reservation to the response cost: `reconcile_budget_reservation` (`litellm/proxy/spend_tracking/budget_reservation.py:356`) and `_set_reserved_entries_actual_cost` (`:1140`). The adjustment is written in one pipeline with the key's increment (`_increment_spend_counters_batched`, `proxy_server.py:3304-3319`).
   - The DB row is written by a batch job every `proxy_batch_write_at + random.randint(0, 5)` seconds per worker (`proxy_server.py:10936`; `PROXY_BATCH_WRITE_AT` defaults to 10, `litellm/constants.py:1877`).
2. **A deleted counter is reseeded from recorded spend (HB-14-control: FAIL → PASS).**
   - Refused reservations still delete the shared counter (`budget_reservation.py:983`, `:1084`): 109–115 times in a slow cycle.
   - The next reservation finds the counter missing and seeds it from the database. The path is `_initialize_reservation_counter` (`budget_reservation.py:947`), then `ensure_spend_counter_initialized` (`proxy_server.py:3830-3837`), then `SpendCounterReseed.coalesced` (`litellm/proxy/db/spend_counter_reseed.py:223`). That function maps `spend:key:{token}` to `LiteLLM_VerificationToken.spend` (line 81).
   - In fail-closed mode, `get_current_spend` also re-reads that floor before admitting and raises a stale-low counter (`proxy_server.py:3003`, `3056-3067`; the floor is cached for 5 s, `:2977`). Arm 1's floor was custom auth's `spend=0`.
   - The floor still excludes in-flight reservations until they settle. In the 10 slow cycles, DB spend already held the in-flight request's 4 credits when the retry succeeded. In the 2 fast cycles (1.1 s and 2.2 s), DB spend was still 0, and the counter kept 8 because no deletion landed after the restore. The database narrows the loss window; it does not close it. A deletion between restoration and the in-flight request's DB write would still lose the reservation. That was not observed in 12 cycles.
3. **Recovery is slower (a behaviour change, not a grade change).**
   - Arm 2 took 116.9–120.6 s to recover in 10 of 12 cycles across the four final runs. Every arm-1 cycle recovered within 90 s: 13 earlier cycles and 3 on the recheck.
   - In LiteLLM's own log, each worker's breaker reopened 60 s after it opened. The half-open probe failed with "Connection closed by server", on a pooled connection the outage had closed, and HALF_OPEN → OPEN resets the 60 s timer (`litellm/caching/redis_cache.py:224-226`). Excerpts: [`evidence/arm2-hb14-breaker-reopen.txt`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/arm2-hb14-breaker-reopen.txt).
   - Why the probe found a stale connection in arm 2 but not in arm 1 is **not established**. One candidate: with a database, LiteLLM starts its scheduled background jobs (`proxy_server.py:1668-1678`). These include an auth-cache invalidation subscriber on the same Redis (`:11062`), which reconnects every 5 s.
4. **Settlement can make the fallback check refuse, but only on timing (HB-13b stays FAIL).**
   - Settlement prices a fallback at the deployment that served it, so the counter rises to 8 per request. LiteLLM's fallback budget check (`litellm/proxy/auth/fallback_budget.py:106-123`) can then refuse a later fallback.
   - That check reads the counter and does not reserve (`:24` says so). In all four final runs, both checks ran before the first settlement. The check times, against the first settlement, were:
     - +0.14 s and +1.12 s against +1.21 s;
     - +0.12 s and +0.55 s against +1.17 s;
     - +0.15 s and +0.75 s against +1.22 s;
     - +0.09 s and +0.11 s against +1.19 s.
     Both fallbacks were billed each time.
   - In the exploratory run, the second fallback was refused and 8 credits were charged. The outcome depends on whether a settlement lands between two fallback checks.
5. **What the database does not change.**
   - Reservations still estimate only the requested group (`_get_request_models`, `budget_reservation.py:1600-1609`).
   - Unpriced requests still skip reservation (`:305-309`) and are recorded at spend 0.
   - A timed-out request is still released to zero by the failure hook (`async_post_call_failure_hook`, `proxy_track_cost_callback.py:264-272`), with a failure row at spend 0.
   - HB-14b still refuses. With Postgres readable, spend can be verified, so the refusal now comes from the failed reservation write (`_raise_reservation_unavailable`, `budget_reservation.py:90`). In arm 1 it came from unverifiable spend (`_raise_budget_unverifiable`, `proxy_server.py:3079`).

### HB-13 settlement

- **Settled to the provider's charge?** Yes, for every request that returned a response: 8 of 8 (HB-13a), 16 of 16 at the fallback price (HB-13b), 6 of 6 (HB-13-control) and 4 of 4 (HB-15-control).
- **Settled when?** In all 28 settlement observations across the four final runs (7 scenarios with a priced request, 4 runs):
  - the Redis counter held the settled value within +0.0–0.2 s of the last response (the limit of the sampling);
  - `LiteLLM_VerificationToken.spend` reached it at +1.3 to +19.9 s, consistent with the batch writer on two workers on a loaded host.
- **A timed-out request?** Not settled to the charge. In HB-15a, LiteLLM released the reservation at the timeout and recorded a failure row with spend 0. The key's recorded spend ended at 8 while the provider charged 12. In HB-15b, only the fallback execution was recorded (8 of 12); the first, committed execution does not appear in LiteLLM's database.
- **Grade under the pack's HB-13 conditions:**
  - Legitimate control **PASS**: two permitted requests completed, and unused reservation was reconciled under the stated contract.
  - Boundary **FAIL**: fallbacks exceeded the cap (13b, 16 credits in all four final runs), and a reachable billable route bypassed reservation (13c).

### Reruns

All four final arm-2 runs agree on every status, and on every credit, counter and HTTP-status value. Run 4 reran the committed revision; it is in [`evidence/arm2-rerun.json`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/arm2-rerun.json). These fields vary:

- `db_settled_at_s` and `settlement_trace` in every row with a watch. DB settlement ranged from +1.3 to +19.9 s; the HB-13c watch is always 0.
- the times inside HB-13b's `request_timing` and `fallback_checks`.
- `cycle_detail` in HB-14-control:

  | Final run | Recovery (s) | Breaker re-openings | Invalidations | DB spend at restore |
  |---|---|---|---|---|
  | 1 | 119.4, 117.4, 117.9 | 2, 2, 2 | 109, 110, 110 | 4, 4, 4 |
  | 2 | 116.9, 2.2, 119.9 | 2, 0, 2 | 111, 4, 115 | 4, 0, 4 |
  | 3 | 120.6, 117.8, 1.1 | 2, 2, 0 | 114, 112, 3 | 4, 4, 0 |
  | 4 | 117.9, 117.9, 117.4 | 2, 2, 2 | 114, 114, 113 | 4, 4, 4 |

- the `observed` text, only where it quotes those values.

The exploratory run with the earlier revision differed in status in two rows (Method, attempt 3). HB-13b and the HB-14 recovery time are therefore timing-dependent. Read them as observed outcomes in these runs, not as constants.

## Limits

- **Arm 1 has no database**, so its settlement-dependent results (HB-13-control NOT EXECUTED, HB-14-control's spend floor of 0) say nothing about a DB-backed deployment. Arm 2 is that deployment in one configuration: Postgres 16, Redis counters, two uvicorn workers and `fail_closed_budget_enforcement: true`, with LiteLLM's default batch-write interval and no spend-log collector. Other configurations were not tested: a single worker without Redis, several pods, `use_redis_transaction_buffer`, a read replica, PgBouncer, or the collector sidecar.
- Arm 2 procedures differ from arm 1 in three places (Method). In HB-13-control, a third request sent before settlement would still see the reserved bound.
- Only the Redis store was made unavailable. A **database outage** was not tested, nor a slow DB, Redis and DB failing together, or a Redis restart, which with a database would reseed counters from recorded spend.
- The settlement watch lasts 35 s after a step's last response. Writes after that were not observed. Recorded spend was stable when each watch ended.
- The cause of arm 2's slow recovery (a breaker re-opened on a stale pooled connection) is identified from LiteLLM's log. Why it affects arm 2 and not arm 1 is not established.
- HB-13b's outcome depends on whether a settlement lands between two fallback checks. Four final runs charged 16; one exploratory run charged 8. Five runs do not give a rate.
- Host load varied from about 6 to 87 across attempts, and from about 6 to 28 during the final runs (other workloads shared the box). That affects every timing in arm 2 and may have affected which worker served what.
- `prisma generate` writes the generated client into the installed `prisma` package. LiteLLM's package files are unmodified. The instrumentation replaces module attributes inside the gateway process. It records calls and returns results unchanged, but it is in-process evidence. The provider ledger, direct Redis reads and database reads are the independent observations.
- The provider is a fixture. Its declared token bound is what makes the 4-credit reservation sufficient, and its charge equals the configured price times tokens, so settlement "matching" the provider is a property of this fixture. A real provider can:
  - charge outside any declared bound (HB-14's unknown condition);
  - bill failed attempts;
  - price differently from the gateway's table;
  - keep separate idempotency spaces per deployment.
  In HB-15b and HB-15-control, both model groups reach one receiver, which is why a forwarded key could deduplicate.
- One host, loopback only. The run does not force which worker serves which request. Neither arm tests:
  - the default fail-open setting;
  - a partitioned or slow store;
  - streaming responses;
  - batch, file or pass-through endpoints;
  - TTL expiry;
  - more than two workers.
- HB-16's subject is a short reference supervisor and a fixture job service with a test-owned clock. It shows the reconciliation pattern and the size of the gap. It is not evidence about any scheduler, runtime or LiteLLM. Its job ownership is by exact job ID chosen by the submitter.
- The seven earlier HB-14 runs in arm 1 used a pre-commit revision of `run.py`, with one outage cycle per invocation. Two of them used a 30 s retry window, and only the last three logged invalidations. Only two losing outputs were preserved, in [`evidence/`](https://github.com/carrotly-ai/rethinkingagents-com/blob/main/docs/research/harness-effects-2026-10/runs/budget/evidence/); the third loss is reported from the session record. The committed `run.py` runs three cycles per invocation and may report PASS when no cycle happens to lose its reservation.
- Whether LiteLLM's documentation presents `max_budget` as a hard cap was not assessed. The FAIL statuses use the spec's fail clauses for a configured 10-credit cap under fail-closed enforcement.

## What should follow

1. Done for one configuration (arm 2). Next, vary what arm 2 held fixed: a database outage, a single worker without Redis, and LiteLLM's Redis transaction buffer. Repeat HB-13b and HB-14-control enough times to state rates rather than single outcomes.
2. Report these upstream as configuration questions, without attaching exploit material:
   - Should reservation estimate across configured fallbacks? The fallback check reads but does not reserve, and the database does not fix that.
   - Should fail-closed mode refuse unpriced routes?
   - Should a timed-out upstream call keep its reservation until reconciled, rather than being recorded at spend 0?
   - Should a failed reservation delete a shared counter that holds other requests' reservations?
   - Why does a half-open breaker probe reuse a connection the outage closed, keeping fail-closed refusals going for two minutes after Redis is back?
3. Teams that rely on a gateway cap can reduce these specific exposures with configuration: price every model group, use fallbacks no pricier than their primary, and set provider-side spend limits. With a database, treat recorded spend after timeouts as a lower bound. With or without one, expect fail-closed refusals to outlast a Redis outage by up to the breaker's recovery time, or a multiple of it.
4. For HB-16, test a real job API that offers cancellation and status (for example, a batch endpoint behind the same gateway) to see whether the gateway or supervisor records remote exposure after local exit.

## Change log

- 2026-10-23: HB-13–HB-16 executed. Added `run.py`, `results.json`, `litellm-freeze.txt`, `evidence/` and this record.
- 2026-10-23: Added configuration arm 2 for HB-13–HB-15: LiteLLM 1.106.2 with Postgres 16, its Prisma schema, a master key and `/key/generate` virtual keys. Arm-1 results are unchanged. Each arm-1 entry in `results.json` gained a `configuration` field, and arm-2 entries were appended. Added `run.py --arm db`, `litellm-db-freeze.txt`, and `evidence/arm2-rerun.json`, `arm2-earlier-runs.json`, `arm2-exploratory-run.json` and `arm2-hb14-breaker-reopen.txt`. Rewrote Limits and What should follow. Arm 2 changes two control grades: HB-13 NOT EXECUTED → PASS and HB-14 FAIL → PASS. No enforcement grade changes.