182 lines
11 KiB
Markdown
182 lines
11 KiB
Markdown
# Usage monitoring, gate & throttle
|
||
|
||
> **Explore-note — verify before trusting.** Distilled map of a subsystem, not authoritative.
|
||
> Last verified against commit `f6cb825` (2026-08-05), plus the uncommitted per-bucket-throttle /
|
||
> draggable-gauge change of 2026-08-06 (this note already describes that newer state).
|
||
> Drift check: `git log --oneline f6cb825..HEAD -- src/ClaudeDo.Worker/Usage src/ClaudeDo.Worker/Queue src/ClaudeDo.Ui/ViewModels/UsagePillViewModel.cs`
|
||
> Stable structure only (no line numbers). See docs/explore-notes/README.md.
|
||
|
||
Covers `src/ClaudeDo.Worker/Usage/`, the queue's throttle/gate integration, per-run token
|
||
accounting, and the UI surfaces (usage pill + usage monitor modal).
|
||
|
||
## Data source
|
||
|
||
`GET https://api.anthropic.com/api/oauth/usage` — an **undocumented** Anthropic endpoint,
|
||
authenticated with the Bearer access token Claude Code itself keeps fresh at
|
||
`~/.claude/.credentials.json`. ClaudeDo reads that token, never refreshes it, never logs it.
|
||
|
||
Because the endpoint is undocumented and can change without notice, **every consumer
|
||
fails open**. That is the single most important invariant here.
|
||
|
||
## Components (`Usage/`)
|
||
|
||
| Type | Role |
|
||
|---|---|
|
||
| `UsageModels` | `UsageBucket` / `UsageLimitRow` / `UsageSnapshot`. `UsageBucket.Utilization` is already a 0–100 percent — compare directly against thresholds, don't rescale. |
|
||
| `ClaudeOAuthUsageClient` | Reads the token, calls the endpoint. Defensive parsing: missing/null buckets → null, missing `limits` → empty list. |
|
||
| `UsageState` | Threadsafe singleton. A failed poll **never** overwrites the last good snapshot — it only sets `LastError`. |
|
||
| `UsageMonitorService` | `BackgroundService`; polls on `usage_poll_interval_seconds` (default 60, clamped to min 15 on config load), one poll at startup. Logs a failure at most once per distinct error message. Broadcasts `HubBroadcaster.UsageUpdated` after **every** tick, success or failure. |
|
||
| `UsageSnapshotBuilder` | Builds the Hub-facing `UsageSnapshotDto` from `UsageState` + `IUsageGate` + `AppSettings`. The one shared place for stale/threshold/gate logic — `WorkerHub.GetUsageSnapshot` and `UsageMonitorService` must not diverge. |
|
||
| `UsageGate` | Hard pause decision → `UsageGateDecision(IsBlocked, Reason)`. |
|
||
| `UsageThrottle` | Pure static staging of parallelism ahead of the gate. |
|
||
| `TranscriptUsageReader` | Aggregates token usage from Claude Code transcripts. |
|
||
|
||
Interfaces in `Usage/Interfaces/`: `IUsageClient`, `ITranscriptUsageReader`, `IUsageGate`.
|
||
|
||
## The gate (hard pause)
|
||
|
||
Thresholds: `AppSettings.UsageGateFiveHourPct` / `UsageGateSevenDayPct` (defaults 80/90).
|
||
Blocked once `five_hour >= UsageGateFiveHourPct` **or** `seven_day >= UsageGateSevenDayPct`
|
||
(`>=`, not `>`). Threshold `0` = that bucket never gates.
|
||
|
||
What it pauses: **only the queue's slot-fill loop** — new queued tasks don't start.
|
||
Unaffected: already-running runs, `RunNow`, `ContinueTask`, interactive ConPTY sessions,
|
||
planning sessions, daily prep (all bypass the queue).
|
||
|
||
**Fail-open**: no snapshot yet, a failed last poll, or an app-settings read error all
|
||
resolve to not-blocked.
|
||
|
||
There is **no persistent pause state**. Recovery is just the queue's 30 s backstop timer
|
||
(`queue_backstop_interval_ms`) re-evaluating the gate on its own once usage drops back
|
||
under the threshold. A blocked↔free transition is logged and broadcast (`WorkerLog`, Warn
|
||
on block / Info on resume) exactly **once per change**, not every tick.
|
||
|
||
## The throttle (staged parallelism)
|
||
|
||
`UsageThrottle.EffectiveSlots(configuredSlots, fiveHourPct, fiveHourThresholds, sevenDayPct,
|
||
sevenDayThresholds)` — pure static, no state. `UsageThresholds(SoftPct, HardPct, GatePct)` is the
|
||
per-bucket triple (same file).
|
||
|
||
Thresholds are **per bucket** (`usage_throttle_five_hour_{soft,hard}_pct` /
|
||
`usage_throttle_seven_day_{soft,hard}_pct`, defaults 50/65 each) because the 5h and 7d windows fill
|
||
at very different rates. Each bucket is staged independently and the **strictest** bucket wins —
|
||
not "whichever is more utilized", so a bucket that is lower but tightly configured can be the one
|
||
that throttles:
|
||
|
||
| Utilization (per bucket) | That bucket's slots |
|
||
|---|---|
|
||
| below soft | full configured `max_parallel_executions` |
|
||
| `>= softPct` | capped at 2 |
|
||
| `>= hardPct` | capped at 1 |
|
||
| `>= gatePct` | 0 — same hard block as `UsageGate` |
|
||
|
||
A threshold of `0` disables that stage for that bucket, and a bucket with no reading (null) never
|
||
throttles. The `0` return is deliberately kept in sync with `UsageGate`'s hard block because both
|
||
read the same gate thresholds — change one, change both.
|
||
|
||
Only **new** slot fills are affected; a run already occupying a slot when the stage tightens
|
||
runs to completion. Same fail-open policy: no snapshot means no throttling.
|
||
|
||
The effective stage (configured vs. effective slots + the decisive bucket, `ThrottleBucket`
|
||
= `"five_hour"` / `"seven_day"`) rides along on `UsageSnapshotDto` purely for UI display.
|
||
It does **not** change what the gate gates on.
|
||
|
||
## Queue integration (`Queue/QueueService`)
|
||
|
||
Per loop tick:
|
||
|
||
1. `GetEffectiveMaxParallelAsync` reads `AppSettings.MaxParallelExecutions` and steps it
|
||
down via `UsageThrottle.EffectiveSlots` against the current `UsageState` snapshot.
|
||
A missing/failed snapshot fails open to the configured value. A **stage change** (not
|
||
every tick) logs once.
|
||
2. Separately, `IUsageGate.EvaluateAsync` — if blocked, the slot-fill loop is skipped
|
||
entirely for that tick.
|
||
|
||
## Per-run token accounting
|
||
|
||
`task_runs` stores four raw token fields: `tokens_in` / `tokens_out` /
|
||
`cache_read_tokens` / `cache_write_tokens`.
|
||
|
||
These are **not** read from the stream-json `result` event's `usage.input_tokens` — that is
|
||
only the uncached remainder of a single API call and undercounts the real prompt size by
|
||
orders of magnitude once caching kicks in.
|
||
|
||
Instead `TaskRunner.ApplyUsageAsync` calls
|
||
`ITranscriptUsageReader.ReadSessionTotalsAsync(sessionId)` — the session transcript's
|
||
cumulative raw totals across every assistant message, located by `{sessionId}.jsonl` — and
|
||
stores the **delta** against prior `task_runs` rows sharing the same `session_id`, so a
|
||
`--resume`'d run doesn't double-count turns already billed to an earlier run.
|
||
|
||
A missing/unreadable transcript leaves all four fields `null`; it never fails the run.
|
||
|
||
### `TranscriptUsageReader` details
|
||
|
||
Reads `~/.claude/projects/**/*.jsonl`, aggregating by date / model / scope (ClaudeDo vs
|
||
Other), deduped by `requestId`, with a per-file length+mtime cache.
|
||
|
||
`ReadAsync` **skips any file whose mtime predates the window start minus one day** — it cannot hold
|
||
a record inside the range, and the full history is large (measured 2026-08-06: 501 files / 230 MB /
|
||
77k lines ≈ 1.7 s to parse cold; a 7-day range touches ~190 files / ~106 MB). The one-day slack
|
||
absorbs local-vs-UTC skew between mtime and record timestamps. `ReadSessionTotalsAsync` is
|
||
unaffected — it looks up a single `{sessionId}.jsonl`.
|
||
|
||
`<synthetic>`-model lines are skipped **everywhere** — they are not real API calls.
|
||
|
||
## UI surfaces
|
||
|
||
- **`UsagePillViewModel`** — one shared instance backs the `UsagePill` control in both the
|
||
footer and the Mission Control header. Loads via `GetUsageSnapshotAsync`, updates live off
|
||
`IWorkerClient.UsageUpdatedEvent`. Dot state priority is mutually exclusive:
|
||
**blocked > stale > warn > normal**. `IsThrottled` (effective slots below configured, and
|
||
not gate-blocked) adds a tooltip line naming effective/configured slots + decisive bucket.
|
||
The pill's click handler (`IslandsShellViewModel.OpenUsageMonitor`) **shows the window before
|
||
loading** (`BeginLoad`) — awaiting the load first made the pill feel like a dead click, because
|
||
the first `GetModelUsage` per worker process scans the whole transcript history.
|
||
- **Draggable stage markers** — each of the two real gauges carries three markers (soft/hard/gate).
|
||
`UsageGaugeBar` (`Views/Controls`) draws them against its own width and does the pointer work;
|
||
the math is a pure static, `UsageThresholdDrag` (in the modal VM's file), which keeps
|
||
soft ≤ hard ≤ gate and treats a neighbour of `0` as off. Release fires the row's
|
||
`CommitCommand` → read-modify-write via `GetAppSettings` + `UpdateAppSettings`, so only the
|
||
dragged bucket's three fields change. Plan-dependent `weekly_scoped` gauges are read-only.
|
||
- **Legend = numeric editor.** Under each adjustable bar sit three legend rows whose colour swatches
|
||
match the markers (soft `TextDimBrush`, hard `StatusReviewBrush`, gate `StatusErrorBrush`), each
|
||
with a `NumericUpDown`. `NumericUpDown` has no commit command, so the box's `Tag`
|
||
(`soft`/`hard`/`gate`) plus two code-behind handlers (`LostFocus`, Enter) call the row's
|
||
`CommitSoft`/`CommitHard`/`CommitGate` command. Those run the typed value through the **same**
|
||
`UsageThresholdDrag.Apply` clamp as a drag, so a box can't invert the order and only the edited
|
||
stage moves. ⚠️ The `KeepLastNumber` converter is mandatory on those bindings — see the
|
||
`NumericUpDown` null gotcha in `src/ClaudeDo.Ui/CLAUDE.md`.
|
||
Rows are updated **in place** on each snapshot (keyed by limit kind) so a poll landing mid-drag
|
||
doesn't replace the bound instance.
|
||
- **`UsageMonitorModalViewModel`** — opened from the pill. Renders one gauge **per row** in
|
||
`UsageSnapshotDto.Limits` — deliberately **dynamic**, because the `seven_day_opus` /
|
||
`seven_day_sonnet`-style buckets the raw API returns are plan-dependent and come back
|
||
`null` on plans that don't have them; a fixed gauge layout would break. Also shows model
|
||
usage (`GetModelUsageAsync`, ClaudeDo-vs-Other split per model) and top-task usage
|
||
(`GetTaskUsageAsync`) over a 7d/30d preset or custom range.
|
||
|
||
## Hub surface
|
||
|
||
- `GetUsageSnapshot() -> UsageSnapshotDto` — percentages/limits/`FetchedAtUtc` are null and
|
||
`IsStale=true` when no snapshot has landed yet. `IsStale` also trips on a failed last poll
|
||
or a snapshot older than 3× `usage_poll_interval_seconds`.
|
||
- `GetModelUsage(from, to)` — thin wrapper over `ITranscriptUsageReader.ReadAsync`.
|
||
- `GetTaskUsage(from, to)` — top consumers from `task_runs` joined to task/list, grouped per
|
||
task. Null token columns count as **0**, never drop the row. `Model` comes from that task's
|
||
most recent run. Sorted by total tokens descending, capped at 100.
|
||
- `UsageUpdated` event carries the same `UsageSnapshotDto`.
|
||
|
||
## Settings columns
|
||
|
||
`app_settings`: `usage_gate_five_hour_pct` / `usage_gate_seven_day_pct` (80/90),
|
||
`usage_throttle_five_hour_{soft,hard}_pct` / `usage_throttle_seven_day_{soft,hard}_pct` (50/65 per
|
||
bucket). All six clamped 0..100 by `AppSettingsRepository.UpdateAsync`, which does **not** enforce
|
||
soft ≤ hard ≤ gate — the ordering is a UI-side drag constraint, and an out-of-order stored config
|
||
degrades instead of throwing. Worker config: `usage_poll_interval_active_seconds` /
|
||
`usage_poll_interval_idle_seconds`.
|
||
|
||
The gate percentages are editable in **two** places that both write the same `app_settings` row:
|
||
Settings → General (typed) and the usage-monitor gauges (dragged). The throttle stages are
|
||
gauge-only — `SettingsModalViewModel` therefore carries them load→save verbatim so saving Settings
|
||
can't reset a dragged value.
|