Merge branch 'claudedo/f359858ac98a439593e459df9c5d0a5d'
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
# ConPTY interactive sessions & launch specs
|
||||
|
||||
> **Explore-note — verify before trusting.** Distilled map of a subsystem, not authoritative.
|
||||
> Last verified against commit `f6cb825` (2026-08-05).
|
||||
> Drift check: `git log --oneline f6cb825..HEAD -- src/ClaudeDo.Worker/Planning src/ClaudeDo.Worker/Hub src/ClaudeDo.Ui/ViewModels/MissionControlViewModel.cs src/ClaudeDo.Ui/Views/InteractiveTerminalView.axaml`
|
||||
> Stable structure only (no line numbers). See docs/explore-notes/README.md.
|
||||
|
||||
Covers `InteractiveLaunchSpecService` and the four kinds of embedded ConPTY session the UI
|
||||
process hosts (real `claude` TUI in a Mission Control tile).
|
||||
|
||||
Autonomous queue tasks are **not** covered here — they stay on the stream-json path via
|
||||
`TaskRunner`. See [worker-task-pipeline.md](worker-task-pipeline.md).
|
||||
|
||||
## The four session kinds
|
||||
|
||||
| Kind | Hub spec method | Notes |
|
||||
|---|---|---|
|
||||
| Task session | `GetInteractiveLaunchSpec` | Effort from the task/list model preset |
|
||||
| Ad-hoc | `GetAdHocLaunchSpec` | Effort from the global default |
|
||||
| Planning | (planning start/resume) | Effort from `PlanningAlias`; uses `--permission-mode default`, **not** `plan` |
|
||||
| List handler | `GetMergeHelperLaunchSpec` | Effort from list config; `--permission-mode auto` (unattended) |
|
||||
|
||||
## ⚠️ Gotcha: never pass task free-text as a CLI argument
|
||||
|
||||
**No ConPTY path ever passes task free-text (title / description / brief) as a CLI argument.**
|
||||
Every one of them writes it to a file first and hands `claude` a single-line kickoff pointing at
|
||||
that file, exposed via `--add-dir`.
|
||||
|
||||
Two independent reasons:
|
||||
|
||||
1. The ConPTY host flattens `Args` into **one command line** to spawn the process, and `claude`
|
||||
re-splits that line on whitespace. Any token starting with `-` in real task text (e.g. `->`,
|
||||
`--abort`) is then misread as an unknown option.
|
||||
2. A raw multi-line positional prompt truncates at its **first newline** regardless.
|
||||
|
||||
A fresh task session's brief lives at `~/.todo-app/task-sessions/<taskId>/brief.md`
|
||||
(`InteractiveLaunchSpecService.BuildFreshTaskArgsAsync`). A task with neither title nor
|
||||
description skips the file **and** the positional arg entirely.
|
||||
|
||||
## Argument ordering
|
||||
|
||||
Every spec passes `--effort <level>` from the relevant model's preset. It **leads** the args —
|
||||
except for a fresh task session with a brief, where `--add-dir <sessionDir>` must come first so
|
||||
`--effort` (a single-value flag) can sit directly before the positional kickoff.
|
||||
|
||||
`--model` is deliberately **NOT** forced on an interactive session — the user can still switch
|
||||
models in the TUI.
|
||||
|
||||
## List handler ("Let Claude handle it")
|
||||
|
||||
`BuildForMergeHelperAsync` uses `--permission-mode auto` so it runs unattended. The
|
||||
`--allowedTools` allowlist is the security boundary:
|
||||
`mcp__claudedo__*,Read,Grep,Glob,Edit,Bash,WebFetch,WebSearch,Skill`.
|
||||
|
||||
`MCP_TOOL_TIMEOUT` is 200 s here — `TaskWaitMcpTools` clamps its own timeout to 170 s to stay
|
||||
comfortably under it (see [external-mcp.md](external-mcp.md)).
|
||||
|
||||
### The host task and its commit range
|
||||
|
||||
The handler run **owns a real ClaudeDo task**, created by `CreateMergeHelperTask` (hub) →
|
||||
`InteractiveLaunchSpecService.CreateMergeHelperTaskAsync`, called by the UI *before* it opens
|
||||
the tile:
|
||||
|
||||
- One new task per run in that list, `Idle` + `IsManual=true` (never queued).
|
||||
- Title/description localized via `missionControl.mergeHelperTaskTitle` /
|
||||
`mergeHelperTaskDescriptionHeader`.
|
||||
- `TaskEntity.HandlerBaseCommit` stamped to the list repo's current HEAD.
|
||||
|
||||
The host task **never gets a worktree of its own** — the handler commits straight into the
|
||||
list's working dir and merges the tasks it handles itself. Consequences:
|
||||
|
||||
- `SubmitTaskForReview` branches on whether the task has a `WorktreeEntity`: with one, it
|
||||
commits the worktree; without one, it stamps `HandlerHeadCommit` to the list repo's current
|
||||
HEAD. Both paths then flip the task `Idle`/`Failed` → `WaitingForReview`.
|
||||
- `GetTaskDiff` and the UI's `DetailsIslandViewModel` / `MergeSectionViewModel` fall back to the
|
||||
`HandlerBaseCommit`..`HandlerHeadCommit` range whenever `Worktree` is null.
|
||||
|
||||
### UI flow
|
||||
|
||||
`MergeHelperSelectionModalViewModel` — checkbox picker over one list's non-terminal, non-manual
|
||||
tasks, pre-ticking the actionable ones. **List-scoped only** (`Configure(listId, listName)`, no
|
||||
global scope). Opened from the list row's context menu, which is hidden when the list has no
|
||||
working dir.
|
||||
|
||||
On confirm: `ListsIslandViewModel` raises `LetClaudeHandleRequested` → shell →
|
||||
`MissionControlViewModel.OpenMergeHelperConPtySessionAsync`, which calls
|
||||
`CreateMergeHelperTaskAsync` and then opens a **task-based** tile (deduped by `TaskId` like
|
||||
`OpenConPtySessionAsync`, **not** `CreateAdHoc`) running the five-phase handler prompt.
|
||||
|
||||
## Tile lifecycle
|
||||
|
||||
`ConPtyPaneViewModel` resolves its own launch spec — the ctor takes a descriptor **factory**,
|
||||
the host wires handlers and then calls `Start()`. So the tile appears **immediately** with its
|
||||
spinner while the worker is still preparing the worktree. A failed launch keeps the tile with an
|
||||
inline error banner instead of the tile never appearing.
|
||||
|
||||
`Ellipse.spinner` (IslandStyles) is the shared indeterminate spinner — used for a starting pane
|
||||
(`InteractiveTerminalViewModel.IsStarting`) and in place of the refine button while
|
||||
`TaskRowViewModel.IsRefining`.
|
||||
|
||||
## Focus / key handling
|
||||
|
||||
`InteractiveTerminalView` lives in `MissionControlWindow`, so the `FocusClearing` Escape handler
|
||||
(scoped to `MainWindow` via `AddClassHandler<MainWindow>`) never runs there — **Escape always
|
||||
reaches the PTY**. See the note in `src/ClaudeDo.Ui/CLAUDE.md`.
|
||||
|
||||
`TaskRowViewModel.HasInteractiveSession` shows an accent "Interactive" chip instead of "Parked";
|
||||
tapping it jumps to that Mission Control pane. `TasksIslandViewModel.SyncInteractiveSessions`
|
||||
mirrors Mission Control's open panes onto the rows.
|
||||
|
||||
## Related hub methods
|
||||
|
||||
`GetInteractiveLaunchSpec`, `GetAdHocLaunchSpec`, `GetMergeHelperLaunchSpec`,
|
||||
`CreateMergeHelperTask`, `SubmitTaskForReview`.
|
||||
|
||||
Planning sessions: `StartPlanningSession`, `ResumePlanningSession`, `DiscardPlanningSession`,
|
||||
`FinalizePlanningSession`, `QueuePlanningSubtasks`, `GetPendingDraftCount`,
|
||||
`GetPlanningAggregate`, `BuildPlanningIntegrationBranch`.
|
||||
@@ -0,0 +1,141 @@
|
||||
# External MCP tool surface
|
||||
|
||||
> **Explore-note — verify before trusting.** Distilled map of a subsystem, not authoritative.
|
||||
> Last verified against commit `f6cb825` (2026-08-05).
|
||||
> Drift check: `git log --oneline f6cb825..HEAD -- src/ClaudeDo.Worker/External`
|
||||
> Stable structure only (no line numbers). See docs/explore-notes/README.md.
|
||||
|
||||
Covers `src/ClaudeDo.Worker/External/` — the always-on MCP tools ClaudeDo exposes to general
|
||||
Claude sessions. Registered explicitly in `Program.cs`'s external app via `.WithTools<T>()`.
|
||||
Server name `claudedo`, registered globally by the installer's `RegisterMcpStep`, so callers
|
||||
need no `--mcp-config`.
|
||||
|
||||
**Scope boundary:** these tools cover *starting* and *observing* sessions plus task/list CRUD
|
||||
and git/merge operations. They deliberately do **not** expose multi-turn control, planning
|
||||
session internals, or app-settings writes. Auth via an optional `X-ClaudeDo-Key` header.
|
||||
|
||||
## Hard conventions (enforced by tests)
|
||||
|
||||
1. **Every optional/filter parameter needs a C# default value** (e.g. `string? status = null`).
|
||||
The MCP schema only marks a parameter optional when it has one — nullability alone does
|
||||
not do it. `ExternalMcpToolSchemaTests` guards this by reflection.
|
||||
2. **No tool returns bare `Task` or a nullable payload directly.** An MCP client cannot tell
|
||||
an empty/omitted response apart from a dropped one.
|
||||
- *Write* tools return a small confirmation record — `{ ok/deleted/removed/reset/started:
|
||||
true, <id>, ... }` (`DeleteListResult`, `RunTaskNowResult`, `ResetFailedTaskResult`,
|
||||
`RemoveAttachmentResult`; `SetListConfigResult` / `SetTaskConfigResult` additionally echo
|
||||
the resulting config so the caller can see which fields were set vs. cleared to null).
|
||||
- *Read* tools that may have nothing to return use an explicit `Found` / `Available` flag
|
||||
alongside the nullable payload (`TaskConfigResult`, `BatchGetTaskResult`, `TaskLogResult`).
|
||||
3. `ExternalMcpExceptionFilter.Wrap` is registered as a call-tool filter so
|
||||
`InvalidOperationException` / `ArgumentException` messages survive as `McpException` —
|
||||
otherwise the SDK's catch-all replaces any non-`McpException` with a generic
|
||||
*"An error occurred invoking 'X'."*
|
||||
|
||||
## Tool classes
|
||||
|
||||
### `ExternalMcpService` — task CRUD, execution, git
|
||||
|
||||
Task: `ListTaskLists`, `ListTasks`, `GetTask`, `AddTask`, `AddSubtask`, `UpdateTask`,
|
||||
`UpdateTaskStatus`, `GetTaskStatusValues`, `ReviewTask`, `RunTaskNow`, `ContinueTask`,
|
||||
`CancelTask`, `DeleteTask`.
|
||||
|
||||
Worktree/git: `GetTaskWorktree`, `GetTaskDiff`, `MergeTask`, `ContinueMerge`, `AbortMerge`,
|
||||
`PreviewMerge`, `PreviewMergeSet`, `RevertMerge`, `ListWorktrees`, `CleanupTaskWorktree`.
|
||||
|
||||
Daily prep: `GetDailyPrepCandidates`, `SetMyDay`.
|
||||
|
||||
### Other classes
|
||||
|
||||
| Class | Tools |
|
||||
|---|---|
|
||||
| `BatchMcpTools` | `BatchGetTasks`, `BatchAddTasks`, `BatchUpdateTaskStatus`, `BatchCancelTasks`, `BatchDeleteTasks`, `BatchSetMyDay`, `BatchCleanupTaskWorktrees` |
|
||||
| `ListMcpTools` | `CreateList`, `UpdateList`, `DeleteList` |
|
||||
| `ConfigMcpTools` | `GetListConfig`, `SetListConfig`, `GetTaskConfig`, `SetTaskConfig` |
|
||||
| `RunHistoryMcpTools` | `ListRuns`, `GetRun`, `GetTaskLog` |
|
||||
| `AgentMcpTools` | `ListAgents` |
|
||||
| `LifecycleMcpTools` | `ResetFailedTask` |
|
||||
| `AppSettingsMcpTools` | `GetAppSettings` (read-only) |
|
||||
| `TaskWaitMcpTools` | `WaitForTaskChange` |
|
||||
| `AttachmentMcpTools` | `AddTaskAttachment`, `ListTaskAttachments`, `RemoveTaskAttachment` |
|
||||
|
||||
## Per-tool behaviour worth knowing
|
||||
|
||||
**`UpdateTaskStatus`** accepts `Idle` / `Queued` / `Cancelled` / `Done` only.
|
||||
- `Cancelled` goes through `TaskStateService.CancelAsync(..., allowFromIdle: true)` — the
|
||||
**only** caller that opts into cancelling from `Idle`. `PlanningChainCoordinator` relies on
|
||||
`Idle` staying a no-op there by default, because a child parked back to `Idle` mid-chain is
|
||||
a manual opt-out signal.
|
||||
- `Done` goes through `TaskStateService.ForceSetStatusAsync` (the same unconditional write the
|
||||
UI's "set status freely" affordance uses) but is **refused** for a task with an active
|
||||
worktree, since that would skip `review_task`'s merge.
|
||||
|
||||
**`ReviewTask`** — `approve` / `reject_rerun` / `reject_park` / `cancel` for a
|
||||
`WaitingForReview` task. Approve is review+merge exactly like the hub's `ApproveReview`: unit
|
||||
merge for parents, worktree merge into optional `targetBranch` for childless tasks. Conflicts
|
||||
are reported in `ReviewTaskResult`.
|
||||
|
||||
**`PreviewMerge`** — non-destructive `git merge-tree --write-tree` mergeability check for one
|
||||
task's worktree branch against `targetBranch` (default: the repo's current branch). Returns
|
||||
status / conflictFiles / changedFileCount / `behind`. Unlike `TaskMergeService.PreviewAsync`'s
|
||||
silent *"unavailable"*, this **throws a clear error** when the task has no worktree, the
|
||||
worktree isn't `Active`, or the list's working dir is missing.
|
||||
|
||||
**`PreviewMergeSet`** — same preview for a batch, plus a file→tasks overlap report built from
|
||||
each task's own diff-stat. ⚠️ That overlap report is a **same-file-name hint only** — it is
|
||||
blind to cross-file collisions (e.g. the CS0103 case that motivated it). A task that fails to
|
||||
preview gets `error` set and is excluded from the overlap instead of aborting the batch.
|
||||
|
||||
**`RevertMerge`** — undoes a previously merged task's merge commit on `targetBranch` via
|
||||
`git revert -m 1`. Always a **new commit**, never a reset/rewrite, because the target working
|
||||
directory is shared with other concurrent sessions.
|
||||
- Requires the task to be `Done` with a `Merged` worktree carrying a recorded
|
||||
`WorktreeEntity.MergeCommit`. A task merged before that field existed has none and is
|
||||
**refused rather than guessed** via `git log`.
|
||||
- On success the task returns to `WaitingForReview` and the worktree moves to `Kept` — not
|
||||
`Active` (its directory/branch are typically already gone from the original merge's cleanup)
|
||||
and not `Merged`/`Discarded` (`WorktreeMaintenanceService` sweeps those).
|
||||
- A conflicting revert is aborted immediately; no half-resolved state is left in the tree.
|
||||
|
||||
**`BatchMcpTools`** — best-effort loops over the `ExternalMcpService` single-entity methods.
|
||||
**Sequential**, because the scoped `DbContext` is not thread-safe. Merge/review stay
|
||||
single-task. Every tool returns a per-item result array (`{ id/index, ok, error?, … }`) — a
|
||||
failing item never aborts the rest — and rejects batches over **100 items**.
|
||||
|
||||
**`GetTaskLog`** — latest run's log, tail-capped at 256 KB.
|
||||
|
||||
**`WaitForTaskChange(taskIds, timeoutSeconds = 60)`** — blocks until any given task leaves
|
||||
`Queued`/`Running`, or times out. Returns immediately for a task already outside those two
|
||||
(unknown ids reported as status `"NotFound"`, also immediate).
|
||||
- Implemented as an **async DB poll** (short-lived `DbContext` per check, 500 ms delay, no
|
||||
held connection, no busy loop) rather than hooking `HubBroadcaster` — deliberately isolated
|
||||
so it can't regress the existing broadcast callers.
|
||||
- `timeoutSeconds` is clamped server-side to `TaskWaitMcpTools.MaxTimeoutSeconds` (170 s),
|
||||
comfortably under the list handler's `MCP_TOOL_TIMEOUT` (200 s, see
|
||||
`InteractiveLaunchSpecService`), so the tool reports `timedOut: true` instead of racing the
|
||||
client's own abort.
|
||||
- Replaced the list handler's old "sleep + poll `get_task` in a loop" Phase 3 instruction.
|
||||
|
||||
**`AttachmentMcpTools`** — re-attaching the same `fileName` overwrites. Add/remove refuse on a
|
||||
`Running` task.
|
||||
|
||||
**`GetDailyPrepCandidates`** — Idle, non-blocked tasks in a git repo **not** excluded by
|
||||
`AppSettings.ReportExcludedPaths` and not already `IsMyDay`, plus the current Idle MyDay tasks
|
||||
and `maxTasks` (= `DailyPrepMaxTasks`). Repo-exclusion logic lives in the `DailyPrepFilter`
|
||||
helper in the same file.
|
||||
|
||||
**`SetMyDay`** — sets `IsMyDay` (+ optional `SortOrder`). A server-side cap-guard rejects
|
||||
turning on MyDay beyond `DailyPrepMaxTasks` open (Idle) MyDay tasks.
|
||||
|
||||
## Model / max-turns on task creation
|
||||
|
||||
Task-generating tools (`AddTask`, planning `CreateChildTask`, `SuggestImprovement`) accept an
|
||||
optional `model`, alias-validated via `ModelRegistry.NormalizeAlias` (`haiku`/`sonnet`/`opus`,
|
||||
blank = inherit), so Claude can assign the cheapest capable model at creation time — the
|
||||
planning/system/improvement prompts instruct it to do so, using
|
||||
`ModelRegistry.ByCostAscending` as the cost order.
|
||||
|
||||
Planning's `CreateChildTask` **additionally** accepts an optional `maxTurns` (positive int;
|
||||
`0`/negative rejected with `ArgumentException`, null = inherit list/global default) so the
|
||||
planner can raise the turn budget for a subtask it knows will run long.
|
||||
`SuggestImprovement` and `AddTask` do **not** expose it.
|
||||
@@ -0,0 +1,174 @@
|
||||
# Review, merge & conflict resolution
|
||||
|
||||
> **Explore-note — verify before trusting.** Distilled map of a subsystem, not authoritative.
|
||||
> Last verified against commit `f6cb825` (2026-08-05).
|
||||
> Drift check: `git log --oneline f6cb825..HEAD -- src/ClaudeDo.Worker/Lifecycle src/ClaudeDo.Worker/State src/ClaudeDo.Worker/Planning src/ClaudeDo.Ui/ViewModels/Conflicts`
|
||||
> Stable structure only (no line numbers). See docs/explore-notes/README.md.
|
||||
|
||||
Covers the review→merge path: `TaskStateService` review transitions, `TaskMergeService`,
|
||||
`PlanningMergeOrchestrator`, the post-merge verify gate, and the UI conflict resolver.
|
||||
|
||||
## Approve = merge the whole unit
|
||||
|
||||
`ApproveReview` (hub) and `review_task` approve (MCP) are the **single** review+merge action.
|
||||
There is no separate "Merge all" entry.
|
||||
|
||||
- **Task with children** → drives `PlanningMergeOrchestrator`: merges the parent worktree if
|
||||
`Active`, then each `Done` child in order, then sets the parent `Done`. A mid-merge conflict
|
||||
pauses for `ContinuePlanningMerge` / `AbortPlanningMerge`.
|
||||
- **Childless task** → `TaskMergeService.ApproveAndMergeAsync`. A conflict keeps the task in
|
||||
`WaitingForReview`.
|
||||
- **No active worktree** (sandbox run) → straight to `Done`.
|
||||
|
||||
Review transitions all live in `TaskStateService`: `SubmitForReviewAsync`,
|
||||
`SubmitForChildrenAsync`, `ApproveReviewAsync`, `RejectToQueueAsync`, `RejectToIdleAsync`,
|
||||
`ClearReviewFeedbackAsync`.
|
||||
|
||||
`ReviewFeedback` (nullable string on `TaskEntity`) is the reviewer's rejection comment: set by
|
||||
`RejectToQueueAsync`, consumed and cleared by `QueueService` on the next re-run, where it
|
||||
becomes the next-turn prompt of the resumed Claude session.
|
||||
|
||||
## Unified parent model
|
||||
|
||||
Every parent — planning **or** improvement — flows
|
||||
`… → WaitingForChildren → WaitingForReview → Done`, advanced by the single
|
||||
`TaskStateService.TryAdvanceParentAsync`. It surfaces any `WaitingForChildren` parent for
|
||||
review once all children are terminal; failed/cancelled children are **annotated on the
|
||||
result, not wedged**.
|
||||
|
||||
- A planning parent enters `WaitingForChildren` at `FinalizePlanningAsync` (or
|
||||
`WaitingForReview` directly if it has no children).
|
||||
- An improvement parent enters it from `TaskRunner.HandleSuccess` when its run spawned children.
|
||||
- Planning/improvement **children** go straight to `Done` — no individual review. Only the
|
||||
parent is reviewed.
|
||||
|
||||
A child that hits a roadblock (fails, or reports `CLAUDEDO_BLOCKED` roadblocks) does **not**
|
||||
advance the parent — the parent stays in `WaitingForChildren` until every child is terminal.
|
||||
The UI surfaces blocked children on the parent's Session tab (`ChildOutcomes` + a "children
|
||||
need attention" band) so the roadblock is visible without forcing a transition.
|
||||
|
||||
## Post-merge verify gate
|
||||
|
||||
A list can set `ListConfigEntity.VerifyCommand` (List Settings modal → Verification).
|
||||
Null/blank (the default) = **no gate**, behavior bit-identical to before the feature existed.
|
||||
|
||||
When set, `TaskMergeService` runs it via `VerifyCommandRunner` (`cmd.exe /c <command>`,
|
||||
10-minute fixed timeout, output tail-captured) in `list.WorkingDir` right after a successful
|
||||
`MergeNoFfAsync` / `ContinueMergeAsync` **and** worktree cleanup, but **before** the task is
|
||||
allowed to reach `Done`.
|
||||
|
||||
| Outcome | Effect |
|
||||
|---|---|
|
||||
| Exit 0 | Unchanged flow — worktree `Merged`, task `Done` if it was `WaitingForReview`. |
|
||||
| Non-zero exit or timeout | The git merge is **deliberately left in place** (no auto-revert — that's a separate, unbuilt feature). The worktree is still marked `Merged` (it's already gone from disk when `removeWorktree` was requested), but the task stays out of `Done`. |
|
||||
|
||||
On failure `MergeResult.Status` comes back `TaskMergeService.StatusVerifyFailed`
|
||||
(`"verify_failed"`) with an output excerpt in `ErrorMessage`. This flows through
|
||||
`MergeResultDto` (hub) and `ReviewTaskResult` (`review_task`) unchanged, because both already
|
||||
treat any non-`blocked`/`conflict` status generically.
|
||||
|
||||
**Serialization:** a process-wide `ConcurrentDictionary<string, SemaphoreSlim>` keyed by
|
||||
`list.WorkingDir` serializes `MergeAsync` / `ContinueMergeAsync` (git ops + verify) per repo,
|
||||
so a verify run can't be interrupted by a second merge landing in the same working dir
|
||||
mid-build.
|
||||
|
||||
## `MergeCommit` and revert
|
||||
|
||||
`WorktreeEntity.MergeCommit` (nullable) is the SHA of the merge commit this worktree's branch
|
||||
produced on the target branch. Stamped by `TaskMergeService` the moment a merge/continue-merge
|
||||
succeeds, written **only** by `WorktreeRepository.SetMergedAsync` (which atomically sets
|
||||
`State=Merged` and stamps the SHA in one update).
|
||||
|
||||
It is the only thing that makes `revert_merge` possible without heuristically searching
|
||||
`git log` — see [external-mcp.md](external-mcp.md) → `RevertMerge`. Null for any worktree
|
||||
merged before the field existed.
|
||||
|
||||
## Review gate in the UI
|
||||
|
||||
**Approve & Merge is gated behind opening the diff.** When there is something to inspect
|
||||
(worktree diff / merged range / children combined diff), the button stays disabled until the
|
||||
diff or combined-diff viewer has been opened once. The gate **re-locks per run** — any state
|
||||
change resets it. Tasks with nothing to inspect are never gated.
|
||||
|
||||
The row-level quick-approve in the task list is an **intentional bypass**.
|
||||
|
||||
Implementation: `MergeSectionViewModel` owns merge-target selection, the mergeability
|
||||
indicator (`MergePreviewPresenter` over `PreviewMergeAsync`), and `OpenDiffAsync` /
|
||||
`ReviewCombinedDiffCommand` — both build a `DiffViewerViewModel`, call `ShowDiffViewer`, and
|
||||
fire the `DiffViewed` callback. `HasReviewableDiff` reports whether anything is inspectable
|
||||
and feeds the gate.
|
||||
|
||||
## Conflict resolver (in-app Rider-style 3-pane merge editor)
|
||||
|
||||
`ConflictResolverViewModel` + `Views/Conflicts/ConflictResolverView`. Handles **both**
|
||||
single-task and planning unit-merge conflicts.
|
||||
|
||||
### Model
|
||||
|
||||
Single-task mode starts the conflict merge, then parses each conflicted file into stable and
|
||||
conflict `MergeFileSegment`s via the worker's `GetMergeConflictDocuments`. Types live in
|
||||
`ConflictModels`: `MergeFile` / `MergeFileSegment` / `MergeConflictBlock`.
|
||||
|
||||
Exposed per active file: `ActiveOursText` / `ActiveResultText` / `ActiveTheirsText`
|
||||
(reconstructed from `MergeFile.OursText/ResultText/TheirsText`; Result seeds unresolved
|
||||
conflicts with Ours), plus `ActiveFile` / `SelectFileCommand` (multi-file switcher),
|
||||
`Current` / `Next` / `Previous` (focused-conflict nav), a per-file `PositionText` readout,
|
||||
per-block `AcceptOurs/Theirs/Both/Base` + `MergeFile.Compose`, and `CanContinue` gated on
|
||||
**every file resolved + no binary**. Each file is written via `WriteConflictResolution`.
|
||||
|
||||
**Planning mode** via `OpenForPlanningAsync(parentId, subtaskId)` loads the current subtask's
|
||||
mid-merge conflicts **without re-starting the merge** and routes continue/abort to
|
||||
`ContinuePlanningMerge` / `AbortPlanningMerge`, so a unit-merge conflict re-opens the editor
|
||||
per subtask via the `PlanningMergeConflict` broadcast.
|
||||
|
||||
### View
|
||||
|
||||
Three **AvaloniaEdit** panes showing the whole file: MAIN/ours (read-only) | editable Result |
|
||||
INCOMING/theirs (read-only). TextMate highlighting by extension (theme `StyleInclude` in
|
||||
`App.axaml`).
|
||||
|
||||
- A code-behind `IBackgroundRenderer` tints each conflict block (unresolved/resolved) across
|
||||
panes. Tints live in `Tokens.axaml` (`Merge*TintBrush`).
|
||||
- An `IReadOnlySectionProvider` + `TextAnchor` regions keep **only conflict spans** editable in
|
||||
Result; edits flow back to the block.
|
||||
- Each unresolved conflict starts **EMPTY** (a thin marker bar).
|
||||
- The between-pane gutter controls **toggle** each side in/out of the result: `›`/`‹` add
|
||||
MAIN/INCOMING in click order (first pick on top), clicking again removes that side — so a
|
||||
conflict can take main, incoming, both, or **neither**.
|
||||
- `FilesSummary` shows how many files still have conflicts. The three panes share a
|
||||
proportional synced vertical scroll.
|
||||
- A conflict overview ruler right of the Result pane (`ConflictMap`) maps every conflict in the
|
||||
file proportionally; click a tick to jump. Useful for long files.
|
||||
|
||||
### Entry points
|
||||
|
||||
Review **Approve** on conflict, and the **Merge** button in the Diff window (a conflicting
|
||||
`MergeTask` hands off via `RequestConflictResolution`).
|
||||
|
||||
## Hub methods
|
||||
|
||||
- Review/merge: `ApproveReview(taskId, targetBranch) -> MergeResultDto`,
|
||||
`ContinuePlanningMerge` / `AbortPlanningMerge`, `PreviewMerge(taskId, targetBranch) ->
|
||||
MergePreviewDto`, `RejectReviewToQueue`, `RejectReviewToIdle`, `CancelReview`, `MergeTask`,
|
||||
`GetMergeTargets`
|
||||
- Single-task conflict resolver: `StartConflictMerge`, `GetMergeConflictDocuments`,
|
||||
`WriteConflictResolution`, `ContinueConflictMerge`, `AbortConflictMerge` — note the
|
||||
service-level `TaskMergeService.ContinueMergeAsync` / `AbortMergeAsync` keep their own names.
|
||||
- Broadcast events: `PlanningMergeStarted`, `PlanningSubtaskMerged`, `PlanningMergeConflict`,
|
||||
`PlanningMergeAborted`, `PlanningCompleted`
|
||||
|
||||
## Diff stack (UI)
|
||||
|
||||
`UnifiedDiffParser` (static) parses `git diff` output into `DiffFileViewModel`s, detecting
|
||||
added/deleted/renamed/binary files and per-line numbers; `Flatten` injects file-header rows for
|
||||
a combined single-pane view. `DiffModels.cs` holds the shared types (`DiffLineViewModel`,
|
||||
`DiffFileViewModel`, `DiffLineKind`, `DiffFileStatus`, `SubtaskDiffRow`,
|
||||
`DiffTreeNodeViewModel`, `DiffTree`).
|
||||
|
||||
`DiffViewerViewModel` is one unified read-only viewer with two modes:
|
||||
- **Files** — dirty worktree / branch-vs-base / commit-range. Loads via `GitService`, folder
|
||||
file-tree left + per-file diff pane right, Merge button for a live branch source.
|
||||
- **Planning** — per-subtask diffs via `GetPlanningAggregateAsync`, subtask list left + flat
|
||||
diff right, combined integration-branch toggle.
|
||||
|
||||
`DiffLinesView` renders per-file content with binary/empty placeholders.
|
||||
@@ -0,0 +1,142 @@
|
||||
# Usage monitoring, gate & throttle
|
||||
|
||||
> **Explore-note — verify before trusting.** Distilled map of a subsystem, not authoritative.
|
||||
> Last verified against commit `f6cb825` (2026-08-05).
|
||||
> Drift check: `git log --oneline f6cb825..HEAD -- src/ClaudeDo.Worker/Usage src/ClaudeDo.Worker/Queue src/ClaudeDo.Ui/ViewModels/UsagePillViewModel.cs`
|
||||
> Stable structure only (no line numbers). See docs/explore-notes/README.md.
|
||||
|
||||
Covers `src/ClaudeDo.Worker/Usage/`, the queue's throttle/gate integration, per-run token
|
||||
accounting, and the UI surfaces (usage pill + usage monitor modal).
|
||||
|
||||
## Data source
|
||||
|
||||
`GET https://api.anthropic.com/api/oauth/usage` — an **undocumented** Anthropic endpoint,
|
||||
authenticated with the Bearer access token Claude Code itself keeps fresh at
|
||||
`~/.claude/.credentials.json`. ClaudeDo reads that token, never refreshes it, never logs it.
|
||||
|
||||
Because the endpoint is undocumented and can change without notice, **every consumer
|
||||
fails open**. That is the single most important invariant here.
|
||||
|
||||
## Components (`Usage/`)
|
||||
|
||||
| Type | Role |
|
||||
|---|---|
|
||||
| `UsageModels` | `UsageBucket` / `UsageLimitRow` / `UsageSnapshot`. `UsageBucket.Utilization` is already a 0–100 percent — compare directly against thresholds, don't rescale. |
|
||||
| `ClaudeOAuthUsageClient` | Reads the token, calls the endpoint. Defensive parsing: missing/null buckets → null, missing `limits` → empty list. |
|
||||
| `UsageState` | Threadsafe singleton. A failed poll **never** overwrites the last good snapshot — it only sets `LastError`. |
|
||||
| `UsageMonitorService` | `BackgroundService`; polls on `usage_poll_interval_seconds` (default 60, clamped to min 15 on config load), one poll at startup. Logs a failure at most once per distinct error message. Broadcasts `HubBroadcaster.UsageUpdated` after **every** tick, success or failure. |
|
||||
| `UsageSnapshotBuilder` | Builds the Hub-facing `UsageSnapshotDto` from `UsageState` + `IUsageGate` + `AppSettings`. The one shared place for stale/threshold/gate logic — `WorkerHub.GetUsageSnapshot` and `UsageMonitorService` must not diverge. |
|
||||
| `UsageGate` | Hard pause decision → `UsageGateDecision(IsBlocked, Reason)`. |
|
||||
| `UsageThrottle` | Pure static staging of parallelism ahead of the gate. |
|
||||
| `TranscriptUsageReader` | Aggregates token usage from Claude Code transcripts. |
|
||||
|
||||
Interfaces in `Usage/Interfaces/`: `IUsageClient`, `ITranscriptUsageReader`, `IUsageGate`.
|
||||
|
||||
## The gate (hard pause)
|
||||
|
||||
Thresholds: `AppSettings.UsageGateFiveHourPct` / `UsageGateSevenDayPct` (defaults 80/90).
|
||||
Blocked once `five_hour >= UsageGateFiveHourPct` **or** `seven_day >= UsageGateSevenDayPct`
|
||||
(`>=`, not `>`). Threshold `0` = that bucket never gates.
|
||||
|
||||
What it pauses: **only the queue's slot-fill loop** — new queued tasks don't start.
|
||||
Unaffected: already-running runs, `RunNow`, `ContinueTask`, interactive ConPTY sessions,
|
||||
planning sessions, daily prep (all bypass the queue).
|
||||
|
||||
**Fail-open**: no snapshot yet, a failed last poll, or an app-settings read error all
|
||||
resolve to not-blocked.
|
||||
|
||||
There is **no persistent pause state**. Recovery is just the queue's 30 s backstop timer
|
||||
(`queue_backstop_interval_ms`) re-evaluating the gate on its own once usage drops back
|
||||
under the threshold. A blocked↔free transition is logged and broadcast (`WorkerLog`, Warn
|
||||
on block / Info on resume) exactly **once per change**, not every tick.
|
||||
|
||||
## The throttle (staged parallelism)
|
||||
|
||||
`UsageThrottle.EffectiveSlots(configuredSlots, fiveHourPct, sevenDayPct, softPct, hardPct,
|
||||
gateFiveHourPct, gateSevenDayPct)` — pure static, no state.
|
||||
|
||||
Thresholds `usage_throttle_soft_pct` / `usage_throttle_hard_pct` (defaults 50/65).
|
||||
Whichever of 5h/7d is **more utilized** decides the stage:
|
||||
|
||||
| Utilization | Effective slots |
|
||||
|---|---|
|
||||
| below soft | full configured `max_parallel_executions` |
|
||||
| `>= softPct` | capped at 2 |
|
||||
| `>= hardPct` | capped at 1 |
|
||||
| `>=` either gate threshold | 0 — same hard block as `UsageGate` |
|
||||
|
||||
A threshold of `0` disables that stage. The `0` return is deliberately kept in sync with
|
||||
`UsageGate`'s hard block because both read the same gate thresholds — change one, change both.
|
||||
|
||||
Only **new** slot fills are affected; a run already occupying a slot when the stage tightens
|
||||
runs to completion. Same fail-open policy: no snapshot means no throttling.
|
||||
|
||||
The effective stage (configured vs. effective slots + the decisive bucket, `ThrottleBucket`
|
||||
= `"five_hour"` / `"seven_day"`) rides along on `UsageSnapshotDto` purely for UI display.
|
||||
It does **not** change what the gate gates on.
|
||||
|
||||
## Queue integration (`Queue/QueueService`)
|
||||
|
||||
Per loop tick:
|
||||
|
||||
1. `GetEffectiveMaxParallelAsync` reads `AppSettings.MaxParallelExecutions` and steps it
|
||||
down via `UsageThrottle.EffectiveSlots` against the current `UsageState` snapshot.
|
||||
A missing/failed snapshot fails open to the configured value. A **stage change** (not
|
||||
every tick) logs once.
|
||||
2. Separately, `IUsageGate.EvaluateAsync` — if blocked, the slot-fill loop is skipped
|
||||
entirely for that tick.
|
||||
|
||||
## Per-run token accounting
|
||||
|
||||
`task_runs` stores four raw token fields: `tokens_in` / `tokens_out` /
|
||||
`cache_read_tokens` / `cache_write_tokens`.
|
||||
|
||||
These are **not** read from the stream-json `result` event's `usage.input_tokens` — that is
|
||||
only the uncached remainder of a single API call and undercounts the real prompt size by
|
||||
orders of magnitude once caching kicks in.
|
||||
|
||||
Instead `TaskRunner.ApplyUsageAsync` calls
|
||||
`ITranscriptUsageReader.ReadSessionTotalsAsync(sessionId)` — the session transcript's
|
||||
cumulative raw totals across every assistant message, located by `{sessionId}.jsonl` — and
|
||||
stores the **delta** against prior `task_runs` rows sharing the same `session_id`, so a
|
||||
`--resume`'d run doesn't double-count turns already billed to an earlier run.
|
||||
|
||||
A missing/unreadable transcript leaves all four fields `null`; it never fails the run.
|
||||
|
||||
### `TranscriptUsageReader` details
|
||||
|
||||
Reads `~/.claude/projects/**/*.jsonl`, aggregating by date / model / scope (ClaudeDo vs
|
||||
Other), deduped by `requestId`, with a per-file length+mtime cache.
|
||||
|
||||
`<synthetic>`-model lines are skipped **everywhere** — they are not real API calls.
|
||||
|
||||
## UI surfaces
|
||||
|
||||
- **`UsagePillViewModel`** — one shared instance backs the `UsagePill` control in both the
|
||||
footer and the Mission Control header. Loads via `GetUsageSnapshotAsync`, updates live off
|
||||
`IWorkerClient.UsageUpdatedEvent`. Dot state priority is mutually exclusive:
|
||||
**blocked > stale > warn > normal**. `IsThrottled` (effective slots below configured, and
|
||||
not gate-blocked) adds a tooltip line naming effective/configured slots + decisive bucket.
|
||||
- **`UsageMonitorModalViewModel`** — opened from the pill. Renders one gauge **per row** in
|
||||
`UsageSnapshotDto.Limits` — deliberately **dynamic**, because the `seven_day_opus` /
|
||||
`seven_day_sonnet`-style buckets the raw API returns are plan-dependent and come back
|
||||
`null` on plans that don't have them; a fixed gauge layout would break. Also shows model
|
||||
usage (`GetModelUsageAsync`, ClaudeDo-vs-Other split per model) and top-task usage
|
||||
(`GetTaskUsageAsync`) over a 7d/30d preset or custom range.
|
||||
|
||||
## Hub surface
|
||||
|
||||
- `GetUsageSnapshot() -> UsageSnapshotDto` — percentages/limits/`FetchedAtUtc` are null and
|
||||
`IsStale=true` when no snapshot has landed yet. `IsStale` also trips on a failed last poll
|
||||
or a snapshot older than 3× `usage_poll_interval_seconds`.
|
||||
- `GetModelUsage(from, to)` — thin wrapper over `ITranscriptUsageReader.ReadAsync`.
|
||||
- `GetTaskUsage(from, to)` — top consumers from `task_runs` joined to task/list, grouped per
|
||||
task. Null token columns count as **0**, never drop the row. `Model` comes from that task's
|
||||
most recent run. Sorted by total tokens descending, capped at 100.
|
||||
- `UsageUpdated` event carries the same `UsageSnapshotDto`.
|
||||
|
||||
## Settings columns
|
||||
|
||||
`app_settings`: `usage_gate_five_hour_pct` / `usage_gate_seven_day_pct` (80/90),
|
||||
`usage_throttle_soft_pct` / `usage_throttle_hard_pct` (50/65). All four clamped 0..100 by
|
||||
`AppSettingsRepository.UpdateAsync`. Worker config: `usage_poll_interval_seconds`.
|
||||
@@ -108,6 +108,20 @@ Offene Entscheidungen dazu:
|
||||
undokumentiert und kann sich ändern; bei Ausfall/Formatänderung ist das Gate wirkungslos
|
||||
(fail-open by design — kein Blocker, aber der Schutz fällt dann aus, ohne dass es auffällt).
|
||||
|
||||
### Nachtrag 2026-08-05: 429-Fix (Poll-Kadenz + Refresh-Button)
|
||||
|
||||
Der 60s-Poll lief in 429s. Neu: aktivitätsabhängige Kadenz (5 Min. solange ein Task `Running`
|
||||
ist, sonst 15 Min.), 429-Backoff mit `Retry-After`, und ein „Jetzt aktualisieren"-Button im
|
||||
Usage-Monitor-Modal (`RefreshUsage` → `UsageMonitorService.RefreshNowAsync`, 10s-Cooldown).
|
||||
Unit-Tests grün, **offen**:
|
||||
|
||||
- **Visueller Pass Refresh-Button** im Modal (Button + Spinner + Hinweiszeile, Dark/Light,
|
||||
en/de) — Teil des oben schon offenen Modal-Passes.
|
||||
- **E2E:** über ≥20 Min. mit und ohne laufenden Task beobachten, dass keine 429s mehr im
|
||||
Worker-Log auftauchen und die Pill trotzdem aktuell bleibt.
|
||||
- **Beachten:** die Pill wird jetzt erst nach 3× 15 Min. als `stale` markiert — ein echter
|
||||
Endpoint-Ausfall fällt vorher nur über `LastError` auf (der `IsStale` sofort setzt).
|
||||
|
||||
## Offene Verifikation (2026-08-05, Max-Turns-Ceiling)
|
||||
|
||||
Build + unit tests grün (`ResolveMaxTurns`-Klemmung, Repository-Backfill von `model_presets`,
|
||||
|
||||
@@ -431,6 +431,8 @@
|
||||
"title": "USAGE MONITOR",
|
||||
"windowTitle": "Usage Monitor",
|
||||
"noGauges": "Noch keine Nutzungslimits gemeldet.",
|
||||
"refresh": "Jetzt aktualisieren",
|
||||
"refreshHint": "Abruf alle 5 Min., während ein Task läuft, sonst alle 15 Min.",
|
||||
"staleFormat": "Werte veraltet (Stand {0})",
|
||||
"staleGateHint": "Das Gate greift in diesem Zustand nicht.",
|
||||
"gateBlockedFormat": "Queue pausiert — {0}",
|
||||
@@ -598,7 +600,10 @@
|
||||
"settingsModal": { "workerOffline": "Worker offline — Einstellungen schreibgeschützt.", "saveFailed": "Speichern fehlgeschlagen: {0}" },
|
||||
"onlineInbox": { "workerOffline": "Worker offline — Konfiguration kann nicht geladen werden.", "saved": "Konfiguration gespeichert.", "saveFailed": "Speichern fehlgeschlagen: {0}", "signedIn": "Erfolgreich angemeldet.", "signedInNoRole": "Angemeldet, aber diesem Konto fehlt die Rolle 'user' in Zitadel — die Online-Synchronisierung wird abgelehnt, bis die Rolle im ClaudeDo-Projekt zugewiesen wird.", "signInFailed": "Anmeldung fehlgeschlagen: {0}", "signedOut": "Abgemeldet.", "signOutFailed": "Abmeldung fehlgeschlagen: {0}" },
|
||||
"weeklyReport": { "invalidRange": "Ungültiger Datumsbereich.", "generating": "Bericht wird erstellt…", "error": "Fehler: {0}" },
|
||||
"usageMonitor": { "loadFailed": "Nutzungsdaten konnten nicht geladen werden: {0}" },
|
||||
"usageMonitor": {
|
||||
"loadFailed": "Nutzungsdaten konnten nicht geladen werden: {0}",
|
||||
"refreshFailed": "Nutzung konnte nicht aktualisiert werden: {0}"
|
||||
},
|
||||
"filesTab": { "workerOffline": "Worker offline.", "noneBundled": "Keine Standard-Agenten mitgeliefert.", "allPresent": "Alle Standard-Agenten bereits vorhanden.", "restored": "{0} Standard-Agent(en) wiederhergestellt.", "restoreFailed": "Wiederherstellung fehlgeschlagen: {0}", "openFailed": "Öffnen fehlgeschlagen: {0}", "resetToDefault": "Auf den mitgelieferten Standard zurückgesetzt." },
|
||||
"sessionSkillsTab": { "installed": "Installiert: {0}", "installFailed": "Installation fehlgeschlagen: {0}", "updated": "Aktualisiert.", "updateFailed": "Aktualisierung fehlgeschlagen: {0}", "removed": "Entfernt.", "removeFailed": "Entfernen fehlgeschlagen: {0}" },
|
||||
"worktreesTab": { "workerOffline": "Worker offline.", "removed": "{0} Worktree(s) entfernt.", "blocked": "Zwangsentfernung nicht möglich: {0} Aufgabe(n) laufen noch. Brich sie zuerst ab.", "removedFrom": "{0} Worktree(s) von {1} Aufgabe(n) entfernt.", "cleanupFailed": "Aufräumen fehlgeschlagen: {0}", "resetFailed": "Zurücksetzen fehlgeschlagen: {0}" },
|
||||
|
||||
@@ -431,6 +431,8 @@
|
||||
"title": "USAGE MONITOR",
|
||||
"windowTitle": "Usage Monitor",
|
||||
"noGauges": "No usage limits reported yet.",
|
||||
"refresh": "Refresh now",
|
||||
"refreshHint": "Polled every 5 min while a task runs, otherwise every 15 min.",
|
||||
"staleFormat": "Values stale (as of {0})",
|
||||
"staleGateHint": "The gate does not apply while values are stale.",
|
||||
"gateBlockedFormat": "Queue paused — {0}",
|
||||
@@ -598,7 +600,10 @@
|
||||
"settingsModal": { "workerOffline": "Worker offline — settings read-only.", "saveFailed": "Save failed: {0}" },
|
||||
"onlineInbox": { "workerOffline": "Worker offline — cannot load config.", "saved": "Config saved.", "saveFailed": "Save failed: {0}", "signedIn": "Signed in successfully.", "signedInNoRole": "Signed in, but this account is missing the 'user' role in Zitadel — online sync will be rejected until the role is granted in the ClaudeDo project.", "signInFailed": "Sign-in failed: {0}", "signedOut": "Signed out.", "signOutFailed": "Sign-out failed: {0}" },
|
||||
"weeklyReport": { "invalidRange": "Invalid date range.", "generating": "Generating report…", "error": "Error: {0}" },
|
||||
"usageMonitor": { "loadFailed": "Couldn't load usage data: {0}" },
|
||||
"usageMonitor": {
|
||||
"loadFailed": "Couldn't load usage data: {0}",
|
||||
"refreshFailed": "Couldn't refresh usage: {0}"
|
||||
},
|
||||
"filesTab": { "workerOffline": "Worker offline.", "noneBundled": "No default agents bundled.", "allPresent": "All default agents already present.", "restored": "Restored {0} default agent(s).", "restoreFailed": "Restore failed: {0}", "openFailed": "Open failed: {0}", "resetToDefault": "Reset to the bundled default." },
|
||||
"sessionSkillsTab": { "installed": "Installed: {0}", "installFailed": "Install failed: {0}", "updated": "Updated.", "updateFailed": "Update failed: {0}", "removed": "Removed.", "removeFailed": "Remove failed: {0}" },
|
||||
"worktreesTab": { "workerOffline": "Worker offline.", "removed": "Removed {0} worktree(s).", "blocked": "Cannot force-remove: {0} task(s) still running. Cancel them first.", "removedFrom": "Removed {0} worktree(s) from {1} task(s).", "cleanupFailed": "Cleanup failed: {0}", "resetFailed": "Reset failed: {0}" },
|
||||
|
||||
@@ -143,6 +143,9 @@ public interface IWorkerClient : INotifyPropertyChanged
|
||||
/// <summary>Raised whenever the worker's usage poller ticks (success or failure).</summary>
|
||||
event Action<UsageSnapshotDto>? UsageUpdatedEvent;
|
||||
Task<UsageSnapshotDto?> GetUsageSnapshotAsync();
|
||||
|
||||
/// <summary>Forces an out-of-band usage poll on the worker and returns the fresh snapshot.</summary>
|
||||
Task<UsageSnapshotDto?> RefreshUsageAsync();
|
||||
Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsageAsync(DateOnly from, DateOnly to);
|
||||
Task<IReadOnlyList<TaskUsageRowDto>> GetTaskUsageAsync(DateOnly from, DateOnly to);
|
||||
}
|
||||
|
||||
@@ -582,6 +582,9 @@ public partial class WorkerClient : ObservableObject, IAsyncDisposable, IWorkerC
|
||||
public Task<UsageSnapshotDto?> GetUsageSnapshotAsync()
|
||||
=> TryInvokeAsync<UsageSnapshotDto>("GetUsageSnapshot");
|
||||
|
||||
public Task<UsageSnapshotDto?> RefreshUsageAsync()
|
||||
=> TryInvokeAsync<UsageSnapshotDto>("RefreshUsage");
|
||||
|
||||
public async Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsageAsync(DateOnly from, DateOnly to)
|
||||
=> await TryInvokeAsync<List<ModelUsageRowDto>>("GetModelUsage", from, to) ?? [];
|
||||
|
||||
|
||||
@@ -40,6 +40,8 @@ public sealed partial class UsageMonitorModalViewModel : ViewModelBase
|
||||
[NotifyPropertyChangedFor(nameof(ModelsEmpty))]
|
||||
private bool _isBusy;
|
||||
|
||||
[ObservableProperty] private bool _isRefreshing;
|
||||
|
||||
[ObservableProperty]
|
||||
[NotifyPropertyChangedFor(nameof(ModelsEmpty))]
|
||||
private IReadOnlyList<ModelUsageDisplayRow> _modelRows = Array.Empty<ModelUsageDisplayRow>();
|
||||
@@ -92,6 +94,28 @@ public sealed partial class UsageMonitorModalViewModel : ViewModelBase
|
||||
|
||||
private void OnUsageUpdated(UsageSnapshotDto snapshot) => Snapshot = snapshot;
|
||||
|
||||
/// <summary>
|
||||
/// Manual refresh: the worker polls usage on a slow cadence (15 min idle / 5 min while a
|
||||
/// task runs) to stay clear of the endpoint's 429s, so this is the way to get a number now.
|
||||
/// </summary>
|
||||
[RelayCommand]
|
||||
private async Task RefreshAsync()
|
||||
{
|
||||
if (IsRefreshing) return;
|
||||
IsRefreshing = true;
|
||||
try
|
||||
{
|
||||
var snapshot = await _worker.RefreshUsageAsync();
|
||||
if (snapshot is not null) Snapshot = snapshot;
|
||||
await LoadUsageDataAsync();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
ErrorReported?.Invoke(Loc.T("vm.usageMonitor.refreshFailed", ex.Message));
|
||||
}
|
||||
finally { IsRefreshing = false; }
|
||||
}
|
||||
|
||||
private void ApplyPresetRange(int days)
|
||||
{
|
||||
var today = DateOnly.FromDateTime(DateTime.Today);
|
||||
|
||||
@@ -60,6 +60,18 @@
|
||||
</Border>
|
||||
</StackPanel>
|
||||
|
||||
<!-- Refresh: the worker polls on a slow cadence to avoid the endpoint's 429s -->
|
||||
<StackPanel DockPanel.Dock="Top" Orientation="Horizontal" Spacing="8"
|
||||
Margin="20,12,20,0" VerticalAlignment="Center">
|
||||
<Button Classes="btn" Content="{loc:Tr modals.usageMonitor.refresh}"
|
||||
Command="{Binding RefreshCommand}"
|
||||
IsEnabled="{Binding !IsRefreshing}"/>
|
||||
<Ellipse Classes="spinner" Width="14" Height="14" VerticalAlignment="Center"
|
||||
IsVisible="{Binding IsRefreshing}"/>
|
||||
<TextBlock Classes="meta" VerticalAlignment="Center"
|
||||
Text="{loc:Tr modals.usageMonitor.refreshHint}"/>
|
||||
</StackPanel>
|
||||
|
||||
<!-- Gauges -->
|
||||
<ItemsControl DockPanel.Dock="Top" Margin="20,12,20,0" ItemsSource="{Binding GaugeRows}">
|
||||
<ItemsControl.ItemsPanel>
|
||||
|
||||
@@ -44,9 +44,18 @@ public sealed class WorkerConfig
|
||||
[JsonPropertyName("online_inbox")]
|
||||
public OnlineInboxConfig OnlineInbox { get; set; } = new();
|
||||
|
||||
/// <summary>Poll interval for the OAuth usage monitor. Clamped to a minimum of 15s on load.</summary>
|
||||
[JsonPropertyName("usage_poll_interval_seconds")]
|
||||
public int UsagePollIntervalSeconds { get; set; } = 60;
|
||||
/// <summary>
|
||||
/// Usage-monitor poll interval while at least one task is Running. Clamped to a minimum
|
||||
/// of 60s on load — the endpoint rate-limits (429) on tighter polling.
|
||||
/// </summary>
|
||||
[JsonPropertyName("usage_poll_interval_active_seconds")]
|
||||
public int UsagePollIntervalActiveSeconds { get; set; } = 300;
|
||||
|
||||
/// <summary>
|
||||
/// Usage-monitor poll interval while nothing is running. Clamped to a minimum of 60s on load.
|
||||
/// </summary>
|
||||
[JsonPropertyName("usage_poll_interval_idle_seconds")]
|
||||
public int UsagePollIntervalIdleSeconds { get; set; } = 900;
|
||||
|
||||
public static string DefaultConfigPath =>
|
||||
Path.Combine(Paths.AppDataRoot(), "worker.config.json");
|
||||
@@ -75,7 +84,8 @@ public sealed class WorkerConfig
|
||||
cfg.SandboxRoot = Paths.Expand(cfg.SandboxRoot);
|
||||
cfg.LogRoot = Paths.Expand(cfg.LogRoot);
|
||||
cfg.CentralWorktreeRoot = Paths.Expand(cfg.CentralWorktreeRoot);
|
||||
cfg.UsagePollIntervalSeconds = Math.Max(15, cfg.UsagePollIntervalSeconds);
|
||||
cfg.UsagePollIntervalActiveSeconds = Math.Max(60, cfg.UsagePollIntervalActiveSeconds);
|
||||
cfg.UsagePollIntervalIdleSeconds = Math.Max(60, cfg.UsagePollIntervalIdleSeconds);
|
||||
|
||||
return cfg;
|
||||
}
|
||||
|
||||
@@ -189,6 +189,7 @@ public sealed class WorkerHub : Microsoft.AspNetCore.SignalR.Hub
|
||||
private readonly Data.Git.GitService? _git;
|
||||
private readonly UsageSnapshotBuilder? _usageSnapshotBuilder;
|
||||
private readonly ITranscriptUsageReader? _usageReader;
|
||||
private readonly UsageMonitorService? _usageMonitor;
|
||||
|
||||
public WorkerHub(
|
||||
QueueService queue,
|
||||
@@ -220,7 +221,8 @@ public sealed class WorkerHub : Microsoft.AspNetCore.SignalR.Hub
|
||||
WorktreeManager? worktreeManager = null,
|
||||
Data.Git.GitService? git = null,
|
||||
UsageSnapshotBuilder? usageSnapshotBuilder = null,
|
||||
ITranscriptUsageReader? usageReader = null)
|
||||
ITranscriptUsageReader? usageReader = null,
|
||||
UsageMonitorService? usageMonitor = null)
|
||||
{
|
||||
_queue = queue;
|
||||
_waker = waker;
|
||||
@@ -252,6 +254,7 @@ public sealed class WorkerHub : Microsoft.AspNetCore.SignalR.Hub
|
||||
_git = git;
|
||||
_usageSnapshotBuilder = usageSnapshotBuilder;
|
||||
_usageReader = usageReader;
|
||||
_usageMonitor = usageMonitor;
|
||||
}
|
||||
|
||||
// Persistence boundary for the session_skills JSON-array columns (task/list/global).
|
||||
@@ -1049,6 +1052,18 @@ public sealed class WorkerHub : Microsoft.AspNetCore.SignalR.Hub
|
||||
return _usageSnapshotBuilder.BuildAsync(Context.ConnectionAborted);
|
||||
});
|
||||
|
||||
/// <summary>
|
||||
/// Manual "refresh now" for the usage monitor. Polls the endpoint out of band and returns the
|
||||
/// fresh snapshot; a refresh inside the monitor's cooldown reuses the last poll's result
|
||||
/// instead of risking a 429.
|
||||
/// </summary>
|
||||
public Task<UsageSnapshotDto> RefreshUsage() => HubGuard(() =>
|
||||
{
|
||||
if (_usageMonitor is null)
|
||||
throw new InvalidOperationException("Usage monitor is not configured.");
|
||||
return _usageMonitor.RefreshNowAsync(Context.ConnectionAborted);
|
||||
});
|
||||
|
||||
public Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsage(DateOnly from, DateOnly to) => HubGuard(async () =>
|
||||
{
|
||||
if (_usageReader is null)
|
||||
|
||||
@@ -209,7 +209,10 @@ builder.Services.AddHttpClient<IUsageClient, ClaudeOAuthUsageClient>(client =>
|
||||
});
|
||||
builder.Services.AddSingleton<IUsageGate, UsageGate>();
|
||||
builder.Services.AddSingleton<UsageSnapshotBuilder>();
|
||||
builder.Services.AddHostedService<UsageMonitorService>();
|
||||
builder.Services.AddSingleton<IRunningTaskProbe, RunningTaskProbe>();
|
||||
// Singleton + hosted service (same instance) so WorkerHub.RefreshUsage can drive a manual poll.
|
||||
builder.Services.AddSingleton<UsageMonitorService>();
|
||||
builder.Services.AddHostedService(sp => sp.GetRequiredService<UsageMonitorService>());
|
||||
|
||||
// Loopback-only bind. Firewall is irrelevant for 127.0.0.1.
|
||||
builder.WebHost.UseUrls($"http://127.0.0.1:{cfg.SignalRPort}");
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using System.Net;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
@@ -36,6 +37,8 @@ public sealed class ClaudeOAuthUsageClient : IUsageClient
|
||||
request.Headers.Add("anthropic-beta", "oauth-2025-04-20");
|
||||
|
||||
using var response = await _http.SendAsync(request, ct);
|
||||
if (response.StatusCode == HttpStatusCode.TooManyRequests)
|
||||
throw new UsageRateLimitedException(ReadRetryAfter(response));
|
||||
if (!response.IsSuccessStatusCode)
|
||||
throw new InvalidOperationException($"Usage request failed with status {(int)response.StatusCode}.");
|
||||
|
||||
@@ -43,6 +46,27 @@ public sealed class ClaudeOAuthUsageClient : IUsageClient
|
||||
return Parse(body);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads the <c>Retry-After</c> header in either form (delta-seconds or HTTP-date).
|
||||
/// A missing/past value returns null — the caller then uses its own backoff.
|
||||
/// </summary>
|
||||
internal static TimeSpan? ReadRetryAfter(HttpResponseMessage response)
|
||||
{
|
||||
var header = response.Headers.RetryAfter;
|
||||
if (header is null) return null;
|
||||
|
||||
if (header.Delta is { } delta)
|
||||
return delta > TimeSpan.Zero ? delta : null;
|
||||
|
||||
if (header.Date is { } date)
|
||||
{
|
||||
var remaining = date - DateTimeOffset.UtcNow;
|
||||
return remaining > TimeSpan.Zero ? remaining : null;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private string ReadAccessToken()
|
||||
{
|
||||
if (!File.Exists(_credentialsPath))
|
||||
@@ -162,3 +186,20 @@ public sealed class ClaudeOAuthUsageClient : IUsageClient
|
||||
? dto
|
||||
: null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The usage endpoint answered 429. Carries the server's <c>Retry-After</c> when it sent one so
|
||||
/// <see cref="UsageMonitorService"/> can honour it instead of guessing a backoff.
|
||||
/// </summary>
|
||||
public sealed class UsageRateLimitedException : InvalidOperationException
|
||||
{
|
||||
public UsageRateLimitedException(TimeSpan? retryAfter)
|
||||
: base(retryAfter is { } r
|
||||
? $"Usage request was rate-limited (429); retry after {(int)r.TotalSeconds}s."
|
||||
: "Usage request was rate-limited (429).")
|
||||
{
|
||||
RetryAfter = retryAfter;
|
||||
}
|
||||
|
||||
public TimeSpan? RetryAfter { get; }
|
||||
}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
namespace ClaudeDo.Worker.Usage.Interfaces;
|
||||
|
||||
/// <summary>
|
||||
/// Tells the usage monitor whether ClaudeDo is currently burning tokens, so it can poll
|
||||
/// often while work is in flight and back off to a slow heartbeat while idle.
|
||||
/// </summary>
|
||||
public interface IRunningTaskProbe
|
||||
{
|
||||
Task<bool> AnyRunningAsync(CancellationToken ct = default);
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
using ClaudeDo.Data;
|
||||
using ClaudeDo.Worker.Usage.Interfaces;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using TaskStatus = ClaudeDo.Data.Models.TaskStatus;
|
||||
|
||||
namespace ClaudeDo.Worker.Usage;
|
||||
|
||||
/// <summary>
|
||||
/// Answers "is anything running?" from the task table rather than the in-memory queue slots,
|
||||
/// so override-slot runs, continued runs, and runs still marked Running after a worker restart
|
||||
/// all count. A read failure reports idle — the usage monitor must never poll harder because
|
||||
/// its activity probe broke.
|
||||
/// </summary>
|
||||
public sealed class RunningTaskProbe : IRunningTaskProbe
|
||||
{
|
||||
private readonly IDbContextFactory<ClaudeDoDbContext> _dbFactory;
|
||||
|
||||
public RunningTaskProbe(IDbContextFactory<ClaudeDoDbContext> dbFactory) => _dbFactory = dbFactory;
|
||||
|
||||
public async Task<bool> AnyRunningAsync(CancellationToken ct = default)
|
||||
{
|
||||
try
|
||||
{
|
||||
await using var context = await _dbFactory.CreateDbContextAsync(ct);
|
||||
return await context.Tasks.AnyAsync(t => t.Status == TaskStatus.Running, ct);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -5,25 +5,43 @@ using ClaudeDo.Worker.Usage.Interfaces;
|
||||
namespace ClaudeDo.Worker.Usage;
|
||||
|
||||
/// <summary>
|
||||
/// Polls <see cref="IUsageClient"/> on <see cref="WorkerConfig.UsagePollIntervalSeconds"/> and keeps
|
||||
/// <see cref="UsageState"/> current. Polls once immediately at startup. A failure is logged as a
|
||||
/// warning at most once per distinct error message, to avoid log spam on a persistent outage.
|
||||
/// Broadcasts <see cref="HubBroadcaster.UsageUpdated"/> after every poll cycle, success or failure,
|
||||
/// so the UI can reflect a stale/blocked state as soon as it happens.
|
||||
/// Polls <see cref="IUsageClient"/> and keeps <see cref="UsageState"/> current. Polls once
|
||||
/// immediately at startup, then on an **activity-dependent** interval: while any task is
|
||||
/// Running it uses <see cref="WorkerConfig.UsagePollIntervalActiveSeconds"/>, otherwise the
|
||||
/// slower <see cref="WorkerConfig.UsagePollIntervalIdleSeconds"/>. A 429 adds exponential
|
||||
/// backoff on top (honouring <c>Retry-After</c> when the server sends one) — the endpoint is
|
||||
/// undocumented and rate-limits aggressively. <see cref="RefreshNowAsync"/> gives the UI a
|
||||
/// manual refresh that also resets the schedule, so the slow idle interval never leaves the
|
||||
/// user staring at a stale number.
|
||||
///
|
||||
/// A failure is logged as a warning at most once per distinct error message, to avoid log spam
|
||||
/// on a persistent outage. Broadcasts <see cref="HubBroadcaster.UsageUpdated"/> after every
|
||||
/// poll cycle, success or failure, so the UI can reflect a stale/blocked state as soon as it
|
||||
/// happens.
|
||||
/// </summary>
|
||||
public sealed class UsageMonitorService : BackgroundService
|
||||
{
|
||||
/// <summary>A manual refresh inside this window of the last poll reuses the last result.</summary>
|
||||
internal static readonly TimeSpan ManualRefreshCooldown = TimeSpan.FromSeconds(10);
|
||||
|
||||
private readonly IUsageClient _client;
|
||||
private readonly UsageState _state;
|
||||
private readonly WorkerConfig _config;
|
||||
private readonly ILogger<UsageMonitorService> _logger;
|
||||
private readonly UsageSnapshotBuilder _snapshotBuilder;
|
||||
private readonly HubBroadcaster _broadcaster;
|
||||
private readonly IRunningTaskProbe _runningProbe;
|
||||
|
||||
// Serializes the background loop against a manual refresh so two polls never overlap.
|
||||
private readonly SemaphoreSlim _pollLock = new(1, 1);
|
||||
private string? _lastLoggedError;
|
||||
private int _rateLimitStrikes;
|
||||
private DateTime _lastPollUtc = DateTime.MinValue;
|
||||
private DateTime _nextPollDueUtc = DateTime.MinValue;
|
||||
|
||||
public UsageMonitorService(
|
||||
IUsageClient client, UsageState state, WorkerConfig config, ILogger<UsageMonitorService> logger,
|
||||
UsageSnapshotBuilder snapshotBuilder, HubBroadcaster broadcaster)
|
||||
UsageSnapshotBuilder snapshotBuilder, HubBroadcaster broadcaster, IRunningTaskProbe runningProbe)
|
||||
{
|
||||
_client = client;
|
||||
_state = state;
|
||||
@@ -31,49 +49,135 @@ public sealed class UsageMonitorService : BackgroundService
|
||||
_logger = logger;
|
||||
_snapshotBuilder = snapshotBuilder;
|
||||
_broadcaster = broadcaster;
|
||||
_runningProbe = runningProbe;
|
||||
}
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
await TickAsync(stoppingToken);
|
||||
// Re-read the due time every iteration: a manual refresh pushes it out, which is
|
||||
// how the loop avoids polling again right after the user hit refresh.
|
||||
var wait = _nextPollDueUtc - DateTime.UtcNow;
|
||||
if (wait > TimeSpan.Zero)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(wait, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await Task.Delay(TimeSpan.FromSeconds(_config.UsagePollIntervalSeconds), stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
await TickAsync(stoppingToken);
|
||||
}
|
||||
}
|
||||
|
||||
internal async Task TickAsync(CancellationToken ct)
|
||||
/// <summary>
|
||||
/// Forces a poll now and returns the resulting snapshot DTO. Within
|
||||
/// <see cref="ManualRefreshCooldown"/> of the last poll the API call is skipped and the
|
||||
/// current state is returned instead, so click-spamming the refresh button can't earn a 429.
|
||||
/// </summary>
|
||||
public async Task<UsageSnapshotDto> RefreshNowAsync(CancellationToken ct = default)
|
||||
{
|
||||
await PollAsync(ct, ManualRefreshCooldown);
|
||||
return await _snapshotBuilder.BuildAsync(ct);
|
||||
}
|
||||
|
||||
internal Task TickAsync(CancellationToken ct) => PollAsync(ct, null);
|
||||
|
||||
private async Task PollAsync(CancellationToken ct, TimeSpan? skipIfPolledWithin)
|
||||
{
|
||||
await _pollLock.WaitAsync(ct);
|
||||
TimeSpan? retryAfter = null;
|
||||
try
|
||||
{
|
||||
var snapshot = await _client.GetUsageAsync(ct);
|
||||
_state.ReportSuccess(snapshot);
|
||||
_lastLoggedError = null;
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_state.ReportFailure(ex.Message, DateTime.UtcNow);
|
||||
// Checked under the lock so a manual refresh that queued behind a background poll
|
||||
// sees that poll's timestamp and reuses its result instead of firing a second call.
|
||||
if (skipIfPolledWithin is { } window && DateTime.UtcNow - _lastPollUtc < window)
|
||||
return;
|
||||
|
||||
if (_lastLoggedError != ex.Message)
|
||||
try
|
||||
{
|
||||
_logger.LogWarning(ex, "UsageMonitorService: failed to fetch usage: {Message}", ex.Message);
|
||||
_lastLoggedError = ex.Message;
|
||||
var snapshot = await _client.GetUsageAsync(ct);
|
||||
_state.ReportSuccess(snapshot);
|
||||
_lastLoggedError = null;
|
||||
_rateLimitStrikes = 0;
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (UsageRateLimitedException ex)
|
||||
{
|
||||
_rateLimitStrikes = Math.Min(_rateLimitStrikes + 1, UsagePollSchedule.MaxStrikes);
|
||||
retryAfter = ex.RetryAfter;
|
||||
RecordFailure(ex);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
RecordFailure(ex);
|
||||
}
|
||||
|
||||
_lastPollUtc = DateTime.UtcNow;
|
||||
_nextPollDueUtc = _lastPollUtc + await NextDelayAsync(retryAfter, ct);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_pollLock.Release();
|
||||
}
|
||||
|
||||
var dto = await _snapshotBuilder.BuildAsync(ct);
|
||||
await _broadcaster.UsageUpdated(dto);
|
||||
}
|
||||
|
||||
private void RecordFailure(Exception ex)
|
||||
{
|
||||
_state.ReportFailure(ex.Message, DateTime.UtcNow);
|
||||
|
||||
if (_lastLoggedError == ex.Message) return;
|
||||
_logger.LogWarning(ex, "UsageMonitorService: failed to fetch usage: {Message}", ex.Message);
|
||||
_lastLoggedError = ex.Message;
|
||||
}
|
||||
|
||||
private async Task<TimeSpan> NextDelayAsync(TimeSpan? retryAfter, CancellationToken ct)
|
||||
{
|
||||
var anyRunning = await _runningProbe.AnyRunningAsync(ct);
|
||||
return UsagePollSchedule.NextDelay(
|
||||
anyRunning,
|
||||
_config.UsagePollIntervalActiveSeconds,
|
||||
_config.UsagePollIntervalIdleSeconds,
|
||||
_rateLimitStrikes,
|
||||
retryAfter);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure poll-interval arithmetic for <see cref="UsageMonitorService"/>: active-vs-idle base
|
||||
/// interval plus 429 backoff. Kept static and side-effect-free so the schedule is testable
|
||||
/// without a running background service.
|
||||
/// </summary>
|
||||
internal static class UsagePollSchedule
|
||||
{
|
||||
/// <summary>Strike count is capped so the exponent can't run away on a long outage.</summary>
|
||||
internal const int MaxStrikes = 4;
|
||||
|
||||
/// <summary>Nothing ever waits longer than this, not even an absurd <c>Retry-After</c>.</summary>
|
||||
internal static readonly TimeSpan MaxDelay = TimeSpan.FromMinutes(30);
|
||||
|
||||
internal static TimeSpan NextDelay(
|
||||
bool anyTaskRunning, int activeSeconds, int idleSeconds, int rateLimitStrikes, TimeSpan? retryAfter)
|
||||
{
|
||||
var baseDelay = TimeSpan.FromSeconds(Math.Max(1, anyTaskRunning ? activeSeconds : idleSeconds));
|
||||
if (rateLimitStrikes <= 0)
|
||||
return baseDelay;
|
||||
|
||||
var backoff = retryAfter ?? baseDelay * Math.Pow(2, Math.Min(rateLimitStrikes, MaxStrikes));
|
||||
// Never poll *sooner* than the normal cadence just because Retry-After was small.
|
||||
if (backoff < baseDelay) backoff = baseDelay;
|
||||
return backoff > MaxDelay ? MaxDelay : backoff;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -39,7 +39,10 @@ public sealed class UsageSnapshotBuilder
|
||||
|
||||
var decision = await _gate.EvaluateAsync(ct);
|
||||
|
||||
var maxAge = TimeSpan.FromSeconds(_cfg.UsagePollIntervalSeconds * 3);
|
||||
// Measured against the *slowest* cadence — the idle interval — so a genuinely idle
|
||||
// worker on its 15-minute heartbeat isn't reported stale just for not polling.
|
||||
var maxAge = TimeSpan.FromSeconds(
|
||||
Math.Max(_cfg.UsagePollIntervalActiveSeconds, _cfg.UsagePollIntervalIdleSeconds) * 3);
|
||||
var isStale = snapshot is null || lastError is not null || (DateTime.UtcNow - snapshot.FetchedAtUtc) > maxAge;
|
||||
|
||||
var limits = (snapshot?.Limits ?? Array.Empty<UsageLimitRow>())
|
||||
|
||||
@@ -153,6 +153,7 @@ public abstract class StubWorkerClient : IWorkerClient
|
||||
public virtual Task ClearOnlineInboxAuthAsync() => Task.CompletedTask;
|
||||
|
||||
public virtual Task<UsageSnapshotDto?> GetUsageSnapshotAsync() => Task.FromResult<UsageSnapshotDto?>(null);
|
||||
public virtual Task<UsageSnapshotDto?> RefreshUsageAsync() => Task.FromResult<UsageSnapshotDto?>(null);
|
||||
public virtual Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsageAsync(DateOnly from, DateOnly to)
|
||||
=> Task.FromResult<IReadOnlyList<ModelUsageRowDto>>(Array.Empty<ModelUsageRowDto>());
|
||||
public virtual Task<IReadOnlyList<TaskUsageRowDto>> GetTaskUsageAsync(DateOnly from, DateOnly to)
|
||||
|
||||
@@ -29,8 +29,19 @@ public class UsageMonitorModalViewModelTests
|
||||
public int ModelUsageCalls;
|
||||
public int TaskUsageCalls;
|
||||
|
||||
public UsageSnapshotDto? RefreshedSnapshot;
|
||||
public int RefreshCalls;
|
||||
public Exception? RefreshThrows;
|
||||
|
||||
public override Task<UsageSnapshotDto?> GetUsageSnapshotAsync() => Task.FromResult(Snapshot);
|
||||
|
||||
public override Task<UsageSnapshotDto?> RefreshUsageAsync()
|
||||
{
|
||||
RefreshCalls++;
|
||||
if (RefreshThrows is not null) throw RefreshThrows;
|
||||
return Task.FromResult(RefreshedSnapshot);
|
||||
}
|
||||
|
||||
public override Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsageAsync(DateOnly from, DateOnly to)
|
||||
{
|
||||
ModelUsageCalls++;
|
||||
@@ -68,6 +79,60 @@ public class UsageMonitorModalViewModelTests
|
||||
isGateBlocked, gateReason, fetchedAtUtc ?? DateTime.UtcNow, isStale, lastError,
|
||||
configuredSlots, effectiveSlots, throttleBucket);
|
||||
|
||||
// ── Manual refresh ──────────────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public async Task Refresh_ReplacesSnapshotAndReloadsTables()
|
||||
{
|
||||
var worker = new FakeWorker
|
||||
{
|
||||
Snapshot = Snapshot(new[] { Limit("session") }),
|
||||
RefreshedSnapshot = Snapshot(new[] { Limit("session"), Limit("weekly_all") }),
|
||||
};
|
||||
var vm = new UsageMonitorModalViewModel(worker);
|
||||
await vm.LoadAsync();
|
||||
var callsAfterLoad = worker.ModelUsageCalls;
|
||||
|
||||
await vm.RefreshCommand.ExecuteAsync(null);
|
||||
|
||||
Assert.Equal(1, worker.RefreshCalls);
|
||||
Assert.Equal(2, vm.GaugeRows.Count);
|
||||
Assert.Equal(callsAfterLoad + 1, worker.ModelUsageCalls);
|
||||
Assert.False(vm.IsRefreshing);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Refresh_NullResult_KeepsPreviousSnapshot()
|
||||
{
|
||||
var worker = new FakeWorker { Snapshot = Snapshot(new[] { Limit("session") }) };
|
||||
var vm = new UsageMonitorModalViewModel(worker);
|
||||
await vm.LoadAsync();
|
||||
|
||||
await vm.RefreshCommand.ExecuteAsync(null);
|
||||
|
||||
Assert.Single(vm.GaugeRows);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Refresh_Failure_ReportsErrorAndClearsBusyFlag()
|
||||
{
|
||||
var worker = new FakeWorker
|
||||
{
|
||||
Snapshot = Snapshot(new[] { Limit("session") }),
|
||||
RefreshThrows = new InvalidOperationException("worker offline"),
|
||||
};
|
||||
var vm = new UsageMonitorModalViewModel(worker);
|
||||
await vm.LoadAsync();
|
||||
string? reported = null;
|
||||
vm.ErrorReported += m => reported = m;
|
||||
|
||||
await vm.RefreshCommand.ExecuteAsync(null);
|
||||
|
||||
Assert.NotNull(reported);
|
||||
Assert.Contains("worker offline", reported);
|
||||
Assert.False(vm.IsRefreshing);
|
||||
}
|
||||
|
||||
// ── Gauge label derivation ──────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -145,6 +145,7 @@ sealed class FakeWorkerClient : IWorkerClient
|
||||
public IReadOnlyList<ActiveTask> GetActiveTasks() => System.Array.Empty<ActiveTask>();
|
||||
|
||||
public Task<UsageSnapshotDto?> GetUsageSnapshotAsync() => Task.FromResult<UsageSnapshotDto?>(null);
|
||||
public Task<UsageSnapshotDto?> RefreshUsageAsync() => Task.FromResult<UsageSnapshotDto?>(null);
|
||||
public Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsageAsync(DateOnly from, DateOnly to)
|
||||
=> Task.FromResult<IReadOnlyList<ModelUsageRowDto>>(System.Array.Empty<ModelUsageRowDto>());
|
||||
public Task<IReadOnlyList<TaskUsageRowDto>> GetTaskUsageAsync(DateOnly from, DateOnly to)
|
||||
|
||||
@@ -44,6 +44,7 @@ public sealed class ClaudeOAuthUsageClientTests : IDisposable
|
||||
public List<HttpRequestMessage> Requests { get; } = new();
|
||||
public HttpStatusCode ResponseStatus { get; set; } = HttpStatusCode.OK;
|
||||
public string ResponseBody { get; set; } = "{}";
|
||||
public RetryConditionHeaderValue? RetryAfter { get; set; }
|
||||
|
||||
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken ct)
|
||||
{
|
||||
@@ -52,6 +53,7 @@ public sealed class ClaudeOAuthUsageClientTests : IDisposable
|
||||
{
|
||||
Content = new StringContent(ResponseBody, Encoding.UTF8, "application/json"),
|
||||
};
|
||||
if (RetryAfter is not null) resp.Headers.RetryAfter = RetryAfter;
|
||||
return Task.FromResult(resp);
|
||||
}
|
||||
}
|
||||
@@ -179,4 +181,43 @@ public sealed class ClaudeOAuthUsageClientTests : IDisposable
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() => client.GetUsageAsync());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetUsageAsync_429_ThrowsRateLimitedWithRetryAfterDelta()
|
||||
{
|
||||
WriteCredentials();
|
||||
var (client, handler) = Build();
|
||||
handler.ResponseStatus = HttpStatusCode.TooManyRequests;
|
||||
handler.RetryAfter = new RetryConditionHeaderValue(TimeSpan.FromSeconds(45));
|
||||
|
||||
var ex = await Assert.ThrowsAsync<UsageRateLimitedException>(() => client.GetUsageAsync());
|
||||
|
||||
Assert.Equal(TimeSpan.FromSeconds(45), ex.RetryAfter);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetUsageAsync_429_WithoutRetryAfter_HasNullRetryAfter()
|
||||
{
|
||||
WriteCredentials();
|
||||
var (client, handler) = Build();
|
||||
handler.ResponseStatus = HttpStatusCode.TooManyRequests;
|
||||
|
||||
var ex = await Assert.ThrowsAsync<UsageRateLimitedException>(() => client.GetUsageAsync());
|
||||
|
||||
Assert.Null(ex.RetryAfter);
|
||||
Assert.Contains("429", ex.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GetUsageAsync_429_WithPastRetryAfterDate_IgnoresIt()
|
||||
{
|
||||
WriteCredentials();
|
||||
var (client, handler) = Build();
|
||||
handler.ResponseStatus = HttpStatusCode.TooManyRequests;
|
||||
handler.RetryAfter = new RetryConditionHeaderValue(DateTimeOffset.UtcNow.AddMinutes(-5));
|
||||
|
||||
var ex = await Assert.ThrowsAsync<UsageRateLimitedException>(() => client.GetUsageAsync());
|
||||
|
||||
Assert.Null(ex.RetryAfter);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
using ClaudeDo.Data.Models;
|
||||
using ClaudeDo.Worker.Tests.Infrastructure;
|
||||
using ClaudeDo.Worker.Usage;
|
||||
using TaskStatus = ClaudeDo.Data.Models.TaskStatus;
|
||||
|
||||
namespace ClaudeDo.Worker.Tests.Usage;
|
||||
|
||||
public sealed class RunningTaskProbeTests : IDisposable
|
||||
{
|
||||
private readonly DbFixture _db = new();
|
||||
|
||||
public void Dispose() => _db.Dispose();
|
||||
|
||||
private async Task SeedTaskAsync(TaskStatus status)
|
||||
{
|
||||
await using var context = _db.CreateFactory().CreateDbContext();
|
||||
var list = new ListEntity { Id = Guid.NewGuid().ToString(), Name = "L", CreatedAt = DateTime.UtcNow };
|
||||
context.Lists.Add(list);
|
||||
context.Tasks.Add(new TaskEntity
|
||||
{
|
||||
Id = Guid.NewGuid().ToString(),
|
||||
ListId = list.Id,
|
||||
Title = $"task-{status}",
|
||||
Status = status,
|
||||
CreatedAt = DateTime.UtcNow,
|
||||
});
|
||||
await context.SaveChangesAsync();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnyRunningAsync_NoTasks_ReturnsFalse()
|
||||
{
|
||||
var probe = new RunningTaskProbe(_db.CreateFactory());
|
||||
|
||||
Assert.False(await probe.AnyRunningAsync());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnyRunningAsync_QueuedTaskOnly_ReturnsFalse()
|
||||
{
|
||||
await SeedTaskAsync(TaskStatus.Queued);
|
||||
var probe = new RunningTaskProbe(_db.CreateFactory());
|
||||
|
||||
Assert.False(await probe.AnyRunningAsync());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task AnyRunningAsync_RunningTask_ReturnsTrue()
|
||||
{
|
||||
await SeedTaskAsync(TaskStatus.Idle);
|
||||
await SeedTaskAsync(TaskStatus.Running);
|
||||
var probe = new RunningTaskProbe(_db.CreateFactory());
|
||||
|
||||
Assert.True(await probe.AnyRunningAsync());
|
||||
}
|
||||
}
|
||||
@@ -32,16 +32,25 @@ public sealed class UsageMonitorServiceTests : IDisposable
|
||||
Task.FromResult(new UsageGateDecision(false, null));
|
||||
}
|
||||
|
||||
private sealed class FakeRunningProbe : IRunningTaskProbe
|
||||
{
|
||||
public bool AnyRunning { get; set; }
|
||||
public Task<bool> AnyRunningAsync(CancellationToken ct = default) => Task.FromResult(AnyRunning);
|
||||
}
|
||||
|
||||
private static UsageSnapshot MakeSnapshot() => new(new UsageBucket(1, null), null, [], DateTime.UtcNow);
|
||||
|
||||
private (UsageMonitorService Service, UsageState State, CapturingHubContext Hub) CreateService(FakeClient client, WorkerConfig? cfg = null)
|
||||
private (UsageMonitorService Service, UsageState State, CapturingHubContext Hub) CreateService(
|
||||
FakeClient client, WorkerConfig? cfg = null, IRunningTaskProbe? probe = null)
|
||||
{
|
||||
var state = new UsageState();
|
||||
var config = cfg ?? new WorkerConfig();
|
||||
var builder = new UsageSnapshotBuilder(state, new FakeGate(), _db.CreateFactory(), config);
|
||||
var hubContext = new CapturingHubContext();
|
||||
var broadcaster = new HubBroadcaster(hubContext);
|
||||
var service = new UsageMonitorService(client, state, config, NullLogger<UsageMonitorService>.Instance, builder, broadcaster);
|
||||
var service = new UsageMonitorService(
|
||||
client, state, config, NullLogger<UsageMonitorService>.Instance, builder, broadcaster,
|
||||
probe ?? new FakeRunningProbe());
|
||||
return (service, state, hubContext);
|
||||
}
|
||||
|
||||
@@ -100,4 +109,94 @@ public sealed class UsageMonitorServiceTests : IDisposable
|
||||
var calls = hubContext.Proxy.Calls.Where(c => c.Method == "UsageUpdated").ToList();
|
||||
Assert.Equal(2, calls.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RefreshNowAsync_Polls_And_Returns_Fresh_Snapshot()
|
||||
{
|
||||
var client = new FakeClient();
|
||||
client.Results.Enqueue(MakeSnapshot);
|
||||
var (service, _, _) = CreateService(client);
|
||||
|
||||
var dto = await service.RefreshNowAsync();
|
||||
|
||||
Assert.Equal(1, client.CallCount);
|
||||
Assert.False(dto.IsStale);
|
||||
Assert.NotNull(dto.FiveHourPercent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RefreshNowAsync_WithinCooldown_ReusesLastPoll()
|
||||
{
|
||||
var client = new FakeClient();
|
||||
client.Results.Enqueue(MakeSnapshot);
|
||||
var (service, _, _) = CreateService(client);
|
||||
|
||||
await service.TickAsync(CancellationToken.None);
|
||||
var dto = await service.RefreshNowAsync();
|
||||
|
||||
// Second call would have thrown "no result queued" had it hit the client.
|
||||
Assert.Equal(1, client.CallCount);
|
||||
Assert.False(dto.IsStale);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RateLimit_Failure_Is_Recorded_As_Error()
|
||||
{
|
||||
var client = new FakeClient();
|
||||
client.Results.Enqueue(() => throw new UsageRateLimitedException(TimeSpan.FromSeconds(30)));
|
||||
var (service, state, _) = CreateService(client);
|
||||
|
||||
await service.TickAsync(CancellationToken.None);
|
||||
|
||||
Assert.NotNull(state.LastError);
|
||||
Assert.Contains("429", state.LastError);
|
||||
}
|
||||
}
|
||||
|
||||
public sealed class UsagePollScheduleTests
|
||||
{
|
||||
[Fact]
|
||||
public void Uses_active_interval_while_a_task_runs()
|
||||
{
|
||||
var delay = UsagePollSchedule.NextDelay(true, 300, 900, 0, null);
|
||||
Assert.Equal(TimeSpan.FromSeconds(300), delay);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Uses_idle_interval_while_nothing_runs()
|
||||
{
|
||||
var delay = UsagePollSchedule.NextDelay(false, 300, 900, 0, null);
|
||||
Assert.Equal(TimeSpan.FromSeconds(900), delay);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backs_off_exponentially_on_repeated_rate_limits()
|
||||
{
|
||||
var first = UsagePollSchedule.NextDelay(true, 300, 900, 1, null);
|
||||
var second = UsagePollSchedule.NextDelay(true, 300, 900, 2, null);
|
||||
|
||||
Assert.Equal(TimeSpan.FromSeconds(600), first);
|
||||
Assert.True(second > first);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Honours_retry_after_when_longer_than_the_base_interval()
|
||||
{
|
||||
var delay = UsagePollSchedule.NextDelay(true, 300, 900, 1, TimeSpan.FromSeconds(420));
|
||||
Assert.Equal(TimeSpan.FromSeconds(420), delay);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Never_polls_sooner_than_the_base_interval_after_a_rate_limit()
|
||||
{
|
||||
var delay = UsagePollSchedule.NextDelay(true, 300, 900, 1, TimeSpan.FromSeconds(5));
|
||||
Assert.Equal(TimeSpan.FromSeconds(300), delay);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Caps_the_backoff()
|
||||
{
|
||||
var delay = UsagePollSchedule.NextDelay(false, 300, 900, 4, TimeSpan.FromHours(4));
|
||||
Assert.Equal(UsagePollSchedule.MaxDelay, delay);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -96,7 +96,7 @@ public sealed class UsageSnapshotBuilderTests : IDisposable
|
||||
public async Task Snapshot_older_than_4x_poll_interval_is_stale()
|
||||
{
|
||||
await SetThresholdsAsync(80, 90);
|
||||
var cfg = new WorkerConfig { UsagePollIntervalSeconds = 60 };
|
||||
var cfg = new WorkerConfig { UsagePollIntervalActiveSeconds = 60, UsagePollIntervalIdleSeconds = 60 };
|
||||
var state = new UsageState();
|
||||
state.ReportSuccess(new UsageSnapshot(
|
||||
new UsageBucket(10, null), null, Array.Empty<UsageLimitRow>(),
|
||||
@@ -113,7 +113,7 @@ public sealed class UsageSnapshotBuilderTests : IDisposable
|
||||
public async Task Fresh_snapshot_is_not_stale()
|
||||
{
|
||||
await SetThresholdsAsync(80, 90);
|
||||
var cfg = new WorkerConfig { UsagePollIntervalSeconds = 60 };
|
||||
var cfg = new WorkerConfig { UsagePollIntervalActiveSeconds = 60, UsagePollIntervalIdleSeconds = 60 };
|
||||
var state = new UsageState();
|
||||
state.ReportSuccess(new UsageSnapshot(
|
||||
new UsageBucket(10, null), null, Array.Empty<UsageLimitRow>(),
|
||||
|
||||
Reference in New Issue
Block a user