fix(usage): stop 429s with an activity-dependent poll cadence + manual refresh

The usage monitor polled the undocumented OAuth usage endpoint every 60s and
earned 429s. It now polls every 5 min while any task is Running and every
15 min while idle (usage_poll_interval_active_seconds / _idle_seconds, both
clamped to >= 60; the old single usage_poll_interval_seconds key is gone).

A 429 comes back as UsageRateLimitedException carrying Retry-After and adds
exponential backoff on top, capped at 30 min and never shorter than the normal
cadence; the strike count resets on the first success. The schedule arithmetic
is the pure static UsagePollSchedule.NextDelay.

Since the idle cadence is slow on purpose, WorkerHub.RefreshUsage drives
UsageMonitorService.RefreshNowAsync behind a Refresh now button in the Usage
Monitor modal: an out-of-band poll that pushes the loop's next-due time out so
no double poll follows, with a 10s cooldown so click-spam can't earn a 429.

Staleness now measures against the slower (idle) interval so an idle worker
isn't flagged stale just for not polling.
This commit is contained in:
mika kuns
2026-08-05 16:40:34 +02:00
parent f6cb8250bb
commit 2700c3d817
24 changed files with 615 additions and 50 deletions
+2 -2
View File
@@ -40,13 +40,13 @@ Design/ — Tokens.axaml (design tokens; merged before styles) + IslandStyle
- **TaskRowViewModel** / **ListNavItemViewModel** — lightweight display VMs (task row: status, planning phase, parent/blocked links, roadblock count, computed `IsDraft`/`IsPlanned`/`IsChild`/`IsPlanningParent`/`CanRefine`; plus `IsManual` (→ MANUAL badge; suppresses `CanSendToQueue`/`CanRefine`/`CanOpenPlanningSession`) and `HasInteractiveSession` (→ accent "Interactive" chip instead of "Parked"; tapping it jumps to that Mission Control pane); list row: kind Smart/Virtual/User, count, icon/dot keys, drop hints, `IsManual`).
- **NotesEditorViewModel** — day navigator + bullet CRUD for daily notes via `INotesApi`.
- **UsagePillViewModel** — one shared instance backs the `UsagePill` control hosted in both the footer and the Mission Control header; loads via `GetUsageSnapshotAsync` and updates live off `IWorkerClient.UsageUpdatedEvent`; derives display text, tooltip, and dot state (normal/warn/stale/blocked, mutually exclusive priority blocked > stale > warn > normal) from the shared `UsageSnapshotDto`. `IsThrottled` (effective slots below configured, and not gate-blocked) adds a tooltip line naming the effective/configured slot count and the decisive bucket (`ThrottleBucket` on the DTO — `"five_hour"`/`"seven_day"`).
- **Modal VMs** — `SettingsModalViewModel` (four tabs: General, Worktrees, Files prompt-paths, Prime Claude incl. `DailyPrepMaxTasks` + prime-schedule rows). General hosts the per-model preset table (`ModelPresets``ModelPresetRowViewModel`: effort + max turns per alias) which **replaced** the single global "Max turns" field, `ListSettingsModalViewModel` (name, working dir, commit type, "manual list" flag, `VerifyCommand` (optional post-merge verify gate, own field/section — not part of `AgentConfigEditorViewModel`), delete list; hosts shared `AgentConfigEditorViewModel` as `Agent` property (scope=List) — save delegates to `Agent.SaveAsync(verifyCommand)`, since both fields land in the same `list_config` row via one `UpdateListConfig` call and would otherwise clobber each other), `RepoImportModalViewModel` (bulk-create lists from git repos found under chosen parents; already-wired repos disabled), `WeeklyReportModalViewModel` (range pickers default "since last standup weekday → today", cached per range, markdown via MarkdownView), `MergeModalViewModel` (single-task merge form, called from the diff modal), `WorktreesOverviewModalViewModel` (global/per-list worktree rows, batch merge + state ops), `UnfinishedPlanningModalViewModel` (Resume/FinalizeNow/Discard for a draft planning session), `MergeHelperSelectionModalViewModel` ("Let Claude handle it": checkbox picker over one list's non-terminal, non-manual tasks, pre-ticks the actionable ones; list-scoped only — `Configure(listId, listName)`, no global scope. Opened from the list row's context menu, which is hidden when the list has no working dir; on confirm `ListsIslandViewModel` raises `LetClaudeHandleRequested` → shell → `MissionControlViewModel.OpenMergeHelperConPtySessionAsync`, which first calls `IWorkerClient.CreateMergeHelperTaskAsync` to create one new ClaudeDo task per run in that list — `Idle`/`IsManual=true` (never queued), title/description localized (`missionControl.mergeHelperTaskTitle`/`mergeHelperTaskDescriptionHeader`), `TaskEntity.HandlerBaseCommit` stamped to the list repo's current HEAD — then opens a **task-based** ConPTY tile for it (deduped by `TaskId` like `OpenConPtySessionAsync`, not `CreateAdHoc`) running the five-phase handler prompt. The handler still merges the tasks it handles itself; the host task never gets a worktree of its own, so "Submit for review" stamps `HandlerHeadCommit` instead of committing a worktree, and the detail pane's `MergeSectionViewModel` falls back to `HandlerBaseCommit`/`HandlerHeadCommit` over the list's working dir for its diff — see `TaskEntity` in `ClaudeDo.Data/CLAUDE.md`), `WorkerConnectionModalViewModel` (offline help), `AboutModalViewModel`, `LogVisualizerViewModel` (worker logs, last 30 min, all levels + a warn/error-only filter; loads via `GetRecentLogsAsync`), `UsageMonitorModalViewModel` (opened from the usage pill; renders one gauge per row in `UsageSnapshotDto.Limits`**dynamic**, since the fixed `seven_day_opus`/`seven_day_sonnet`-style buckets the raw Anthropic API can return are plan-dependent and come back `null` on plans that don't have them, so a fixed gauge layout would break; also shows model usage (`GetModelUsageAsync`, ClaudeDo-vs-Other split per model) and top-task usage (`GetTaskUsageAsync`) tables over a 7d/30d preset or custom date range).
- **Modal VMs** — `SettingsModalViewModel` (four tabs: General, Worktrees, Files prompt-paths, Prime Claude incl. `DailyPrepMaxTasks` + prime-schedule rows). General hosts the per-model preset table (`ModelPresets``ModelPresetRowViewModel`: effort + max turns per alias) which **replaced** the single global "Max turns" field, `ListSettingsModalViewModel` (name, working dir, commit type, "manual list" flag, `VerifyCommand` (optional post-merge verify gate, own field/section — not part of `AgentConfigEditorViewModel`), delete list; hosts shared `AgentConfigEditorViewModel` as `Agent` property (scope=List) — save delegates to `Agent.SaveAsync(verifyCommand)`, since both fields land in the same `list_config` row via one `UpdateListConfig` call and would otherwise clobber each other), `RepoImportModalViewModel` (bulk-create lists from git repos found under chosen parents; already-wired repos disabled), `WeeklyReportModalViewModel` (range pickers default "since last standup weekday → today", cached per range, markdown via MarkdownView), `MergeModalViewModel` (single-task merge form, called from the diff modal), `WorktreesOverviewModalViewModel` (global/per-list worktree rows, batch merge + state ops), `UnfinishedPlanningModalViewModel` (Resume/FinalizeNow/Discard for a draft planning session), `MergeHelperSelectionModalViewModel` ("Let Claude handle it": checkbox picker over one list's non-terminal, non-manual tasks, pre-ticks the actionable ones; list-scoped only — `Configure(listId, listName)`, no global scope. Opened from the list row's context menu, which is hidden when the list has no working dir; on confirm `ListsIslandViewModel` raises `LetClaudeHandleRequested` → shell → `MissionControlViewModel.OpenMergeHelperConPtySessionAsync`, which first calls `IWorkerClient.CreateMergeHelperTaskAsync` to create one new ClaudeDo task per run in that list — `Idle`/`IsManual=true` (never queued), title/description localized (`missionControl.mergeHelperTaskTitle`/`mergeHelperTaskDescriptionHeader`), `TaskEntity.HandlerBaseCommit` stamped to the list repo's current HEAD — then opens a **task-based** ConPTY tile for it (deduped by `TaskId` like `OpenConPtySessionAsync`, not `CreateAdHoc`) running the five-phase handler prompt. The handler still merges the tasks it handles itself; the host task never gets a worktree of its own, so "Submit for review" stamps `HandlerHeadCommit` instead of committing a worktree, and the detail pane's `MergeSectionViewModel` falls back to `HandlerBaseCommit`/`HandlerHeadCommit` over the list's working dir for its diff — see `TaskEntity` in `ClaudeDo.Data/CLAUDE.md`), `WorkerConnectionModalViewModel` (offline help), `AboutModalViewModel`, `LogVisualizerViewModel` (worker logs, last 30 min, all levels + a warn/error-only filter; loads via `GetRecentLogsAsync`), `UsageMonitorModalViewModel` (opened from the usage pill; renders one gauge per row in `UsageSnapshotDto.Limits`**dynamic**, since the fixed `seven_day_opus`/`seven_day_sonnet`-style buckets the raw Anthropic API can return are plan-dependent and come back `null` on plans that don't have them, so a fixed gauge layout would break; also shows model usage (`GetModelUsageAsync`, ClaudeDo-vs-Other split per model) and top-task usage (`GetTaskUsageAsync`) tables over a 7d/30d preset or custom date range. A **Refresh now** button (`RefreshCommand`/`IsRefreshing`) drives `RefreshUsageAsync` — the worker deliberately polls slowly (5 min while a task runs, 15 min idle) to stay clear of the endpoint's 429s, so this is the only way to pull a current number on demand).
- **Diff stack** — `UnifiedDiffParser` (static; parses `git diff` output into `DiffFileViewModel`s, detecting added/deleted/renamed/binary files and per-line numbers; `Flatten` injects file-header rows for a combined single-pane view). `DiffModels.cs` holds shared types: `DiffLineViewModel`, `DiffFileViewModel`, `DiffLineKind`, `DiffFileStatus`, `SubtaskDiffRow`, `DiffTreeNodeViewModel`, `DiffTree`. `DiffViewerViewModel` is a single unified read-only diff viewer with two modes: **Files** (dirty worktree / branch-vs-base / commit-range — loads via GitService, shows a folder file-tree on the left + per-file diff pane on the right, Merge button for live branch source) and **Planning** (per-subtask diffs via `GetPlanningAggregateAsync`, subtask list left + flat diff right, combined integration-branch toggle). The Merge button opens the merge form, which routes to `ConflictResolverViewModel` on conflict. `DiffLinesView` renders per-file diff content with binary/empty placeholders.
- **Conflicts** — `ConflictResolverViewModel` (in-app **Rider-style 3-pane merge editor** for both single-task and planning unit-merge conflicts: single-task starts the conflict merge, parses each conflicted file into stable/conflict `MergeFileSegment`s via the worker's `GetMergeConflictDocuments`; exposes the active file's three reconstructed documents — `ActiveOursText` / `ActiveResultText` / `ActiveTheirsText` (from `MergeFile.OursText/ResultText/TheirsText`; Result seeds unresolved conflicts with Ours) — plus `ActiveFile`/`SelectFileCommand` (multi-file switcher), `Current`/`Next`/`Previous` (focused-conflict nav), a per-active-file `PositionText` readout, per-block `AcceptOurs/Theirs/Both/Base` + `MergeFile.Compose`, and `CanContinue` gated on every file resolved + no binary; writes each file via `WriteConflictResolution`, continue/abort; **planning mode** via `OpenForPlanningAsync(parentId, subtaskId)` loads the current subtask's mid-merge conflicts without re-starting the merge and routes continue/abort to `ContinuePlanningMerge`/`AbortPlanningMerge`, so a unit-merge conflict re-opens the editor per subtask via the `PlanningMergeConflict` broadcast). The view (`Views/Conflicts/ConflictResolverView`) shows the whole file in three **AvaloniaEdit** panes — MAIN/ours (read-only) | editable Result | INCOMING/theirs (read-only) — with TextMate highlighting by extension (theme `StyleInclude` in `App.axaml`); a code-behind `IBackgroundRenderer` tints each conflict block (unresolved/resolved) across panes, an `IReadOnlySectionProvider` + `TextAnchor` regions keep only conflict spans editable in Result (edits flow back to the block); each unresolved conflict starts EMPTY (a thin marker bar); the between-pane gutter controls **toggle** each side in/out of the result — ``/`` add MAIN/INCOMING in click order (first pick on top), clicking again removes that side — so a conflict can take main, incoming, both, or neither; a `FilesSummary` readout shows how many files still have conflicts, and the three panes share a proportional synced vertical scroll. A conflict overview ruler right of the Result pane (`ConflictMap`) maps every conflict in the file proportionally (click a tick to jump) — handy for long files. Conflict block tints live in `Tokens.axaml` (`Merge*TintBrush`). The editor is reached from review **Approve** on conflict and from the **Merge** button in the Diff window (a conflicting `MergeTask` hands off to the resolver via `RequestConflictResolution`).
## Services
- **WorkerClient** / **IWorkerClient** — SignalR client connecting to `http://127.0.0.1:47821/hub`, auto-reconnect with exponential backoff. The surface tracks `WorkerHub` (see `src/ClaudeDo.Worker/CLAUDE.md` for the canonical method/event list); groups: task execution (RunNow/Cancel/Continue/Reset/SetTaskStatus), review (`ApproveReviewAsync(taskId, targetBranch) -> MergeResultDto`, reject-to-queue/idle, cancel review, `PreviewMergeAsync -> MergePreviewDto`), planning sessions (start/resume/discard/finalize, queue subtasks, pending draft count, refine), embedded ConPTY launch specs (`GetInteractiveLaunchSpecAsync`/`GetAdHocLaunchSpecAsync`), planning aggregate/integration-branch diffs, unit-merge continue/abort, single-task conflict resolving (start/get-conflict-documents/write-resolution/continue/abort), worktrees (overview, set state, force remove, cleanup, reset all), agents, app settings, lists/config, weekly report, daily notes, daily prep (`RunDailyPrepNowAsync`, `ClearMyDayAsync`, `GetLastPrepLogAsync`), prime schedules, recent worker logs (`GetRecentLogsAsync`), usage monitoring (`GetUsageSnapshotAsync -> UsageSnapshotDto?`, `GetModelUsageAsync(from, to)`, `GetTaskUsageAsync(from, to)`). Events mirror `HubBroadcaster` (task/worktree/list/run updates, prep events, planning-merge events, refine events, worker log, `UsageUpdatedEvent`). Lifecycle (`StartAsync`/`StopAsync`) and a few admin methods live only on the concrete `WorkerClient`.
- **WorkerClient** / **IWorkerClient** — SignalR client connecting to `http://127.0.0.1:47821/hub`, auto-reconnect with exponential backoff. The surface tracks `WorkerHub` (see `src/ClaudeDo.Worker/CLAUDE.md` for the canonical method/event list); groups: task execution (RunNow/Cancel/Continue/Reset/SetTaskStatus), review (`ApproveReviewAsync(taskId, targetBranch) -> MergeResultDto`, reject-to-queue/idle, cancel review, `PreviewMergeAsync -> MergePreviewDto`), planning sessions (start/resume/discard/finalize, queue subtasks, pending draft count, refine), embedded ConPTY launch specs (`GetInteractiveLaunchSpecAsync`/`GetAdHocLaunchSpecAsync`), planning aggregate/integration-branch diffs, unit-merge continue/abort, single-task conflict resolving (start/get-conflict-documents/write-resolution/continue/abort), worktrees (overview, set state, force remove, cleanup, reset all), agents, app settings, lists/config, weekly report, daily notes, daily prep (`RunDailyPrepNowAsync`, `ClearMyDayAsync`, `GetLastPrepLogAsync`), prime schedules, recent worker logs (`GetRecentLogsAsync`), usage monitoring (`GetUsageSnapshotAsync -> UsageSnapshotDto?`, `RefreshUsageAsync -> UsageSnapshotDto?` (manual out-of-band poll), `GetModelUsageAsync(from, to)`, `GetTaskUsageAsync(from, to)`). Events mirror `HubBroadcaster` (task/worktree/list/run updates, prep events, planning-merge events, refine events, worker log, `UsageUpdatedEvent`). Lifecycle (`StartAsync`/`StopAsync`) and a few admin methods live only on the concrete `WorkerClient`.
- **INotesApi** / **WorkerNotesApi** — daily-note CRUD (`ListAsync(day)`, `AddAsync`, `UpdateAsync`, `DeleteAsync`); UI DTO `DailyNoteDto(Id, Date, Text, SortOrder)`.
- **IPrimeScheduleApi** — prime-schedule CRUD (`ListAsync`, `UpsertAsync`, `DeleteAsync`).
- **UpdateCheckService** — polls releases, exposes `LastCheckStatus`/`LatestVersion`/`CheckNowAsync` (feeds the shell's update banner).
@@ -143,6 +143,9 @@ public interface IWorkerClient : INotifyPropertyChanged
/// <summary>Raised whenever the worker's usage poller ticks (success or failure).</summary>
event Action<UsageSnapshotDto>? UsageUpdatedEvent;
Task<UsageSnapshotDto?> GetUsageSnapshotAsync();
/// <summary>Forces an out-of-band usage poll on the worker and returns the fresh snapshot.</summary>
Task<UsageSnapshotDto?> RefreshUsageAsync();
Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsageAsync(DateOnly from, DateOnly to);
Task<IReadOnlyList<TaskUsageRowDto>> GetTaskUsageAsync(DateOnly from, DateOnly to);
}
+3
View File
@@ -582,6 +582,9 @@ public partial class WorkerClient : ObservableObject, IAsyncDisposable, IWorkerC
public Task<UsageSnapshotDto?> GetUsageSnapshotAsync()
=> TryInvokeAsync<UsageSnapshotDto>("GetUsageSnapshot");
public Task<UsageSnapshotDto?> RefreshUsageAsync()
=> TryInvokeAsync<UsageSnapshotDto>("RefreshUsage");
public async Task<IReadOnlyList<ModelUsageRowDto>> GetModelUsageAsync(DateOnly from, DateOnly to)
=> await TryInvokeAsync<List<ModelUsageRowDto>>("GetModelUsage", from, to) ?? [];
@@ -40,6 +40,8 @@ public sealed partial class UsageMonitorModalViewModel : ViewModelBase
[NotifyPropertyChangedFor(nameof(ModelsEmpty))]
private bool _isBusy;
[ObservableProperty] private bool _isRefreshing;
[ObservableProperty]
[NotifyPropertyChangedFor(nameof(ModelsEmpty))]
private IReadOnlyList<ModelUsageDisplayRow> _modelRows = Array.Empty<ModelUsageDisplayRow>();
@@ -92,6 +94,28 @@ public sealed partial class UsageMonitorModalViewModel : ViewModelBase
private void OnUsageUpdated(UsageSnapshotDto snapshot) => Snapshot = snapshot;
/// <summary>
/// Manual refresh: the worker polls usage on a slow cadence (15 min idle / 5 min while a
/// task runs) to stay clear of the endpoint's 429s, so this is the way to get a number now.
/// </summary>
[RelayCommand]
private async Task RefreshAsync()
{
if (IsRefreshing) return;
IsRefreshing = true;
try
{
var snapshot = await _worker.RefreshUsageAsync();
if (snapshot is not null) Snapshot = snapshot;
await LoadUsageDataAsync();
}
catch (Exception ex)
{
ErrorReported?.Invoke(Loc.T("vm.usageMonitor.refreshFailed", ex.Message));
}
finally { IsRefreshing = false; }
}
private void ApplyPresetRange(int days)
{
var today = DateOnly.FromDateTime(DateTime.Today);
@@ -60,6 +60,18 @@
</Border>
</StackPanel>
<!-- Refresh: the worker polls on a slow cadence to avoid the endpoint's 429s -->
<StackPanel DockPanel.Dock="Top" Orientation="Horizontal" Spacing="8"
Margin="20,12,20,0" VerticalAlignment="Center">
<Button Classes="btn" Content="{loc:Tr modals.usageMonitor.refresh}"
Command="{Binding RefreshCommand}"
IsEnabled="{Binding !IsRefreshing}"/>
<Ellipse Classes="spinner" Width="14" Height="14" VerticalAlignment="Center"
IsVisible="{Binding IsRefreshing}"/>
<TextBlock Classes="meta" VerticalAlignment="Center"
Text="{loc:Tr modals.usageMonitor.refreshHint}"/>
</StackPanel>
<!-- Gauges -->
<ItemsControl DockPanel.Dock="Top" Margin="20,12,20,0" ItemsSource="{Binding GaugeRows}">
<ItemsControl.ItemsPanel>