fix(usage): stop 429s with an activity-dependent poll cadence + manual refresh

The usage monitor polled the undocumented OAuth usage endpoint every 60s and
earned 429s. It now polls every 5 min while any task is Running and every
15 min while idle (usage_poll_interval_active_seconds / _idle_seconds, both
clamped to >= 60; the old single usage_poll_interval_seconds key is gone).

A 429 comes back as UsageRateLimitedException carrying Retry-After and adds
exponential backoff on top, capped at 30 min and never shorter than the normal
cadence; the strike count resets on the first success. The schedule arithmetic
is the pure static UsagePollSchedule.NextDelay.

Since the idle cadence is slow on purpose, WorkerHub.RefreshUsage drives
UsageMonitorService.RefreshNowAsync behind a Refresh now button in the Usage
Monitor modal: an out-of-band poll that pushes the loop's next-due time out so
no double poll follows, with a 10s cooldown so click-spam can't earn a 429.

Staleness now measures against the slower (idle) interval so an idle worker
isn't flagged stale just for not polling.
This commit is contained in:
mika kuns
2026-08-05 16:40:34 +02:00
parent f6cb8250bb
commit 2700c3d817
24 changed files with 615 additions and 50 deletions
@@ -1,3 +1,4 @@
using System.Net;
using System.Net.Http.Headers;
using System.Text.Json;
using System.Text.Json.Nodes;
@@ -36,6 +37,8 @@ public sealed class ClaudeOAuthUsageClient : IUsageClient
request.Headers.Add("anthropic-beta", "oauth-2025-04-20");
using var response = await _http.SendAsync(request, ct);
if (response.StatusCode == HttpStatusCode.TooManyRequests)
throw new UsageRateLimitedException(ReadRetryAfter(response));
if (!response.IsSuccessStatusCode)
throw new InvalidOperationException($"Usage request failed with status {(int)response.StatusCode}.");
@@ -43,6 +46,27 @@ public sealed class ClaudeOAuthUsageClient : IUsageClient
return Parse(body);
}
/// <summary>
/// Reads the <c>Retry-After</c> header in either form (delta-seconds or HTTP-date).
/// A missing/past value returns null — the caller then uses its own backoff.
/// </summary>
internal static TimeSpan? ReadRetryAfter(HttpResponseMessage response)
{
var header = response.Headers.RetryAfter;
if (header is null) return null;
if (header.Delta is { } delta)
return delta > TimeSpan.Zero ? delta : null;
if (header.Date is { } date)
{
var remaining = date - DateTimeOffset.UtcNow;
return remaining > TimeSpan.Zero ? remaining : null;
}
return null;
}
private string ReadAccessToken()
{
if (!File.Exists(_credentialsPath))
@@ -162,3 +186,20 @@ public sealed class ClaudeOAuthUsageClient : IUsageClient
? dto
: null;
}
/// <summary>
/// The usage endpoint answered 429. Carries the server's <c>Retry-After</c> when it sent one so
/// <see cref="UsageMonitorService"/> can honour it instead of guessing a backoff.
/// </summary>
public sealed class UsageRateLimitedException : InvalidOperationException
{
public UsageRateLimitedException(TimeSpan? retryAfter)
: base(retryAfter is { } r
? $"Usage request was rate-limited (429); retry after {(int)r.TotalSeconds}s."
: "Usage request was rate-limited (429).")
{
RetryAfter = retryAfter;
}
public TimeSpan? RetryAfter { get; }
}