releases/ClaudeDo
2
0
Fork 0
Code Issues 1 Pull Requests 2 Actions Packages Projects Releases 39 Wiki Activity

feat(worker): Handler-Session vor dem Usage-Limit warnen und Handoff bis zum Reset zurueckhalten

Browse Source 
Eine PowerUse-Handler-Session soll am 5h-Limit nicht mehr mitten im Merge
sterben, sondern vorher selbst sauber uebergeben:

- HandlerUsageGuard (neu) warnt die laufende Session genau einmal pro Runde,
  sobald FiveHour.Utilization den Schwellwert PowerUseHandoffPct erreicht.
  Die Warnung wird per HandlerUsageWarning-Broadcast an Mission Control
  geschickt und dort in das Pane der Handler-TaskId getippt.
- handoff_list_handler broadcastet den Handoff dann nicht sofort, sondern parkt
  ihn bis ResetsAt + 60 s; das Ergebnis traegt deferredUntil. Ohne ResetsAt
  wird wie bisher sofort gesendet.
- Der geparkte Handoff liegt im HandlerRoundState (handler-rounds.json) und
  wird vom Guard-Tick ausgeloest, also auch nach einem Worker-Neustart wieder
  eingeplant bzw. sofort gesendet, wenn er ueberfaellig ist.
- Review-Befund aus #272: HandlerRestartCoordinator wartet nicht mehr selbst
  bis ResetsAt, sondern parkt den Neustart im selben Slot. AutoRestartUsed wird
  erst gesetzt, wenn der Broadcast wirklich rausging - ein Worker-Absturz
  waehrend der Wartezeit laesst die Runde nicht mehr dauerhaft tot zurueck.
  Zurueckgehaltener Handoff und Auto-Neustart teilen sich einen Slot, also
  oeffnet nie beides dieselbe Runde. UsageLimitProofPct entfaellt zugunsten von
  PowerUseHandoffPct.
- PtyTerminalSession.SendLineAsync schreibt in die laufende Session. Gegen eine
  echte claude-Session verifiziert: Text und Enter muessen zwei getrennte
  Writes sein, die Enter-Bytes kommen von Terminal.GenerateKeyInput.
This commit is contained in:
mika kuns
2026-09-02 21:35:29 +02:00
parent c3a418f077
commit fe8186514f
19 changed files with 851 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ClaudeDo.Worker/Runner/HandlerRestartCoordinator.cs
+30 -48
View File
@@ -20,17 +20,14 @@ namespace ClaudeDo.Worker.Runner;
///
/// Fails CLOSED, unlike the usage gate: every missing signal means "don't relaunch". An interactive
/// session that pops back up at 3am uninvited is worse than one that stays dead.
///
/// It does not wait here. The restart is <b>parked in <see cref="HandlerRoundState"/></b> and fired
/// by <see cref="HandlerUsageGuard"/> — the same slot a session that handed off cleanly before the
/// limit uses, so exactly one of the two reopens a round. Parking it also survives a worker restart
/// during the hours-long wait, which an in-process <c>Task.Delay</c> did not.
/// </summary>
public sealed class HandlerRestartCoordinator
{
// The corroborating signal that this exit really was the usage limit. A ConPTY session gives us
// nothing but an exit code -- there is no stdout to match against, so TaskRunner's
// IsUsageLimitMessage regex (which reads the CLI's own "session limit reached" text) has nothing
// to read here. The 5h utilization at the moment of death is the one independent witness we do
// have, and it is pinned at 100% when the limit is what stopped the session. A crash, a manual
// Ctrl-C, or a bad launch at 40% utilization is therefore NOT restarted.
internal const int UsageLimitProofPct = 95;
private readonly IDbContextFactory<ClaudeDoDbContext> _dbFactory;
private readonly UsageState _usageState;
private readonly HandlerRoundState _rounds;
@@ -51,14 +48,10 @@ public sealed class HandlerRestartCoordinator
_logger = logger;
}
/// <summary>Test seams — a real 5h wait is not a test.</summary>
internal Func<DateTimeOffset> NowUtc { get; set; } = () => DateTimeOffset.UtcNow;
internal Func<TimeSpan, CancellationToken, Task> Delay { get; set; } = (d, ct) => Task.Delay(d, ct);
/// <summary>Handles a ConPTY session's process exit. Returns true when a restart was claimed and
/// carried out (the returned task completes only after the wait, so a caller that wants
/// fire-and-forget must not await it). Any non-handler task, clean exit, non-usage-limit death,
/// PowerUse-off list, or already-used round is a silent no-op.</summary>
/// <summary>Handles a ConPTY session's process exit. Returns true when a restart was parked for
/// the reset (HandlerUsageGuard fires it). Any non-handler task, clean exit, non-usage-limit
/// death, PowerUse-off list, already-used round, or round that already has a handoff parked is a
/// silent no-op.</summary>
public async Task<bool> OnSessionExitedAsync(string taskId, int exitCode, CancellationToken ct = default)
{
if (exitCode == 0 || string.IsNullOrEmpty(taskId)) return false;
@@ -71,7 +64,6 @@ public sealed class HandlerRestartCoordinator
var snapshot = _usageState.Snapshot;
if (snapshot?.FiveHour is not { } fiveHour) return false;
if (fiveHour.Utilization < UsageLimitProofPct) return false;
if (fiveHour.ResetsAt is not { } resetsAt) return false;
int number;
@@ -85,8 +77,18 @@ public sealed class HandlerRestartCoordinator
// Both PowerUse keys, exactly as ComposeHandlerPromptAsync gates the PowerUse prompt
// addendum: the global switch alone must not make an interactive run on some unrelated
// list restart itself overnight.
if (!(await new AppSettingsRepository(ctx).GetAsync(ct)).PowerUseEnabled) return false;
var settings = await new AppSettingsRepository(ctx).GetAsync(ct);
if (!settings.PowerUseEnabled) return false;
if ((await new ListRepository(ctx).GetConfigAsync(task.ListId, ct))?.PowerUseAllowed != true) return false;
// The corroborating signal that this exit really was the usage limit. A ConPTY session
// gives us nothing but an exit code -- there is no stdout to match against, so
// TaskRunner's IsUsageLimitMessage regex (which reads the CLI's own "session limit
// reached" text) has nothing to read here. The 5h utilization at the moment of death is
// the one independent witness we do have. Same threshold the warning and the withheld
// handoff use (PowerUseHandoffPct) so all three agree on where "at the limit" starts;
// a crash or a manual Ctrl-C well below it is NOT restarted.
if (fiveHour.Utilization < Math.Clamp(settings.PowerUseHandoffPct, 50, 99)) return false;
}
catch (Exception ex)
{
@@ -94,36 +96,16 @@ public sealed class HandlerRestartCoordinator
return false;
}
// One restart per round, claimed before the wait -- a restart that dies again lands back
// here and must stop, not relaunch.
if (!_rounds.TryClaimAutoRestart(taskId)) return false;
// One restart per round, and never on top of a handoff the session itself already parked.
// The budget is NOT spent here -- HandlerUsageGuard spends it when the broadcast goes out,
// so a worker that dies during the wait doesn't leave the round marked "restart used".
var dueAt = resetsAt + HandlerUsageGuard.ResetGrace;
if (!_rounds.TryScheduleAutoRestart(taskId, dueAt, round.Phase, round.TaskIds)) return false;
try
{
await _broadcaster.WorkerLog(
$"List handler #{number} died at the usage limit (5h {fiveHour.Utilization:0}%) — " +
$"restarting phase {round.Phase} after the reset at {resetsAt.ToLocalTime():HH:mm}",
WorkerLogLevel.Warn, DateTime.UtcNow);
var wait = resetsAt - NowUtc();
if (wait > TimeSpan.Zero) await Delay(wait, ct);
await _broadcaster.HandoffRequested(taskId, round.TaskIds, round.Phase);
await _broadcaster.WorkerLog(
$"Auto-restarting list handler #{number} at phase {round.Phase} after usage-limit reset",
WorkerLogLevel.Info, DateTime.UtcNow);
return true;
}
catch (OperationCanceledException)
{
return false;
}
catch (Exception ex)
{
// Callers fire-and-forget this (WorkerHub.ReportInteractiveSessionExited), so an
// unobserved throw here would be invisible.
_logger.LogError(ex, "HandlerRestartCoordinator: auto-restart of handler task {TaskId} failed", taskId);
return false;
}
await _broadcaster.WorkerLog(
$"List handler #{number} died at the usage limit (5h {fiveHour.Utilization:0}%) — " +
$"restarting phase {round.Phase} after the reset, at {dueAt.ToLocalTime():HH:mm}",
WorkerLogLevel.Warn, DateTime.UtcNow);
return true;
}
}
src/ClaudeDo.Worker/Runner/HandlerRoundState.cs
+93 -11
View File
@@ -9,17 +9,26 @@ namespace ClaudeDo.Worker.Runner;
/// <c>HandoffRequested</c> broadcast that is the only other place these ids exist.</summary>
/// <param name="Phase">A <see cref="MergeHelperPhase"/> value, or "triage" for the initial session
/// (which has no handoff spec and therefore cannot be reopened — see HandlerRestartCoordinator).</param>
/// <param name="AutoRestartUsed">Claimed once per round by HandlerRestartCoordinator, so a handler
/// that dies again after its auto-restart stays dead instead of relaunching all night.</param>
/// <param name="DeferredHandoffAt">Reserved for the "hand off cleanly before the usage limit"
/// follow-up, which parks a handoff until the 5h reset and needs to survive a worker restart.</param>
/// <param name="AutoRestartUsed">Spent once per round, and only once the deferred
/// <c>HandoffRequested</c> broadcast has actually gone out — a restart that is merely *scheduled*
/// must not count, or a worker that dies during the hours-long wait leaves the round permanently
/// marked "restart used" without one ever having happened.</param>
/// <param name="DeferredHandoffAt">Due time of a handoff parked until the 5h usage window resets.
/// One slot per round, shared by both producers: a session that handed off cleanly just before the
/// limit (HandlerUsageGuard) and a session that died at it (HandlerRestartCoordinator) — so the two
/// can never reopen the same round twice. Fired by HandlerUsageGuard, which re-picks it up after a
/// worker restart because it lives in this file rather than in a timer.</param>
public sealed record HandlerRound(
string HandlerTaskId,
string Phase,
IReadOnlyList<string> TaskIds,
DateTime StartedAtUtc,
bool AutoRestartUsed = false,
DateTimeOffset? DeferredHandoffAt = null);
DateTimeOffset? DeferredHandoffAt = null,
string? DeferredPhase = null,
IReadOnlyList<string>? DeferredTaskIds = null,
bool DeferredIsAutoRestart = false,
bool UsageWarningSent = false);
/// <summary>
/// File-backed store for <see cref="HandlerRound"/>, one entry per handler task. Deliberately a
@@ -74,22 +83,95 @@ public sealed class HandlerRoundState
}
}
/// <summary>Takes this round's single auto-restart budget. False when there is no round for the
/// task or its budget is already spent — the same "never twice" discipline
/// <c>TaskRepository.TryClaimUsageLimitAutoContinueAsync</c> enforces for headless runs.</summary>
public bool TryClaimAutoRestart(string handlerTaskId)
public IReadOnlyList<HandlerRound> All()
{
lock (_lock)
return Rounds().Values.ToList();
}
/// <summary>Claims this round's single "warn the session before the usage limit" shot. True only
/// the first time, so a monitor that keeps ticking above the threshold does not retype the same
/// warning into the session every 30 seconds.</summary>
public bool TryMarkUsageWarned(string handlerTaskId)
{
lock (_lock)
{
var rounds = Rounds();
if (!rounds.TryGetValue(handlerTaskId, out var round) || round.AutoRestartUsed) return false;
if (!rounds.TryGetValue(handlerTaskId, out var round) || round.UsageWarningSent) return false;
rounds[handlerTaskId] = round with { AutoRestartUsed = true };
rounds[handlerTaskId] = round with { UsageWarningSent = true };
Persist(rounds);
return true;
}
}
/// <summary>Parks a handoff the running session asked for until <paramref name="dueAt"/>.
/// Overwrites any existing slot: the live session's own request is the most recent word on what
/// the next round should be.</summary>
public void SetDeferredHandoff(
string handlerTaskId, DateTimeOffset dueAt, string phase, IReadOnlyList<string> taskIds)
=> WriteDeferred(handlerTaskId, dueAt, phase, taskIds, isAutoRestart: false);
/// <summary>Schedules this round's single auto-restart (the session died at the limit). False
/// when there is no round, the restart budget is already spent, or a handoff is *already* parked
/// for this round — the deferred handoff and the auto-restart are one trigger, not two.
/// <see cref="HandlerRound.AutoRestartUsed"/> is deliberately NOT set here; only
/// <see cref="ClearDeferredHandoff"/> spends it, once the broadcast really went out.</summary>
public bool TryScheduleAutoRestart(
string handlerTaskId, DateTimeOffset dueAt, string phase, IReadOnlyList<string> taskIds)
{
lock (_lock) // Monitor is reentrant — WriteDeferred takes the same lock.
{
var rounds = Rounds();
if (!rounds.TryGetValue(handlerTaskId, out var round)) return false;
if (round.AutoRestartUsed || round.DeferredHandoffAt is not null) return false;
WriteDeferred(handlerTaskId, dueAt, phase, taskIds, isAutoRestart: true);
return true;
}
}
/// <summary>Clears the parked handoff. <paramref name="spendAutoRestart"/> marks the round's
/// restart budget as used — pass the slot's own <see cref="HandlerRound.DeferredIsAutoRestart"/>,
/// so a clean pre-limit handoff never costs the next session its restart.</summary>
public void ClearDeferredHandoff(string handlerTaskId, bool spendAutoRestart)
{
lock (_lock)
{
var rounds = Rounds();
if (!rounds.TryGetValue(handlerTaskId, out var round)) return;
rounds[handlerTaskId] = round with
{
DeferredHandoffAt = null,
DeferredPhase = null,
DeferredTaskIds = null,
DeferredIsAutoRestart = false,
AutoRestartUsed = round.AutoRestartUsed || spendAutoRestart,
};
Persist(rounds);
}
}
private void WriteDeferred(
string handlerTaskId, DateTimeOffset dueAt, string phase, IReadOnlyList<string> taskIds, bool isAutoRestart)
{
lock (_lock)
{
var rounds = Rounds();
if (!rounds.TryGetValue(handlerTaskId, out var round)) return;
rounds[handlerTaskId] = round with
{
DeferredHandoffAt = dueAt,
DeferredPhase = phase,
DeferredTaskIds = taskIds.ToList(),
DeferredIsAutoRestart = isAutoRestart,
};
Persist(rounds);
}
}
public void Remove(string handlerTaskId)
{
lock (_lock)
src/ClaudeDo.Worker/Runner/HandlerUsageGuard.cs
+207
View File
@@ -0,0 +1,207 @@
using ClaudeDo.Data;
using ClaudeDo.Data.Models;
using ClaudeDo.Data.Repositories;
using ClaudeDo.Worker.External;
using ClaudeDo.Worker.Hub;
using ClaudeDo.Worker.Usage;
using Microsoft.EntityFrameworkCore;
namespace ClaudeDo.Worker.Runner;
/// <summary>
/// Hands a PowerUse list-handler round over cleanly across the 5h usage limit, instead of letting it
/// die mid-merge:
///
/// 1. <b>Warn</b> — once per round, as soon as the 5h window crosses
/// <c>AppSettings.PowerUseHandoffPct</c>, a message is typed straight into the live ConPTY session
/// (<c>HandlerUsageWarning</c> → Mission Control) telling it to finish what it is doing and call
/// <c>handoff_list_handler</c> itself. A session that hands off on its own leaves no half-merged
/// tree behind; an auto-restart after the fact cannot clean one up.
/// 2. <b>Withhold</b> — the resulting handoff is not broadcast immediately (the next session would
/// open straight into the same exhausted window) but parked in <see cref="HandlerRoundState"/>
/// until the window resets. Same slot the auto-restart of a session that *did* die uses
/// (<see cref="HandlerRestartCoordinator"/>), so the two can never reopen one round twice.
/// 3. <b>Fire</b> — this service's tick is also what releases a parked handoff. Deliberately a poll
/// over the persisted state rather than a timer: a worker restart mid-wait re-picks the parked
/// handoff up on the next tick, and an overdue one goes out immediately, for free.
/// </summary>
public sealed class HandlerUsageGuard : BackgroundService
{
internal static readonly TimeSpan TickInterval = TimeSpan.FromSeconds(30);
// Small cushion past ResetsAt: the reported reset time is the server's, and reopening a session
// one second early buys nothing but another dead tile.
internal static readonly TimeSpan ResetGrace = TimeSpan.FromSeconds(60);
private readonly IDbContextFactory<ClaudeDoDbContext> _dbFactory;
private readonly UsageState _usageState;
private readonly HandlerRoundState _rounds;
private readonly HandoffRoundTracker _handoffRounds;
private readonly HubBroadcaster _broadcaster;
private readonly ILogger<HandlerUsageGuard> _logger;
public HandlerUsageGuard(
IDbContextFactory<ClaudeDoDbContext> dbFactory,
UsageState usageState,
HandlerRoundState rounds,
HandoffRoundTracker handoffRounds,
HubBroadcaster broadcaster,
ILogger<HandlerUsageGuard> logger)
{
_dbFactory = dbFactory;
_usageState = usageState;
_rounds = rounds;
_handoffRounds = handoffRounds;
_broadcaster = broadcaster;
_logger = logger;
}
/// <summary>Test seam — a real 5h wait is not a test.</summary>
internal Func<DateTimeOffset> NowUtc { get; set; } = () => DateTimeOffset.UtcNow;
/// <summary>PowerUse verdict for one handler task: both PowerUse switches on, a 5h bucket
/// available, plus the configured threshold and the task's display number.</summary>
internal sealed record Policy(int HandoffPct, UsageBucket FiveHour, int Number);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await TickAsync(stoppingToken);
await Task.Delay(TickInterval, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "HandlerUsageGuard: tick failed");
await Task.Delay(TickInterval, stoppingToken);
}
}
}
internal async Task TickAsync(CancellationToken ct)
{
foreach (var round in _rounds.All())
{
await FireIfDueAsync(round);
await WarnIfNearLimitAsync(round, ct);
}
}
/// <summary>Called by <c>handoff_list_handler</c>. Returns the time the handoff was parked until
/// when it must be withheld, or null when it should go out now (PowerUse off for this list, well
/// below the threshold, or no reset time to wait for).</summary>
public async Task<DateTimeOffset?> TryDeferHandoffAsync(
string handlerTaskId, IReadOnlyList<string> survivingTaskIds, string phase, CancellationToken ct)
{
var policy = await EvaluateAsync(handlerTaskId, ct);
if (policy is null) return null;
// The warning flag matters on its own: utilization can dip back under the threshold between
// the warning and the session actually finishing its merge, and a handoff produced *because*
// of the warning must still wait for the reset.
var warned = _rounds.Get(handlerTaskId)?.UsageWarningSent == true;
if (policy.FiveHour.Utilization < policy.HandoffPct && !warned) return null;
if (policy.FiveHour.ResetsAt is not { } resetsAt)
{
_logger.LogWarning(
"HandlerUsageGuard: 5h window at {Pct}% has no reset time; handing off {TaskId} immediately",
policy.FiveHour.Utilization, handlerTaskId);
return null;
}
var dueAt = resetsAt + ResetGrace;
_rounds.SetDeferredHandoff(handlerTaskId, dueAt, phase, survivingTaskIds);
await _broadcaster.WorkerLog(
$"List handler #{policy.Number}: handoff to phase {phase} withheld until " +
$"{dueAt.ToLocalTime():HH:mm} (5h window at {policy.FiveHour.Utilization:0}%)",
WorkerLogLevel.Warn, DateTime.UtcNow);
return dueAt;
}
private async Task FireIfDueAsync(HandlerRound round)
{
if (round.DeferredHandoffAt is not { } dueAt || dueAt > NowUtc()) return;
if (round.DeferredPhase is not { Length: > 0 } phase || round.DeferredTaskIds is not { Count: > 0 } taskIds)
{
_rounds.ClearDeferredHandoff(round.HandlerTaskId, spendAutoRestart: false);
return;
}
// Cleared BEFORE the broadcast so a throw on the way out can't leave a slot that re-fires on
// every tick; the restart budget is spent here too, which is the point of the whole
// rework -- it is now only spent once a handoff really went out.
_rounds.ClearDeferredHandoff(round.HandlerTaskId, spendAutoRestart: round.DeferredIsAutoRestart);
await _broadcaster.HandoffRequested(round.HandlerTaskId, taskIds, phase);
await _broadcaster.WorkerLog(
$"Reopening list handler at phase {phase} after the usage-limit reset",
WorkerLogLevel.Info, DateTime.UtcNow);
}
private async Task WarnIfNearLimitAsync(HandlerRound round, CancellationToken ct)
{
if (round.UsageWarningSent) return;
// A sealed chain has no further handoff to ask for -- telling it to call
// handoff_list_handler would only earn the session an exception.
if (_handoffRounds.GetState(round.HandlerTaskId).Sealed) return;
var policy = await EvaluateAsync(round.HandlerTaskId, ct);
if (policy is null || policy.FiveHour.Utilization < policy.HandoffPct) return;
if (!_rounds.TryMarkUsageWarned(round.HandlerTaskId)) return;
await _broadcaster.HandlerUsageWarning(
round.HandlerTaskId, BuildWarning(round.HandlerTaskId, round.Phase, policy.FiveHour.Utilization));
await _broadcaster.WorkerLog(
$"List handler #{policy.Number}: warned the session at 5h {policy.FiveHour.Utilization:0}% " +
$"(handoff threshold {policy.HandoffPct}%)",
WorkerLogLevel.Warn, DateTime.UtcNow);
}
// Plain ASCII on purpose: this is typed character-by-character into a PTY, not rendered by us.
internal static string BuildWarning(string handlerTaskId, string phase, double utilization) =>
$"USAGE LIMIT IMMINENT ({utilization:0}% of the 5h window). Finish or abort the merge you are " +
"in - never leave conflict markers in the tree - then call " +
$"handoff_list_handler(taskId: \"{handlerTaskId}\", survivingTaskIds: <every task not yet Done>, " +
$"nextPhase: \"{NextPhaseFor(phase)}\") right now. Do not start anything new. " +
"The next session opens automatically after the limit resets.";
// The session should resume where it is, so the warning names its own phase back. The opening
// triage round is the exception: "triage" is not a handoff phase, and what follows triage is
// always the wait phase.
private static string NextPhaseFor(string phase) =>
MergeHelperPhase.All.Contains(phase, StringComparer.Ordinal) ? phase : MergeHelperPhase.Wait;
/// <summary>Both PowerUse switches, exactly as <c>ComposeHandlerPromptAsync</c> and
/// <see cref="HandlerRestartCoordinator"/> gate them — the global toggle alone must not make an
/// unrelated list's handler session get typed into. Null = leave this round alone.</summary>
private async Task<Policy?> EvaluateAsync(string handlerTaskId, CancellationToken ct)
{
if (_usageState.Snapshot?.FiveHour is not { } fiveHour) return null;
try
{
await using var ctx = await _dbFactory.CreateDbContextAsync(ct);
var task = await new TaskRepository(ctx).GetByIdAsync(handlerTaskId, ct);
if (task is null) return null;
var settings = await new AppSettingsRepository(ctx).GetAsync(ct);
if (!settings.PowerUseEnabled) return null;
if ((await new ListRepository(ctx).GetConfigAsync(task.ListId, ct))?.PowerUseAllowed != true) return null;
return new Policy(Math.Clamp(settings.PowerUseHandoffPct, 50, 99), fiveHour, task.Number);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "HandlerUsageGuard: could not evaluate handler task {TaskId}", handlerTaskId);
return null;
}
}
}