ClaudeDo/docs/explore-notes/worker-task-pipeline.md

Explore-note — verify before trusting. Distilled map of a subsystem, not authoritative. Last verified against commit cc90600 (2026-08-06). Drift check: git log --oneline cc90600..HEAD -- src/ClaudeDo.Worker Stable structure only (no line numbers). See docs/explore-notes/README.md.

Worker: Task Execution Pipeline

How a task moves Queued → Running → terminal, across src/ClaudeDo.Worker (Queue, Runner, Lifecycle, State, Agents, Worktrees, Hub).

End-to-End Flow (Queued → Terminal)

1. Enqueue — ITaskStateService.EnqueueAsync() (State/TaskStateService.cs)

   • Idle → Queued, then wakes the dispatcher via IQueueWaker.Wake().

2. Dispatch — QueueService loop (Queue/QueueService.cs)

   • BackgroundService; waits for a wake signal or a backstop timer.
   • Reads the max-parallel limit from settings; claims a free slot if under limit.

3. Atomic Claim — IQueuePicker.ClaimNextAsync() (Queue/QueuePicker.cs)

   • Raw SQL UPDATE ... RETURNING in one transaction: picks an eligible Queued task (unblocked, due or unscheduled; sorted by sort_order/created_at), sets status→Running
     • started_at, returns the row. Prevents two workers claiming the same task (TOCTOU).

4. Slot Execution — QueueService.RunInSlotAsync() (Queue/QueueService.cs)

   • For review feedback: resume the prior session if one exists, else fold feedback into the prompt. Calls TaskRunner.RunAsync() / ContinueAsync() with alreadyClaimed=true.

5. Run Preparation — TaskRunner.RunAsync() (Runner/TaskRunner.cs)

   • Loads task, list config, subtasks, attachments from the DB.
   • StartRunningAsync() (only if not pre-claimed): atomic claim to Running, before any resource is created. A rejected claim (task already Running) bails out immediately — no worktree, no MCP token file. Broadcasts TaskStarted.
   • PrepareRunDirectoryAsync(): worktree (via WorktreeManager) if the list has a WorkingDir, else sandbox. Generates a per-run MCP token, writes MCP config to disk.

6. Claude Execution — TaskRunner.RunOnceAsync() (Runner/TaskRunner.cs)

   • Creates a TaskRunEntity, points the task at the run's log path.
   • Builds claude CLI args (ClaudeArgsBuilder), spawns the process via IClaudeProcess.RunAsync() with prompt + working dir + streaming callback.
   • Stream lines → NDJSON log + broadcast via TaskMessage. MCP tools (AskUser, SuggestImprovement) are scoped by the per-run token.

7. Result Handling — TaskRunner.HandleSuccess() / MarkFailed() (Runner/TaskRunner.cs)

   • Success (exit 0 + result markdown): if worktree, commit + broadcast WorktreeUpdated; then transition to Done / WaitingForReview / WaitingForChildren (CompleteAsync / SubmitForReviewAsync / SubmitForChildrenAsync).
   • Failure: if a session exists, auto-retry once via ContinueAsync; else MarkFailed → FailAsync.
   • All terminal writes use CancellationToken.None so a task is never left Running.

8. Terminal States — ITaskStateService transitions (State/TaskStateService.cs)

   • Done CompleteAsync (Running → Done) — top-level success.
   • WaitingForReview SubmitForReviewAsync (Running → WaitingForReview) — review gate.
   • WaitingForChildren SubmitForChildrenAsync (Running → WaitingForChildren) — blocks on children. Advances to WaitingForReview via TryAdvanceParentAsync once every remaining child is terminal (Done/Failed/Cancelled) — including zero children left, e.g. after the last child is deleted (WorkerHub.DeleteTask / ExternalMcpService.DeleteTask both call it).
   • Failed FailAsync (Running/Queued → Failed).
   • Cancelled CancelAsync (Running/Queued/WaitingForReview/WaitingForChildren → Cancelled).

Model, effort & max-turns resolution

(section added at commit f6cb825, 2026-08-05; resolver extraction added same day)

The resolution below lives in Runner/EffectiveRunConfigResolver.Resolve (not inlined in TaskRunner anymore) so TaskRunner.ResolveConfigAsync and the read-only get_effective_run_config MCP tool (External/ConfigMcpTools.cs) share one codepath and can't report different numbers for the same task. The tool additionally surfaces, per field, whether it came from the task/list/preset/global layer, and — for max turns — the raw requested value plus whether it was clamped.

Step 6 builds the CLI args. Model and turn budget resolve like this:

1. Effective model — task override → list config → AppSettings.DefaultModel.
2. Preset row — ModelPresets.For(global.ModelPresets, model, global.DefaultMaxTurns). The model string is resolved through ModelRegistry.TryNormalizeAlias first, so a full CLI model id (e.g. claude-sonnet-4-6, not just the bare sonnet/opus/haiku/fable aliases) still hits its alias's preset row instead of missing every lookup. Only a model that normalizes to nothing recognized falls back to a synthesized row using AppSettings.DefaultMaxTurns — never a hardcoded number, and it never throws: an unknown model must not block a run.
3. The preset supplies --effort and the global max-turns default. Task/list MaxTurns overrides still win over it.
4. Ceiling clamp — TaskRunner.ResolveMaxTurns hard-clamps the resolved value to AppSettings.MaxTurnsCeiling (default 80). An override above the ceiling still starts, just capped, and a Warn logs the task id + requested + effective value.

⚠️ Trap: if app_settings.model_presets is somehow null, the fallback path decides the turn budget — which is why AppSettingsRepository.GetAsync backfills shipping defaults on the first read after null. Ship preset turns are low (haiku 20, sonnet 30, opus 40, fable 25), so a task that genuinely needs a long run must set its own MaxTurns.

Prompt composition: TaskPromptComposer.Compose injects attachment absolute paths as a read-only "## Reference files" section.

Component Responsibilities

Queue/

• QueueService — main dispatch loop; slot limit; decides when to start tasks.
• QueuePicker — atomic Queued→Running claim via raw SQL.
• QueueWaker — semaphore for non-blocking, idempotent wake signals.
• OverrideSlotService — owns the RunNow / ContinueTask slot (bypasses the queue).

Runner/

• TaskRunner — orchestrates the run (prepare, execute, handle result).
• WorktreeManager — creates/manages git worktrees; self-heals stale branches.
• ClaudeProcess — spawns the claude CLI subprocess; manages streams/logs.
• TaskRunMcpService — runtime MCP tools (AskUser, SuggestImprovement).
• TaskRunTokenRegistry — per-run MCP identity for tool-access control.
• InteractiveLaunchSpecService — config for the task's claude run.

State/

• TaskStateService — all task status transitions; guards preconditions; signals queue/hub.

Lifecycle/ (startup recovery)

• StaleTaskRecovery — tasks stuck Running after a crash/restart → Failed. The underlying TaskStateService.RecoverStaleRunningAsync bulk-flips Running→Failed, then re-runs the same chain/parent-advance side effects as a normal FailAsync (per recovered id, best-effort) so a crash mid-chain-child or mid-improvement-child doesn't leave a successor blocked forever or a WaitingForChildren parent wedged.
• OrphanRecovery — dequeues children whose parent is no longer planning (stays attached).
• AttachmentOrphanRecovery — cleans orphaned attachment files.
• TaskResetService — manual reset to Idle.
• TaskMergeService — conflict resolution for worktree merges.

Hub/

• HubBroadcaster — single SignalR broadcast point (TaskStarted/TaskUpdated/TaskMessage/WorktreeUpdated…).
• WorkerHub — SignalR hub + client methods.

Agents/

• AgentFileService — file I/O for custom agents.
• DefaultAgentSeeder — seeds built-in agents on startup.

Worktrees/

• WorktreeMaintenanceService — cleanup, state tracking, overview reporting.

Entry Points & Call Chain

Program.cs (DI setup)
 ├─ QueueService (BackgroundService) → ExecuteAsync loop
 │   ├─ waits: IQueueWaker.WaitAsync() or timer
 │   ├─ claims: IQueuePicker.ClaimNextAsync()
 │   └─ runs:  TaskRunner.RunAsync() / ContinueAsync()
 ├─ Hub clients → WorkerHub methods
 │   ├─ Enqueue     → ITaskStateService.EnqueueAsync() → Wake()
 │   ├─ RunNow      → OverrideSlotService.RunNow()      → TaskRunner.RunAsync()
 │   ├─ ContinueTask→ OverrideSlotService.ContinueTask()→ TaskRunner.ContinueAsync()
 │   └─ CancelTask  → QueueService.CancelTask()
 ├─ Lifecycle recovery (startup): StaleTaskRecovery / OrphanRecovery / AttachmentOrphanRecovery
 └─ State transitions → HubBroadcaster.TaskUpdated()

Invariants & Conventions

• Atomic claiming — QueuePicker's UPDATE ... RETURNING makes Queued→Running atomic.
• Slot limit — respects MaxParallelExecutions; a backstop timer wakes even if a Wake() is missed.
• Pre-claimed tasks — the dispatcher pre-claims via the picker; the override slot (RunNow/ContinueTask) must call StartRunningAsync if a task is not pre-claimed.
• Claim before create — TaskRunner.RunAsync's unclaimed path calls StartRunningAsync before PrepareRunDirectoryAsync. RunNow racing the picker for the same Queued row used to create the worktree first and only claim afterwards, so the losing dispatch could hit WorktreeManager's branch-collision self-heal and force-remove the winner's live worktree mid-run. OverrideSlotService.RunNow also fast-rejects a task already Running in the DB (defense in depth; the picker's atomic SQL claim is the real arbiter either way). RunCancellationRegistry.Register refuses (and logs) a second registration for the same task id instead of silently overwriting the first, so a losing dispatch's cleanup can't unregister the winner's CTS out from under it.
• Terminal writes — use CancellationToken.None; a task is never left Running after crash/cancel.
• Per-run MCP tokens — each run gets a unique token scoping tool access; unregistered on end.
• Auto-retry — one automatic retry if a session exists and the first run failed.
• Worktree self-heal — on branch collision, remove phantom worktrees, prune, delete branch, retry add.
• Review feedback — stored on the task; consumed once a run reaches a terminal state; a re-queued task resumes the session or folds feedback into the prompt.
• Child tasks — planning creates draft children; finalization requires no Queued children remain; OrphanRecovery dequeues children if the parent is not planning.
• Lifecycle recovery runs at startup: stale-Running → Failed; orphaned children → dequeued but attached.