Why Long-Running Coding Agents Need Durable Execution

Long-running coding agents are often discussed as if they have one main problem: eventually, they run out of context.

That problem is real. A coding agent working for hours may forget an earlier constraint, lose track of why a decision was made, repeat repository discovery, or declare the work complete before every requirement has been verified. Larger context windows, compaction, progress files, and clean handoffs all help. Current guidance for long-running agent work rightly emphasizes clear outcomes, constraints, completion criteria, incremental progress, and durable handoff artifacts. [1] [2]

But context loss is only one failure mode.

The worker running the agent can crash after creating a branch but before recording the branch name. A request to create a ticket can time out after the ticket was created. A browser session can disappear after an expensive bug reproduction. A human can approve a proposed change, then the target branch can move before the workflow resumes. A new application version can be deployed while old agent runs are still waiting for input.

None of those problems is fixed by letting the model remember more tokens.

An agent can remember exactly what it intended to do and still execute the same side effect twice. It can retain a perfect summary and still not know whether a timed-out request succeeded. It can preserve its full conversation and still lose the process that was holding its next timer, retry, or approval wait in memory.

Once coding-agent work lasts long enough to cross process failures, deployments, external APIs, or human response times, it becomes a distributed-systems problem. The missing capability is not only memory. It is durable execution.

Context continuity and execution continuity are different problems

Context engineering is the work of deciding what information should be visible to the model for its next decision. That includes instructions, relevant files, tool descriptions, conversation history, plans, summaries, and retrieved repository knowledge. Context is finite, and simply filling more of the window can reduce the signal available to the model. [3]

Execution engineering asks different questions:

• Which step has actually completed?
• Which external effects may already have happened?
• What should be retried after a crash?
• What must not be repeated?
• What is the workflow waiting for?
• Which evidence proves the current state?
• Can a new worker resume without guessing?

Those questions belong to four different continuity surfaces:

These surfaces support each other, but they are not interchangeable.

A progress file can say that a branch should be created. It cannot make branch creation atomic with writing that progress file. A conversation summary can say that a ticket was probably updated. It cannot resolve whether the update reached the issue tracker before the network connection failed. An event history can say that the workflow is waiting for approval. It should not be copied into the model context in full every time the agent makes a decision.

The architecture becomes much easier to reason about once those responsibilities stay separate:

• context supplies the model's working set
• the work contract defines what the run is allowed to accomplish
• the workflow records how far execution has progressed
• reconciliation establishes what is true in external systems

Long-running work needs all four.

This complements docs-first product development, which preserves product meaning and repository knowledge, and living plans, which define the active work contract. Those artifacts help a resumed agent understand the task; persisted execution state records which operations have actually completed and which outcomes still need reconciliation.

A coding-agent workflow fails in more places than the agent loop

Consider an agentic bug-repair pipeline for a web application. A report arrives in natural language. The system turns it into a structured ticket, prepares a repository and runnable environment, reproduces the problem in a browser, collects evidence, diagnoses the likely cause, attempts a bounded repair, replays the reproduction, and creates a draft pull request for human review.

At a high level, that sounds like one agent task:

bug report
-> reproduce
-> diagnose
-> repair
-> verify
-> create draft pull request

Operationally, it is a sequence of external interactions:

receive report
-> create or update ticket
-> clone repository
-> resolve revision
-> install dependencies
-> start application
-> seed fixtures
-> wait for readiness
-> run browser reproduction
-> store screenshots and trace
-> ask model for diagnosis
-> modify files
-> run targeted checks
-> replay browser procedure
-> create branch and commit
-> push branch
-> create draft pull request
-> attach evidence
-> wait for human review

Nearly every arrow crosses a failure boundary.

The model provider can rate-limit a request. The dependency registry can be temporarily unavailable. The application can start but never become ready. The browser worker can crash. The Git host can accept a push while the client loses the response. A human review may take two days. The orchestrator itself may be redeployed several times during that wait.

If this whole pipeline lives in one process, one shell script, or one agent conversation, then the process lifetime becomes the workflow lifetime. A crash near the end may throw away hours of useful work. Blindly restarting from the beginning is not safe either, because some of the earlier steps may have already changed external systems.

This is why a long-running agent harness and a durable workflow solve related but different problems. Harness work has shown the value of incremental tasks, structured handoffs, clean repository state, and end-to-end browser verification. It has also shown that compaction alone does not guarantee coherent multi-session progress. [2] [4]

The amount of harness scaffolding should also change as models improve. Anthropic's later experiments removed the sprint structure when a stronger model could sustain coherent work with less decomposition. [4] Better models and context management can simplify that harness.

Durable execution complements those improvements. It preserves the execution state that decides what happens after a worker, process, or machine disappears. Context continuity and execution continuity both contribute to reliable work.

What durable execution means for a coding agent

Durable execution means that workflow progress is persisted independently of the process currently doing the work. When a worker fails, another worker can reconstruct the last recorded state and continue from there instead of replaying the entire task from memory.

Durable workflow systems commonly record an event history and rebuild workflow state by replaying the deterministic orchestration logic against those recorded events. External and failure-prone operations are separated into activities or tasks whose results become part of that history. Temporal calls this distinction Workflows and Activities. [5] [6] Microsoft's Durable Task uses orchestrators and activities. [7] The terminology varies, but the responsibility split is the important part.

For coding agents, the split should look roughly like this:

The model belongs on the activity side of this boundary.

That point matters. An LLM call is nondeterministic, failure-prone, externally hosted work. It may return a different answer if it is repeated. The model can propose the next action, diagnose a failure, or choose among allowed tools, but it should not be the only place where the workflow's retry budget, approval state, or completion rules exist.

The durable workflow is not the agent's brain. It is the control system around the agent.

In conceptual pseudocode, the bug-repair workflow might look like this:

workflow repair_bug(report):
    ticket = run_activity(ensure_ticket, operation_id="ticket")
    environment = run_activity(prepare_environment, operation_id="environment")
    reproduction = run_activity(
        reproduce_bug, environment, operation_id="reproduction"
    )

    if reproduction.status != "reproduced":
        run_activity(update_ticket_with_inconclusive_evidence)
        return "needs-human-triage"

    for attempt in bounded_repair_attempts:
        proposal = run_activity(diagnose_and_propose_fix)
        result = run_activity(
            apply_and_verify_fix, environment, proposal, operation_id=attempt.id
        )

        if result.regression_passed:
            draft = run_activity(ensure_draft_pull_request)
            review = wait_for_human_review(draft, result.evidence)
            return run_activity(act_on_review_with_preconditions, review, draft)

    run_activity(update_ticket_with_failed_attempts)
    return "repair-budget-exhausted"

The workflow decides the allowed state transitions. Activities perform the work that can fail. The event history records which results were accepted. Evidence is stored durably and referenced by the workflow. These records establish a recovery point after a crash. Continuing safely also requires recoverable workspaces and activity contracts that handle partial or ambiguous effects. The review activity must enforce the approved preconditions when it performs the mutation, as described below.

Checkpoint facts, not just conversation

A common first attempt at resumability is to save the conversation or ask the agent to write a summary before the session ends. That is useful for model continuity, but it is a weak execution checkpoint.

A summary is prose. It may omit a stable identifier, describe an intention as if it were completed, or collapse an ambiguous result into a confident sentence. The next agent still has to interpret it.

An execution checkpoint should instead preserve typed, externally verifiable facts such as:

workflow_id: bug-repair-1842
workflow_version: 3
phase: verifying-fix
target_revision: 8f15a4c
repair_attempt: 2
ticket:
  id: BUG-1842
  operation_id: bug-repair-1842:create-ticket
environment:
  id: preview-9187
  recovery_spec: artifacts/environment.json
  last_observed_status: ready
reproduction:
  status: reproduced
  procedure_artifact: artifacts/reproduction.json
  browser_trace: artifacts/trace.zip
candidate_change:
  commit: 12c0c71
  patch_artifact: artifacts/attempt-2.patch
  verification_status: pending
pending_approval: null

This state does not try to preserve every token the model saw. It records the minimum execution facts required to make the next transition safely.

The full browser trace, command output, screenshots, and patch do not need to be embedded in the workflow history. Large artifacts can live in repository storage or an artifact store. The workflow should retain stable references, content hashes where useful, and enough metadata to determine whether the artifact belongs to the current run.

This separation helps with context budget, storage limits, auditability, and privacy at the same time. The agent can retrieve the specific evidence needed for its next decision without receiving every old log line. The workflow can resume without depending on an agent-generated summary.

Make the workspace recoverable

A recorded activity result does not preserve the machine that produced it. Replaying the result of prepare_environment will not recreate a lost checkout, uncommitted edits, browser session, or application process. Its output should identify durable recovery inputs, rather than only a path on the original worker. Each activity that uses the workspace must restore or validate it at the start of an execution attempt.

Keep the base revision and candidate changes available outside the worker, as reachable commits, patch artifacts, or workspace snapshots. Record the setup instructions, dependency versions, and fixtures needed to reconstruct the environment. Browser procedures must define how to recreate any required session state. A saved last_observed_status: ready describes an earlier observation; the next worker must check readiness again.

apply_and_verify_fix also needs a recovery contract for a crash after only some files have changed. One approach is to rebuild an isolated workspace from the recorded base revision and apply the persisted candidate patch before rerunning verification. Another is to resume from a verified snapshot. Blindly applying the patch again to a partly modified checkout is unsafe. Only accept results whose evidence identifies the candidate revision actually tested.

Retries are a policy, not a generic loop

Durability makes retries possible. It does not make every retry safe.

Before retrying a failed activity, the system needs to know what kind of failure occurred:

Treating all exceptions as transient is one of the easiest ways to turn a recoverable workflow into an automated incident.

A syntax error in an agent-generated patch may justify another bounded repair attempt. A missing repository permission will not become valid because the same request is sent twenty times. A model refusal should not necessarily consume the same budget as a network timeout. A browser assertion failure is usually domain evidence, not an infrastructure failure to hide with retries.

Durable workflow engines usually provide declarative retry policies around failure-prone activities. Temporal, for example, distinguishes deterministic workflow logic from API calls, LLM invocations, and other failure-prone activities, and supports backoff, time limits, attempt limits, and non-retryable error types. [8]

The framework cannot choose the semantic policy for you. That still belongs to the application.

For a coding-agent workflow, I prefer explicit budgets at several levels:

• a transport retry budget for rate limits and network failures
• a tool retry budget for a command that may succeed after environmental repair
• a reasoning or repair-attempt budget for genuinely different candidate fixes
• a workflow time budget for the whole outcome
• a human escalation boundary when uncertainty remains

Those budgets should not be collapsed into one counter. Retrying the same HTTP request is not the same kind of work as asking the model to develop a new repair hypothesis.

Idempotency is what makes retries safe

An idempotent operation can be repeated without creating additional side effects beyond the intended result. Long-running agent systems need idempotency because an activity can finish its external work and still fail before the workflow records the result.

Imagine that the workflow sends create draft pull request. The Git host creates it, but the response is lost. From the workflow's perspective, the activity failed. If the retry blindly sends another create request, the repository may end up with duplicate pull requests.

The safer contract gives the logical operation a stable identifier:

bug-repair-1842:create-draft-pr

Every retry carries the same identifier or uses another stable external key, such as the workflow ID plus the intended branch. The parameters must also describe the same intent. Reusing an operation identifier with different parameters should fail rather than silently treating a new request as a retry.

An identifier alone does not prevent duplicates. If an activity checks whether a resource exists and then creates it, two attempts can both observe its absence and both create it. A request that timed out may also still be running, or its result may not yet be visible to a lookup.

Duplicate prevention must be enforced where the mutation happens: through a provider-supported idempotency key, an atomically enforced unique resource key, or a conditional write with the appropriate precondition. AWS's idempotent API guidance explicitly requires recording the request token and performing the associated mutations atomically. A repeated request can then return the accepted result without performing the mutation again. [9]

A local idempotency table cannot make a remote mutation atomic with a local state update. Where the external API lacks a suitable guarantee, checking for existing resources helps with reconciliation, but an unresolved outcome must remain ambiguous. It does not justify a blind retry or a promise that duplicate effects are impossible.

Coding-agent workflows have many operations that benefit from this design:

• create a ticket
• provision a preview environment
• create a branch
• push a generated commit
• create a draft pull request
• add an evidence comment
• request a human review
• tear down an environment

Not every underlying tool is naturally idempotent. git commit, for example, creates a new object based on current repository state. The workflow can still make publication idempotent by durably retaining the intended commit and using a conditional branch update that requires the expected previous remote revision. On retry, it can recognize that the branch already points to the intended commit. A matching tree alone does not identify the intended commit, and a separate lookup cannot prevent the branch from changing before the update.

The realistic goal is usually not exactly-once physical execution. Networks and process crashes make that promise difficult across independent systems. The useful goal is an effectively-once business outcome: activities may be attempted more than once while the external system enforces one logical result. That guarantee depends on the mutation contract; stable identifiers and reconciliation alone cannot provide it for every integration.

Ambiguous outcomes require reconciliation

Some failures do not tell you whether the operation happened.

Suppose the push command sends all objects and updates the remote branch, then the connection breaks before the client receives confirmation. The error says only that the client did not observe success. It does not prove failure.

This is an ambiguity boundary.

A timeout alone does not justify repeating the mutation. When the receiver supports an idempotent retry within its documented guarantees, resending the same operation identifier may be safe. Otherwise, query external truth before deciding how to proceed. For the Git push:

1. Inspect the remote branch.
2. Compare its revision with the intended commit.
3. If they match, record the original operation as successful.
4. If the branch does not exist, retry with a conditional create that fails if another operation has created it in the meantime.
5. If it exists but points somewhere else, stop and escalate the conflict.

That pattern applies far beyond Git:

• Query the issue tracker for the operation marker before creating another ticket.
• Query the deployment platform for the workflow-owned environment before provisioning another one.
• Inspect the existing pull request by branch and workflow ID before creating a duplicate.
• Check whether an evidence comment with the same content hash already exists before posting it again.

For these lookups, finding the intended result can resolve uncertainty. Not finding it is generally insufficient: the original request may still complete, or the lookup may lag behind the mutation. Retrying safely still requires the external system to enforce duplicate prevention, or evidence that the original attempt can no longer produce an effect together with protection against concurrent creation. Otherwise, preserve the ambiguous state.

Reconciliation is not an exceptional recovery script. In a distributed workflow, it is a normal state transition.

When external truth cannot be determined safely, the workflow should become explicitly inconclusive or needs-human-review. Guessing success risks skipping necessary work. Guessing failure risks duplicating an irreversible effect. Durable execution is valuable partly because it gives ambiguity a place to live without forcing the process to invent certainty.

Timeouts must describe what stopped making progress

One giant timeout around the whole agent run is rarely useful. It says that the workflow took too long, but not what failed.

Long-running workflows need time boundaries that match distinct failure modes:

• Queue timeout: no worker started the activity in time.
• Attempt timeout: one execution attempt did not complete in time.
• Heartbeat timeout: a long activity stopped reporting progress.
• Total activity timeout: all attempts together exceeded their budget.
• Workflow deadline: the overall outcome is no longer useful after this point.

Those distinctions change the correct recovery.

If a browser reproduction is queued for too long, the browser-worker pool may be unavailable. If it starts but stops heartbeating, the worker may have crashed or the browser may be stuck. If several valid attempts cannot reproduce the report, the correct outcome may be inconclusive, not another infrastructure retry.

Heartbeats are especially useful for work that cannot be divided into very small activities. A dependency installation, repository analysis, or browser procedure can periodically report progress and a resumable checkpoint. That lets the system detect missing progress reports and resume from a recorded substate when the tool supports it. A missing heartbeat does not prove that the worker stopped.

Temporal's activity model illustrates another important property: cancellation is cooperative. Activities receive cancellation and get a chance to release resources, but the workflow must decide whether to wait for that cleanup or continue. [6]

A timed-out activity may still be executing while its replacement starts. The original worker might have lost contact with the orchestrator while retaining access to the Git host, issue tracker, or deployment API. Expiring a lease or requesting cancellation does not itself revoke that access.

Where overlapping attempts could conflict, the resource receiving the mutation must reject stale attempts. Depending on the system, that can mean enforcing a fencing token that identifies the current ownership generation, or a conditional write against the expected resource version. Idempotency protects repeated requests for the same logical operation; stale workers must also be prevented from making conflicting changes. If the integration cannot enforce that boundary, keep the operation unresolved until it is safe to proceed.

Cancellation and cleanup are part of the workflow

Long-running agent systems create resources: working directories, processes, containers, preview environments, temporary credentials, browser sessions, branches, and artifacts. Stopping the main task does not make those resources disappear.

Cancellation therefore needs an explicit state model:

running
-> cancellation-requested
-> stopping-active-work
-> preserving-required-evidence
-> cleaning-owned-resources
-> cancelled | cleanup-incomplete

Cleanup is itself external work. It can fail, time out, or be retried. The same rules apply:

• tag resources with a stable workflow ID
• delete only resources owned by that workflow
• make cleanup idempotent
• prevent late activity attempts from recreating resources after cleanup
• retain evidence that must survive the environment
• record incomplete cleanup as an actionable state
• never broaden a cleanup target because the first attempt failed

This is not secondary operational polish. A workflow that reliably creates preview environments but unreliably removes them is not durable; it is a resource leak with a persistent scheduler.

There is also an important difference between cancellation and termination. Cancellation asks the workflow and its activities to stop coherently. Termination stops the workflow immediately when coherent cleanup is no longer possible or desirable. Systems need both, but termination should make the remaining uncertainty visible rather than reporting a clean cancellation.

Human approval should be a durable wait

Human review is not an edge case in agentic engineering. It is often the authority boundary that prevents an agent from publishing, merging, deploying, or applying an irreversible change on its own.

A weak implementation keeps a worker process open while polling a database or waiting for a webhook. That ties workflow continuity to infrastructure that may not survive the wait.

A durable implementation persists the waiting state, releases the worker, and resumes when an approval, rejection, correction, or timeout arrives. Durable workflow systems expose message primitives for this kind of interaction. For example, Temporal distinguishes asynchronous signals, state queries, and tracked updates that can return a result to the caller. [10]

The approval event should include more than yes:

• who approved
• which proposed action was approved
• which revision or artifact hash was reviewed
• which evidence was available
• when the approval expires
• which policy version governed the decision

Most importantly, the workflow should revalidate external preconditions after the wait.

A reviewer may approve a draft based on commit abc123. Before the workflow acts, the target branch may advance, permissions may change, a vulnerability may be disclosed, or another agent may modify the same branch. The old approval does not automatically authorize a materially different action.

Re-reading the state still leaves a race before the action executes. Where supported, the external system must enforce the approved revision or version as a precondition of the mutation itself. For example, GitHub's pull-request merge API accepts a sha parameter and rejects the merge if the PR head no longer matches it. That protects the head revision; approval conditions concerning the base branch require separate enforcement. [11]

If the target system cannot enforce a required precondition, revalidation alone cannot guarantee that the approved state is the state acted on. The workflow must stop or hand control to a human through a process that can enforce the required constraint.

The resume sequence should therefore be:

receive approval
-> validate approval identity and scope
-> re-read current external state
-> compare it with the approved state
-> request renewed approval or stop if the action has materially changed
-> execute with the approved preconditions enforced by the target system
-> reconcile an ambiguous result; revalidate after a precondition failure

Human-in-the-loop is not merely a pause in the agent loop. It is a durable state transition with a time-of-check/time-of-use problem.

Progress, logs, and evidence should not be the same artifact

Durable execution makes it possible to inspect what a workflow believes. It does not automatically prove that belief is correct.

A useful agentic workflow keeps three record types separate:

1. Execution history records state transitions, activity attempts, waits, retries, and outcomes.
2. Operational logs help diagnose processes, workers, commands, and infrastructure.
3. Evidence artifacts prove domain claims such as reproduction, regression behavior, or the exact change proposed for review.

Those records have different retention, privacy, and presentation needs.

The event history might record that browser reproduction completed and returned artifact sha256:.... The logs might show browser startup and network requests. The evidence bundle might contain a concise reproduction procedure, screenshots, a trace, observed-versus-expected behavior, and the application revision.

Pasting all logs into the workflow state makes histories large and pushes noise into later model context. Keeping only a final status discards the proof needed for review. The better design stores evidence separately and links it from the state transition that accepted it.

For each significant activity, I want to be able to answer:

• What input and external revision did it operate on?
• Which worker and tool version executed it?
• How many attempts occurred?
• What result did the workflow accept?
• Where is the supporting evidence?
• Did a human approve the next authority boundary?

This is observability for decisions, not only infrastructure.

Workflow code changes while workflows are still running

A workflow that lasts hours or days will eventually overlap a deployment. The code that resumes it may not be the code that started it.

That creates a versioning problem.

Event-history-based systems depend on deterministic replay: given the recorded events, the workflow code must reconstruct compatible decisions before it can continue. If a deployment changes past branching logic, reorders activities, or interprets stored state differently, old histories may no longer replay safely. Temporal documents replay as the mechanism that reconstructs a workflow after failure and checks generated commands against the existing history. [5]

Even without a replay-based engine, the same underlying issue exists. A database row written by workflow version 2 may not contain the fields that version 3 now requires. A new prompt or model may propose different actions for an old pending state. A tool schema may have changed while a run was waiting for approval.

Durable agent systems therefore need versioned contracts for:

• workflow definitions
• persisted state schemas
• activity inputs and outputs
• prompt and policy bundles
• tool schemas
• evidence formats
• approval payloads

There are several valid deployment strategies:

• let old workers drain old workflow versions
• route each run to a compatible worker version
• add explicit version markers in workflow logic
• migrate persisted state through a tested transition
• terminate and restart only when external effects can be reconciled safely

The unsafe strategy is to deploy new code and assume that every waiting run can restart from the top.

Historical workflow data is also useful as a test corpus. Replaying representative histories against new workflow code can reveal incompatible transitions before deployment. That is more valuable than testing only fresh happy-path runs, because the dangerous state is often the half-completed workflow crossing a version boundary.

Durable execution does not require one particular engine

Temporal and Durable Task are current examples of platforms that expose durable workflow primitives for state persistence, recovery, retries, timers, and human interaction. Microsoft's current Durable Task documentation explicitly includes fault-tolerant, scalable AI agents and deterministic agent orchestration among the workloads the platform is intended to support. [12]

That does not mean every coding-agent workflow needs one of those platforms.

A smaller system can implement the same responsibilities with ordinary components:

• a relational database containing workflow and step state
• a queue containing runnable activities
• leases that let crashed work become eligible again
• a transactional outbox for publishing work after state commits
• stable operation IDs and an idempotency table
• an artifact store for evidence
• a reconciler that detects stale, ambiguous, or abandoned work
• an API for approvals, cancellation, status, and resume

That approach can be entirely reasonable when the workflow is small and the team already operates those components well.

The cost is that the team now owns the difficult parts: atomic state transitions, duplicate delivery, timers, leases, retry scheduling, concurrent messages, schema evolution, workflow inspection, and recovery tooling. The first version often looks deceptively simple because those problems appear only after real failures.

A dedicated durable-execution engine becomes more attractive when several of these conditions are true:

• work regularly lasts longer than one process lifetime
• workflows wait for humans or external events
• activities have costly or irreversible side effects
• many workflow instances run concurrently
• retries and cancellation need distinct policies
• exact state and audit history matter operationally
• the same reliability machinery would otherwise be rebuilt by several teams

The decision should follow the failure model, not the current popularity of an agent framework.

A practical architecture for durable coding-agent work

The smallest useful architecture keeps orchestration, execution, evidence, and human authority distinct:

The workflow stores references to external resources and evidence. Workers are replaceable when the required workspace can be restored. The systems receiving mutations must also prevent conflicting changes from superseded attempts. Human review enters through a persisted message rather than an in-memory callback. External systems remain the source of truth for effects they own.

That design also limits what must enter model context. A diagnosis activity may need the ticket, repository slice, reproduction procedure, trace summary, and current attempt history. It does not need the scheduler's complete event history or every infrastructure retry. The workflow can construct a focused input from durable facts.

This is the point where context engineering and durable execution meet without becoming the same thing: durable state preserves the facts; context engineering selects the facts needed for the next decision.

Test recovery, not only the happy path

A durable workflow should be tested by breaking it at the boundaries where production will break it.

These recovery checks complement the targeted verification loops and guardrails that establish whether a code change behaves correctly. Recovery tests also check whether the workflow can preserve and act on those results after a crash.

For the bug-repair example, I would include tests that:

• kill a worker after a ticket is created but before the result is recorded
• overlap two attempts that both observe a missing resource and verify that the external system admits only one logical creation
• lose the response after a branch push and verify that reconciliation finds the existing revision
• deliver the same approval twice
• cancel while a browser or build activity is heartbeating
• restart with a different worker after an activity timeout
• keep the original worker alive after its timeout and verify that stale writes are rejected, including attempts to recreate resources after cleanup
• lose the worker's filesystem after a partially applied patch and restore the candidate on a fresh worker before verification
• exhaust the repair budget without creating a pull request
• approve one commit, move the target revision, and verify that the workflow requests renewed review
• change the approved revision after revalidation and verify that the mutation's precondition rejects the change
• deploy a compatible workflow change while old runs are waiting
• detect and report cleanup that cannot complete
• resume after all in-memory caches have been cleared

These are not chaos tests for show. They prove the core promise: the workflow can make progress without relying on a particular process having survived.

The final assertions should inspect external outcomes as well as workflow state:

• exactly one logical ticket exists
• exactly one intended branch exists
• no draft pull request exists before the evidence and authority gates pass
• the accepted evidence belongs to the correct repository revision
• cancellation leaves no unowned environment running
• a failed or inconclusive reproduction is reported honestly

That last point matters. Durability should preserve negative and inconclusive outcomes, not convert every workflow into an infinite loop that eventually claims success.

A reproducible crash-recovery experiment

The standalone recovery experiment exercises one specific failure boundary: a worker creates a ticket, then dies before recording the activity result. It uses two separate SQLite files. One represents a ticket service that atomically stores a ticket and its unique operation ID; the other stores the workflow checkpoint. There is no transaction spanning the two stores.

The first worker is killed with SIGKILL immediately after the ticket commit. A fresh worker then retries with the same operation ID and parameters. The simulated ticket service returns the existing ticket, and the new worker records its ID in the workflow checkpoint. Assertions inspect both stores and verify that the checkpoint refers to the original ticket.

Download the script and run it with Node.js 24 on a POSIX system:

node durable-execution-recovery.mjs

The experiment produced this output on Linux with Node.js 24.18.1 on 21 October 2026:

Before crash: tickets=0, checkpoints=0
After worker crash: tickets=1, checkpoints=0
After retry: tickets=1, checkpoints=1, ticket_id=1
PASS: one logical ticket, recorded by a fresh worker.

The checkpoint alone could not tell the replacement worker whether the ticket existed. Recovery succeeded because the receiver enforced duplicate prevention and retained the original result. This is a local process-crash demonstration; it does not test network partitions, machine loss, overlapping workers, or a real issue tracker's API guarantees. The broader recovery tests above still matter. The script makes no network calls and removes its temporary databases after checking the result.

Start with the boundaries where ambiguity is expensive

It is possible to overengineer durable execution.

A read-only repository question that finishes in two minutes does not need a workflow engine. A small local refactor with fast tests may be safer to rerun than to persist. Adding event sourcing, idempotency storage, and distributed workers to every agent command would create more operational surface than it removes.

I would adopt durability incrementally:

1. Choose one workflow whose failure currently loses expensive work or risks duplicate side effects.
2. Enumerate its explicit states and terminal outcomes.
3. Mark every external mutation and every ambiguity boundary.
4. Give logical mutations stable operation identifiers and establish how the receiving systems enforce duplicate prevention.
5. Classify failures as transient, permanent, ambiguous, or policy-bound.
6. Persist results at activity boundaries and retain the inputs needed to restore workspaces after partial execution.
7. Turn human review into a durable wait with scoped approval data and enforce required preconditions when the approved action executes.
8. Design cancellation, stale-worker protection, and cleanup before calling the workflow production-ready.
9. Store proof separately and reference it from accepted state transitions.
10. Kill workers deliberately and verify recovery against external truth.

This is usually enough to reveal whether a database-backed state machine remains simple or whether a dedicated workflow engine has become justified.

The important move is not adopting a particular platform. It is refusing to let conversation state impersonate execution state.

Conclusion: make progress survive the process

Bigger context windows improve what a coding agent can consider. Compaction, plans, and handoff files improve how it carries understanding across sessions. Those are necessary parts of long-running agent work.

They do not answer whether an external operation already happened, whether a retry is safe, whether an approval still applies, whether cleanup completed, or whether a new worker can resume after the old one disappeared.

Durable execution makes workflow state independent of process state. It separates deterministic control from failure-prone work, persists accepted results, and gives ambiguity an explicit state. Safe continuation also depends on recoverable workspaces and external systems that enforce the required mutation contracts. Retries, human waits, cancellation, evidence, and versioning all need those boundaries to be explicit.

The useful distinction is simple:

• context helps the agent decide what to do next
• a work contract defines what done means
• durable execution preserves what has actually happened
• reconciliation checks that belief against the outside world

If a coding-agent task is important enough to run for hours, change external systems, survive deployments, or wait for human authority, then its progress should not depend on one model context or one worker process staying alive.

The goal is not merely a longer-running agent.

The goal is work that can survive the agent, the worker, and the inevitable failure between two steps.

References

1. OpenAI, "Long-running work."
2. Anthropic, "Effective harnesses for long-running agents."
3. Anthropic, "Effective context engineering for AI agents."
4. Anthropic, "Harness design for long-running application development."
5. Temporal, "Temporal Workflow Execution overview."
6. Temporal, "Activity Execution."
7. Microsoft, "Programming model overview."
8. Temporal, "What is a Temporal Retry Policy?"
9. AWS Builders' Library, "Making retries safe with idempotent APIs."
10. Temporal, "Workflow message passing: Signals, Queries, and Updates."
11. GitHub, "Merge a pull request."
12. Microsoft, "Durable Task documentation."