Study track
Do these in order, each against the running lab worker, watching the UI. The lab repository is
lerta-orchestration-lab; every command below is in its README.
- Durability. Start
--fail 3, kill the worker between attempts, restart it. The run finishes. Nothing in the client noticed. Then kill the client mid-run and re-attach withorchestrator.handle(type, RunId("...")): the result was waiting. - Retries are the engine's, not yours.
RetryPolicyinRunOptionsis the whole retry story. Look at the attempt list in the UI. ThrowNonRetryableTaskErrorand watch it stop on attempt 1. - Ids are idempotency keys. Run
--id same-idtwice while the first is in flight:RunAlreadyExists. Switch toUSE_EXISTING: the second caller gets the first run's handle. Decide whichReusePolicyan OTP send wants and which a nightly recompute wants. - Cancellation is cooperative. It is delivered at the next heartbeat. A handler that never heartbeats cannot be cancelled, only timed out.
- Timeouts are two budgets.
attemptTimeoutandoverallTimeout. Unbounded retries without an overall budget is rejected; find out why by removing the check. - The agentic part. Run the agent flow with
--silent, open the run in the UI, read the event history. Kill the worker while the run is parked on the approval, restart it, send the approval withlertha flow signal. The flow resumes at thereceivecall with all its local state intact. The worker that finishes the run is not the one that started it. - Break determinism on purpose. Put a clock read or a
Randomin a flow body, do the kill-and-restart trick, and watch for the non-determinism error. Then run two builds of the worker on one queue and see version skew for yourself.
Lessons the first cohort learned the hard way: keep one worker tab, and make the approval wait long enough for a human.