Durable execution at the speed of the disk

What a step costs

Durable execution lets you write a long-running process as ordinary code. A checkout that quotes a price, charges a card, waits ten minutes and asks for a review is one function with four awaits. If the machine dies halfway through, the function carries on from where it stopped instead of starting over. Temporal made the idea mainstream; Restate, DBOS, Inngest, Hatchet and others each have their own take.

Underneath, every one of them keeps a journal. When a step finishes, the engine records its result somewhere durable before the code moves on. After a crash, it runs the code again from the top and hands back the recorded results instead of running those steps again, so the code takes the same path with the same values.

Recording a result durably is the whole job, and it’s where engines differ by orders of magnitude. Published figures run from about a millisecond to a few hundred milliseconds per step (Fig. 1). The thing that actually makes bytes survive a power cut, one flush on a datacenter NVMe drive, takes about ten microseconds. Nearly all of a step’s cost is something other than the disk.

That cost shapes how people use these systems. At several milliseconds a step, you make steps coarse. You keep durability for the checkout and leave it off the agent loop that makes four hundred tool calls. You put a queue in front of anything that runs once per message. A step that costs what the disk costs is a different tool: you can make every call durable and stop rationing it.

Fusee is a durable execution engine built on that premise, an open-source library for Rust, TypeScript, Python, Go and Java. On a server with local NVMe, a step that depends on the one before it costs 14 µs, against 12 µs for the disk write it waits for, and it stays at 14 µs up to 51,000 steps a second. At 6.5 million steps a second, its p99 is still under a millisecond. This post is about how it gets there, and how we tried to break it.

Where the time goes

Follow one step through a typical server-based engine. A worker process polls a task queue and receives the step. It runs the step’s function, then calls the orchestrator to report the result. The orchestrator writes the result to its database in a transaction, which ends in that database’s own flush and often in replication to other machines. Then it puts the next task on a queue, and a worker polls for it.

Count what isn’t the disk. There are network hops to the orchestrator and back. There’s a database transaction per step, and a flush per transaction, because the database can’t know that a thousand other steps are about to commit too. And there’s polling between steps. Each is reasonable on its own. Together, they’re why a step costs milliseconds.

None of this is a mistake. A central orchestrator buys real things: workers in any language, one view of every workflow, one place to operate. If your workflows span a fleet, some of that distance is the job. But a great many workflows run inside a single service, and they pay for distance they never use.

So the first decision was the simplest one. Fusee is a library. Workflows are async code in your process, and the engine writes a log next to it. There’s no orchestrator to call, no database between the code and the disk, and no queue between steps. The log is the database. When you do want many processes to share one log, it can live in a server of its own, as we’ll see in section 10.

That leaves the disk. Here is what one flush costs on a range of drives, measured with fio by Mark Callaghan at Small Datum in January 2026:

A drive with power-loss protection can acknowledge a flush as soon as the data is in its capacitor-backed cache, so it’s fast. A consumer drive has to reach the flash, so it’s slow, which makes a laptop the worst place to benchmark any of this. On macOS, plain fsync doesn’t even ask the drive to empty its cache; only F_FULLFSYNC does, it takes milliseconds, and it’s what Fusee uses there. Every figure in this post comes from servers. fusee disk-check <dir> measures yours.The vendors’ figures in Fig. 1 come from Vercel’s benchmark, Hatchet’s documentation, Windmill’s, Temporal’s, Restate’s and DBOS’s own write-ups.

Write, then wait

The obvious engine is the one you’d draw on a whiteboard. When a step returns, append its result to the log, flush, then hand the result to the code. It’s plainly correct: nothing the code does next can depend on a result the disk doesn’t have.

It’s also slow in a particular way. Every step waits for its own flush, and flushes on one drive happen one at a time. With one workflow running, a step costs one flush. With a thousand, each waits behind the others’ flushes, and at one flush per record a consumer drive tops out at a few hundred to a few thousand steps per second. You can watch it fall behind in Fig. 3: choose “Flush each record” and raise the load.

The textbook fix is group commit: wait a little, collect whatever arrives, flush it all at once. Databases have done this for decades. The question is how long to wait. A fixed window adds itself to every step when the system is quiet, which is exactly when you’d expect it to be fastest; libDSE, a recent research system we’ll meet again, batches every 10 ms. Under load, a window is either too short to fill or too long to keep latency flat.

There’s a deeper problem with write-then-wait. It makes the code wait even when nothing it’s about to do leaves the process. Between two steps, workflow code is deterministic. It computes, it calls ctx.now(), maybe it draws a number from ctx.random(), and every one of those is journaled. None of it is visible to anyone. Why should it wait for the disk?

Run ahead of the disk

It shouldn’t, and this isn’t our idea. In Rethink the Sync (OSDI 2006), Nightingale and colleagues showed that a file system can offer synchronous durability without blocking the application, as long as nothing externally visible depends on data that isn’t durable yet. They called it external synchrony. Cloudflare’s Durable Objects apply the same rule with output gates. Netherite lets durable orchestrations run ahead of persistence. libDSE generalizes it to speculation across services. What we did was take it all the way down to the disk’s floor inside an embeddable library, and test it harder than felt reasonable.

In Fusee, appending a record makes it visible to the code at once, and the log makes records durable in the background. Exactly two kinds of things wait for the disk:

1. A step’s function waits for the records before it. The function is where a workflow talks to the world, for example by charging a card, so the inputs it sees must not be something a crash could take back. We call this the step’s gate.
2. Outputs wait for the records behind them. Acknowledgments, results returned to callers, refusals like “this workflow has already finished”, and messages to other workflows are all released only once the records they depend on are durable.

Everything else runs ahead. Fig. 2 follows one workflow through it.

Notice what this does and doesn’t buy on a chain of dependent steps. Each step’s function still waits for the flush of the step before it. That’s the floor, and we can’t remove it without changing the contract: speculating through external calls and rolling back when something fails, as libDSE does, is a different guarantee, and Fusee doesn’t make it. What running ahead removes is everything else. The code’s own work overlaps the flush, the result is handed over without a round trip, and, as the next two sections show, every workflow on the machine shares the same flush.

There is one way to give a step’s gate back. A step the code marks read-only changes nothing outside the process (a lookup, a fetch, a model call), so it runs at once, before the records ahead of it are durable. If a crash then takes those records back, the read runs again with different inputs, which is harmless for a read and wrong for anything else; that’s why it’s opt-in, step by step. The read’s own result is recorded like any other, and the next step with effects waits for it. Two reads followed by a charge cost one flush instead of two.

On a server, the effect is easy to see, because the step time lands on the flush time. In a chain of a thousand dependent steps, a step costs 14 µs at the median, on a drive whose flush takes 12 µs. In memory mode, with no disk at all, a step costs 0.22 µs of CPU. That’s the engine; the rest is the disk.

One flush in flight

Group commit without a window turns out to be simple. Keep exactly one flush in flight. When it finishes, the next one starts immediately and takes everything appended since. If nothing else is happening, a lone record is flushed the moment it’s appended, without waiting for company. If the engine is busy, the batch is whatever arrived during the previous flush, so batches grow with load on their own.

A record waits between one and two flush times at any load the disk can carry: at most one for the flush in progress to finish, and one for its own. Throughput is however many records fit in a batch. Fig. 3 simulates all three designs. With “One flush in flight”, raise the load and watch the wait barely move while the records per flush climb.

This is where the name comes from. A fusee is the cone-shaped pulley in a mechanical clock that evens out the mainspring’s pull, so the clock keeps the same pace whether the spring is fully wound or nearly run down. Fusee makes the same promise about load: the same step time whether the engine is idle or busy, with p99 close to p50. In our runs, the median step didn’t move at all between 3,200 and 51,000 steps a second. Raising the offered load 512-fold, to 1.6 million steps a second, moved it by 37 µs, while the records in each flush grew from 1.0 to 8.6.

One detail took longer than it should have. When the disk is fast and no flush is in flight, the thread that appended a record writes it itself instead of handing it to a writer thread, because two thread handoffs cost more than the flush does. A lone workflow on a fast drive pays for the disk and very little else.

The disk counts writes

Fusee uses more than one core through shards. A workflow ID hashes to a shard, and each shard is a single thread that owns its workflows and their timers. In our first design, each shard also owned a log, with a flush of its own in flight. On a laptop, that looked ideal. On a cloud server with eight shards, it fell apart under load in a way we hadn’t predicted.

An NVMe drive doesn’t only have a bandwidth limit. It also has a limit on how many small synchronous writes it can finish each second; the instance storage we measured topped out at around 165,000. Eight shards with a flush in flight each put eight small writes on the drive at once, and the drive answered every one of them more slowly than it would have answered a single write carrying all eight shards’ records. Past a couple of hundred thousand steps per second, the engine was spending the drive’s write budget, not its bandwidth. Batches grew, but so did the wait, and the slowest steps became more than ten times slower.

So shards stopped owning files. Every shard on a drive now appends to one shared log. Each shard encodes its records into a buffer of its own, and whoever writes next takes every shard’s pending records and writes them as one batch: a header, then a section per shard. Under load, a log thread starts the next write the moment the drive can take one, with up to two in flight; when the log is idle, the thread that appended still writes inline. The drive sees one stream of writes, never more than two deep, instead of eight streams queued behind each other, and a record waits for about one write instead of several. Fig. 4 puts both layouts on the same simulated drive. At low load they’re the same; push the load up and watch where the writes queue.

Rebuilding the log was also a chance to lower the floor itself. By default, Linux writes through the page cache and then asks for a flush: a copy and two system calls. Writing with O_DIRECT, each write carrying RWF_DSYNC, is one call that goes straight to the device, and on the drives we measured it took almost a third off the cost of a durable write. Fusee calibrates the method for each device the first time it opens a directory, and falls back to buffered writes where the filesystem refuses direct I/O.

The new format also lets the log carry on from where it stopped. Every batch includes a checksum of the batch before it, so the log is a chain, and the position its writer had already acknowledged when the batch was submitted. Recovery follows the chain and stops at the first batch that doesn’t continue it. Bytes left by a torn write, or by a recycled file’s earlier life, can never join the chain. If a well-formed batch turns up past the end, written after something beyond the end had been acknowledged, then acknowledged data was lost, and Fusee refuses to open the directory rather than guess. The rule is small enough to model-check, so we checked it in TLA+. It’s also what lets old log files be recycled instead of zero-filled, so opening a directory takes milliseconds however many times the process has restarted.

Replay

Running ahead only works if recovery can rebuild exactly what the code saw. Fusee’s journal numbers every operation a workflow makes through ctx in the order the code calls it: steps, sleeps, waits for signals, children, now(), random(), the results of update handlers. Results are recorded in the order they’re handed to the code. After a crash, the code runs again from the top, and the journal hands back the recorded results in that same order. The code takes the same path, and step functions whose results are recorded don’t run.

The order matters most where things race. A signal, a timer and a child’s result can arrive in any order while a workflow runs. The journal records the one order the code saw, and replay delivers them in it, not in whatever order they’d happen to arrive the second time.

When replay can’t match, Fusee says exactly where. If the code calls a different operation than the journal recorded at that position, the run stops with an error that names both. The run isn’t marked failed; it waits for code that matches. Fig. 5 walks through one.

const quote = await ctx.step("quote", () => pricing.quote(order));const charge = await ctx.step("charge", () => payments.charge(quote, key));const refund = await ctx.step("refund", () => payments.refund(order));await ctx.sleep("cool-off", "10m");return charge.id;

Two features exist to keep you away from that error. Versions pin each run to the code it started with, so a deploy changes only new runs. Patching, ctx.patched("refund-rounding"), lets a workflow that has been asleep for three months pick up a fix at the point its code reaches it, without shifting its journal. And before a deploy, fusee export --sample 1000 copies recent journals, and a Replayer runs the new code against them in CI, reporting the first difference by step name.

What a crash takes back

Running ahead means the process holds state the disk doesn’t have yet. A crash takes it back. By construction, that’s fine: nobody outside the process saw it.

“By construction” is carrying the whole design. It has to be true for every way information leaves: every acknowledgment, every refusal, every query answer, every message to another workflow and every input a step’s function sees. Get one wrong and the engine works almost all the time, because flushes are fast and crashes are rare. Then one day it charges a card twice.

Fig. 6 is a two-step workflow: quote a price, then charge it with an idempotency key. Drag the crash anywhere. With the gate, no crash makes the payment service see two different inputs for one key. Now remove the gate and crash just after the charge goes out.

The numbers in that figure aren’t made up. They’re what our simulator reported when we removed the gate on purpose: service key wf-5/7/3 was called with input 855791 and later with 1001727: a step ran on state a crash took back. The quote’s result was in memory but not on disk, the charge saw it, and the crash took it back. On recovery, the quote ran again, the pricing service answered differently, and the charge went out a second time with the same key and a different amount. An idempotency key only makes identical requests safe.

Our own history has a quieter example. For a while, a refusal (“this workflow has already finished”) could reach a caller before the record of the finish was durable. A crash then took the finish back, and the workflow ran on after its caller had been told it was done. The simulator produced the crash between the refusal and the flush.

Trying to break it

Fusee’s guarantee fits in a paragraph:

An acknowledged start, signal or step result is never lost. A recorded step result never changes. A step’s function runs at least once, and again only if the process failed before its result was durable (or, for a step marked read-only, before the records ahead of it were); ctx.key() gives a stable key that makes its effect exactly once at services that accept one. Nothing leaves the process based on state a crash could take back.

These are the journaling semantics of DBOS, Restate and Temporal activities. Fusee doesn’t weaken them anywhere; it changes their cost. The rest of this section is how we test that claim, starting with the method we lean on most.

A hostile, repeatable world

The core test is a deterministic simulator, in the tradition of FoundationDB and TigerBeetle. The real shard runtime, log, journal machine and Rust SDK run in one thread under a seeded scheduler, so a seed reproduces a run exactly. Everything around them is hostile:

• The disk, at a crash, loses the writes since the last flush, or lands them whole, or tears them into random sectors (sometimes garbage). Writes in flight land in any order, recycled log files hold stale bytes that look valid, and flushes fail at a configured rate.
• The process crashes at random times, and messages between shards die with it.
• An external service logs every call with its idempotency key and input, and answers differently every time. A step that runs again gets a different value, which turns a lost result from silently equivalent into visible.

When a flush fails, the engine stops and the process restarts; it never retries the flush. After a failed fsync, the kernel may already have marked the dirty pages clean, so a retry can succeed without the data reaching the disk. PostgreSQL’s developers learned this the hard way in 2018, and Rebello and colleagues (ATC 2020) found widely used applications mishandling it. Recovering from the log is the only answer we trust.

After every recovery and at the end of every run, checkers compare what happened with what’s allowed. Every acknowledged start and signal exists. A reported result never changes. The service never saw one key with two inputs. An accepted update ran once. Every query answer is a state the journal recorded. A schedule never fired twice for one time. A limit never had more permits out than its size. And once the faults stop, every workflow finishes.

Every change to the log runs here before it touches a real disk. For this release, the simulator ran 300,000 seeds, 2.1 million crash-recoveries and 19.9 million workflows without a single violation.

Checking the checkers

We also tested the simulator itself by deliberately introducing eight bugs, one at a time. It caught all eight.

Several of those tests reproduce bugs we encountered in the engine. The simulator also found torn headers on freshly activated log files, checkpoints that dropped messages whose records weren’t durable yet, batches torn across a file boundary, and a finished run’s leftover timer loading it back and hiding a newer run with the same ID. Each needed a crash at exactly the wrong moment, the kind a test suite on real machines might hit once a year.

Real processes, real disks

The simulator runs our code against a model of the world, so we also run it against the world. fusee-crashtest kills a real engine process with SIGKILL under load, over and over, then checks the same properties on real files: in 20 kills under load, 1,521 acknowledged workflows, every workflow finished, and every step that ran a second time saw the same inputs. One layer down, LazyFS, a filesystem built to lose unsynced data on command, and dm-log-writes, which records every block an ext4 filesystem writes so it can be replayed up to any cut, test what survives a power cut: 20 kills with LazyFS dropping its unsynced data at each one recovered all 864 acknowledged starts, and four block-replay cuts recovered 9, 69, 187 and 401, with every acknowledged workflow retained.

The cross-SDK conformance suite runs the same scenarios in Rust, TypeScript, Python, Go and Java and requires identical journals. Fuzzers cover record decoding, log scanning, protocol frames, the run store, checkpoints and the journal matcher. Six TLA+ models, of the output rule, the log’s acceptance rule, the replication protocol, read leases, membership change and checkpoint installation, pass TLC with no invariant violation.

One thing we haven’t done is pull the power on a drive with power-loss protection. LazyFS and block replay stand in for it, and the guarantee rests on the drive honoring its flushes.

Three copies

Local mode survives a process crash, and power loss if the drive honors its flushes. It doesn’t survive losing the machine, and on a cloud that matters more than it sounds. The fast drives there are instance storage, which disappears with the host. The volumes that outlive a host sit across the network, where one flush takes from half a millisecond to a millisecond.

So on a cloud, the fastest write that survives losing a machine isn’t a write to any one disk. It’s a write to the local NVMe of a majority of three machines nearby: one network round trip plus one flush, about 50 µs in a placement group, which is faster than a single flush on a network volume. Replicated mode is that. Three or five instances of your application form a group, workflows run on the primary, and when the primary dies, a backup takes over from its own disk. On a cloud, it’s the mode we’d run in production.

The protocol is Viewstamped Replication, in the style of Liskov and Cowling’s Viewstamped Replication Revisited (2012). The primary sends each batch of the log to the backups as soon as it’s appended, before its own flush. Backups flush and report how far they’re durable. A batch commits once any quorum holds it durably in the current view, whether or not the primary is part of that quorum, so a slow disk on the primary doesn’t slow the group. Step gates and outputs wait for the commit position instead of the local flush. Fig. 7 shows a failover.

Because the unit of replication is a batch of the shared log, one group carries every shard on the primary. On each node, a single dedicated thread reads the network, runs the protocol and writes the log, so a commit never waits on a handoff between threads. A backup can keep its engine open while it waits, so a promoted backup reopens nothing and only replays, from memory, what its shards recorded since their checkpoints: from a primary’s death to its successor’s first new commit takes 30 ms at the median. And a primary cut off from its quorum steps down instead of answering from stale state.

Replicas also have to survive their own disks. Batches form a hash chain, and a replica that finds a damaged record in the middle of its log doesn’t truncate there, which could throw away committed work. It fetches the range from its peers, verifies it against the chain and rewrites its log. This is protocol-aware recovery, from Alagappan and colleagues (FAST 2018).

Replication is where the simulator earned its keep: most of the protocol’s eleven rules started as bugs it found. Our favourite: a restarted primary must never lead its old view again, because entries that a quorum of backups committed may be missing from its own disk. Another: a log may only be replaced atomically, once the complete new log is in hand, and the saved “last normal view” must be written after the log it vouches for. With those rules in place, 5,000 seeds, 12.8 million committed entries, 65,743 crashes and 13,142 repairs of damaged entries ran clean through dropped, duplicated and reordered messages, partitions and torn writes. On real machines, we killed whichever process was primary 1,000 times; every acknowledged workflow finished, and no service key was ever called with two different inputs.

That makes three ways to run the same engine, chosen when you open it:

Server mode puts the log in a process of its own, fusee server, and lets many workers share it, over shared memory on one host or over TCP across hosts. Shards are leased one at a time, so every worker runs workflows, and a worker that hangs loses only its own shards. It’s how Fusee grows past one process, and it’s the same log as local mode, one hop away.

The rest of a workflow engine

Fast steps aren’t a workflow engine. Teams that run Temporal in production rely on a long list of features, and Fusee has them in all five SDKs: queries; updates and update-with-start; step heartbeats; completing a step later with a token; continue-as-new; patching; terminate and reset; schedules with time zones, overlap rules and backfill; search attributes; concurrency and rate limits; payload encryption; replay testing; and workflow ID and child close policies.

Each one is a record type in the journal plus an SDK call, with the same durability rules as steps, the same simulator checkers and the same cross-SDK scenarios. That uniformity did most of the design work:

• A query writes nothing, but its answer leaves the process, so the answer waits until the state it read is durable.
• An accepted update is appended with its ID and delivered at its place in the journal. A retried call with the same ID gets the recorded result instead of running the handler twice.
• A schedule is itself a Fusee workflow built on durable timers, so there’s no separate scheduler. Each run’s ID is the schedule’s name plus its scheduled time, so one time never fires twice, even across crashes and backfills.
• Limits aren’t journaled at all. A step waiting for a permit holds nothing else, and after a crash it simply asks again.
• Payloads are encrypted with AES-256-GCM before they become records, so the log, the replicas and a Fusee server hold only ciphertext.

ctx.onUpdate("addItem", {
  validate: (item) => (item.qty > 0 ? undefined : "quantity must be positive"),
  handle: async (item) => {
    const price = await ctx.step("price", () => pricing.quote(item));
    cart.push({ ...item, price });
    return cart.length;
  },
});
ctx.onQuery("cart", () => cart);
const approved = await ctx.awaitCompletion("approval", (cb) => mail.sendApproval(cb.url ?? cb.token));
if (ctx.journalSize() > 1000) return ctx.continueAsNew(state);

Around the engine are the things you need to operate it: OpenTelemetry traces and metrics from the Rust, TypeScript and Python SDKs, a web UI over every run and its journal, TLS between workers, servers and replicas, continuous backup of the log to object storage with point-in-time restore, and worker threads that let one Node process use every core.

And the tool that found our bugs is yours to use. Fusee’s simulator can run your own workflows under its seeded scheduler, with time that skips ahead and the same injected crashes and disk faults. A test of a three-month workflow with a crash in the middle runs in milliseconds and replays exactly from its seed. The guide shows how.

Measured at matched durability

Benchmarks of durable execution are easy to get wrong, usually in the author’s favour. The common mistake is to compare an engine that makes every step durable before the next one starts with one that writes asynchronously, or holds results in memory until a batch fills. So we hold every comparison to one rule: both engines make a step’s result durable before the next step starts, release outputs only after that, and survive the same failures. Configurations that persist asynchronously, such as Temporal’s local activities or Inngest with checkpointing, are in the full results, labeled, and never in a headline.

Every engine ran on the same class of machine with local NVMe, five trials each, in rotated order, because run order alone can flip rankings at low load. Timestamps are taken inside the step functions, in the process that runs them. Open-loop latency is measured from each workflow’s scheduled start, so queueing is never hidden.

On one machine, a Fusee step is 20 µs at the median, 3 µs above the write it waits for; the nearest engine at the same durability, DBOS on SQLite, takes 0.83 ms, about forty times as long. On three machines, Fusee’s replicated step is 132 µs against 1.6 ms for Restate, about twelve times less. The lead is thinnest in the tail of Fusee’s own server mode, whose p99 on this host ranged from 59 µs to 838 µs between trials, and wherever the disk itself is slow: on a network volume with a 1 ms flush, a Fusee step costs the flush plus 21 to 39 µs, and the engine stops being what you wait for.

The harness and every engine’s driver and configuration are public, in bench/, with the method in the benchmark notes. If we’ve configured your engine badly, tell us, and we’ll rerun it.

Some things won’t get faster. A step that calls a model for 800 ms spends 800 ms in the model, and the engine’s overhead disappears from its latency. It doesn’t disappear from the bill. Fusee spends 9.7 µs of CPU on a step at 100,000 steps per second, and a parked workflow, one waiting on a timer or a signal, costs about 50 bytes of memory, so ten million of them fit in under 500 MiB.

Try it

Fusee is open source, under the MIT or Apache 2.0 license, at github.com/bitsinfra/fusee. It’s at 1.0: the guarantees in this post are tested as stated, and its APIs follow semantic versioning.

The installation guide covers Rust, TypeScript, Python, Go and Java. Here’s the checkout in TypeScript.

import { Engine, signal, workflow } from "@bitsinfra/fusee";

const review = signal<boolean>("review");

const checkout = workflow("checkout", async (ctx, order: Order) => {
  const quote = await ctx.step("quote", () => pricing.quote(order));
  const charge = await ctx.step("charge", (s) => payments.charge(quote, { idempotencyKey: s.key, signal: s.signal }), {
    retry: { attempts: 5, max: "1m" },
  });
  await ctx.sleep("cool-off", "10m");
  const approved = await ctx.waitFor(review, { timeout: "1d" });
  return approved ? charge.id : await ctx.step("refund", () => payments.refund(charge.id));
});

const engine = await Engine.open({ dir: "./data", workflows: [checkout] });
const run = await engine.start(checkout, "order-123", order);
await run.signal(review, true);
console.log(await run.result());

If you’re running a workflow where the engine is the slow part, we’d like to hear what you measure. You can share a benchmark or report a problem in the Fusee issue tracker.