Persistencealpha

How moxzid stores actor state, what survives a restart or a crash, and how to back it up.

With -s DIR, moxzid keeps every actor's state in DIR and restores it on startup. State survives a clean shutdown, a crash, and kill -9. Without -s, state is held in memory and lost when the process exits.

Surviving a crash#

moxzid -m actors.json -s ./state

for i in 1 2 3; do
  curl -s -w '\n' -d '[]' http://127.0.0.1:7000/call-json/counter/inc
done

[1]
[2]
[3]

Kill the server without letting it shut down, then start it again:

pkill -9 moxzid

moxzid -m actors.json -s ./state
curl -s -w '\n' -d '[]' http://127.0.0.1:7000/query-json/counter/get

[3]

How it works#

moxzid writes each update message to the actor's log, and flushes it to disk, before running it. A message that was accepted is never lost.

It also writes a snapshot of each actor's memory. On startup it loads the latest snapshot and replays the log entries made after it. A message that trapped the first time traps again during replay, so the result is the same.

Non-deterministic inputs are recorded rather than repeated, so replay produces the same state:

Small actors are snapshotted after every message, so their log is usually empty and startup is fast. Actors larger than --snapshot-below (default 50 MB) are snapshotted less often and rely on the log.

Two kinds of work are always snapshotted, whatever the actors' sizes: a message that calls other actors, and a timer that fires between messages. Every actor such work reached is snapshotted when it finishes, so it survives a crash exactly once and is never replayed. Expect a snapshot of each large actor such work touches.

With --snapshot-below 0, no actor is snapshotted after its own messages, and recovery replays them from the log:

moxzid -m actors.json -s ./state --snapshot-below 0
curl -s -w '\n' -d '[]' http://127.0.0.1:7000/call-json/counter/inc
curl -s -w '\n' -d '[]' http://127.0.0.1:7000/call-json/counter/inc
pkill -9 moxzid
moxzid -m actors.json -s ./state
curl -s -w '\n' -d '[]' http://127.0.0.1:7000/query-json/counter/get

[4]
[5]
[5]

An upgrade always writes a new snapshot, so log entries are never replayed against different code.

Timers are saved too. A timer set before a crash fires after the restart:

import Timer "mo:core/Timer";

actor {
  var fired : Nat = 0;

  public func schedule(seconds : Nat) : async () {
    ignore Timer.setTimer<system>(#seconds seconds, func() : async () {
      fired += 1;
    });
  };

  public query func count() : async Nat {
    fired
  };
};

curl -s -w '\n' -d '[3]' http://127.0.0.1:7000/call-json/reminder/schedule

pkill -9 moxzid
moxzid -m actors.json -s ./state

sleep 4
curl -s -w '\n' -d '[]' http://127.0.0.1:7000/query-json/reminder/count

[1]

What's on disk#

find state -type f | sort

state/16556e94d03598adc98c3c04c8dc20a4e870491d996352c07908e35503/actor.json
state/16556e94d03598adc98c3c04c8dc20a4e870491d996352c07908e35503/device.log
state/16556e94d03598adc98c3c04c8dc20a4e870491d996352c07908e35503/history.jsonl
state/16556e94d03598adc98c3c04c8dc20a4e870491d996352c07908e35503/module.ref
state/16556e94d03598adc98c3c04c8dc20a4e870491d996352c07908e35503/outcalls.log
state/16556e94d03598adc98c3c04c8dc20a4e870491d996352c07908e35503/snapshot.bin
state/16556e94d03598adc98c3c04c8dc20a4e870491d996352c07908e35503/wal.log
state/child-allocator.bin
state/modules/d39e35b2d74f953a036cf60ba3f0004650bff51c48eb40bc59601f0151995746.wasm
state/principal.seed
state/runtime.lock

What isn't kept#

An actor whose module can't be loaded at startup is reported as LOST ACTOR_ID REASON and skipped. The other actors start normally.

Shutting down#

SIGTERM or Ctrl-C (SIGINT) shuts down cleanly: the current message finishes, no new ones start, every actor is snapshotted, and the process exits with status 0. The next start reads snapshots and replays nothing.

Startup recovery runs one actor at a time. Requests other than health checks get 503 until it finishes, so wait for /health/ready. To keep restarts fast, shut down with SIGTERM rather than killing the process.

Backups#

Stop the server (Ctrl-C, or systemctl stop moxzid if it runs as a service), then archive the state directory:

tar -czf backup.tar.gz state

To restore, stop the server, replace the directory with the archive's contents, and start the server:

tar -xzf backup.tar.gz
moxzid -m actors.json -s ./state
curl -s -w '\n' -d '[]' http://127.0.0.1:7000/query-json/counter/get

[5]

Don't copy the directory while the server is running; you can capture a half-written file.

Many idle actors#

Each running actor holds its memory even when idle. --live-cap N keeps at most N actors loaded; beyond that, the least recently used actor is snapshotted and unloaded, and loaded again by its next message. The first message to an unloaded actor takes tens of milliseconds longer. Use it for many mostly idle actors, not a few busy ones. --live-cap requires -s.

Limitations#