PostgreSQL ARM64 on Nanos

Jerboa can run a real PostgreSQL server on an ARM64 Nanos guest, but it cannot use an unmodified upstream PostgreSQL binary. Nanos is a single-application unikernel: it does not provide Unix process creation (fork/exec) or the ordinary System V shared-memory/process model on which stock PostgreSQL relies.

This repository therefore pins the experimental threaded PostgreSQL fork at commit d5933309617388a7eb6b2e195352054edf438cb7 (PostgreSQL 11devel, October 2018). That commit is still the fork’s master HEAD: there is no newer maintained code to upgrade to. Status: experimental compatibility fixture. It is not a supported production database.

What the fixture patch changes

tests/fixtures/postgres-arm64/nanos.patch and openssl.patch contain the PostgreSQL delta. They do not patch Jerboa or the Nanos kernel.

Build recipe

The complete, reproducible input is in tests/fixtures/postgres-arm64:

initdb runs in the Linux ARM64 Docker build stage, not inside Nanos, because it starts helper programs. It initializes the cluster with:

pg_hba.conf contains only scram-sha-256 entries, with hostssl for all TCP connections, and the build fails if trust, password or md5 appear. Plaintext TCP sessions are rejected.

The superuser password is a BuildKit secret. It is neither a build argument nor a layer, and only its SCRAM verifier is stored in the cluster. Supply a server certificate and matching private key as separate BuildKit secrets. Unlike the password, the private key is deliberately installed in the packaged /db tree and seeded volume. Keep those artifacts private. Secret contents do not invalidate the build cache, so pass a fresh CLUSTER_NONCE whenever any credentials change:

docker build --platform linux/arm64 \
  --secret id=pgpass,src=/path/to/password-file \
  --secret id=pgcert,src=/path/to/server.crt \
  --secret id=pgkey,src=/path/to/server.key \
  --build-arg CLUSTER_NONCE="$(openssl rand -hex 16)" \
  --output type=local,dest=stage tests/fixtures/postgres-arm64

The resulting /db tree is packaged and used to seed a named Jerboa volume. The test client expects a certificate with the DNS SAN postgres.jerboa.test and receives its trusted PEM certificate through PGCA_BASE64. It never disables certificate or hostname verification. Use a certificate valid for the actual server name with other clients.

Clean shutdown

A cooperative stop works as follows:

  1. jerboa stop asks the HVF VMM to shut down, with a 35-second timeout that forces termination when it expires.
  2. The VMM presses the virtio-input power button. Nanos sends SIGTERM to the process and arms a 30-second shutdown timer.
  3. With shutdown.patch, PostgreSQL performs a fast shutdown:
    • open sessions are aborted and clients receive terminating connection due to administrator command;
    • the checkpointer writes a shutdown checkpoint, and the statistics file is written;
    • the postmaster logs database system is shut down and calls exit().
  4. Nanos syncs storage and the VM exits. The next boot logs database system was shut down at ... and runs no WAL recovery.

If PostgreSQL does not finish in time:

The patch is needed because the threaded fork keeps signal handlers per thread, but pqsignal() installs one process-wide sigaction. Nanos delivers a process-directed signal to an arbitrary thread, so the shutdown SIGTERM could run a backend’s handler, or none, instead of the postmaster’s. The patch makes four changes:

A process-directed signal that arrives before the postmaster registers its thread keeps the old behaviour.

Validation on 2026-09-14, Apple M5, reviewed bundle, stopping by VM ID:

The later integrated validation on the same M5 uses the newly built 0.0.0-integrated-closure bundle, including the daemon’s stop-by-name fix and the VMM storage-error latch. It stops by name, not the historical ID workaround:

These paths are under dist/integrated-closure/. Both runs use the same final PostgreSQL binary and fixture hashes. This remains experimental qualification, not approval for production or physical power-loss certification.

Statistics and autovacuum

Earlier fixtures ran with track_counts=off. Every boot logged could not resolve "localhost": Name or service not known, then disabling statistics collector for lack of working socket and autovacuum not started because of misconfiguration.

The cause was name resolution, not Nanos networking. PostgreSQL’s statistics collector binds a UDP socket to the result of getaddrinfo("localhost"), and the package had no /etc/hosts, so that lookup failed.

The fixture now ships /etc/hosts (127.0.0.1 and ::1 for localhost). A boot with only that file started the collector without warnings. The fixture also ships /etc/nsswitch.conf (hosts: files dns), which only makes the lookup order explicit. The acceptance script maps both into the package. No PostgreSQL or Nanos change was needed: the loopback UDP send, select() and receive self-test of the collector passes on Nanos. Both settings are pinned on on the command line.

The acceptance test checks that collection and autovacuum really work, rather than only that the warnings are gone:

Acceptance and durability test

Thread identities and query execution

SQL statistics and pg_backend_pid() use positive, non-recycled 32-bit IDs. PGPROC.thread_id separately holds the real pthread_t. Signal delivery resolves the public ID while holding ProcArrayLock, keeping the backend slot alive until the signal has been sent. Permission checks for SQL cancellation are preserved. pg_cancel_backend, pg_terminate_backend, protocol CancelRequest and cancellation of a blocking autovacuum worker use this separation. BackendKeyData carries the same ID as SQL; a wrong cancellation key or an expired ID cannot signal a backend. Signal-mask variables are explicitly initialized for each new pthread. A CancelRequest arrives before that connection has a PGPROC, so it retries conditional lock acquisition without joining a wait queue; startup timeout and termination signals are blocked only while attempting or holding the lock.

The same distinction applies to lock-group membership, serializable-transaction wakeups and buffer-pin waiters. A separate public leader ID fixes the early lost connection to parallel worker failure, but repeated parallel aggregates still hung in this old fork. The Nanos port therefore uses PostgreSQL’s existing leader-only fallback in CreateParallelContext. It does not launch parallel query workers, even if a client raises the parallel settings. Other operating systems retain the original parallel execution path.

The package defaults max_parallel_workers and max_parallel_workers_per_gather to zero to avoid unnecessary parallel plans. Multiple sessions still execute concurrently, and autovacuum remains enabled. The acceptance test deliberately overrides these settings, requires an EXPLAIN ANALYZE with zero launched workers and verifies repeated aggregates. This is a containment of the fork’s executor limitation, not a repair or qualification of parallel query execution.

The acceptance test exercises SQL cancel/terminate, permission denial, correct and incorrect protocol cancellation keys, stale IDs, and an actual autovacuum worker blocking an access-exclusive table lock. The server must remain usable.

Running the suite

Run on Apple Silicon macOS with Docker available. Explicitly select an updated bundle so an old VMM is not accidentally qualified:

JERBOA_TEST_BIN="/path/to/updated-bundle/bin" \
JERBOA_TEST_GO="$PWD/../firecracker-macos/experiments/hvf/build/guest-tools/go/bin/go" \
python3 scripts/test-postgres-arm64.py

Optional overrides:

The test uses its own temporary HOME, daemon socket, network, volumes, VM names, random password and two-day self-signed certificate. The password is redacted from retained evidence; the private key, package store and seeded data trees are excluded. The temporary build artifacts themselves contain the key and must remain private.

It runs these phases and fails on any violated criterion:

  1. SQL, settings and authentication. A separate ARM64 guest checks:

    • fsync, synchronous_commit, full_page_writes and data_checksums are all on;
    • password_encryption is scram-sha-256 and wal_sync_method is fdatasync;
    • all *_flush_after settings are 0, and SET backend_flush_after = 16 is rejected.

    It then runs CREATE/INSERT/UPDATE/DELETE, a unique-violation error with session recovery, ROLLBACK, an index scan, PL/pgSQL DO, VACUUM ANALYZE and CHECKPOINT. Empty and wrong passwords must fail with password authentication failed. The client refuses any server that accepts it without SCRAM, which catches a regression to trust. It verifies pg_stat_ssl, rejects plaintext startup, TLS below 1.2, an untrusted certificate and an incorrect server name, and opens concurrent TLS sessions during configuration reloads. It then runs the statistics and autovacuum checks described above.

  2. Cooperative stop. The server VM is stopped normally, removed, and recreated on the same volume, and the data is read back. The server is stopped by VM ID. With shutdown.patch, the next boot logs database system was shut down at without crash recovery; the combined local run on 2026-09-14 stopped in 0.311 seconds. The statistics and autovacuum restart checks then run. RESULT.json records whether the stop was clean and whether the counters persisted. A clean stop requires the counters to have persisted.

  3. Abrupt stop. A client commits one row per autocommit transaction and prints ACK n only after PostgreSQL acknowledges the COMMIT. It also runs CHECKPOINT every 250 commits. After a random number of acknowledgements, jerboa stop --force kills the VMM process. The killed server never sends a TCP reset over the userspace network, so the test captures the client’s ACK lines and then force-stops the client too. A replacement server on the same volume then runs crash recovery. The criteria after recovery are:

    • every commit whose acknowledgement the host observed is present;
    • every payload is intact;
    • a full heap scan passes page-checksum verification;
    • index and heap counts agree;
    • amcheck bt_index_check passes on both indexes;
    • the earlier SQL-phase data is intact;
    • the server log contains no could not flush dirty data, PANIC:, could not fsync, invalid page or checksum failure;
    • the recovered server’s collector and autovacuum launcher are running.

    Autovacuum, with a 1-second naptime, runs during these rounds, so a kill can land during automatic maintenance.

  4. Negative control. The same abrupt stop runs with fsync=off on a separate volume. The outcome is recorded, not asserted. This shows whether the kill test can detect a durability violation at all. In the local run on 2026-09-13, the fsync=off cluster could not recover: startup failed with PANIC: replication checkpoint has wrong magic 0 instead of 307747550. The fsync=on rounds on the same kill path recovered every acknowledged commit.

Evidence and a RESULT.json with per-round numbers are retained under dist/claude-closure/postgres/acceptance-* on success or failure.

Durability model and limits

The storage chain on macOS is:

  1. PostgreSQL fsync/fdatasync.
  2. Nanos commits page-cache writes and the TFS metadata log.
  3. Nanos sends a virtio-blk FLUSH request.
  4. The HVF VMM (firecracker-macos/src/hvf-vmm/native/devices.c) writes through pwrite() and answers FLUSH only after host fcntl(F_FULLFSYNC) succeeds. Interrupted calls are retried; unsupported operations and I/O errors are reported to the guest, without silently falling back to weaker fsync().

What this does and does not cover:

Use matching updated VMM binaries; an older bundle does not gain this behaviour by updating the fixture alone.

Other limitations:

Background on the port and threaded fork: