Running a skaidb cluster

skaidb is leaderless: every node serves reads and writes, data is placed on a consistent-hash ring (virtual nodes), and each key is replicated to the next replication_factor nodes clockwise. Writes wait for a tunable quorum; reads gather from replicas and resolve by last-writer-wins (with a deterministic tie rule — an exact cross-node HLC stamp collision resolves the same way on every replica, never by arrival order). Clustered UPDATEs additionally serialize per row through a per-key consensus round, so concurrent read-modify-writes of one row never lose updates — see the UPDATE semantics under Statements in QUERY_SYNTAX.md. There is no special "primary" to configure.

This guide covers running multiple skaidb server nodes. For the mechanics of how membership changes and data rebalances, see RESHARDING.md; for install, see INSTALL.md.

Contents

• How a node decides it's clustered
• Ports
• Form a cluster (static seed list)
• Three nodes on three machines
• Three nodes on one machine (local test)
• Config-file and env-var equivalents
• Replication factor & consistency
• Per-table placement (RF overrides & pins)
• Verify the cluster
• Add or remove nodes at runtime (online resharding)
• Anti-entropy: repair & space reclamation
• Internode security
• Witness nodes
• Operational notes & limitations

How a node decides it's clustered

• seeds is empty (the default) → the node runs standalone: a single local engine, no internode networking.
• seeds is non-empty → the node runs as a cluster member. The seed list is the full membership: every entry is a member's internode address host:internode_port, and the list must include this node itself.

A node's identity on the ring is bind_addr:internode_port. For the node to be part of the ring, that exact string must appear in seeds. bind_addr must therefore be the address other nodes use to reach this one — a routable IP/host on a real cluster, not 0.0.0.0. All nodes should be given the same seed list.

Ports

Each node listens on three ports, all on bind_addr:

Purpose	Config	Default	Who connects
Client binary (fast path)	server.quic_port	7000	applications / drivers
Client REST	server.rest_port	7080	curl, HTTP clients
Internode RPC	cluster.internode_port	7100	other skaidb nodes
Prometheus metrics	observability.prometheus_port	9090	scrapers

The internode port must be reachable between every pair of nodes — open it in your firewall/security group across the cluster. Client ports only need to be reachable by clients.

Form a cluster (static seed list)

The supported way to stand up a cluster is to give every node the same seeds list at startup. Each node must also be told bind_addr + internode_port so its own identity (bind_addr:internode_port) matches its entry in seeds, and all nodes must share the same replication_factor.

Three nodes on three machines

Machines 10.0.0.1, 10.0.0.2, 10.0.0.3, internode port 7100 on each, RF 3:

# On 10.0.0.1
skaidb --data-dir /var/lib/skaidb \
  --bind-addr 10.0.0.1 --quic-port 7000 --rest-port 7080 \
  --internode-port 7100 \
  --seeds 10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100 \
  --replication-factor 3 \
  --default-read-consistency QUORUM --default-write-consistency QUORUM

# On 10.0.0.2 — identical, but --bind-addr 10.0.0.2
skaidb --data-dir /var/lib/skaidb --bind-addr 10.0.0.2 \
  --quic-port 7000 --rest-port 7080 --internode-port 7100 \
  --seeds 10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100 \
  --replication-factor 3 \
  --default-read-consistency QUORUM --default-write-consistency QUORUM

# On 10.0.0.3 — identical, but --bind-addr 10.0.0.3
skaidb --data-dir /var/lib/skaidb --bind-addr 10.0.0.3 \
  --quic-port 7000 --rest-port 7080 --internode-port 7100 \
  --seeds 10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100 \
  --replication-factor 3 \
  --default-read-consistency QUORUM --default-write-consistency QUORUM

Only --bind-addr differs between nodes; the --seeds list is identical everywhere. Start order doesn't matter — a node tolerates peers that aren't up yet (writes/reads just need their quorum).

Three nodes on one machine (local test)

Same idea with 127.0.0.1 and distinct ports + data dirs per node:

SEEDS=127.0.0.1:7100,127.0.0.1:7101,127.0.0.1:7102

skaidb --data-dir ./n1 --bind-addr 127.0.0.1 --quic-port 7000 --rest-port 7080 \
  --internode-port 7100 --seeds $SEEDS --replication-factor 3 &
skaidb --data-dir ./n2 --bind-addr 127.0.0.1 --quic-port 7001 --rest-port 7081 \
  --internode-port 7101 --seeds $SEEDS --replication-factor 3 &
skaidb --data-dir ./n3 --bind-addr 127.0.0.1 --quic-port 7002 --rest-port 7082 \
  --internode-port 7102 --seeds $SEEDS --replication-factor 3 &

Each node's 127.0.0.1:<internode_port> matches one entry in $SEEDS.

Config-file and env-var equivalents

Every flag is also a TOML key and an environment variable. A per-node config file (skaidb --config /etc/skaidb.toml):

[server]
bind_addr = "10.0.0.1"      # this node's reachable address
quic_port = 7000
rest_port = 7080
data_dir  = "/var/lib/skaidb"

[cluster]
seeds = ["10.0.0.1:7100", "10.0.0.2:7100", "10.0.0.3:7100"]
internode_port = 7100
replication_factor = 3
vnodes_per_node = 256
default_read_consistency = "QUORUM"
default_write_consistency = "QUORUM"

Or env vars (handy for containers; CLI flags override these, which override the file):

export SKAIDB_BIND_ADDR=10.0.0.1
export SKAIDB_INTERNODE_PORT=7100
export SKAIDB_SEEDS=10.0.0.1:7100,10.0.0.2:7100,10.0.0.3:7100
export SKAIDB_REPLICATION_FACTOR=3
export SKAIDB_DEFAULT_READ_CONSISTENCY=QUORUM
export SKAIDB_DEFAULT_WRITE_CONSISTENCY=QUORUM
skaidb --data-dir /var/lib/skaidb

Confirm the resolved settings on any node with skaidb --print-config.

Replication factor & consistency

• replication_factor (RF) — how many nodes hold each key. RF 3 tolerates one node down at QUORUM. RF must be ≤ the number of nodes (it's capped at the node count otherwise). Use the same RF on every node.
• Consistency (ONE / QUORUM / ALL), set as the cluster defaults:
• ONE — ack after one replica (fast, weak).
• QUORUM — majority of replicas (floor(RF/2)+1).
• ALL — every replica.
• Strong consistency when read CL + write CL > RF (e.g. QUORUM+QUORUM with RF 3 → R2 + W2 > 3). With weaker levels, the remaining replicas are updated in the background and converge via anti-entropy.
• Per session: SET CONSISTENCY ONE|QUORUM|ALL on a binary-protocol connection (or \consistency in skaidbsh) overrides the defaults for that session's statements. REST is stateless and rejects it.

ONE on a full-copy cluster is a local pass — use it for analytics

When RF ≥ member count every node holds every row, so a read at ONE is served by a single pass over the local replica instead of gathering the same rows from several replicas and last-writer-wins-merging them on the coordinator. This applies to every read shape: point reads, COUNT(*), filtered counts, DISTINCT, ordered reads, and plain and grouped row gathers (GROUP BY, time_bucket).

The difference is large on aggregate reads. Measured on a 3.69M-row table over a 24-hour window (~40k matching rows), RF=3 over 3 members:

	QUORUM	ONE
count(*)	1612 ms	116 ms
GROUP BY service	2050 ms	115 ms
GROUP BY service, name + ORDER BY avg()	—	164 ms
time_bucket hourly buckets	—	109 ms

Dashboards and reporting queries are the intended users: they read data every node holds in full and do not need read-your-writes. The trade is that ONE returns one replica's view, which may lag an in-flight write by a beat — so do not use it for a read that immediately follows a write. A resyncing or partially-backfilled node never serves these paths (see Resync state), so ONE cannot read an incomplete copy.

Foldable grouped aggregations (COUNT/SUM/AVG/MIN/MAX, plain args) stream at any consistency: rows fold into per-group state as the gather produces them — O(groups) memory, exempt from scan_row_budget — so a GROUP BY over a window bigger than the row budget answers whether the coordinator takes the ONE local pass or the QUORUM merge. ONE is still the faster choice here (it skips the cross-replica merge entirely — see the 2050 ms vs 115 ms row above, which is coordination cost, not a memory difference), but it is not the only consistency that can answer a large grouped window. Non-foldable aggregates (PERCENTILE, COUNT(DISTINCT …)) still materialize at every consistency.

Note consistency is a property of the session/connection, so a client that pools connections generally needs a separate pool for its analytics reads rather than a per-query flag.

Per-table placement (RF overrides & pins)

The cluster RF is the default, not the law. Any table can override it at CREATE or later:

CREATE TABLE metrics_cold (PRIMARY KEY (id)) WITH (replication = 1)
CREATE TABLE hot_config  (PRIMARY KEY (k))  WITH (nodes = ['skai2'])
ALTER  TABLE metrics_cold SET (replication = 2)   -- online transition
ALTER  TABLE hot_config   SET (nodes = ['skai3']) -- online pin move

• replication = n — ring placement at a per-table copy count (n >= member count behaves as a full copy, like cluster-wide RF does). Quorums for that table's reads and writes derive from its replica count, not the cluster default.

  Upgrade order matters. A node running a build from before per-table placement ignores the catalog's replication/nodes fields (unknown fields deserialize to the default) and places those rows at the CLUSTER RF instead. A mixed-version cluster therefore disagrees about which nodes own a row: writes land on one replica set and reads look on another, so a read can legitimately miss a row that was acked. Finish rolling every member onto the feature version before the first WITH (replication = …) / WITH (nodes = …) DDL — the same constraint class as any placement-affecting format change. Rolling back a member below the feature version after such a table exists has the same effect and needs the table reverted to cluster RF first.

• nodes = ['<alias-or-id>', ...] — the whole table lives on exactly those members (aliases resolve to stable internode ids at DDL time; renames never move data). Every pin holds every row; a non-pin coordinator routes reads and writes to the pins. Pins are a durability trade the operator owns: a pinned node down means quorum errors for that table until it returns, and ALTER CLUSTER REMOVE NODE refuses to remove a pinned member until it is re-pinned away. Mutually exclusive with replication.
• Changing placement is online. The ALTER opens a dual-placement window: the table's old and new placement are both live, and every read/write addresses the union — the per-table twin of the membership-change dual ring — so quorum reads stay correct while new owners are still empty. A background driver (the sorted union's first member) repairs until every member has completed a full anti-entropy pass that began after the change, then finalizes automatically. SHOW TABLES shows transition = true (the UI data tab shows → moving) while the window is open. One transition per table at a time; if the driver node is down the window just stays open — safe, merely wider than needed — and the operator escape is REPAIR CLUSTER followed by ALTER TABLE t SET (placement_finalized = true). After finalize, RECLAIM trims the copies the new placement no longer owns.
• GLOBAL-index entry tables follow their base table's placement; system tables refuse placement options; sharded tables (RF below member count) are scatter-pulled and merged by witness nodes, so mirrors stay complete for any placement.

Partition co-location (distribute_by)

By default a row's ring position hashes its whole primary key, so two rows sharing a key prefix (a tenant and its records) land on unrelated replica sets once RF is below the member count. distribute_by places the table by the encoded prefix of its leading primary-key columns instead:

CREATE TABLE events (PRIMARY KEY (tenant_id, id))
WITH (replication = 1, distribute_by = (tenant_id))

• Every row with the same distribute_by values lands on the same replica set — the whole partition is co-located. (Reads are served from any coordinator exactly as before; co-location changes where rows live, not how queries are planned.)
• The column list must be a leading prefix of the PRIMARY KEY and is fixed at CREATE (ALTER refuses — changing it would re-home every row; recreate and copy instead).
• Placement is the only thing that changes: quorums, repair, rebalancing and resharding all follow the prefix, and at RF ≥ member count the behavior is indistinguishable from whole-key placement.
• Write partition columns with one consistent type: values that encode differently (1 vs 1.0) are different prefixes and land on different owner sets.
• One skew trade-off to own: a partition is only as spread as its prefix cardinality — a single huge tenant concentrates on one replica set.
• Search index scatter fast paths (sharded aggregations/top-k) decline on distribute_by tables and fall back to the row gather — correct, just not the sharded shortcut.

Single-shard transactions (BEGIN ATOMIC PARTITION)

Co-location is what makes multi-row atomicity affordable: because a whole partition shares one replica set, a transaction over it needs one consensus round, not two-phase commit. Over a binary-driver connection:

BEGIN ATOMIC PARTITION (tenant_id = 't1');
INSERT INTO org (tenant_id, id, name) VALUES ('t1', 1, 'acme');
INSERT INTO settings (tenant_id, id, theme) VALUES ('t1', 1, 'dark');
COMMIT;

• Statements inside may touch only rows whose distribution columns equal the pinned literals — across any number of tables distributed by those columns. A statement that would touch another partition errors; the transaction stays open.
• Writes buffer in the session (reads see the session's own overlay; other sessions see nothing until COMMIT).
• COMMIT serializes on the partition through the same per-key consensus round the linearizable-UPDATE path uses, taken on the partition's row in a hidden per-database __txnlog__ manifest table whose committed value embeds the write set — that round is the atomic point. The rows then apply as ordinary replicated writes at the manifest entry's timestamp; if the coordinator dies mid-apply, every replica's sweeper completes the entry (recover-forward, idempotent), so a commit is never partial.
• Isolation is atomicity only: all-or-nothing per commit, last-writer-wins between concurrent transactions and against non-transactional writes — the embedded engine's semantics, made cluster-safe. There is no read-set validation and no interactive locking.
• Refused inside a transaction: DDL, USE, time-series writes, and writes to tables with UNIQUE indexes, GLOBAL indexes, value-TOAST (each of those write legs lives on other replica sets) or a FOREIGN KEY on either side (the buffered apply cannot check the other side). The buffered write set is capped at 8 MB.

Foreign keys on a cluster

A FOREIGN KEY is enforced by the coordinator, wherever the rows live: a child write reads its parent at QUORUM before writing, a parent delete or key change reads the referencing children (through the constraint's supporting index) and applies the ON DELETE / ON UPDATE action to them as ordinary replicated writes. ADD CONSTRAINT and VALIDATE CONSTRAINT stream every child row cluster-wide and refuse the statement — nothing applied on any node — on the first violation; NOT VALID skips that scan. The constraints travel with the schema: a member that missed the DDL adopts them through anti-entropy schema sync, with the same last-writer-wins stamps as tables and indexes.

Statements racing on opposite sides of a constraint are serialized per referenced value through a hidden replicated guard table (__fkref__<child>__<constraint>, placed and replicated like the parent, one row per referenced tuple, updated by CAS). A child write claims the tuples it references before it probes the parent and releases them once its rows are acked; a parent delete or key change fences the tuple, waits for the claims in flight to drain, waits for the child's index maintenance to settle on every member, and only then runs its RESTRICT probe or CASCADE / SET NULL fan-out, lowering the fence when the statement completes. Child writes arriving behind a fence wait for it (long cascades refresh their fence every page). The outcome is the standalone one: a child insert racing a parent delete either lands before the delete and is seen by it, or is refused after it. Guard rows are transient — a statement that dies mid-flight (a coordinator crash) leaves a claim or a fence that the other side resets after a 10 s grace period; a child write whose put stalls longer than that between its claim and its release (a partition during the quorum round-trip) is the one shape the guard does not cover. The two counters skaidb_cluster_fk_guard_waits_total and skaidb_cluster_fk_guard_stale_resets_total say how often writers waited and how often dead state was reset; a steady rate of stale resets on a healthy cluster is a bug report. Two statements cascading into each other's tables in opposite orders can wait on each other's fences; the statement deadline (storage.statement_timeout_secs) resolves that, as it does a CAS livelock.

Rows the guard does not see: ADD CONSTRAINT / VALIDATE CONSTRAINT scan existing children without fencing, so a child insert racing the validation of a not-yet-valid constraint is checked by the insert alone; writes that bypass the executor — internode repair, resync and PITR replay — reproduce rows as they were written and do not re-check the constraint. VALIDATE CONSTRAINT finds any orphan either leaves behind.

Verify the cluster

Write through one node and read it back through another — leaderless means any node accepts both:

# Create + insert via node 1 (REST on :7080)
curl -X POST 10.0.0.1:7080/query -d "CREATE TABLE users (PRIMARY KEY (id))"
curl -X POST 10.0.0.1:7080/query -d "INSERT INTO users (id, name) VALUES (1, 'ada')"

# Read via node 2 — sees the replicated row
curl -X POST 10.0.0.2:7080/query -d '{"sql":"SELECT * FROM users WHERE id = 1"}'

# Per-node metrics
curl 10.0.0.3:7080/metrics

Each node's startup log prints its endpoints (binary endpoint listening on …, REST endpoint listening on …).

Add or remove nodes at runtime (online resharding)

The ring can change while serving traffic — a node can join (and receive its share of the keyspace) or be gracefully decommissioned (drain its keys first). The full mechanics — pending-ranges dual-write during a join, single-sender migration, epoch'd membership, throttling/resume — are in RESHARDING.md.

Drive them with skaidbsh, the unified shell/admin client (shipped alongside skaidb). It talks to any node's REST endpoint over an authenticated POST /admin/* control plane (RBAC: the role needs Admin on the whole cluster; membership changes are serialized server-side, one at a time):

# Point it at any node (REST port defaults to 7080, override with --rest-port);
# add --user/--password if the server requires auth.
skaidbsh --host 10.0.0.1 cluster status            # show the ring, epoch, members, RF

skaidbsh --host 10.0.0.1 cluster add-node 10.0.0.4:7100    # join: migrates its share in
skaidbsh --host 10.0.0.1 cluster remove-node 10.0.0.3:7100 # decommission: drains, then leaves
skaidbsh --host 10.0.0.1 cluster repair            # anti-entropy: converge all replicas
skaidbsh --host 10.0.0.1 cluster reclaim           # free space former owners no longer own

status prints JSON like:

{ "clustered": true, "node_id": "10.0.0.1:7100", "epoch": 3,
  "replication_factor": 3,
  "configured": ["10.0.0.1:7100", "10.0.0.2:7100", "10.0.0.3:7100"],
  "self_in_ring": true,
  "members": ["10.0.0.1:7100", "10.0.0.2:7100", "10.0.0.4:7100"],
  "peers": [
    { "id": "10.0.0.2:7100", "in_config": true, "in_ring": true,
      "reachable": true, "hints_pending": 0, "lag_ms": 4 },
    { "id": "10.0.0.4:7100", "in_config": false, "in_ring": true,
      "reachable": true, "hints_pending": 0, "lag_ms": 7 }
  ],
  "discrepancies": {
    "configured_not_in_ring": ["10.0.0.3:7100"],
    "ring_not_configured": ["10.0.0.4:7100"]
  } }

Configured vs. actual. configured is what seeds says membership should be; members is the live ring the coordinator actually routes and replicates to. They diverge in normal operation: cluster add-node admits a node that was never in anyone's seeds (it shows up under ring_not_configured), and a seed that has not (yet) been admitted to the ring shows up under configured_not_in_ring. The latter is exactly the trap to watch for: a node started with seeds pointing at the cluster will pull data via background catch-up (it sees peers, so anti-entropy runs) without ever joining the ring — it serves stale-but-converging reads while no one routes writes to it. Such a node appears in configured_not_in_ring until you run cluster add-node for it. If the joining node's own seeds omit itself (it lists only peers), peers never learn of it at all — there is no gossip — so the tell is on the node itself: its self_in_ring is false, meaning it is coordinating/catching-up but owns no ring tokens. The unauthenticated GET /status carries the same configured / self_in_ring / configured_not_in_ring / ring_not_configured fields (without the per-peer liveness probe); \cluster adds reachable, hints_pending, and lag_ms per peer (see METRICS.md for the matching Prometheus gauges).

Under the hood these call the coordinator: add-node broadcasts the new ring, bootstraps the joiner's schema, and streams it the keys it now owns (dual-writing during the move so concurrent writes stay correct); remove-node drains the leaving node's keys to their new owners before dropping it from the ring. The same operations are also available as raw HTTP (POST /admin/status, /admin/add-node with {"addr":"…"}, /admin/remove-node with {"id":"…"}, /admin/repair, /admin/reclaim), as plain SQL from any client — SHOW CLUSTER, ALTER CLUSTER ADD NODE 'host:7100' / REMOVE NODE 'id', REPAIR CLUSTER, RECLAIM, plus SHOW CONFIG [LIKE] / SET CONFIG and SHOW SLOW QUERIES (identical RBAC and audit as the HTTP endpoints) — and as skaidb_cluster::Node library methods.

A long migration keeps the skaidbsh request open until it finishes; tune the push rate per node with the migration throttle (see RESHARDING.md). Run one membership change at a time.

A join that fails partway (e.g. the joiner became unreachable during its schema bootstrap) leaves the ring in its dual-placement phase — /status shows "resharding": true indefinitely. Recovery is automatic on the joiner's next announce (restart the joining node: the re-announce finalizes the pending transition); or remove and re-add the node.

Backups on a cluster

BACKUP TO '/path' backs up the answering node's shard (a crash-consistent copy of its data directory — each node backs up its own). RESTORE FROM is refused on a live cluster: swapping one node's data underneath quorum reads would silently diverge replicas. To restore a node: stop it, restore its data directory offline, start it, and let repair converge it.

Point-in-time recovery on a cluster

With storage.wal_archive_dir set, each node keeps every sealed WAL segment and RESTORE FROM '<path>' TO TIMESTAMP '<when>' replays them on top of a backup — undoing a bad write without losing the work done since the last backup. See QUERY_SYNTAX.md for the statement.

On a ring this is an operator procedure, not a statement:

1. Stop every member. A node restored under a live ring has its recovered rows immediately overwritten by peers that still hold the bad write.
2. Run the same RESTORE … TO TIMESTAMP on each node's data directory offline, with an identical target. Replicas converge only if they replay to the same instant.
3. Start the ring. Anti-entropy closes whatever a node that was down during the window missed.

No coordination protocol is involved, and that is deliberate: HLC stamps are cluster-wide and last-writer-wins is deterministic, so independent replay of the same window on each node reaches the same state. What it needs is a stopped ring, not a distributed algorithm.

In practice, per node (RESTORE is a statement, so the node must be up — just not in the ring):

1. Swap in a standalone config: same data_dir, same keys, same archive, but seeds = [] (standalone mode is what accepts RESTORE) and statement_timeout_secs = 0 (replay can exceed any interactive timeout).
2. Under systemd, clear the unit's filesystem sandbox for the window. ProtectSystem=full plus StateDirectory/ReadWritePaths bind-mount the data directory into the service's namespace, and RESTORE renames the live directory aside — which fails with EBUSY against a bind mount and EACCES against a root-owned parent. A runtime drop-in (removed afterwards) is enough: printf '[Service]\nProtectSystem=off\nStateDirectory=\nReadWritePaths=\n' > /run/systemd/system/skaidb.service.d/restore.conf, systemctl daemon-reload, and make the data directory's PARENT writable by the service user for the window.
3. Start the service, wait for the LISTENER (a large data directory opens long after systemd reports active), run RESTORE FROM '<backup>' TO TIMESTAMP '<target>', stop, undo steps 1–2.

Coordinated snapshots make the target exact. BACKUP CLUSTER TO '<path>' backs up every member and mints ONE cut instant after all backups complete (recorded in each backup's CLUSTER_CUT); restore every node TO TIMESTAMP that instant and the ring lands on a single consistent cut instead of "roughly the same moment".

The previous data directory is left beside the restored one as <data_dir>.pre-restore. It is a full copy — DELETE IT (and the backup) once the restore is verified, or a tight node runs out of disk and the inflow watermark starts refusing writes.

Each node archives its own shard's segments, so the archive is per-node like the backup. A witness mirrors data but is not a recovery point: it applies the bad write too.

Resolution is uniform across table kinds. Row tables seal on the wal_archive_max_lag_secs timer (5 minutes by default), and TIME-SERIES tables ROLL their WAL segment on the same timer — a roll, not a flush, so the head, blocks and compaction cadence are untouched; sealed segments are copied to the archive and stay in place for crash replay until their checkpoint retires them.

Cost. wal_archive_max_lag_secs rotates the WAL on a timer; it does NOT flush, so the LSM's flush cadence and SSTable shape are unchanged. The archive holds roughly what the node writes: measured on a production node taking ~46 MB/h of writes, a 6-hour window is about 0.3 GB. Put the archive on the same filesystem as data_dir (segments are then hard-linked rather than copied) but OUTSIDE it, or BACKUP will copy the archive into every backup.

Anti-entropy: repair & space reclamation

Replicas converge automatically through read-repair (a quorum read writes the winning version back to stale replicas — pushed in the background, so the reader's answer never waits on a stale peer; an unreachable one gets a hint) and hinted handoff (a write to a down replica is buffered and replayed when it returns). Hints are held in memory up to a per-replica cap and spill to a per-replica on-disk log beyond it — so a replica that stays down or keeps shedding for a long time loses no writes (bounded memory, durable across restarts) rather than dropping the overflow. A drain that aborts (peer busy, restart mid-pass) is retried by a 60-second ticker while any backlog exists, with delivered work logged — a large backlog deferred at an unreachable peer logs its size instead of waiting silently. The logs also look after themselves: hints older than cluster.hint_max_age_secs (default a day — many times the anti-entropy ceiling, so repair has long since covered them) are expired instead of replayed; a log for a replica that left the ring is deleted; a drain claim orphaned by a crash re-adopts onto the live log; and each replica's log is capped at cluster.hint_max_disk_mb (both knobs live-mutable via SET CONFIG; default 1024 — at the cap new hints fall through to anti-entropy, counted and reported, exactly the pre-spill behavior but loud). Every such drop lands on skaidb_cluster_hints_expired_total. Tables created WITH (memory = true) are excluded from hinted handoff's durability story and from repair and reshard data motion entirely: they are ephemeral by contract (empty on restart, repopulated by their writers). For a full sweep — e.g. after a node was down a long time — run an active repair (Node::repair/repair_cluster), which reconciles every co-replica pair in both directions, including the catalog: databases, tables, and indexes are synced both ways, so a node that missed a DDL broadcast while it was down gets the missing schema too. Schema reconciles by last-writer-wins with tombstones — every DDL is HLC-stamped and a DROP leaves a versioned tombstone — so a drop that happened while a node was down propagates to it on rejoin, and a lagging node holding the now-dropped object does not resurrect it (the tombstone's newer stamp wins). A genuinely newer re-CREATE still wins over an older drop.

Continuous anti-entropy. Beyond read-repair (on reads) and hinted handoff (for writes to a briefly-down replica), each node runs a full repair pass on a timer — cluster.anti_entropy_interval_secs (default 60s, 0 disables) — so a node that missed a broadcast while it was up (e.g. a DDL that committed at quorum while this node was momentarily behind) converges on its own, with no operator action. Passes are staggered per-node so the cluster doesn't repair in lockstep, and a pass is skipped while the node is shedding — repair is the largest transient allocator in the process and is pure background work, so one deferred interval costs nothing next to pushing a node already under memory pressure into an OOM. The skip is logged.

Adaptive cadence (cluster.anti_entropy_adaptive = true, off by default). The fixed timer has a structural problem: a pass's cost scales with data, its value scales with how much has actually diverged, and divergence is bursty — caused by restarts, missed writes and membership changes, not by the passage of time. With adaptivity on, the interval becomes a bounded variable that follows what passes actually find:

• a pass that reconciled rows snaps the interval back to anti_entropy_interval_secs (the floor);
• a pass that converged (found nothing) relaxes it ×1.5, up to anti_entropy_max_interval_secs (0 = auto, 4 × the floor);
• a deferred or failed pass changes nothing — a skip is not evidence of convergence.

The ceiling doubles as the guaranteed full-sweep bound: adaptive rest is bounded exposure, never unbounded trust. Divergence events pull the next pass in rather than waiting out a relaxed interval — a hint stored (a replica just missed a write), a hint replay draining, a member joining — debounced so events move the next pass earlier but never schedule more passes than the floor already permits. All three knobs are live: SET CONFIG cluster.anti_entropy_adaptive = 'true' per node, no restart.

Watch it through skaidb_ae_interval_seconds (the live interval), skaidb_ae_passes_total{outcome=…} and skaidb_ae_rows_reconciled_total (see METRICS.md); alert on time() - skaidb_ae_last_pass_at_seconds exceeding the ceiling, which is what distinguishes a healthy backed-off loop from a stuck one.

Digest-gated passes. A repair pass first exchanges a compact XOR digest (4096 buckets over key ‖ hlc ‖ op, ~32 KB) per (table, peer) pair; equal digests prove the pair converged and skip the full paged compare, so a steady-state pass ships digests instead of tables. Digest computation itself is cheap: it scans value-free stamps — each SSTable carries a <file>.stamps sidecar holding just (key, hlc, op) per entry, so no row value is even decompressed. Encrypted tables have a sidecar too (sealed like the data blocks); tables written before the sidecar existed — or before encrypted ones got theirs — fall back to data blocks until compaction rewrites them. On full-copy clusters (replication factor ≥ member count) each node also caches its digest per table, keyed on a (schema stamp, write-sequence) version, so an idle table's digest is served from memory — a fully converged pass touches no table data at all. The cache can never mask divergence: any write bumps the version on the node that has it, forcing a fresh digest — and therefore a mismatch — on at least one side of the pair.

Incremental content digests. On multi-node deployments, every full-copy table also carries an engine-maintained digest: the same 4096-bucket XOR fold a repair scan computes, kept current by the write path itself (each write folds the key's previous winner out and the new one in, at the single choke point every write variant funnels through), built once by a paged background scan, and invalidated whenever the merged view changes outside the write path (a compaction purging tombstones or TTL-expired rows, RECLAIM's row drops) — the invalidated table simply falls back to scans until the background rebuild finishes. The digest gate answers in O(1) where a full verify would re-scan millions of stamps per pass. And the exchange happens as of a cutoff (~30 s in the past): each side omits keys whose current winner is stamped past it, so writes still in flight are excluded on BOTH sides and two converged replicas match even mid-ingest — without the cutoff, any write landing between the two computations forced the full paged compare every pass. Divergence younger than the cutoff is invisible for exactly one pass. Because the digest is trusted to skip work, a missed write site would read as permanent convergence — so every pass re-verifies ONE digest-backed table (rotating) by full scan and compares; a mismatch on a quiet table is counted and logged as a bug, and either way the digest is dropped and rebuilt. TTL tables never carry a digest (their expiry reclaims on wall time); standalone nodes never build one and their write path never pays the digest's prior-stamp lookup.

Hot-table backoff. A table under continuous ingest defeats the versioned digest cache: every write voids it, and because the two sides scan at different instants, in-flight writes make even converged replicas' plain digests mismatch — forcing the full paged compare on every pass (measured: one 4.4M-row live-ingest table cost ~400 s of every hourly pass, per node). The as-of digests above remove most of that cost; the backoff remains as belt-and-suspenders and for tables without a digest. A table that changed since its last clean verify (a pass where every peer agreed and nothing needed repair) is deferred, and re-verified every 4th pass — bounded divergence-detection lag (4 × the anti-entropy interval, worst case) in exchange for passes that stop re-verifying data the quorum write path and hints already delivered. The backoff never applies where it could mask divergence someone is waiting on: an explicit REPAIR CLUSTER sweeps everything (on every member), a resyncing node verifies everything, a table in a placement transition is verified every pass (the finalize protocol counts on full passes), a verify that repaired a real volume of rows re-verifies every pass until it comes back clean (a handful of "repairs" per pass is tolerated as in-flight write noise — the merge-join's two sides scan at different instants, so a live-ingest table never verifies at exactly zero), and a deferral is honored only while cluster membership matches the peer set the clean verify covered (a join/decommission re-verifies everything). A table whose own verify keeps failing (its peer busy/unreachable) never earns a deferral and is re-verified every pass — but one table's busy peer does not disable the backoff for the rest (under continuous ingest most passes have some transiently busy pair, and those are exactly the passes worth keeping cheap). A pass that deferred anything logs how many tables it skipped. An idle table whose peer diverged is never deferred — deferral requires the local copy to have changed — so one-sided divergence is still caught on the next pass.

Automatic catch-up on (re)join. When a node starts and finds peers, it runs a catch-up pass in the background as soon as a peer is reachable — the same repair (schema + data) — so a node that was down converges on its own without an operator running anything. This covers schema that quorum-DDL couldn't reach while the node was offline, plus any row writes beyond what hinted handoff replayed. (A brand-new node added with cluster add-node is bootstrapped explicitly with schema + its share of the data.)

Automatic join (self-announce). A node that starts with seeds pointing at an existing cluster but that the cluster doesn't yet know about will announce itself to a reachable seed, which runs add_member and broadcasts the new membership to every node — so you no longer have to run cluster add-node by hand, and you avoid the half-join trap (a node that pulls data via catch-up but was never admitted to the ring). The announce is a no-op when the seed already lists the node (symmetric seeds), and is rejected if the joiner's replication factor doesn't match the cluster's — fix the RF and restart rather than form a cluster whose coordinators disagree on each key's replica set. Joins are still serialized; do one at a time. \cluster cross-checks each peer's membership view and flags membership_disagreement when a peer you route to doesn't list you.

After resharding, reclaim (Node::reclaim/reclaim_cluster) physically frees space for keys a former owner no longer holds. See RESHARDING.md.

/status never blocks on the engine. The unauthenticated /status serves cluster identity (name, aliases, witness rows) from a cache a background thread refreshes every few seconds — it runs no SQL inline, so a health probe answers in milliseconds regardless of what holds the engine lock. Staleness-age fields (seen_age_secs, sync_age_secs) are computed from row timestamps at render time, so they stay accurate even from cached rows. Point external monitors at it freely; a slow /status now means the process itself is unhealthy, not that a repair pass is running.

Repair rides out a slow peer. A repair RPC that draws a busy reply, an I/O timeout, or the local circuit breaker's refusal is retried with backoff rather than failing the pass — a receiver applying a merge chunk holds its engine read guard, so being briefly late is normal, and one late chunk used to abort an entire time-series leg. Retrying is safe because repair requests are idempotent: a merge is keyed by timestamp, so re-sending a chunk whose acknowledgement was merely lost changes nothing.

Time-series repair reconciles in BOTH directions. A series' primary compares per-series (count, checksum) summaries with each replica; when they differ it pulls the replica's samples for a window, merges them locally, and pushes the union back. Pushing alone cannot converge — a replica holding samples the primary lacks would stay different after every push, and because the summaries still disagreed the primary would re-push the series' entire history on every later pass, indefinitely. Samples are immutable facts keyed by timestamp, so the union is the authoritative view and merging is idempotent on both sides.

Internode security

By default internode traffic is unauthenticated (internode_auth = "none") — fine on an isolated/trusted network, but anything that can reach a node's internode port can read data and change membership. Two modes lock it down; every node must use the same mode and material:

Token — a shared secret. Peers prove knowledge of it with a mutual HMAC-SHA256 challenge-response, so the secret never crosses the wire and each connection uses a fresh nonce (no replay). No encryption.

[auth]
internode_auth = "token"
internode_token = "a-long-random-shared-secret"      # or:
# internode_keyfile = "/etc/skaidb/cluster.token"     # file holding the secret

Cert — mutual TLS. Every node presents a certificate signed by a shared CA, and the channel is encrypted. Node certificates must carry the SAN DNS:skaidb (how peers verify each other without per-node hostnames) and extendedKeyUsage = serverAuth, clientAuth.

Generate the CA and per-node leaf certs with the built-in helper (no OpenSSL incantations, no locale footguns):

skaidbsh certs gen --out ./skaidb-certs --nodes 3
# writes ca.crt, ca.key, and node1..node3.{crt,key}, all 0600

Give each node its own leaf (node<i>.crt/node<i>.key) plus the shared ca.crt; keep ca.key off the nodes (it's the issuing root — store it offline for future cert minting). Then:

[auth]
internode_auth = "cert"
internode_tls_cert = "/etc/skaidb/node.crt"   # this node's leaf (SAN: skaidb)
internode_tls_key  = "/etc/skaidb/node.key"
internode_tls_ca   = "/etc/skaidb/ca.crt"     # CA that signs every node's cert

Both are also settable via flags/env (--internode-auth, SKAIDB_INTERNODE_*). A node that can't satisfy the configured mode is dropped at the handshake, before any RPC. Rollout: there's no mixed-mode window — a cert node and a none/token node cannot talk, so turn the mode on with the same material on every node and restart them together (a brief flag-day; clients fail over and retry). The effective mode is surfaced at GET /status as "internode_auth": "none" | "token" | "cert", so a monitoring check can catch a node that silently came up unauthenticated. Client auth is separate — SCRAM on the binary endpoint and HTTP Basic on REST, plus RBAC; see the README.

Encryption note: only cert mode encrypts the internode channel. none and token are plaintext on the wire (token authenticates but does not encrypt).

Rotating internode certificates (no flag day)

Unlike turning cert mode on, replacing certificates does have a mixed window, because internode_tls_ca may hold several CAs — every certificate in that file is added to the trust store, for both the peers a node dials and the peer certificates it accepts. Concatenating the old and new CA gives an overlap in which nodes on either certificate interoperate.

mTLS is mutual, and that dictates the order. A node handed the new CA alone rejects the old certificates its peers are still presenting — its own certificate being valid does not help. Deploying "new CA + new cert" to one node first therefore partitions that node. Bundles go everywhere before any certificate is swapped:

1. Bundle, node by node. Set internode_tls_ca to cat old-ca.crt new-ca.crt, leave each node's existing cert/key alone, restart. Safe in any order: a bundled node still accepts untouched peers.
2. Verify every node reports "internode_auth": "cert" on GET /status and the cluster is ready with the full member count.
3. Swap certificates, node by node. Replace internode_tls_cert / internode_tls_key with the new-CA-issued pair, restart. Mixed old/new nodes interoperate for as long as this takes — both sides still hold the bundle.
4. Drop the old CA once every node presents a new certificate: set internode_tls_ca back to the new CA alone and restart node by node. Old certificates stop being trusted, which is what completes the rotation.

Each step is a rolling restart, so quorum holds throughout at RF≥3. The four trust states above — including the two failure modes in step 3's warning — are pinned by cert_rotation_via_ca_bundle in skaidb-cluster, so the property this procedure depends on cannot regress silently.

Client TLS (driver ↔ cluster)

The binary (7000) and REST (7080) ports can be TLS-wrapped independently of internode mode. Set [encryption]:

[encryption]
client_tls = "opportunistic"   # off | opportunistic | required
tls_cert_file = "/etc/skaidb/server.crt"
tls_key_file  = "/etc/skaidb/server.key"

• off (default) — plaintext only.
• opportunistic — one port serves both: a TLS ClientHello is wrapped, a plaintext connection is served as before. Use this to migrate — point every client at TLS, confirm, then switch to required.
• required — plaintext connections (including probes on the REST port) are refused. TLS only.

Clients authenticate with SCRAM/HTTP-Basic inside the TLS channel (client certs are not required — this is one-way server TLS). A misconfiguration (client_tls on but cert/key unset) fails the listener startup loud, never silently plaintext; the effective mode shows at GET /status as client_tls. The server cert can be any TLS cert; the cluster CA from skaidbsh certs gen works (its node certs carry SAN skaidb). Connect with:

skaidbsh -H node --tls --tls-ca ca.crt          # verify against the cluster CA
skaidbsh -H node --tls                          # verify against the PUBLIC-CA roots
skaidbsh -H node --tls --tls-insecure           # self-signed / dev (INSECURE)
# --tls-server-name <name>  overrides the verified SAN (default: skaidb)

Bare --tls verifies against the public-CA roots (Mozilla's bundle, compiled in) — right for a server behind a public certificate, and a certificate error against a cluster-CA server (pass --tls-ca, or --tls-insecure for dev). The Rust driver's TlsVerify::System is the same policy.

The driver takes Client::connect_many_tls(endpoints, user, pw, Some(tls)).

At-rest encryption

Encrypt every table's and index's WAL and SSTables on disk with AES-256-GCM. The scheme is envelope: a KEK from a keyfile wraps a per-file DEK that seals the data, so key rotation is cheap and the KEK never touches data.

skaidbsh keyfile gen --out /etc/skaidb/at-rest.key   # 32 bytes, 0600

[encryption]
at_rest_enabled = true
at_rest_kek_source = "keyfile"          # kms: not supported
at_rest_keyfile = "/etc/skaidb/at-rest.key"

• New files encrypt; existing plaintext files stay readable (mixed migration). To fully encrypt an existing node, do a rolling per-node resync: wipe the node's data dir and let it rebuild from peers onto the encrypted engine — one node at a time, RF keeps the cluster serving (the same shape as re-encrypting any replicated store).
• A missing or bad keyfile fails startup loud — the node never comes up silently unencrypted. at_rest_enabled is restart-scoped. The effective state shows at GET /status as at_rest.
• Back up the keyfile off-box before enabling. Losing it makes all encrypted data unrecoverable — it is operator-critical.
• The WAL, SSTables and the stamps sidecar are ciphertext on disk. The sidecar is sealed with the same per-file key in a separate nonce domain, so encrypted tables keep the value-free repair fast path (they used to have none, which made every digest scan decode full data blocks).

Rotating the at-rest KEK (no data rewrite)

Rotation rewraps each file's small wrapped-DEK header under the new KEK — the sealed data, and the DEK itself, are never touched, so rotating a multi-GB node costs seconds of I/O, not a rewrite. Every encrypted file carries a kek_id (a fingerprint of the KEK that wrapped its DEK); a background sweep finds files tagged by a retired KEK and patches their headers in place, ~130 files per minute-tick per node, crash-safe (a journal sidecar replays an interrupted patch at the next open).

The keyring dictates the order, exactly like the certificate rotation above. A node given the new keyfile alone cannot open a single file wrapped by the old KEK — it fails startup loud. The old KEK must therefore be in the ring before the primary is swapped, and leave it only when no file references it:

1. Generate the new keyfile (skaidbsh keyfile gen --out /etc/skaidb/at-rest-2.key) and back it up off-box, like the first one.
2. Swap with the old key in previous, node by node:

toml [encryption] at_rest_keyfile = "/etc/skaidb/at-rest-2.key" # new primary at_rest_previous_keyfiles = ["/etc/skaidb/at-rest.key"] # old, unwrap-only

Restart. The node opens everything (old files unwrap via the previous key), writes every new file under the new KEK, and the sweep starts rewrapping old headers immediately. 3. Watch skaidb_storage_kek_stale_files to zero on each node. This gauge counts every file (SSTable, WAL, sealed WAL segment) still referencing a non-primary KEK. SSTables converge by in-place patch; a busy table's WAL converges at its next flush, a quiet (empty) WAL is re-keyed directly by the sweep. 4. Retire the old keyfile once the gauge reads zero: remove it from at_rest_previous_keyfiles, restart, then destroy the old keyfile — at zero, no byte on disk needs it (the node would now start without it).

Steps 2 and 4 are rolling restarts; quorum holds at RF≥3. Every state above — including that step 2's ordering is mandatory and that the gauge reaching zero really does mean the old KEK is unreferenced — is pinned by kek_rotation_rewrap_pins_every_rollover_state in skaidb-storage.

Resilience

• Damaged table files quarantine the table, not the node. A torn or corrupt SSTable/WAL (bad footer, failed checksum at replay) used to refuse the whole database open. Now the node starts, every other table serves, and the damaged table is quarantined: statements against it error with the cause (its catalog entry is kept, so nothing can silently re-create over the damage), the startup log names the file error, and skaidb_storage_quarantined_tables counts it — alert on non-zero. Remediation: restore the table's directory from a backup, wipe-resync the node from its peers (RF≥2), or DROP TABLE to discard it. This covers encrypted files too: each file's kek_id proves whether the right KEK is configured, so an authentication failure under the correct key reads as the disk rot it is and quarantines — while a wrong or mis-rotated keyring still fails the open loud (a key mistake must never masquerade as one damaged table). Environment errors (permissions, I/O) also still fail the open loud.
• A slow or unresponsive replica cannot hang a write. Every internode connection has a bounded read/write timeout, so a peer that is up but not answering (thrashing under memory pressure, a kernel that accepted the socket while the process is stalled) is failed fast — the coordinator meets the write quorum from the responsive replicas and hints the slow one for handoff, rather than blocking on it. (A refused connection already failed fast on connect; this covers the connected-but-silent case.)
• A quorum read is never failed by a suspended-but-healthy peer. The per-peer circuit breaker makes a flapping peer cheap by failing calls to it fast, but a peer that has just restarted — or that briefly declined reads as busy — stays suspended on every coordinator for the cooldown after it is serving again. Counting those peers as non-responders would fail QUORUM reads on a cluster where every member is up, which is exactly the shape a rolling upgrade produces. So when, and only when, a read is about to fall short of its quorum, the coordinator gives each unanswered peer one breaker-bypassing probe: a bounded (2 s) connect/read/write attempt that ignores the breaker. A success both satisfies the read and closes that peer's circuit for every other caller, so recovery is detected in seconds rather than waited out. This covers point reads, paged scans (LIMIT pushdown, candidate resolution, streaming aggregation) and time-series gathers. Healthy reads never pay for it, and a genuinely dead peer costs the bounded timeout at most once per statement — the alternative being an error. When the probe also fails, the "read quorum not met" error names each peer and why it failed.
• Memory-pressure release and load shedding. A node watches its memory against its limit (cgroup when set, else system RAM), measuring non-reclaimable usage — the cgroup charge minus reclaimable file-backed page cache (mmap'd SSTable/WAL/search segments the kernel evicts before OOM), so a cache-filled node isn't falsely shed while it still has real headroom. Two tiers:
• Past 75% it actively releases: flushes table/index memtables (≥4 MB) and commits every dirty search-index writer (Tantivy holds indexed-but-uncommitted documents in heap buffers; a node that stops taking writes otherwise never commits them and rides its limit until the fault storm or the OOM killer gets it — observed in production). Release actions are paced (at most every 10 s).
• Past 85% it also sheds writes — rejecting new writes (client and inbound-replica) with a retryable "memory pressure" error — and the release pass turns aggressive (flushes memtables down to 64 KB). The release is driven by the memory sampler, not a client write — otherwise a shedding node would deadlock (it rejects the very writes that would trigger a flush) — and covers every memtable, since pressure spread thin across many tables leaves each below the per-engine flush threshold while the sum pins the node. The flag clears at 70% (hysteresis).

Shedding is loud: entering it logs the anon/file split plus jemalloc's allocated/resident/retained numbers, a distress line repeats every 60 s while it persists ("releases are not freeing enough; OOM risk"), and recovery logs the episode's duration. Anti-entropy passes log their duration (and allocator stats) whenever they reconcile rows or take ≥60 s. The packaged unit also sets MemoryHigh=85% so the kernel throttles and reclaims the service before a hard OOM kill (which costs a restart + full search-index rebuild). Reads and DDL are never shed; a coordinator that gets a shed rejection from a replica hints it and proceeds at quorum. Watch skaidb_memory_shedding_writes / skaidb_memory_used_bytes (METRICS.md); a node stuck shedding is undersized for its workload. - Graceful shutdown. SIGTERM/SIGINT (what systemd's stop/restart send) flush memtables and commit search-index writers before exit, so the next start replays almost nothing — an unclean kill costs a full search-index rebuild from the last committed watermark. The flush waits at most ~10 s for the engine lock, so a wedged node still exits inside systemd's kill window. - Full-copy counts are local. When replication_factor >= members (every node holds every row), unfiltered COUNT(*) is answered from the local engine's key statistics — no cluster gather (which would materialize the whole merged table on the coordinator). The local answer is O(keys) with no value decode, at the same freshness trade the search paths already make. - Compaction commits before it deletes. Retired SSTables are removed only after the manifest durably points at their replacement, so a kill can never leave the manifest naming deleted files (which would make the engine refuse to open). A manifest entry that fails to open is logged with the exact file and manifest path. - Bulk index builds stream the table. Building or rebuilding a search index (CREATE SEARCH INDEX, startup catch-up, or an automatic rebuild) reads the source table one row at a time rather than gathering the whole shard into memory first, so indexing a large table stays within a bounded footprint (the writer heap, sized from memory_target, plus one row) and does not OOM a small node. DB workload must never OOM a node.

Witness nodes

A witness is a standalone node (never a ring member, never counted in quorums) that mirrors chosen databases from a primary cluster on its own schedule — a cross-region, pull-based backup. Configure [witness] in its toml (primary SQL + internode addresses, witness-scoped credentials, databases) and pair it with server.read_only = true (drivers may connect for read-only queries; every mutation is refused for non-superusers). Pulls ride the internode protocol near-live: per-table write_seq change hints gate incremental stamps-walked delta pages (interval_secs, default 60 s), with a full sweep every full_sweep_interval_secs (default 24 h) as the anti-entropy backstop; pulls self-pace to at most witness.duty_pct (default 50 %) of the serving member's capacity. The primary's witnesses registry drives the overview-tab sync detail and holds tombstone GC back (up to witness_gc_config.grace_period_secs) until every live witness has pulled the deletes — per table: each table's delete markers are held only for the witnesses that actually mirror it, and only back to their watermark for that table, so one witness stuck on one table does not pin every other table's tombstones.

Two independent controls decide what a witness holds. The primary opts a table out for all witnesses with WITH (witness = false) / ALTER TABLE t SET (witness = false). Each witness then narrows what is left with witness.tables / witness.exclude_tables (see below) — the data owner decides what may leave the cluster, each backup decides what it does carry. A witness can never widen past the primary's flag. Time-series tables mirror too: their samples move via time-windowed TsQuery pulls (scattered over every member — series placement shards them on RF < members primaries — unioned, deduped, merged any-aged), with the same watermark + full-sweep-backstop ladder as row tables. On the witness they are recreated as plain TIMESERIES tables with the source's series key but NO retention (a backup keeps what the primary ages out); rollups mirror as plain time-series tables (the rollup→source link is derived state).

Setting one up

On the primary, create the witness's scoped role. It needs SELECT as well as the writes — registration reads the row back before deciding between an INSERT and an UPDATE, so a role granted only INSERT, UPDATE fails on its first beat:

CREATE USER nasw_witness WITH PASSWORD '<generated>';
GRANT SELECT, INSERT, UPDATE ON witnesses TO nasw_witness;

That grant is the witness's entire reach on the primary. It reads table schemas and rows over the pull path, which is authenticated separately by [auth] — the SQL role exists only to keep its registry row current.

On the witness host, run a standalone node — no seeds, never a ring member — with [witness] pointed at the primary's SQL and internode addresses. Both lists matter: the control plane (registration, schema listing) is SQL, the bulk pull is internode, and a firewall that passes only one produces a witness that registers and never mirrors.

[server]
read_only = true                     # the copy is a backup, not a replica to write to
data_dir  = "/var/lib/skaidb"

[cluster]
seeds = []                           # a witness never joins the ring

[auth]                               # must satisfy the primary's internode auth
internode_auth      = "cert"
internode_tls_cert  = "/etc/skaidb/tls/node.crt"
internode_tls_key   = "/etc/skaidb/tls/node.key"
internode_tls_ca    = "/etc/skaidb/tls/ca.crt"

[witness]
enabled                  = true
witness_id               = "nasw"    # stable identity; pairs with the primary's registry
region                   = "nas"
databases                = ["agencik"]
primary_sql_addrs        = ["10.0.0.1:7000", "10.0.0.2:7000", "10.0.0.3:7000"]
primary_internode_addrs  = ["10.0.0.1:7100", "10.0.0.2:7100", "10.0.0.3:7100"]
user                     = "nasw_witness"
password                 = "<generated>"
primary_tls              = true      # required if the primary sets client_tls = "required"
interval_secs            = 60

Mirroring part of a database

databases is the coarse control. Within those databases, tables is an allowlist and exclude_tables a denylist, both naming db.table with an optional trailing * for a prefix. Empty tables (the default) means every table; the denylist is applied after the allowlist.

[witness]
databases      = ["agencik"]
tables         = ["agencik.core_*", "agencik.gmail_emails"]  # only these
exclude_tables = ["agencik.core_scratch"]                    # minus this

Use it when one backup should not carry everything: a cheap off-site box that mirrors the small operational tables but not a 200 GB vector table, or two witnesses splitting a large database between them. Notes:

• Value-TOAST companions follow their base table. __toast__<t> holds a table's out-of-line blobs; it is never matched against these lists on its own, because a mirror holding a table without its blobs resolves those fields to nothing.
• A deselected table stops holding tombstones for this witness on its next cycle — it drops out of the heartbeat's watermarks, which is exactly how the primary learns to stop waiting for it. Deletes to that table may then be collected without this witness ever seeing them, so re-adding a table later is a full-resync-and-rebuild, not a resume.
• Data already mirrored is not removed when a table leaves the selection; the local copy simply stops advancing. Drop it on the witness if you want the space back.
• Selection is the witness stating what it carries, not an access control: the primary serves any table the witness's [auth] credentials can reach. Use the SQL role's grants for a boundary that has to hold.

The witness's certificate must be signed by the same CA the primary's internode port trusts — it is authenticated as a peer, not as a client. primary_tls covers the separate SQL connection; getting one right and the other wrong is the usual reason a witness starts, registers, and then mirrors nothing.

Verify it, in this order — the two questions have different answers and different sources:

# On the witness: has a cycle ever COMPLETED?
curl -s http://<witness>:7081/status | jq '.role, .last_pull_at, .databases'

# On any member: is it alive, and is it current?
curl -sk https://<member>:7443/status | jq '.witnesses[]'

last_pull_at is null until the first cycle finishes, and the first cycle copies everything — on a large database over slow storage that can take hours while the mirror is working perfectly. seen_age_secs answers "alive" and keeps moving throughout; sync_age_secs answers "current" and stays null until that first cycle closes. Alert on the second, not the first: a witness that is up but failing every cycle heartbeats on schedule, so only sync_age_secs (or last_pull_at) exposes it.

Registered witnesses are part of the admin surface even though they are outside the ring: RESET OOM COUNTER includes every registered witness (and accepts a witness id as its ON '<node>' target) by stamping the request on the witness's registry row — the witness applies it on its next pull cycle, exactly once per stamp, offline gaps included. The result row reports absorbed = NULL with a "stamped" status, since the apply happens on the witness's own schedule.

Seeing witnesses from outside. A witness is not a member, but the cluster does know it, and both sides say so on the unauthenticated GET /status. Each member publishes its registry:

"witnesses": [
  { "witness_id": "onetw", "alias": "onetw", "region": "slug",
    "registered_at": 1785251756333, "last_seen_at": 1785338156333,
    "seen_age_secs": 61, "sync_age_secs": 7204 }
]

sync_age_secs (oldest per-table watermark; null before the first cycle) is the one to alert on — a witness heartbeats on schedule, so seen_age_secs stays small even while one table is stuck. The matching Prometheus gauges are skaidb_witness_last_seen_age_seconds and skaidb_witness_oldest_sync_age_seconds (see METRICS.md).

The witness's own /status self-describes, so its witness_id pairs with the entry above and a monitor can file it under the cluster it backs up instead of as an unrelated standalone server:

{ "clustered": false, "ready": true, "role": "witness",
  "witness_id": "onetw", "region": "slug", "databases": ["onet"],
  "primary_sql_addrs": ["10.0.0.1:7000"],
  "interval_secs": 60, "duty_pct": 25,
  "last_pull_at": 1785338096128 }

It reports the primary by endpoint, not name: a witness mirrors only its configured databases, not default where the cluster's name lives, and its scoped role is granted the witnesses table alone. last_pull_at is this node's own last SUCCESSFUL cycle (null until one completes) — that is the tell for a witness that is up but failing every cycle, which never gets to send the heartbeat the primary-side gauge watches. Judge liveness against duty_pct, not interval_secs: at 25 % a long cycle rests roughly three times as long as it ran, so gaps far exceeding the interval are normal.

Operational notes & limitations

• Same RF and seed list on every node. A node's own bind_addr:internode_port must be in seeds, and bind_addr must be reachable by peers (not 0.0.0.0).
• Distinct data_dir per node (and distinct ports when co-located).
• No membership gossip/consensus. Static membership is via seeds; runtime changes are best-effort broadcasts ordered by an epoch and persisted, so a restart reloads the live ring. A fresh node auto-announces to a seed to get admitted (above), but there is no continuous gossip — a node that missed a membership broadcast while up needs it re-sent, and two concurrent topology changes aren't linearizable. Do one membership change at a time.
• Transactions are single-node. BEGIN/COMMIT/ROLLBACK work on the embedded engine and over driver connections to a STANDALONE server (per-connection transaction state, crash-atomic commits via a redo journal); the cluster coordinator autocommits per statement and refuses transaction control — there is no distributed transaction coordinator.
• Joins gather to the coordinator. A single-table WHERE is pushed to the shards, but a SQL JOIN pulls the tables to the coordinating node.
• Bulk-apply QoS. Inbound bulk batch appliers (drain, rebalance, hint replay, repair) run under admission control — at most half the cores (clamped 1–4) apply batches concurrently, so a migration flood can't monopolize the CPU and starve foreground queries (measured during a decommission: cpu PSI ~80% with io ~0 — pure CPU saturation from FTS-indexing inbound rows). The admission wait is bounded (2 s): a saturated node answers "busy" fast instead of parking excess connection threads on the gate — an unbounded queue would let a rejoining node facing every peer's catch-up flood accumulate thousands of abandoned threads (senders time out at 10 s and retry on fresh connections) while making no progress. Rejected/timed-out senders degrade to hints, which repair backstops. "Busy"/shedding replies count toward the sender's circuit breaker like transport errors (probes still bypass and close it), so a saturated peer gets a cooldown instead of a full-rate retry hammer — without this the rejecting node would burn cores decoding a flood it keeps refusing. Inbound repair scans (ScanPage/LocalScan) take the engine read lock with a bounded wait (1.5 s) and answer "busy" rather than parking behind a catch-up write flood; the repairing peer retries a busy answer with backoff (500 ms; a peer that stays busy trips the breaker within a few attempts), then treats it as unreachable for that pass and retries next interval. TS repair merges and summaries (TsMerge/TsSummary) instead queue for the lock: they arrive serially (one in flight per pass), so parked threads can't pile up, and a bounded poll starved under continuous ingest — a writer is almost always waiting, and waiting writers block new read attempts — failing passes that should have converged. The drain also pauses migration_pause_ms (floor 10 ms) between chunks. node_stats carries cpu_pressure_pct (PSI) so saturation is visible per node.
• Broad filters resolve in one merged scan. A pushed-down WHERE (or an index scan) gathers candidate keys per member and re-reads them for the authoritative LWW version. Few candidates re-read as quorum point reads; past ~256 the coordinator switches to a single paged, LWW-merged pass over the table intersected with the candidate set — same read-quorum guarantee, one scan instead of one RPC fan-out per key (a count(*) over a 100k-row match would otherwise issue 100k sequential quorum reads).
• A flapping peer is circuit-broken. A zombie node (TCP up, application unresponsive) would otherwise make every replicated write burn the full internode I/O timeout — worse than a cleanly-down peer. After 3 consecutive failures a peer's circuit opens: calls to it fail fast (the coordinator hints immediately) for a 10 s cooldown, then one call re-tests. Liveness probes bypass the breaker and close it on success.
• Disk-spilled hints drain in bounded pages, only to live peers. The hinted-handoff overflow log replays a page (1024 records) at a time and is left untouched while its peer is down, so a large log never balloons a restarting node's memory. The drain also runs after a restart for logs inherited from the previous process.
• Unfiltered scans page, merge, and honour LIMIT. SELECT … FROM t with no WHERE gathers every shard last-writer-wins, paged so the coordinator holds a few pages at a time rather than whole shards. A plain LIMIT n (no ORDER BY) is pushed into the gather: sources are paged in lockstep and a row is emitted once every still-active replica has scanned past it, so the scan stops after the first n rows are sealed instead of materialising the whole table.
• A PK pinned by =/IN is a point-read set. When every primary-key column is pinned by an equality or a literal IN list (bound array parameters included), the coordinator resolves the exact candidate keys (≤ 1000; composite keys cross-multiply) and routes each to its replica set — the "fetch these N ids" shape never scatters a filter.
• ORDER BY <indexed> LIMIT k at QUORUM is a distributed sorted top-k (k ≤ 10000): every member returns its local index-ordered top candidates (4× overfetched to absorb per-shard staleness in the sort column and boundary ties), the bounded union is re-read at quorum, and the executor re-sorts — ~members × 4k row reads instead of gathering the whole match set. A multi-key ORDER BY <indexed>, <more…> rides the same leading-column windows under a coordinator-side completeness proof: the top-k is served bounded when at least k rows sort strictly before every truncated window's weakest candidate on the leading column (ties there are decided by the later keys, which the full re-sort applies); a leading column with too few distinct values fails the proof and the exact gather answers. Conservative by construction: if any member is unreachable or cannot walk in order, or fewer matches than k resolve, the exact full gather answers instead; every returned row is quorum-fresh. (At consistency ONE on a full-copy cluster the ordered read serves entirely from the local replica's index walk.)
• Schema repair converges index definitions, not just names. Every repair pass exchanges the full catalog as stamped idempotent DDL; a replayed CREATE … IF NOT EXISTS whose schema stamp advances replaces a differing search/vector index definition and rebuilds it — a node that missed the DDL that widened an index while down does not keep the stale, narrower definition forever. SHOW INDEXES' local column (ok/building/missing) exposes each node's live index state for cross-ring comparison.

See RESHARDING.md for the deeper design and the current edges.