skaidb metrics & observability

This is the contract dashboards and monitoring agents build against. Every metric skaidb exposes is listed here with its type, labels, and meaning. The registry renders the Prometheus text exposition format with a correct # TYPE and # HELP per metric — counters use rate()/increase(), gauges are instantaneous, histograms expose _bucket{le=…} + _sum + _count.

Endpoints

skaidb serves these over HTTP/1.1. They are unauthenticated and read-only, so orchestrators, load balancers, and scrapers need no credentials and no admin rights:

Path	Method	Purpose
/metrics	GET	Prometheus scrape. Pull-model gauges are refreshed on each scrape.
/health, /healthz	GET	Liveness — 200 ok whenever the process is up.
/ready, /readyz	GET	Readiness — 200 ready when the node will serve reads AND accept writes (engine open, not shedding, disk not blocked); 503 with the reasons in the body otherwise. ?strict=1 also waits out resharding migration, resync backfill, and heavy search/index/vector rebuilds — gate rolling operations on it.
/status	GET	Low-privilege topology read: ring/epoch/members and default consistency, no secrets. Unauthenticated by default; set server.status_auth_required = true to require MONITOR on * (the probes below stay open regardless).
/admin/slow	POST	Sample of recent (masked) slow queries. Requires Admin.
/admin/status	POST	Full topology incl. member ids. Requires Admin.

The /metrics, /health, /ready, /status routes are served both on the REST port (server.rest_port, default 7080) and on a dedicated metrics listener (observability.prometheus_port, default 9090) when that port differs from the REST port. Point your scraper at prometheus_port to keep it off the data plane.

SHOW TABLES / SHOW INDEXES (see QUERY_SYNTAX.md) let a tool enumerate the catalog without /query data access.

The ops shortlist (page/panel on these first)

The ~15 metrics an external monitor should build against, before anything else on this page. "Page" = wake someone; "panel" = graph and alert lazily.

Metric	What it means	Guidance
skaidb_up	1 while the process serves.	Page on scrape failure / 0.
skaidb_iwm_state	Workload-manager state: 0 normal, 1 elevated, 2 shedding writes, 3 disk-blocked.	Page on ≥2 sustained >5 min; panel 1.
skaidb_memory_shedding_writes	1 while writes are rejected under memory pressure.	Page on sustained 1 (starts at 85% of limit, clears at 70%).
skaidb_memory_used_bytes / skaidb_memory_limit_bytes	Sampled usage vs the node's (cgroup) limit.	Panel the ratio; alert >90% for 10 min.
skaidb_iwm_disk_write_block (+ skaidb_iwm_disk_rejections_total)	1 while disk-growing writes are refused for lack of space.	Page on 1 — the node is read-only until space frees.
skaidb_storage_disk_bytes	Data-dir bytes actually on disk.	Panel vs volume size; alert >80%.
skaidb_cluster_hints_pending	Writes buffered for currently-unreachable replicas (also per-peer in /status peers[].hints_pending + lag_ms).	Alert on non-zero >15 min: a peer is down or unreachable. Must drain to ~0 before each step of a rolling upgrade.
skaidb_cluster_hints_expired_total	Hints dropped after cluster.hint_max_age_secs — those writes now reach the replica only via anti-entropy.	Page on increase: replicas were down longer than the hint window.
skaidb_ae_last_pass_at_seconds	Unix time of the last completed anti-entropy pass.	Alert when now - value exceeds the AE interval ceiling — repair has stalled.
skaidb_wal_archive_backlog_segments	WAL segments not yet archived (PITR).	Alert on non-zero sustained: the archive is stalled and the recovery window is eroding.
skaidb_cluster_search_shard_missing_total	Search queries answered with a shard missing (partial results).	Page on increase — silent wrong answers to users.
skaidb_storage_quarantined_tables	Tables quarantined for torn/corrupt files.	Page on non-zero.
skaidb_flusher_last_tick_age_seconds	Age of the last completed background maintenance tick.	Page >600 s: the flusher thread is wedged, wherever it stalled.
skaidb_storage_flushes_total / skaidb_storage_compactions_total	Background write-path health.	Panel the rates; a flatline under write load with rising skaidb_storage_memtable_max_bytes means the flusher is not keeping up.
skaidb_witness_oldest_sync_age_seconds	Staleness of the least-synced witness.	Alert when it exceeds a few pull intervals (witness recovery data going stale).
skaidb_maintenance_longest_job_seconds	Longest currently-running maintenance job.	Panel; alert on hours-long jobs (stuck rebuild).
skaidb_build_info	version/git_sha labels, always 1.	Panel for deploy tracking; no alert.

Conventions

• Counters end in _total (with a few standard exceptions like skaidb_*_seconds_count). Only ever increase; reset to 0 on restart.
• Gauges are absolute, can go up or down (e.g. skaidb_queries_in_flight).
• Histograms (skaidb_query_duration_seconds) bucket observations; query the _bucket/_sum/_count series with histogram_quantile().
• Cardinality is bounded. Per-table metrics are opt-in (observability.per_table_metrics = true) because the table count is unbounded. Error and consistency labels come from small fixed sets.
• Every series can be attributed to a node via skaidb_node_info (or by tagging the scrape target).

Build / runtime

Metric	Type	Labels	Meaning
skaidb_up	gauge	—	1 while the server is up.
skaidb_build_info	gauge	version, git_sha, rustc	Always 1; build metadata for tracking deploys.
skaidb_node_info	gauge	node_id, role	Always 1; node identity so federated scrapes are distinguishable at the source.
skaidb_start_time_seconds	gauge	—	Unix time the process started.
skaidb_uptime_seconds	gauge	—	Seconds since start.

git_sha/rustc come from the SKAIDB_GIT_SHA / SKAIDB_RUSTC build-time env vars; unset → "unknown".

Query path

Metric	Type	Labels	Meaning
skaidb_queries_total	counter	type	Statements executed (select/insert/update/delete/ddl/tx/other).
skaidb_query_duration_seconds	histogram	type	Execution latency. Use histogram_quantile(0.99, …).
skaidb_queries_in_flight	gauge	—	Statements currently executing.
skaidb_query_errors_total	counter	class	Failed statements by class: parse, constraint, storage, timeout, permission, other.
skaidb_rows_returned_total	counter	—	Rows returned to clients.
skaidb_rows_written_total	counter	—	Rows written (inserted/updated/deleted) — the write-throughput signal a bulk import shows up in (queries/s counts statements, so a multi-row batch is one query).
skaidb_rows_scanned_total	counter	—	Result cells examined (rows × width) — a proxy for result volume.
skaidb_slow_queries_total	counter	—	Statements slower than slow_query_ms.
skaidb_transactions_total	counter	kind	begin/commit/rollback (embedded engine).
skaidb_authz_denied_total	counter	—	Statements denied by RBAC.
skaidb_logins_total / skaidb_login_failures_total	counter	—	Auth outcomes.
skaidb_admin_total	counter	op	Admin control-plane ops (status/add_node/remove_node/repair/reclaim/slow).

Connections

Metric	Type	Labels	Meaning
skaidb_connections_active	gauge	endpoint	Open connections by endpoint (binary/rest/mqtt).
skaidb_connections_total	counter	endpoint	Connections accepted by endpoint.
skaidb_connections_refused_total	counter	endpoint, reason	Connections turned away at admission; reason="over_capacity" means server.max_connections was already reached.
skaidb_auth_refused_total	counter	endpoint, reason	Authentications refused before the verifier ran; reason="rate_limited" means the source IP was over auth.failed_login_rate_limit.

MQTT broker

Present when [mqtt] is enabled (see MQTT.md).

Metric	Type	Labels	Meaning
skaidb_mqtt_packets_total	counter	type, dir	Control packets by type (publish, puback, subscribe, …) and direction (in/out).
skaidb_mqtt_connect_total	counter	outcome	Connection admissions: accepted, rate_limited, over_capacity.
skaidb_mqtt_messages_dropped_total	counter	reason	Dropped deliveries: outbox_full, queue_full, retained_full, expired, too_large, acl, store_error.
skaidb_mqtt_sessions_expired_total	counter	—	Stored sessions discarded by the expiry sweep.
skaidb_mqtt_sink_rows_total	counter	—	Messages captured by [[mqtt.sink]] rules.
skaidb_mqtt_sink_dropped_total	counter	reason	Sink failures: error, ooo (out-of-order-window TS drops).
skaidb_mqtt_cluster_forward_failed_total	counter	—	Peer fan-out RPCs that failed (state self-heals via _mqtt.*).
skaidb_mqtt_cluster_forward_dropped_total	counter	—	Fan-out events dropped on queue overflow.

Storage / LSM

Pulled from the engine snapshot at scrape time (aggregated across all table and index storage engines).

Metric	Type	Labels	Meaning
skaidb_storage_tables	gauge	—	Tables in the catalog.
skaidb_storage_indexes	gauge	—	Secondary + vector indexes.
skaidb_storage_memtable_bytes	gauge	—	Approx live memtable footprint (sum across all engines).
skaidb_storage_memtable_max_bytes	gauge	—	Largest single-engine memtable footprint (active + frozen), excluding memory = true tables (their memtable is the table by design). Stays at or under skaidb_storage_flush_threshold_bytes when the background flusher is healthy: the maintenance sweep freezes any over-threshold engine even if no writes arrive. Alert on the ratio of the two sitting well above 1: the flusher is wedged.
skaidb_storage_sstables	gauge	—	On-disk SSTable count across levels.
skaidb_storage_disk_bytes	gauge	—	On-disk bytes across all SSTables.
skaidb_storage_flushes_total	counter	—	Memtable flushes installed as SSTables. With memory-plan-sized thresholds a flush can legitimately be a rare event — compare memtable_max_bytes to flush_threshold_bytes rather than expecting steady flush cadence.
skaidb_storage_build_failures_total	counter	—	SSTable builds (flush or compaction) that failed. A committed write is never lost to one — the memtable keeps the rows and the build is retried after a backoff — but a rising rate means the node cannot write its memtables out (a full disk, typically) and memory is holding what should be on disk. Alert on this.
skaidb_storage_enospc_total	counter	—	Storage I/O errors that were No space left on device.
skaidb_storage_flush_threshold_bytes	gauge	—	The per-engine flush threshold in force on this node (32 MB default; the memory plan sizes it up to 1 GB under memory_target). The healthy ceiling for memtable_max_bytes.
skaidb_maintenance_longest_job_seconds	gauge	—	Age of the longest in-flight background job (SHOW MAINTENANCE). Digest builds and backfills finish in minutes; an hours-old job is stuck.
skaidb_flusher_last_tick_age_seconds	gauge	—	Seconds since the background flusher/maintenance thread completed a full tick (clustered nodes only). The thread-liveness heartbeat: a growing age is a wedge, wherever the thread stalled.
skaidb_storage_compactions_total	counter	—	Compaction passes completed.
skaidb_storage_legacy_block_bytes	gauge	—	On-disk bytes in SSTables written under an older (smaller) block-size target. Compaction rewrites them at the current target, so this trends to zero; cold tables are finished by a background sweep (one small table per maintenance pass). Zero = block-size migration complete.
skaidb_storage_legacy_block_files	gauge	—	File count behind skaidb_storage_legacy_block_bytes.
skaidb_storage_quarantined_tables	gauge	—	Tables whose files were damaged (torn/corrupt SSTable or WAL) and were quarantined at startup instead of refusing the whole open. Statements on them error with the cause; restore the table's directory (backup or wipe-resync from peers) or DROP TABLE to discard. Non-zero should alert.
skaidb_storage_kek_stale_files	gauge	—	KEK rotation progress: files (SSTables, WALs, sealed WAL segments) whose data key is still wrapped by a KEK other than the current primary. A background sweep rewraps them in place; zero means no file references a previous KEK and its keyfile can be retired (see CLUSTERING.md, "Rotating the at-rest KEK").
skaidb_storage_compaction_bytes_total	counter	—	Bytes written by compaction.
skaidb_batch_spliced_rows_total	counter	—	Rows executed through the spliced executemany fast path (plain INSERT batches run as multi-row statements instead of once per row). Zero while drivers send batches = the per-row fallback is carrying them.
skaidb_wal_bytes	gauge	—	Live write-ahead log size.
skaidb_wal_archived_segments_total	counter	—	Sealed WAL segments copied into the point-in-time-recovery archive before deletion. Zero forever = storage.wal_archive_dir is unset.
skaidb_wal_archived_bytes_total	counter	—	Bytes of those segments.
skaidb_wal_archive_backlog_segments	gauge	—	Segments the archive REFUSED, so the node did NOT delete them. Non-zero = the archive is behind (full or unmounted volume) and the data directory is growing. Deliberate: a gapped archive would still look restorable. Alert on this.
skaidb_wal_fsyncs_total	counter	—	WAL fsyncs issued (compare to writes to see group-commit coalescing).
skaidb_cache_hits_total / skaidb_cache_misses_total / skaidb_cache_evictions_total	counter	—	Read-cache effectiveness.
skaidb_cache_entries	gauge	—	Live read-cache entries.
skaidb_cluster_search_shard_missing_total	counter	—	Ranked searches (vector ANN / full-text relevance) that ran with a shard missing because a peer could not answer. These degrade rather than fail by design — refusing every search over one peer blip would trade a quality dip for an outage — so this counter is the only signal that results were computed from fewer shards. A rising rate means searches are quietly returning less than they should.
skaidb_jobs_runs_total	counter	—	Scheduled job runs executed by this node (any outcome). Jobs are singly-owned, so this is per-owner, not per-cluster.
skaidb_jobs_failures_total	counter	—	Job runs that ended in error. Also visible per job in SHOW JOBS (last_status, last_error, failures) and in _job_runs. Alert on a sustained rate: a job retries with exponential backoff, so a broken one goes quiet rather than loud.
skaidb_jobs_deferred_total	counter	—	Job ticks the workload manager deferred: the run did not start and was rescheduled. Not a failure, and invisible in the job's own state — a job that is being shed looks exactly like a healthy quiet one. A rising rate means scheduled work is losing to load.
skaidb_iwm_state	gauge	—	IWM node state: 0 normal, 1 elevated (memory in its release tier), 2 overloaded (shedding writes), 3 critical (disk writes blocked). Alert on dwell time at ≥1.
skaidb_iwm_disk_write_block	gauge	—	1 while disk-growing writes are blocked for lack of free space ([iwm] watermarks; hysteresis).
skaidb_iwm_disk_rejections_total	counter	—	Writes rejected by the low-disk guard.
skaidb_iwm_disk_enospc_trips_total	counter	—	Times a real ENOSPC write failure tripped the disk guard ahead of the free-space sampler. Any non-zero value means the volume actually hit zero free bytes — the watermark was not enough headroom for this node's compaction outputs; raise iwm.disk_low_watermark.
skaidb_iwm_disk_reserve_present	gauge	—	1 while the guard's reserve file (iwm.disk_reserve) is on disk; 0 while it has been released to give deletes/compaction room (writes are blocked), or when the reserve is off.
skaidb_iwm_sentinel_kills_total	counter	—	Statements killed by the Query Sentinel (iwm.sentinel = auto, fires only while shedding).
skaidb_queries_disconnect_kills_total	counter	—	Statements cancelled because their driver client closed the connection mid-statement.
skaidb_iwm_conn_rejections_total	counter	—	Connections refused by the IWM rate gate (iwm.conn_rate).
skaidb_iwm_ops_rejections_total	counter	—	Statements refused by the Overloaded ops ceiling (iwm.ops_rate_overloaded).
skaidb_query_stream_total{path}	counter	path	Streamed queries by server path: keyset paged (node holds one page) or buffered (node materialised, then chunked).
skaidb_stream_events_lost_total	counter	—	Change-stream events that could not be appended to their log. The source write still succeeded, so any non-zero value means a stream's log is behind its table and a replay of that window has a hole in it. Should be 0.
skaidb_iwm_heavy_scans_running	gauge	—	Heavy read scans holding an admission slot right now (iwm.max_concurrent_scans).
skaidb_iwm_scan_admission_refusals_total	counter	—	Heavy scans refused because every admission slot was taken.
skaidb_block_cache_bytes	gauge	—	Decompressed bytes resident across all tables' block caches. The first place to look when node memory ramps during scan-heavy periods (anti-entropy sweeps, witness pulls).
skaidb_block_cache_hits_total / skaidb_block_cache_misses_total / skaidb_block_cache_evictions_total	counter	—	Block-cache effectiveness.
skaidb_bloom_negative_lookups_total	counter	—	Point reads resolved absent by the Bloom/SSTable layer.

Host system (per node)

Sampled from /proc (plus df for the data-directory filesystem) at each scrape; each node reports its own host. Memory is cgroup-aware — a container reports its own limit and usage, not the host's. CPU% and disk throughput are computed over the window since the previous sample.

Metric	Type	Labels	Meaning
skaidb_host_cpu_percent	gauge	—	Busy CPU as % of all cores over the last sampling window.
skaidb_host_cpus	gauge	—	Logical CPU count.
skaidb_host_mem_total_bytes	gauge	—	Total memory (cgroup limit when one applies).
skaidb_host_mem_used_bytes	gauge	—	Used memory (cgroup memory.current when limited, else MemTotal - MemAvailable).
skaidb_host_rss_bytes	gauge	—	The skaidb process's resident set size.
skaidb_host_disk_read_bytes_total / skaidb_host_disk_written_bytes_total	counter	—	Whole-host disk IO since boot (physical devices; partitions/loop/dm excluded).
skaidb_host_disk_total_bytes / skaidb_host_disk_available_bytes	gauge	—	The filesystem holding the data directory.

Per-table (opt-in)

Enabled with observability.per_table_metrics = true. Each carries db and table labels — only turn this on when the table set is small and known.

Metric	Type	Labels	Meaning
skaidb_table_live_keys	gauge	db, table	Live keys (full merged scan; O(rows) at scrape).
skaidb_table_tombstones	gauge	db, table	Tombstones awaiting compaction.
skaidb_table_disk_bytes	gauge	db, table	On-disk bytes for the table — every table kind, TIME-SERIES included.
skaidb_ts_table_series	gauge	db, table	Series count per time-series table.
skaidb_ts_table_samples_appended_total	counter	db, table	Samples appended per time-series table.
skaidb_ts_table_samples_rejected_total	counter	db, table	Samples rejected (OOO / series limit) per time-series table.
skaidb_ts_table_blocks	gauge	db, table	On-disk block count per time-series table — watch it FALL during a compaction-backlog drain.
skaidb_ts_table_maintenance_errors_total	counter	db, table	Best-effort retention/compaction failures (maintenance never fails an append).

Vector index

Metric	Type	Labels	Meaning
skaidb_vector_indexes	gauge	—	HNSW indexes.
skaidb_vector_indexed_total	gauge	—	Total vectors held in memory.
skaidb_vector_rebuild_seconds	gauge	—	Time to rebuild vector indexes on the last open.

Cluster / replication

Present only when the node is clustered (cluster.seeds set). Pulled from the coordinator at scrape time.

Metric	Type	Labels	Meaning
skaidb_membership_epoch	gauge	—	Membership epoch; alert on changes.
skaidb_cluster_members	gauge	—	Members visible from this node.
skaidb_cluster_resharding	gauge	—	1 while a join/decommission dual-write window is open.
skaidb_cluster_writes_total	counter	consistency	Coordinated writes by level (one/quorum/all).
skaidb_cluster_reads_total	counter	consistency	Coordinated reads by level.
skaidb_cluster_quorum_failures_total	counter	kind	Operations that failed to reach quorum (read/write).
skaidb_cluster_read_repairs_total	counter	—	Read-repair writes pushed to lagging replicas.
skaidb_alloc_purges_total	counter	—	jemalloc arena purges on IWM tier relax (iwm.jemalloc_purge_on_relax, off by default). Each purge logs the before/after allocator split — the evidence for whether the knob earns its keep.
skaidb_ae_passes_total	counter	outcome	Anti-entropy pass attempts: reconciled (fixed rows), converged (found nothing), deferred (pressure/peer mid-pass), failed.
skaidb_ae_rows_reconciled_total	counter	—	Rows fixed by anti-entropy — the divergence actually found, and the adaptive cadence's feedback signal.
skaidb_ae_interval_seconds	gauge	—	Current effective anti-entropy interval. Fixed unless cluster.anti_entropy_adaptive is on; then floor..ceiling, relaxing while passes converge, snapping to the floor when one reconciles.
skaidb_ae_last_pass_seconds	gauge	—	Duration of the last completed pass (deferred attempts excluded).
skaidb_ae_last_pass_at_seconds	gauge	—	Unix time the last completed pass finished. Alert on time() - this > ceiling — it is what distinguishes a healthy backed-off loop from a stuck one.
skaidb_cluster_hints_stored_total	counter	—	Hinted-handoff writes buffered for unreachable replicas.
skaidb_cluster_hints_replayed_total	counter	—	Hinted-handoff writes successfully replayed.
skaidb_cluster_hints_expired_total	counter	—	Hints dropped without delivery: older than cluster.hint_max_age_secs, over the per-replica cluster.hint_max_disk_mb cap, or logged for a replica that left the ring. Anti-entropy repair covers what they carried; a growing rate means a replica is persistently behind or gone.
skaidb_cluster_hints_pending	gauge	—	Hints currently buffered (all peers).
skaidb_cluster_hints_pending_peer	gauge	peer	Hints buffered per peer — exact replication backlog for that node.
skaidb_cluster_cas_rounds_total	counter	—	Per-key consensus (CAS) rounds started by clustered UPDATEs.
skaidb_cluster_cas_retries_total	counter	—	CAS rounds retried on contention (ballot bumps / lost quorums).
skaidb_cluster_cas_contention_failures_total	counter	—	UPDATE statements that exhausted CAS retries; alert if growing — a pathologically hot key.
skaidb_cluster_fk_guard_waits_total	counter	—	Backoff rounds foreign-key writers spent waiting on the other side of a constraint (a child insert behind a parent delete's fence, or a parent delete draining child claims in flight). A steady rate means hot parent rows contended from both sides.
skaidb_cluster_fk_guard_stale_resets_total	counter	—	Foreign-key guard state (a claim or a fence) reset because its holder stopped heartbeating for the 10 s grace period — a coordinator died mid-statement. A steady rate on a healthy cluster is a bug; the constraint stays enforced, but the statement that died may have left an orphan VALIDATE CONSTRAINT finds.
skaidb_cluster_index_union_incomplete_total	counter	—	Index-bounded reads that fell back to an unbounded scan because a peer could not contribute its candidate keys (unreachable, write-locked, breaker-open). Answers stay correct; a rising count means they are being served the slow way — look at peer health.
skaidb_memory_shedding_writes	gauge	—	1 while the node is shedding writes under memory pressure (rejecting new writes so it can drain instead of being OOM-killed). Alert on sustained 1.
skaidb_memory_used_bytes / skaidb_memory_limit_bytes	gauge	—	Sampled memory usage vs. the node's limit (cgroup when set, else system RAM); shedding starts at 85% and clears at 70%.
skaidb_memory_anon_bytes / skaidb_memory_file_bytes	gauge	—	Cgroup anon vs file split. The production memory-wedge signature is anon ratcheting up while file collapses toward zero — graph these together.
skaidb_alloc_allocated_bytes / skaidb_alloc_resident_bytes / skaidb_alloc_retained_bytes	gauge	—	jemalloc live heap / resident pages / OS-unreturned address space. resident − allocated ≈ fragmentation + unpurged dirty pages: distinguishes "something holds memory" from "the allocator won't give it back".
skaidb_cluster_replication_lag_ms	gauge	peer	Approx. ms between this node's HLC frontier and the latest write it has confirmed peer applied.
skaidb_cluster_peer_requests_total	counter	—	Internode RPCs issued by the coordinator.
skaidb_cluster_peer_errors_total	counter	—	Internode RPCs that errored or timed out.
skaidb_witness_last_seen_age_seconds	gauge	witness	Seconds since a registered witness last heartbeat its registry row.
skaidb_witness_oldest_sync_age_seconds	gauge	witness	Seconds since the LEAST caught-up mirrored table was synced. Alert on this one (e.g. > 3600).

Witness staleness: alert on sync age, not last-seen. A witness heartbeats its registry row every cycle, so last_seen_age stays small even while one table is stuck — a witness once failed every cycle for three days with a perfectly healthy heartbeat. oldest_sync_age is derived from the per-table watermarks and is the honest signal. Both are emitted by members (a witness is not a member and does not emit them about itself); the registry read behind them is throttled to once a minute. Absent until a witness registers, and oldest_sync_age is absent until its first cycle heartbeats a watermark. Set the threshold against the witness's duty_pct, not its interval_secs — at 25% a long cycle rests roughly three times as long as it ran, so gaps well past the interval are normal. The same two clocks are on GET /status as seen_age_secs / sync_age_secs per witness (see CLUSTERING.md).

These anti-entropy and quorum signals are correctness-critical — without them, read-repair, hinted handoff, and quorum failures are invisible.

Reading replication health per peer. skaidb_cluster_hints_pending_peer is the exact backlog — writes this node has buffered for a peer it couldn't reach. skaidb_cluster_replication_lag_ms is an estimate: it only advances a peer's baseline when a write is confirmed to it, so it climbs while a peer is unreachable and falls once hinted handoff/anti-entropy catch it up. A peer with no confirmed write yet (freshly added, or down since startup) is absent from replication_lag_ms — rely on hints_pending_peer and the reachable flag in \cluster for those. Both are emitted per current peer (ring ∪ configured seeds).

The same per-peer backlog and lag are surfaced without a scraper: GET /status carries a peers array (id, in_ring, in_config, hints_pending, lag_ms), and the UI members panel renders a backlog column (buffered writes owed to that node — nonzero is flagged) and a lag column (that node's replication lag), so you can see at a glance how far behind each node is.

The node_stats table (replicated host statistics)

With observability.node_stats (default on), every node INSERTs its own host statistics — CPU, load, memory, disk I/O and space, uptime, restart count, and cgroup OOM kills — into the replicated node_stats table every node_stats_interval_secs (default 1 s, live-mutable): one row per node, keyed on the node id, stamped with the sample time (ts, epoch ms). The row replicates like any write, so any member serves the whole cluster's picture from a local read, and it is plain SQL:

SELECT node, ts, mem_used_bytes, restarts, oom_kills FROM node_stats;
-- memory-ramp composition (anon vs file vs allocator view):
SELECT node, mem_anon_bytes, mem_file_bytes,
       alloc_allocated_bytes, alloc_resident_bytes, alloc_retained_bytes
FROM node_stats;

oom_kills counts kernel OOM kills in the node's cgroup since the last RESET OOM COUNTER [ON '<node>'] (ADMIN) — or, absent one, since container boot. After an incident is investigated, reset it so the next nonzero reading is unambiguous fresh signal.

The UI's stats NODES table reads this (falling back to live probes for members without a fresh row, e.g. mid rolling-upgrade) and shows each row's age — a silently struggling node dims and its age climbs, instead of one missed probe flapping a live node to "unreachable". Node restarts log their start number, and when the cgroup's OOM-kill count advanced since the previous start, the log says the prior run likely died to the OOM killer. Probe-loss/recovery transitions and circuit-breaker events are logged on the coordinator as well.

Logs

Audit/query/login logs are written to stderr in human-readable text by default. Set observability.log_format = "json" to emit one JSON object per line (event, elapsed_ms, error, sql, …) so a log agent can parse them reliably. Query logs are masked (literals → ?) unless query_log_masked is disabled. A bounded, masked sample of recent slow queries is available at POST /admin/slow.

Log files

By default logs go to stderr. Set observability.log_file to a path to write all audit logs to a file instead (created if absent, appended otherwise):

[observability]
log_file = "/var/log/skaidb/audit.log"

Each log category can be split into its own file with a per-category override; an empty override falls back to log_file, and an empty log_file falls back to stderr:

Key	Stream
observability.query_log_file	executed-statement log
observability.slow_query_log_file	slow-query log
observability.error_log_file	error log
observability.login_log_file	login/auth log

[observability]
log_file = "/var/log/skaidb/audit.log"   # everything not overridden below
error_log_file = "/var/log/skaidb/error.log"
slow_query_log_file = "/var/log/skaidb/slow.log"

Categories pointed at the same path share one file handle, so their lines interleave safely. File sinks are write-behind: executed-statement ([query]) lines are buffered and flushed within 250 ms, while slow-query, error and login/auth lines are flushed as they are written; every buffer is also flushed on clean shutdown (SIGTERM/SIGINT) and before a config set swaps a log file. A tail -f of the query log therefore lags a statement by at most a quarter second, and a line is always written whole. All of these keys are runtime-mutable (config set, --* flags, and SKAIDB_*_LOG_FILE env vars), and a path that can't be opened falls back to stderr with a one-line warning rather than failing startup.

Every log line carries a timestamp: text-format lines (and every operational skaidb: line in the server log) are prefixed with an ISO-8601 UTC instant (2026-07-20T18:42:13.123Z …); JSON-format lines (observability.log_format = "json") carry it as a ts field instead, so each line stays independently parseable.

Query/slow-query/error lines carry the execution context: the authenticated user, the session database, and the access surface — via=driver (binary-protocol/driver connections), rest, es, ui, prom (PromQL evaluations, which log like queries), or internal (background/system statements):

2026-07-20T18:42:13.123Z [query] 3ms user=agencik db=agencik via=driver SELECT … WHERE id = ?
2026-07-20T18:42:14.001Z [query] 1ms user=pi_air_quality db=pi_air_quality via=prom promql query_range avg(pm25{}[?m])

JSON format carries the same as user/db/via fields.

REST request activity

• skaidb_rest_requests_total{path="query"|"insert"|"es"|"prom"|"ui"|"ops"|"admin"|"other"} — REST requests served, per path class, timed end to end.
• skaidb_rest_request_duration_us_total{path=…} — total serving time in microseconds; divide by skaidb_rest_requests_total for the average response time (the overview tab's REST-activity card shows exactly this).