The advisor — findings from the cluster's own signals

Every node emits the raw material of a diagnosis: the slow log, host statistics, the workload manager's state, anti-entropy timings, witness freshness, and the free-text maintenance lines in its log. The advisor turns those into findings — what is wrong, the evidence, and what to do — and shows each finding to the people it concerns.

What it does

Collects signals. Every member (and a standalone server) feeds _advisor.signals: log lines that match a known pattern (anti-entropy legs, digest self-checks, store-lock holds, at-rest open failures, resyncs, membership repairs, panics, refused index mirrors), slow statements aggregated per minute by masked shape, failed statements aggregated by message, and sampled state (pending hints, quorum failures, the IWM level, the maintenance thread's tick age, OOM kills, restarts, disk headroom, CPU pressure, witness staleness). Only what a rule looks at is written; signals live 30 days.
Runs rules. One node — the advisor node — reads the last six hours of signals from every node and runs deterministic detectors over them. Each rule knows a failure mode of this product: which signals it shows up as, who it concerns, what usually causes it, and which document explains it. A detection becomes a finding, deduplicated by a stable fingerprint, so a problem that persists updates one row's last_seen and occurrences instead of piling up.
Shows findings by audience.

Category	About	Who sees it
`database`	one database's tables, indexes, TTLs, statement shapes	anyone with a privilege on that database, or `ADVISOR ON DATABASE d`
`os`	a host or the whole cluster: memory, disk, I/O, CPU, replication, witnesses	`ADVISOR ON *` and superusers
`bug`	the software may be wrong: digest mismatches, panics, open failures, wedged threads	`ADVISOR ON *` and superusers

A database finding names its tables and statement shapes and never carries host detail or other databases' names, so showing it to the database's users leaks nothing about the rest of the cluster.

The advisor is advisory: nothing is applied. Actions are typed (sql, config, os, docs) and only an allow-list of statements is marked safe (reading a document, SHOW, EXPLAIN, CREATE INDEX, ALTER TABLE … SET (ttl = …), live-mutable SET CONFIG).

Statements

SHOW ADVICE [FOR DATABASE d] [LIMIT n]
  -- id, category, severity, status, scope, occurrences, first_seen,
  -- last_seen, title, summary, suggestion, actions (JSON), evidence
  -- (signal ids), note. Filtered by the caller's grants; no privilege
  -- needed beyond what already shows a database.
ALTER ADVICE '<id>' SET STATUS 'acknowledged' | 'dismissed' | 'fixed' | 'false_positive' | 'new' [NOTE '…']
  -- on a finding the caller can see. A dismissed or false_positive finding
  -- stays that way when re-detected; a fixed one that recurs reopens. A
  -- status set while a rules pass runs stands: the pass changes a status
  -- only to reopen a finding that is still fixed or resolved.
SHOW ADVISOR
  -- settings, this node, whether it runs the rules, the last run, signal
  -- and open-finding counts (MONITOR).
ALTER ADVISOR SET (enabled = true|false, schedule = '5m'|'off', model_schedule = '30m'|'off',
                   categories = 'database,os,bug', node = '<member id>',
                   provider = '<model provider>'|'none',
                   data_sharing = 'shapes'|'none'|'samples',
                   token_budget_per_day = 300000, max_runs_per_hour = 4,
                   on_observation = false, docs_grounding = true,
                   sample_databases = 'shop,crm')
  -- ADMIN. schedule minimum 60 s; node empty = the lowest member id.
CALL advisor.run()
  -- ADMIN: the rules now, on this node, then the model stage when a
  -- provider is configured; returns SHOW ADVICE.
CALL advisor.run('<database>')
  -- for anyone who may read that database: a scoped model run that
  -- reports database findings for it only; returns SHOW ADVICE FOR DATABASE.
CALL advisor.export_bug_report('<id>')
  -- ADMIN: a Markdown bug report for one finding, written by the
  -- provider from the finding, its evidence rows, the cluster facts and
  -- the doc sections it cites; stored on the finding (bug_report) and
  -- returned. It never leaves the cluster on its own.
CALL advisor.digest()
  -- ADVISOR ON * or ADMIN: the digest the model stage would send now, as
  -- one JSON row, for an external agent (the MCP pull mode).
CALL advisor.report('<findings json>')
  -- ADVISOR ON * or ADMIN: findings an external agent produced from the
  -- last pulled digest, validated exactly like the model's and stored
  -- with origin 'external'; returns SHOW ADVICE.
CREATE MODEL PROVIDER [IF NOT EXISTS] <name> [TYPE 'anthropic'|'openai'|'openai_compatible'|'ollama'|'embedding'|'rerank']
  WITH (endpoint = 'https://…', model = '…', api_key = 'env:NAME'|'file:PATH'|'cmd:PATH',
        timeout_secs = 120, max_output_tokens = 8000, fallbacks = true, tls_verify = 'system'|'insecure',
        reasoning_effort = 'none'|'minimal'|'low'|'medium'|'high'|'xhigh'|'max',
        dim = 768, batch_size = 32, active = true)   -- the last three: embedding / rerank types
ALTER MODEL PROVIDER <name> SET (...)
DROP MODEL PROVIDER [IF EXISTS] <name>       -- clears the advisor's provider if it was this one
SHOW MODEL PROVIDERS                         -- MONITOR; shows the key's reference, never its value
GRANT ADVISOR ON DATABASE d TO r     -- see d's findings without data access
GRANT ADVISOR ON * TO r              -- see every category

Findings whose rule stops firing and that nobody touched for a day are marked resolved automatically (only from new or acknowledged). Every status change is recorded in _advisor.feedback with the caller and the note.

The tables

_advisor is a server database like _bi: created by the primaries (and by a standalone server), replicated, mirrored to witnesses, hidden from SHOW DATABASES's default listing.

Table	Holds
`signals`	one row per collected signal: `ts`, `node`, `kind`, `sig`, `severity`, `metric`, `body` (masked), `db`, `tbl`, `meta`; TTL 30 days
`findings`	one row per fingerprint: category, scope, severity, status, title, summary, suggestion, actions, evidence, first/last seen, occurrences, note
`feedback`	every `ALTER ADVICE` with who, verdict and note
`runs`	every rules run: node, cause, signals read, findings produced
`settings`	`ALTER ADVISOR` options

Reading the tables directly (SELECT … FROM _advisor.findings) follows the database's normal grants; SHOW ADVICE is the audience-filtered view.

The rules

Rule	Category	Fires on	Points at
`ae_leg_slow`	database	an anti-entropy row leg ≥ 300 s naming the same slowest table on two or more passes (one slow pass is a startup catch-up or a repair)	a digest-less or hot table; CLUSTERING.md anti-entropy
`slow_shape`	database	≥ 20 slow executions of one shape, or one ≥ 3 s	`EXPLAIN`; INDEXING.md
`node_stalled_statements`	os	slow statements that do no work of their own (`USE`, `SHOW`, `SET`, `BEGIN`) or the server's own `_`-database traffic, per node	the node's lock holds and I/O at those times
`toast_index_refused`	database	a mirror refusing `CREATE INDEX` on an out-of-line table	re-mirror or accept
`error_burst`	database	≥ 50 failures with one message in a database	the retrying client
`store_lock_hold`	os	a flush/checkpoint holding a store lock ≥ 2 s	the host's I/O
`iwm_pressure`	os	the workload manager shedding or blocking	memory target, scan budget, disk
`oom_kills`, `restart_loop`	os	kills and repeated restarts in `node_stats`	memory sizing; the error log
`disk_low`	os	under 15% free	reclaiming space
`cpu_pressure`	os	sustained PSI above 50%	attributing the host's load
`hints_pending`	os	hints held for a peer through the whole window	the peer's reachability
`write_quorum_failures`	os	writes that failed quorum	a member down or mid-restart
`witness_stale`	os	a witness unseen for 5 minutes or more	its health and its log
`resync`	os	a node backfilling from its peers	whether anyone wiped it
`flusher_wedged`	bug	the maintenance thread's tick age ≥ 600 s	`SHOW MAINTENANCE`, a stack
`digest_mismatch`	bug	the digest self-check disagreeing on a quiet table	report with the table and versions
`at_rest_open_failed`	bug	a file the node's key cannot open	one damaged file, or a rotated key
`panic`, `backfill_failed`, `membership_healed`, `quarantined`	bug	the matching log lines	report with the lines attached

In the UI

The overview tab carries an advisor card when there is something to show: one row per finding the viewer may see (severity, category, scope, title, status, occurrences, last seen), the counts per category in the card's title, and under a row on click the summary, suggestion and actions, the model's explanation and confidence when it added one, the bug report when one was exported (else the statement that exports it), and the acknowledgement statement. The card is served from /ui/advice with the same grant filtering as SHOW ADVICE. For a viewer with ADVISOR ON * or ADMIN a runs sub-card (from /ui/advisor) lists the last runs — rules passes, model runs, bug reports, external reports — with their cause, provider and model, status, duration, tokens, kept and dropped counts and the failure detail; the drop reasons show on hover.

The model stage

With a MODEL PROVIDER configured, the advisor node sends the window to a language model after each scheduled rules pass and on CALL advisor.run(). Without one (or with data_sharing = 'none') nothing leaves the cluster: the rules alone produce the findings.

Providers. CREATE MODEL PROVIDER stores the endpoint, the model name and a reference to the secret — env:NAME (an environment variable of the node), file:PATH (a file read at call time) or cmd:PATH (a program whose stdout is the key, the at_rest_kek_command pattern). A literal key is refused; the catalog, SHOW MODEL PROVIDERS, backups and witness mirrors carry the reference only, so a node resolves it only where its own environment holds the secret. Types: anthropic (the Messages API, default model claude-opus-5, fallbacks = true opts into the server-side refusal fallback), openai, openai_compatible and ollama (the chat-completions shape, which covers OpenAI, Azure, Ollama, vLLM and llama.cpp; endpoint may name a full path). tls_verify = 'insecure' is for a self-signed local server. reasoning_effort bounds how much the model thinks before it answers — sent as reasoning_effort on the chat-completions shape and as the effort setting on the Anthropic one (where thinking cannot be switched off, so none means low). A local model that thinks by default can spend the whole max_output_tokens on its reasoning and return no answer; give it reasoning_effort = 'none' or a much larger output budget (and a server context that holds the ≈ 16k-token prompt plus that budget).

What the model sees. One request per run: a byte-identical system prefix (the role, the three audiences and their rules, the output contract, and a playbook rendered from the rule catalog — the prefix is cache-friendly) and the digest as the user turn. The digest (JSON, at most 40 KB) holds the cluster facts (members, versions, replication, the databases), the rules' observations — the 40 most severe, each with up to four of the signal rows it cites, shown under short aliases (sig-N) that the validation maps back to the rows' ids — the window's other log-derived signals, per-node metric summaries (min/p50/max), the slow statement shapes as the slow log masks them (never a literal), recent events, the live-mutable config keys, and the findings already open with their status and feedback notes (the judged ones first, at most 60). Row values, statement literals, secrets and config values are not in it; when the window is busier than the cap, the extra signals, events, shapes, unjudged open findings and the least severe observations go first, in that order. CALL advisor.run('db') sends a digest reduced to that database and asks for database findings only.

What comes back. The answer is a JSON document of a fixed schema (structured output); a document that is not the schema fails the run and nothing is stored. Each finding is then checked here: evidence ids must exist in the digest (a finding with none is dropped); a database finding must name a database from the digest, and one whose text mentions a node or another database is downgraded to os (dropped in a scoped run); safe survives only on CREATE INDEX, ALTER TABLE … SET (ttl = …), config actions on a live-mutable key and docs actions; a confidence below 0.4 keeps the question and loses its actions; at most 10 findings, ranked by severity then confidence. A finding that reuses an open rule finding's fingerprint attaches its text to that row as explanation (plus confidence, llm_actions) instead of replacing the rule's; a finding on its own fingerprint is stored with origin = 'llm' under the same status merge as the rules (dismissed and false_positive stay, fixed and resolved reopen). What was dropped, and why, is in the run's row (_advisor.runs.dropped_detail).

Budgets and failure. The model runs on its own cadence, model_schedule (default 30 minutes; the rules keep their 5-minute schedule, which costs nothing), and max_runs_per_hour (default 4) and token_budget_per_day (default 300000, midnight UTC; 0 = no limit, the right setting for a local model) gate the scheduled runs; CALL advisor.run() is not gated. A provider error (HTTP status, a refusal, an answer cut at max_output_tokens, malformed JSON) fails the run, records it in _advisor.runs with the reason, and suspends calls for 10 minutes — SHOW ADVISOR shows provider_health, the tokens spent today, the runs in the last hour and the last model run. Only the advisor node calls out.

When it runs. On model_schedule, and — with on_observation = true — early, when a rules pass raises a finding that was not in the table before, no sooner than 10 minutes after the last model run. Every run is a row in _advisor.runs (cause = schedule, observation, call, bug_report or report).

Documentation grounding. Every node keeps the shipped documentation (ADVISOR, CLUSTERING, INDEXING, METRICS, QUERY_SYNTAX) sectioned by heading in _advisor.docs, loaded once per binary version by a primary and mirrored to witnesses — only the sections whose text changed are rewritten (a version bump that touched no document writes nothing but the version), and never while the member is still in its startup catch-up (the load waits for the next scheduled run). With docs_grounding = true (the default) a digest carries up to eight excerpts: the sections the observations' doc references name and the best keyword match for each observation's title, each with its ref (FILE.md#heading) so a finding's doc_refs can cite it.

Row samples. data_sharing = 'samples' adds, for the databases named in sample_databases only, up to three rows of up to three tables per database — the tables that database's slow statement shapes name — with values cut at 80 characters and any column whose name suggests a secret (token, password, key, credential, cookie, session) redacted. shapes (the default) carries no row values at all.

Pull mode

An external agent — one running through the MCP server, or anything that can run SQL as a role with ADVISOR ON * — can do the model stage's job from outside without the cluster calling anyone: CALL advisor.digest() returns the same digest the model stage would send, and CALL advisor.report('<json>') lands the agent's findings (the answer schema of the model stage: {findings: [...], notes: ''}) under the same validation and the same fingerprints, with origin = 'external'. The MCP server offers these as the advisor_digest and advisor_report tools at its admin tier, and advisor_findings (SHOW ADVICE) at every tier. A report is validated against the digest that was last pulled on that node, so pull, analyse and report in one session.

Embedding and rerank providers

TYPE 'embedding' and TYPE 'rerank' providers are the [inference] block as statements: CREATE MODEL PROVIDER emb TYPE 'embedding' WITH (endpoint = 'https://…/v1/embeddings', model = '…', dim = 768, api_key = 'env:NAME', batch_size = 32, active = true). The active provider of each type (one at a time; ALTER MODEL PROVIDER … SET (active = true) moves it) is installed as the node's embedder or reranker — on the node that ran the statement at once, on every other member within a minute, from the replicated table — and replaces what [inference] configured. They cannot be the advisor's model; ALTER ADVISOR SET (provider = …) refuses them. See VECTOR.md for the wire contracts.