Design Records

• 1: Turning PostgreSQL CSV Logs into Durable Metrics
• 2: Caching pgBackRest Metrics Outside the Scrape Path
• 3: The PostgreSQL Observability Product We Chose Not to Build
• 4: One Endpoint, Several Sources: The Composite Exporter Contract
• 5: Snapshot Histograms Are Gauges, Not Counters

1 - Turning PostgreSQL CSV Logs into Durable Metrics

Decision status: Implemented in a development line; not present in main, a tag, or a published package as of 2026-08-28.
Decision date: 2026-08-24.
Applies to: the default-off PostgreSQL 14+ CSV log collector inside the Composite exporter.
Supersedes: the separate observability-product direction recorded in The PostgreSQL Observability Product We Chose Not to Build.
Release boundary: implementation and source-level tests exist; merge, release, package, Pigsty deployment, and production canary remain separate gates.

PostgreSQL logs contain operational facts that SQL snapshots cannot reconstruct: deadlocks, authentication failures, canceled autovacuum workers, checkpoint phases, temporary-file removals, client disconnects, and the duration of statements selected by logging policy. The design question was not whether those records were useful. It was how to expose a small reliable subset without turning PG Exporter into a log platform.

The final boundary is deliberately narrow:

PG Exporter continuously reads one local PostgreSQL CSV log directory, persists a bounded aggregate state, and publishes low-cardinality counters and histograms on the existing /metrics endpoint. It does not store, search, or send raw logs.

Product boundary

The collector belongs to the existing pg_exporter binary, package, service, target, and /metrics endpoint. It is enabled only by a non-empty log directory:

pg_exporter \
  --pg-log-dir=/var/log/postgresql \
  --pg-log-state-file=/var/lib/pg_exporter/pglog-state.json \
  --pg-log-poll-interval=1s

The environment equivalents are PG_EXPORTER_PG_LOG_DIR, PG_EXPORTER_PG_LOG_STATE_FILE, and PG_EXPORTER_PG_LOG_POLL_INTERVAL. The default directory is empty, the default state path is /var/lib/pg_exporter/pglog-state.json, and the default poll interval is one second.

When disabled, no worker, file scan, state lock, state file, log metric family, or extra scrape coordination exists. --dry-run and --explain do not start the worker.

The feature explicitly does not provide JSONLOG, stderr, syslog, journald, cloud log APIs, other component logs, log shipping, OTLP, Kafka, VictoriaLogs, Loki, full-text search, a TUI, session timelines, arbitrary regular expressions, a reset endpoint, or automatic root-cause analysis. PostgreSQL logging settings remain ordinary SQL/deployment configuration; there are no pg_log_setting_* metrics.

Why CSV and why complete records

PostgreSQL 14 through 18 document the same 26 CSV columns. They include timestamp, user, database, process and session identity, per-session line number, severity, SQLSTATE, message fields, application, backend type, parallel leader PID, and query ID. PostgreSQL’s sample import schema uses (session_id, session_line_num) as a primary key.

CSV is not one record per physical line. Query, detail, hint, and context fields may contain commas, quotes, carriage returns, and embedded newlines. A bufio.Scanner, tail -F | regex, or strings.Split implementation will eventually split one logical record into several false records.

The collector uses a complete CSV framing state machine and the standard CSV decoder. An unfinished record at the active file’s end is retained as pending input; its offset is not committed until the complete logical record parses. A single record is bounded at 16 MiB. Malformed or oversized data enters a bounded resynchronization path and increments explicit parse and gap counters rather than silently advancing as if nothing happened.

The first release targets PostgreSQL 14+ because that gives one fixed 26-column schema. A repeated schema mismatch degrades the component instead of guessing a different format.

Polling, file identity, and rotation

The worker periodically reconciles the directory instead of relying on fsnotify. Polling is easier to make portable and auditable across rename rotation, missed notifications, restarts, and directories that already contain several generations.

Each cursor records a stable file identity, generation, hashed path identity, byte offset, size, a small content fingerprint, EOF/disappearance state, and resynchronization state. The durable file does not retain the clear log path.

Rename rotation is followed by identity: the old file can be completed after its pathname changes while the new file begins at its own cursor. If a file becomes smaller than the committed offset, the collector treats it as truncation, resets that generation to offset zero, and increments both rotation and input-gap metrics. Copy-truncate can overwrite unread bytes before any reader observes them, so the design reports the unavoidable gap rather than promising impossible exactly-once delivery.

Missing unfinished files and fingerprint identity changes are also explicit gaps. Completed disappeared cursors become evictable tombstones. The state tracks at most 256 file identities; it will not evict active work merely to stay under the limit.

On first startup without state, existing files are baselined at their current EOF. This avoids treating an arbitrary retained history as new counters. The baseline increments state_resets_total and an input-gap reason so operators can see that metrics begin at a new epoch. Deliberate historical backfill is outside the first contract.

The durable commit protocol

Counters and Histograms derived from log records must survive a PG Exporter restart. Advancing a cursor without its aggregates loses metrics; publishing aggregates without the matching cursor duplicates them after restart. The design commits them as one versioned state object.

One background cycle follows this order:

scan and read complete records
    -> clone previous state
    -> update cursors, counters, histograms, and self-state
    -> validate monotonicity, labels, limits, and metric families
    -> write a 0600 temporary state file
    -> fsync temporary file
    -> atomic rename
    -> fsync parent directory where supported
    -> publish the matching immutable metric snapshot

The state file is capped at 4 MiB and must be a regular file with mode 0600; symlinks and unsafe permissions are rejected. A non-blocking state lock prevents two processes from owning the same cursor. The directory identity stored in state must match the configured source.

The publication point matters. A crash before rename leaves the old state and old snapshot. A crash after rename but before the next scrape leaves the new durable state, which reconstructs the same families on restart. The request path reads only an atomic pointer; it never scans files, parses CSV, writes state, or calls fsync.

Each pass is bounded to 100,000 records or 64 MiB. If backlog remains, the worker immediately runs another pass instead of sleeping for the normal poll interval.

Metrics and cardinality

The core families are:

pg_log_records_total{severity}
pg_log_errors_total{severity,sqlstate_class}
pg_log_query_duration_seconds{kind}
pg_log_events_total{category,event}

The regular default surface also covers bounded exact SQLSTATE, checkpoint and restartpoint counts/durations/WAL activity, autovacuum outcomes and durations, lock waits, temporary-file counts and sizes, connections, and session durations.

Unlike the SQL Snapshot Histogram, log Histograms are durable cumulative observations. Their bucket, count, and sum state is persisted together with the cursor, so ordinary Prometheus counter-Histogram queries such as rate() apply until an explicit state reset creates a visible new epoch.

Labels are fixed enumerations. Query duration, for example, uses statement, execute, parse, bind, or other. Exact SQLSTATE is restricted to official codes, capped at 128 observed code series, with custom and unknown values folded into other. The complete log surface has a hard limit of 1,200 series.

Raw query, message, detail, hint, context, database, user, application, relation, query ID, client address, PID, session, transaction ID, and file path never enter labels or durable state. A record may contribute to several safe aggregates, but it can produce at most one primary classified event.

These decisions trade forensic detail for predictable monitoring. The exporter can alert that deadlocks or authentication failures increased; it cannot show the SQL text that caused them.

Self-observability and trust

Business metrics are insufficient without evidence that the reader is healthy. The collector adds:

pg_exporter_log_bytes_read_total
pg_exporter_log_parse_errors_total{reason}
pg_exporter_log_files_watched
pg_exporter_log_last_record_timestamp_seconds
pg_exporter_log_state_persist_timestamp_seconds
pg_exporter_log_rotations_total{type}
pg_exporter_log_input_gaps_total{reason}
pg_exporter_log_state_resets_total

It also participates in pg_exporter_component_*{component="postgres_log"}. Input, permission, parser, state, limit, or family-conflict failures mark only the log component down. The last committed business snapshot remains available when safe, while up, last-success time, and error counters show that it is stale or incomplete.

--disable-intro removes exporter/component/log-reader self-metrics but retains pg_log_* business metrics, matching the existing distinction between exporter introspection and collected domain data.

Logging policy changes metric meaning

The collector measures records PostgreSQL chose to emit. pg_log_query_duration_seconds is therefore a conditional distribution determined by log_duration, log_min_duration_statement, sampling, and protocol behavior. It must never be described as the distribution of every query unless PostgreSQL was configured to log every query duration.

Checkpoint, autovacuum, connection, disconnection, lock-wait, and temporary-file families are similarly present only when the corresponding server settings produce those messages. PG Exporter documents these prerequisites but does not duplicate settings as log-derived metrics.

PostgreSQL logs may contain sensitive statements and client data. The service account needs read access to the selected directory, but files should not be made world-readable. The collector stores only bounded aggregates and hashed source identity; it does not copy raw records to state.

Alternatives rejected

• Reading logs during /metrics was rejected because parse and persistence latency would enter the PostgreSQL scrape path.
• fsnotify was rejected as the correctness authority because notifications can be missed and do not replace reconciliation after restart.
• Processing all historical files on first start was rejected because retention policy would redefine counter origin unpredictably.
• At-least-once publication was rejected because duplicate alert counters are not an acceptable recovery strategy.
• Dynamic labels and user regular expressions were rejected because input data could control cardinality and durable schema.
• A new endpoint or binary was rejected because the feature is a bounded metric source, not a second product.

Acceptance and remaining gates

Source-level acceptance covers PostgreSQL 14, 16, and 18 fixtures; quoted commas, quotes, CRLF, multiline and partial records; rename rotation, truncation, restart restoration, state locks and permissions; crash points around state persistence; event-catalog fixtures; series budgets; family conflicts; disabled fast path; overlapping scrape behavior; shutdown; race tests; and package metadata.

Those tests establish an implementation candidate, not a public release. Before delivery, the exact merged commit must pass repository CI and package builds, then run under the final service user against a disposable PostgreSQL instance with real rotation and restart. Pigsty targets, dashboards, recording rules, alerts, package upgrades, and production canary results require separate evidence.

The Composite coordination contract is documented in One Endpoint, Several Sources. PostgreSQL remains the authority for the CSV schema and logging prerequisites: PostgreSQL 14 logging and PostgreSQL 18 logging.

2 - Caching pgBackRest Metrics Outside the Scrape Path

Decision status: Implemented in a development line; not present in main, a tag, or a published package as of 2026-08-28.
Decision date: 2026-08-23.
Applies to: the optional --pgbackrest cached component inside the Composite exporter.
Release boundary: source-level behavior has been implemented and tested; public release, packaging, deployment, and retirement of a standalone exporter remain unverified.

pgBackRest exposes rich backup state through pgbackrest info --output=json, but it exposes that state through a command, not a low-latency metrics endpoint. The command may inspect local configuration, contact remote repositories, wait on storage, emit large JSON, or fail for reasons unrelated to PostgreSQL SQL metrics.

Executing it during every Prometheus request would couple the availability and latency of the primary database scrape to backup infrastructure. The selected design therefore treats pgBackRest as a cached component:

background worker
    -> pgbackrest version
    -> pgbackrest info --output=json
    -> strict validation and metric construction
    -> atomic immutable snapshot

/metrics request
    -> load current snapshot
    -> merge below PostgreSQL, Patroni, and PgBouncer

The command path and scrape path never wait for one another.

Bounded command execution

The collector executes the configured binary directly, without a shell. The production defaults are:

The worker first detects the pgBackRest version, preferring numeric output and falling back to the older text form when necessary. It then runs info --output=json. Version and JSON shape are validated together because supported fields and known compatibility cases vary across pgBackRest releases.

Limits are part of the security and reliability contract. A corrupt repository, unexpected command, or hostile wrapper must not allocate unbounded memory or leave a child process running after overflow. The runner also uses a short wait delay after cancellation so process cleanup cannot stall indefinitely.

Validate before publication

A successful command is not yet a successful refresh. The JSON must satisfy the expected document shape, bounded object counts, numeric constraints, stanza and repository relationships, and metric-name and label rules. The resulting Prometheus families are normalized before publication.

Only a complete valid candidate replaces the current snapshot. Prometheus requests can then use a cheaper merge path: the cached families have already passed full client-library validation, so each database scrape checks ownership and headers without revalidating the entire backup payload.

The Composite ownership order remains:

PostgreSQL > Patroni > PgBouncer > pgBackRest > PostgreSQL Log

A pgBackRest family that conflicts with a higher-priority owner is omitted and marks the component down. It cannot append samples to the existing family and cannot make the PostgreSQL gather fatal.

Last-good semantics

After one successful refresh, a later command, timeout, parse, or limit failure keeps the previous business families. The component reports up=0, increments a bounded error reason, and preserves the timestamp of the last success.

This is not “pretend the backup source is healthy”. It is a deliberate split between two facts:

• the newest refresh failed;
• the last valid backup state is still useful if its age is visible.

Relative backup-age gauges continue to advance during a failure. The worker retains the last valid JSON and version and may rebuild only those time-relative values in the background. /metrics itself never reparses JSON or executes pgBackRest.

Before the first successful refresh there is no last-good business snapshot to serve. Health stays down and the endpoint still returns PostgreSQL plus any other valid components.

Health and overlapping scrapes

The cached component participates in the common health surface with component="pgbackrest". Its failure reasons include command execution and resource limit in addition to connection-independent parse and gather failures.

An overlapping Prometheus request does not queue behind a full Composite scrape. It reads the same atomic pgBackRest snapshot and includes it when its families do not conflict. Cached data therefore remains available during overlap without starting a second command or mutating component health from the request path.

Why not run on every scrape

Running the command on demand would provide superficially fresher data, but would produce several undesirable contracts:

• Prometheus timeout would become a backup-command timeout.
• Concurrent scrapes could execute concurrent repository inspections.
• Repository latency could hide PostgreSQL metrics.
• A scrape storm could amplify load on local and remote storage.
• Command output size and process cleanup would become HTTP-handler concerns.

A separate exporter process provides stronger process isolation, but retains another port, target, package, and health model. Composite mode offers a migration option without claiming that every environment must retire the standalone process immediately.

A universal command-plugin framework was also rejected. pgBackRest’s JSON, version compatibility, security, metrics, and last-good semantics are domain-specific. Treating it as arbitrary shell output would weaken validation while making an unstable plugin API part of the product.

Operational consequences

Enabling the component grants the PG Exporter service permission to execute pgBackRest and read its configuration. The official PG Exporter package does not need to bundle the pgBackRest binary; the executable and repository credentials remain owned by the host’s backup installation.

Operators must alert on component health and last-success age, not merely the continued presence of backup metrics. They must also validate the final package and service user against real repositories before replacing a standalone exporter. Source tests, a tagged release, a packaged binary, a Pigsty target change, and production parity are separate acceptance gates.

The surrounding coordinator and failure rules are documented in One Endpoint, Several Sources.

3 - The PostgreSQL Observability Product We Chose Not to Build

Decision status: Superseded on 2026-08-24 by the in-process PostgreSQL CSV log metrics decision.
Research date: 2026-08-13.
What remains valid: CSV framing, rotation, SQLSTATE, privacy, and bounded-cardinality findings.
What no longer applies: a new product, repository, binary, brand, interactive query tool, TUI, agent, or log-delivery pipeline.

Before PostgreSQL log metrics were scoped, we explored a much larger product: a local-first incident workbench that could inspect structured logs, reconstruct session timelines, offer interactive filters and a terminal UI, and eventually run as a node signal agent.

The exploration was useful precisely because the product was not built. It separated facts that belonged to any correct PostgreSQL log parser from ambitions that would have changed PG Exporter’s product category and operating model.

The original hypothesis

The user problem was credible. During an incident, a DBA often has local PostgreSQL logs but no prepared query in a central platform. A tool that understood PostgreSQL CSV records, SQLSTATE, sessions, rotation, and multi-line fields could produce a useful timeline without uploading SQL text or deploying a new backend.

The proposed workbench emphasized:

• correct PostgreSQL 14+ CSV parsing rather than line-oriented regular expressions;
• filters over time, severity, SQLSTATE, database, user, application, and session;
• cross-file context through rename rotation;
• a terminal-first workflow for machines without a web service;
• local privacy by default;
• low-cardinality Prometheus metrics derived from the same parser;
• a possible future daemon with durable cursors and multiple outputs.

This was a coherent product idea. It was also far larger than the immediate need.

What the research established

Several conclusions survived the pivot.

PostgreSQL 14 through 18 document the same 26-field CSV layout, including session_id, session_line_num, backend_type, leader PID, and query ID. PostgreSQL’s sample import table uses (session_id, session_line_num) as a primary key. CSV values may contain commas, quotes, and newlines, so a physical-line scanner is not a correct parser.

SQLSTATE is more stable than localized message text and is therefore the best first classifier for errors. Message grammars remain useful for bounded PostgreSQL events such as checkpoint, autovacuum, lock wait, temporary-file, and connection messages, but they must be versioned and tested.

Raw query text, detail, hint, context, user, database, application, client address, PID, session, transaction ID, and file path are unsafe default metric labels. Prometheus is a good destination for bounded counts and distributions, not a substitute for a log index.

Rotation, partial writes, restart recovery, and cursor persistence are product requirements, not parser implementation details. A tool that silently double-counts or skips bytes after restart produces worse evidence than no tool.

Why the larger product was rejected

The decisive problem was not technical feasibility. It was scope and responsibility.

A workbench would require an independent command vocabulary, output schema, UX, packaging, documentation, support surface, and release lifecycle. A durable agent would add service management, upgrades, queues, backpressure, retry, disk retention, security policy, and delivery SLOs. A TUI and log index would optimize for investigation, while a Prometheus exporter optimizes for bounded machine-readable state. Treating all of them as one MVP would delay the smallest useful outcome and make its reliability harder to prove.

The new product also lacked a validated reason to exist separately. The immediate request was not “search every log” or “send logs to another backend”. It was “derive a small set of reliable operational metrics from PostgreSQL’s structured log”. PG Exporter already owned the target identity, /metrics, packaging, component health, and Pigsty integration needed for that job.

Brand and repository work were therefore distractions. Naming a speculative product could make the architecture feel more committed than the user value. The design review chose to remove the entire public naming surface rather than polish it.

The replacement decision

On 2026-08-24 the goal became deliberately smaller:

Add one default-off background PostgreSQL CSV log collector to the existing PG Exporter process, and expose only bounded operational metrics on the existing /metrics endpoint.

The replacement excluded log shipping, OTLP, VictoriaLogs, Kafka, a web UI, TUI, full-text search, session timelines, arbitrary user regular expressions, automatic root-cause analysis, and other component logs.

It retained the hard parts that determine trust: complete CSV framing, file identity, rename and truncation handling, a durable cursor plus aggregates, atomic persistence, immutable snapshots, fixed labels, series limits, and explicit gaps.

The resulting engineering contract is documented in Turning PostgreSQL CSV Logs into Durable Metrics.

Why keep a superseded record

Deleting the exploration would hide why apparently attractive features are absent. Publishing the raw research would be equally misleading: it contained a large naming exercise, market snapshots, and product recommendations that were intentionally overturned.

This calibrated record preserves the useful causal chain:

credible incident workflow
    -> correct CSV and session research
    -> oversized workbench and agent proposal
    -> scope and ownership review
    -> bounded log metrics inside PG Exporter

Future maintainers should not reopen the larger product merely because interactive investigation sounds useful. Reconsider it only with evidence that users need a local query experience the existing log stack cannot provide, that more than one real consumer validates the event model, and that someone is prepared to own a separate product lifecycle.

Sources

The stable external facts retained here are grounded in PostgreSQL’s own CSV log definitions for PostgreSQL 14 and PostgreSQL 18. Product and branding snapshots from the original research are intentionally not reproduced.

4 - One Endpoint, Several Sources: The Composite Exporter Contract

Decision status: Implemented in a development line; not present in main, a tag, or a published package as of 2026-08-28.
Decision date: 2026-08-11; extended by cached components on 2026-08-23 and 2026-08-24.
Applies to: the optional Composite coordinator for PostgreSQL, PgBouncer, Patroni, pgBackRest, and PostgreSQL CSV log metrics.
Release boundary: design and implementation evidence exist; merge, tag, package, deployment, and production replacement remain separate gates.

PG Exporter began as a declarative SQL exporter for one PostgreSQL-compatible target. A Pigsty node, however, also exposes useful state through PgBouncer’s admin database, Patroni’s Prometheus endpoint, pgBackRest’s local command, and PostgreSQL’s structured logs. Running one exporter per source is operationally simple at first, but it creates several targets, ports, labels, health conventions, and lifecycle owners for what operators understand as one database instance.

The Composite design allows one PG Exporter process to expose those sources on the existing /metrics endpoint. Its purpose is not merely to concatenate metrics. Its purpose is to make the failure and ownership rules explicit enough that adding optional components can never weaken the original PostgreSQL contract.

The non-negotiable invariant

PostgreSQL remains mandatory and authoritative. Only the PostgreSQL collector’s gather error may become the fatal error returned by the composite gatherer. PgBouncer, Patroni, pgBackRest, or log-metric failures may remove their own data and set their own health, but they must not turn an otherwise valid PostgreSQL response into HTTP 500.

This rule addresses the most dangerous failure mode of a combined endpoint: a TLS error in an optional Patroni request, a slow backup repository, or a malformed log record must not hide the database metrics that operators need to diagnose the incident.

All optional components are disabled by default. When their flags and URLs are empty, PG Exporter does not create their collectors, issue requests, execute commands, scan files, register component health, or add a coordination lock to the old scrape path.

One coordinator, three execution models

The sources do not have the same latency or freshness contract, so they should not share one generic adapter:

Prometheus /metrics
        |
        v
Composite coordinator
  |-- PostgreSQL SQL       live, authoritative
  |-- PgBouncer SQL        live, optional
  |-- Patroni HTTP         live, optional
  |-- pgBackRest snapshot  cached, optional
  `-- PostgreSQL log       cached, optional

PostgreSQL keeps the established real-time SQL path. PgBouncer also runs SQL on each scrape because its queries are cheap and the existing exporter semantics are already useful. Patroni is fetched live because role, leader, DCS, and timeline state can change immediately.

pgBackRest and PostgreSQL logs use background workers and immutable snapshots. Executing a backup command or scanning log files inside a Prometheus request would make database metric availability depend on disk, repository, parser, and persistence latency. Their detailed contracts are recorded in Caching pgBackRest Metrics Outside the Scrape Path and Turning PostgreSQL CSV Logs into Durable Metrics.

Live work is bounded and concurrent

PgBouncer and Patroni run concurrently under one sidecar timeout, which defaults to 10 seconds. PostgreSQL runs in parallel but is not canceled by that optional deadline. This prevents a sidecar budget from becoming a new timeout for the authoritative query path.

The coordinator also refuses to queue overlapping Prometheus requests behind optional live work. If another full Composite scrape is already running, the overlapping request gathers PostgreSQL, process metrics, and the lock-free cached snapshots. It marks or omits live optional work instead of waiting for the earlier request.

This is a degradation policy, not an optimization detail. A scrape storm should reduce optional completeness before it increases latency or goroutine queues on the primary path.

Metric-family ownership

Combining registries requires a deterministic rule for identical family names. The selected order is:

PostgreSQL > Patroni > PgBouncer > pgBackRest > PostgreSQL Log

Process and HTTP self-metrics are merged behind PostgreSQL as well. A lower-priority source may add a family only when no higher-priority source owns that name or any reserved Histogram/Summary derivative. It may not append samples to an existing family merely because Help, type, and labels happen to match.

This whole-family rule avoids a subtle invalid state in which two components partially co-own one metric. It also blocks collisions such as a literal foo_count beside a higher-priority foo Histogram. Non-conflicting families from a partially rejected optional component remain useful; the conflict marks that component down with reason gather but does not alter the PostgreSQL result.

Patroni is parsed, not blindly proxied

Patroni already exposes Prometheus metrics, but byte concatenation would bypass validation and create an invalid response if it emitted conflicting metadata, OpenMetrics-only syntax, excessive data, or a duplicate family. The Composite collector therefore fetches, parses, validates, and re-encodes supported classic Prometheus semantics.

The design preserves Patroni family names, Help, types, labels, and sample values. It deliberately does not rename every metric or add a component label. Source identity belongs in the target labels and component-health surface, not in every business series.

Outbound HTTPS uses system roots plus an optional CA file. The design does not add insecure_skip_verify; a certificate failure is visible as a Patroni component failure instead of silently weakening transport verification.

Health is not one Boolean

Each enabled component has a bounded health surface:

pg_exporter_component_enabled{component="..."}
pg_exporter_component_up{component="..."}
pg_exporter_component_scrape_duration_seconds{component="..."}
pg_exporter_component_scrape_errors_total{component="...",reason="..."}
pg_exporter_component_last_success_timestamp_seconds{component="..."}

These signals answer different questions. enabled is configuration state. up is the latest component outcome. last_success shows staleness. Error counters preserve bounded reasons such as connect, timeout, TLS, parse, gather, command execution, state, or source failure.

They do not replace PostgreSQL’s existing /up, /health, /primary, or /replica semantics. Those routes continue to describe the PostgreSQL target. Turning them into “all components healthy” would break routing and failover users that never opted into a composite availability definition.

--disable-intro suppresses exporter and component self-metrics, not business metrics. This preserves the existing meaning of that flag.

Alternatives rejected

The design intentionally refused several broader abstractions:

• A generic plugin registry would make the first integration harder to audit and would expose lifecycle and precedence as public extension APIs before real components stabilized them.
• Sequential collection would let one optional source consume the full request budget before the next source ran.
• Byte-level Patroni passthrough would avoid a parse step but give up conflict, protocol, and resource validation.
• A component label on every family would change established metric contracts and multiply series without resolving name ownership.
• A global “all sources must succeed” policy would make optional integrations reduce availability.
• Replacing every standalone exporter and Pigsty target in the same change would mix source correctness with deployment migration and remove rollback options.

The selected design is explicit rather than universal: named components, a fixed ownership order, separate live and cached paths, and PostgreSQL-first failure semantics.

Consequences and release gates

Composite mode can reduce target and process count while preserving existing namespaces. In return, one process now owns more credentials, network clients, parsers, background workers, and health states. Operators must grant only the permissions needed by enabled components and must treat family conflicts as configuration or compatibility defects.

The implementation existing in a development tree is not evidence that a stable release, Pigsty target migration, dashboard update, recording rule, alert, or old-process retirement has happened. Those gates must be verified independently when the feature reaches a public branch and release.

5 - Snapshot Histograms Are Gauges, Not Counters

Decision status: Shipped in v1.4.0.
Decision date: 2026-07-11; amended on 2026-07-17 before release.
Applies to: HISTOGRAM columns and the bundled pg_xact_age collector in PG Exporter v1.4.0 and later.
Release boundary: this article describes a released source and metric contract; dashboards and recording rules remain consumer responsibilities.

PG Exporter originally mapped each SQL result cell to a scalar Gauge or Counter. That model could answer questions such as “how old is the oldest transaction?”, but it could not preserve the shape of the current population. A database with one 30-minute transaction and a database with one hundred 18-second transactions could produce the same maximum while requiring very different action.

The first Histogram design added a distribution without turning PG Exporter into a stateful event processor. Its defining decision is simple:

A PG Exporter Histogram is a distribution rebuilt from the rows returned by one real SQL query execution. It describes the current population, not observations accumulated since process start.

That sentence determines the exposition type, cache behavior, failure semantics, PromQL, and bucket policy.

The temporal contract

The reference population is open transactions. A transaction appears when it starts, moves to older buckets as time passes, and disappears when it commits or rolls back. The number of observations and every cumulative bucket may rise or fall between scrapes.

Ordinary instrumented Prometheus histograms are cumulative counters: their bucket, count, and normally their sum series increase over time. Prometheus therefore teaches users to apply rate() before computing request rates or time-window distributions. That assumption is wrong for a fresh SQL snapshot.

PG Exporter consequently emits the logical Histogram as ordinary Gauge series:

pg_xact_age_seconds_bucket{datname="app",le="10"} 5
pg_xact_age_seconds_bucket{datname="app",le="30"} 9
pg_xact_age_seconds_bucket{datname="app",le="+Inf"} 11
pg_xact_age_seconds_count{datname="app"} 11
pg_xact_age_seconds_sum{datname="app"} 214

The familiar names and cumulative le buckets preserve compatibility with histogram_quantile(). The Gauge type preserves the truth that all values may decrease. This is a deliberate semantic compromise: the output looks like a classic histogram family but is not a counter histogram.

Never apply rate(), irate(), increase(), or counter-reset logic to these series. Query the current distribution directly:

histogram_quantile(
  0.95,
  sum by (datname, le) (pg_xact_age_seconds_bucket)
)

The current mean is equally direct:

sum by (datname) (pg_xact_age_seconds_sum)
/
sum by (datname) (pg_xact_age_seconds_count)

Configuration and SQL contract

The design adds one user-facing usage, HISTOGRAM:

- seconds:
    usage: HISTOGRAM
    bucket: [1, 3, 10, 30, 100, 300, 1000, 3000]
    description: Open transaction age snapshot in seconds

Each non-NULL value in the configured column is one observation. Rows with the same complete label tuple belong to the same distribution. Multiple Histogram columns in one query aggregate independently.

Buckets are inclusive finite upper bounds. Configuration loading rejects an empty list, duplicates, non-increasing order, NaN, and infinities. PG Exporter appends +Inf; users must not configure it. The generated le label and the derived _bucket, _count, and _sum names are reserved, so collisions fail before scraping.

scale is applied before bucket assignment and summation. Timestamp and Boolean values follow the existing scalar conversion path and are exempt from scaling. An explicit default converts NULL into an observation; otherwise NULL is ignored.

Query, cache, and atomicity

A real SQL execution starts with empty accumulators. PG Exporter groups observations, assigns finite buckets, produces cumulative counts, and materializes the result only after the complete result set is valid.

If the query fails, a required column is missing, or any observation cannot be converted to a finite number, no scalar or Histogram family from that execution is published. The failure is atomic at the collector-query boundary.

A cache hit reuses the previous immutable result. It does not add the same observations again. A subsequent real query creates a new snapshot from zero; no Histogram state survives between executions. Reloading configuration discards the old collector and its cached snapshot, including an old bucket layout.

These rules keep Histogram semantics aligned with the declarative collector model: SQL remains the source of truth, TTL controls query execution, and the exporter does not invent a second time domain.

Why explicit Gauge series won

Several apparently simpler alternatives were rejected:

• prometheus.NewConstHistogram creates counter-like Histogram metadata. It can encode the numbers but misstates their temporal behavior.
• A formal OpenMetrics GaugeHistogram would express the semantics more precisely, but PG Exporter served the classic Prometheus text format and did not require an OpenMetrics-only mode for one collector type.
• Native Histograms solve different storage and resolution problems. They do not turn a changing SQL population into a cumulative event stream.
• Pre-aggregating buckets in SQL would duplicate grouping logic in every collector and create two configuration contracts.
• Retaining observations across scrapes would change “current database state” into “events seen by this exporter process”, with restart, persistence, and double-counting obligations.

The selected design is narrower: raw SQL observations in, one current distribution out.

Reference collector and bucket evolution

The first implementation was refined before release around pg_xact_age, a per-database distribution of open transaction age and idle-in-transaction age. The final collector filters to client backends and uses logarithmic-style bucket grids suitable for instant operational questions: current quantiles, population above a threshold, and current mean.

The following pgbench workload was used to create several query and hold-time bands for live acceptance. It is reproduced here because the original design directory is no longer the documentation authority:

\set query_band random(1, 4)
\if :query_band = 1
  \set query_ms 300
\elif :query_band = 2
  \set query_ms 3000
\elif :query_band = 3
  \set query_ms 10000
\else
  \set query_ms 30000
\endif

\set hold_band random(1, 4)
\if :hold_band = 1
  \set hold_ms 300
\elif :hold_band = 2
  \set hold_ms 3000
\elif :hold_band = 3
  \set hold_ms 10000
\else
  \set hold_ms 30000
\endif

BEGIN;
SELECT pg_current_xact_id(), pg_sleep(:query_ms / 1000.0);
\sleep :hold_ms ms
COMMIT;

The important evidence was not a particular benchmark number. It was that bucket populations moved and disappeared as transactions advanced, cache hits did not accumulate them, and reload installed a new layout without retaining old samples.

Consequences

The design gives users aggregatable current distributions without a stateful exporter. In return, users must understand that the _bucket, _count, and _sum suffixes do not imply counter temporality here. Documentation, dashboards, and alerts must state that distinction explicitly.

The complete collector syntax is maintained in Collector Configuration, and the shipped implementation is summarized in the v1.4.0 release note. Prometheus’ documentation remains the authority for ordinary counter histograms and explains why their usual rate()-based queries depend on monotonic counts: Histograms and summaries.