Skip to content

Troubleshooting

A symptom-driven runbook for startup, connectivity, missing metrics, slow scrapes, reloads, and packaging

Check the process, HTTP access, database connection, then the failing collector. pg_up 1 means the last scrape reached the main target and completed its fatal collectors. Other collectors or discovered databases can still have errors.

Initial Checks

# Process and logs
systemctl status pg_exporter --no-pager
journalctl -u pg_exporter -n 100 --no-pager

# HTTP response and version
curl -fsS http://127.0.0.1:9630/version
curl -i http://127.0.0.1:9630/up

# Core metrics
curl -fsS http://127.0.0.1:9630/metrics \
  | grep -E '^(pg|pgbouncer)_(up|version|in_recovery)(\{| )'

# Collection plan and runtime statistics
curl -fsS http://127.0.0.1:9630/explain
curl -fsS http://127.0.0.1:9630/stat
Observation Layer to investigate
Unit exits immediately Arguments, config path, telemetry path, listen address, file permissions
Port answers but /up is 503 Target URL, network, authentication, pg_hba.conf, TLS, startup probe state
/up is 200 but a metric family is absent Dynamic plan, version/role/tags/predicates, auto-discovery, disabled collector
/metrics returns an error or Prometheus times out Slow query, scrape timeout, duplicate or invalid metric samples
Exporter is healthy but Prometheus target is down Prometheus address, scheme, auth, TLS, firewall, telemetry path

Process Will Not Start

no valid config path

The config search order is --config, PG_EXPORTER_CONFIG, ./pg_exporter.yml, /etc/pg_exporter.yml, then /etc/pg_exporter. Confirm the file exists and is readable by the runtime user:

sudo -u prometheus test -r /etc/pg_exporter.yml
sudo -u prometheus /usr/bin/pg_exporter \
  --config=/etc/pg_exporter.yml \
  --dry-run >/dev/null

--dry-run parses and explains the raw configuration without needing a live database. A config directory is non-recursive and loads only .yaml / .yml files in alphabetic order; a duplicate top-level branch name from a later file overrides the earlier definition.

Invalid telemetry path

Since v1.4.0, --web.telemetry-path must be a canonical literal path beginning with /. It cannot contain a query, fragment, Go ServeMux wildcard, or collide with built-in endpoints such as /up, /reload, or /version.

Use /metrics unless there is a concrete need to change it, and update the Prometheus metrics_path at the same time.

Address already in use

lsof -nP -iTCP:9630 -sTCP:LISTEN

Stop the conflicting service or choose another --web.listen-address.

pg_up Is 0 or /up Returns 503

For the packaged service, add --log.level=debug to PG_EXPORTER_OPTS in /etc/default/pg_exporter, then restart the service. This preserves its user and connection settings:

sudo systemctl restart pg_exporter
sudo journalctl -u pg_exporter -f

For a foreground process, use the real target URL and supply the password through .pgpass or a secret file:

PG_EXPORTER_URL='postgres://[email protected]:5432/postgres?sslmode=verify-full&sslrootcert=/etc/pg_exporter/ca.crt' \
pg_exporter --config=/etc/pg_exporter.yml --log.level=debug

Check, in order:

  1. DNS and TCP reachability to the advertised host and port.
  2. Database name and login role.
  3. pg_hba.conf source address, authentication method, and reload state.
  4. Password source: URL, .pgpass, or PG_EXPORTER_URL_FILE.
  5. .pgpass mode 0600, hostname match, and the actual runtime user’s HOME.
  6. TLS CA path, hostname, and sslmode.
  7. Connection limit on the monitoring role and server-wide connection exhaustion.
  8. --connect-timeout: the default is 100 ms for connection and metadata probes. Increase it if normal network latency exceeds this budget.

After diagnosis, restore --log.level=info and restart the service.

The packaged service runs as prometheus; testing as your shell user can therefore succeed while the service fails.

Non-blocking startup is normal: while the target is unavailable, the HTTP server remains up and the background probe retries. Use --fail-fast only when the orchestrator should treat initial database unreachability as a process startup failure.

Metrics Are Missing

Scrape /metrics once to build the plan. /explain shows collectors selected for the main target. For a missing collector, compare its YAML with the target version, role, and tags; debug logs report why a branch was rejected. Check for:

  • server version outside [min_version, max_version);
  • primary/replica role mismatch;
  • missing extension:, schema:, dbname:, or username: fact;
  • missing custom tag or matching not: tag;
  • predicate query returning false;
  • skip: true;
  • database excluded from auto-discovery.

The include/exclude lists apply to additional discovered databases. The database in the connection URL is always monitored, even if its name appears in the default exclusion template0,template1,postgres. Predicates can skip a selected collector, and a successful query can return no rows; check /stat and the SQL result as well.

One Collector Fails

Use /stat for the main target. Use exporter self-metrics, labeled by datname and query, for all monitored databases:

pg_exporter_query_scrape_error_count > 0

This value counts errors since the last plan was built; a nonzero value can describe an earlier failure. Scrape again and check whether it increases. Read the log for the error text.

Common causes include:

  • monitoring role lacks access to an optional view or function;
  • extension/schema exists in a different database;
  • custom SQL result columns do not match the YAML metrics list;
  • a configured LABEL column is absent (the exporter emits no metrics from that collector for this call);
  • the query exceeds its timeout;
  • an extension or pre-release PostgreSQL view changed shape.

Run the collector SQL manually as the monitoring user in the same database. Preserve the result column names exactly, including zero-row result schemas.

Scrapes Are Slow or Time Out

/stat reports query durations in milliseconds for the main target. pg_exporter_query_scrape_duration reports seconds for each datname and query. Error counts are available in both; error text is in the logs.

Typical remedies:

  1. Optimize or narrow the SQL.
  2. Increase ttl so expensive results are reused across scrapes.
  3. Set a realistic per-query timeout below the Prometheus scrape timeout.
  4. Disable optional high-cost collectors with skip: true.
  5. Reduce database auto-discovery scope with include-database / exclude-database.
  6. Control per-object metric cardinality on databases with thousands of tables or indexes.

Queries and discovered databases are scraped sequentially, so budget for their combined duration. Keep the full scrape below Prometheus scrape_timeout and scrape_interval. The HTTP server also has a 30-second write timeout.

Reload Fails

curl -i -X POST http://127.0.0.1:9630/reload
  • 200: the new configuration passed validation and was loaded; the next scrape will rebuild the collection plans.
  • 500: the response contains the parse, schema, label-conflict, or config-path error. The old active query set remains in place.
  • 405: use GET or POST; POST is recommended.

Restart the service after changing the target URL, listen address, logging, namespace, discovery flags, or the path passed to --web.config.file. Exporter-toolkit reads the contents of that web configuration file dynamically; certificate or Basic Auth changes at the same path do not use /reload.

Collector YAML can also be reloaded with SIGHUP; Unix builds additionally accept SIGUSR1. A reload rebuilds plans on the next scrape and resets per-query statistics.

Docker and Package Pitfalls

Symptom Likely cause and fix
TLS verification fails in Docker scratch image has no CA bundle; mount the CA and set sslrootcert
.pgpass is ignored in packaged service Check the prometheus account home or PGPASSFILE; give the service user ownership and use mode 0600
RPM automation cannot find v1.4.1 artifact Artifact prefix changed from pg_exporter to pg-exporter
Config edits disappear after container replacement Mount /etc/pg_exporter.yml from durable configuration
Container cannot reach host PostgreSQL localhost means the container; use the database service DNS name or an explicit host route

Evidence to Include in a Bug Report

  • pg_exporter --version or curl http://127.0.0.1:9630/version output, and installation method
  • Operating system and architecture
  • PostgreSQL or PgBouncer exact version
  • Redacted target URL (retain scheme, host class, port, database, and sslmode)
  • Relevant collector YAML and SQL, with secrets removed
  • Debug log around the failure
  • /explain and /stat entries for the collector
  • Whether the problem reproduces with the bundled v1.5.0 config

Remove passwords, private keys, customer identifiers, and sensitive SQL text, then submit the evidence to pgsty/pg_exporter Issues.