Probe diagnostics v10.6

Probe diagnostics gives you a fleet-wide view of probe health: which probes are failing, degraded, or stale — including probes left stale by an unreachable agent — and how long each one takes to run. It has two pages: a probe-health list across every target, and a per-probe detail page with charts and a per-target breakdown.

To open it, select Management > Probes > Probe Diagnostics.

Prerequisites

• You need the pem_manage_probe privilege to see the tab at all. Without it, PEM shows an access-denied message and the underlying data endpoints also refuse the request.
• Probe telemetry must be enabled. It's controlled by the probe_telemetry_enabled configuration parameter, which ships on by default. If an administrator turns it off, Probe Diagnostics shows no data until it's turned back on.
• An agent must support probe telemetry to appear in the diagnostics data. Unsupported agents are listed by name in the dashboard itself, so you don't need to check versions manually.

Probe health

The list page shows one row per probe, aggregated across every target it runs on:

Filters

A fixed set of target-type buttons (Agent, Tool, Server, Database, Schema, Extension) is always shown on the toolbar, even when no probe of that type falls in the current window. Use Columns to choose which columns are visible, and Refresh to reload the list. Column definitions opens a short legend explaining Target Status, Targets, Success % / Runs, and Execution Time.

Target status

Each target a probe runs on is in exactly one of five health states, checked in this order — the first match wins:

Status reflects the selected Prior to window, not necessarily right now — a historical window shows each target's state as of that point in time, so viewing the past won't show every target as stale.

Select Target status legend on either page to see this same reference without leaving the dashboard.

Probe detail

Select a probe from the list to open its detail page. It has eight KPI tiles, all scoped to the selected time window and target: Target status, Success %, Runs, Failures, average execution time, 95th percentile execution time, maximum execution time, and Metric writes.

Charts

The window is split into roughly 100 time buckets for these charts, so the bucket size adjusts automatically to how wide a window you select.

Per-target breakdown

Use Select target to scope the whole page to one target, or leave it on All targets. The breakdown table lists every target the probe runs on:

Use Columns to show or hide the optional columns.

Targets that are dropped or unregistered fall off this list — that's expected, not data loss. The list always reflects what PEM currently monitors; history for a target that no longer exists stops contributing to the probe's aggregate numbers on the list page too.

Configuring a probe for one target

Select Configure probe for this target to open the inline configuration form for a specific target, without leaving the diagnostics page.

Two things affect whether you can use this form:

• Without the pem_config_probe privilege, the form is read-only.
• If a profile manages the target's configuration, the read-only form notes that profile changes affect every assigned target, with a link to the profile. If you're missing pem_config_probe, that note isn't shown — no-privilege takes precedence over profile-managed.

The Manage Profile link itself is hidden unless you also hold pem_manage_profile.

Privileges and access control

Holding pem_manage_probe doesn't automatically show every target in your fleet. Each target is additionally checked against your team's visibility on the underlying agent and server — a target whose agent or server your team can't see doesn't appear in Probe Diagnostics, even though you can otherwise use the feature.

Data source and retention

Probe Diagnostics reads from pemhistory.probe_telemetry, populated by agents that support probe telemetry. History retention isn't a single fleet-wide setting — it's configured per probe, the same History retention (days) setting described above, defaulting to 30 days and adjustable from 1 to 365.

Troubleshooting