Webhooks v10.4

You must configure the PEM Server to use webhooks to receive notification of alert events on threshold value violations in your configured applications.

Creating a webhook

PEM monitors your system for conditions that require user attention. You can use a webhook to create the endpoints to receive a notification if current values deviate from threshold values specified in an alert definition. Based on the events triggered, PEM sends a notification to multiple webhook endpoints or to specific target webhook endpoints.

Use the Webhooks tab to configure endpoint recipients. To open the Webhooks tab, select Management > Alerts > Manage Alerts. From the Manage Alerts tab, on the Quick Links toolbar, select Webhooks.

The Webhooks tab displays a list of the currently defined recipient applications as endpoints. Select an endpoint and select Edit at the far left end of the row to modify an existing endpoint.

To define a new webhook, select the plus sign (+) in the upper-right corner of the table. You can then use the General tab to define the basic details of the webhook:

• Provide a name for the webhook in the Name field.
• Specify a webhook URL to deliver all the notifications to in the URL field.
• Set the request method type used to make the call in the Request Method field: POST or PUT.
• By default, webhooks are enabled. To disable a webhook, set Enable? to No.

Defining webhook SSL configurations

You can define the webhook SSL parameters in the respective agent configuration file or registry in Windows. You can find the list of webhook SSL parameters in PEM agent configuration parameters. If you add or remove any of the agent configuration parameters, you must restart the agent to apply them.

• On Windows systems, PEM registry entries for webhooks are located in HKEY_LOCAL_MACHINE\Software\EnterpriseDB\PEM\agent\WEBHOOK.
• On Linux systems, PEM configuration options for webhooks are stored in the agent.cfg file, located by default in /usr/edb/pem/agent/etc.

Defining webhook headers

Use the HTTP Headers tab to define the header parameters to pass while calling the webhook endpoints:

• Specify all the values as a key and value pair.
• Specify a key parameter in the Key field and a value in the Value field.
• To add HTTP headers, select the plus sign (+) in the upper-right corner of the HTTP Headers table.
• To delete HTTP headers, select Delete to the left of Key. The header remains in the list but in strike-through font. Select Save to permanently delete the headers.
• To edit the HTTP headers, select Edit to the left of Key.

Defining webhook payloads

Use the Payload tab to define the JSON data to send to the endpoint when an alert is triggered:

• Type specifies data to send in the format type, that is, JSON.

• Use Template to configure JSON data sent to endpoints. In the template, you can use placeholders for the following:

  • %AlertID% — The id of the triggered alert.
  • %AlertName% — The name of the triggered alert.
  • %ObjectName% — The name of the server or agent on which the alert was triggered.
  • %ObjectType% — The type on which the alert was generated.
  • %ThresholdValue% — The threshold value reached by the metric when the alert triggered.
  • %CurrentValue% — The current value of the metric that triggered the alert.
  • %CurrentState% — The current state of the alert.
  • %OldState% — The previous state of the alert.
  • %AlertRaisedTime% — The time that the alert was raised or the most recent time that the alert state changed.
  • %AgentID% — The id of the agent by which the alert was generated.
  • %AgentName% — The name of the agent by which the alert was generated.
  • %ServerID% — The id of the server on which the alert was generated.
  • %ServerName% — The name of the server on which the alert was generated.
  • %ServerIP% — The ip or address of the server on which the alert was generated.
  • %ServerPort% — The the port of the server on which the alert was generated.
  • %DatabaseName% — The name of the database on which the alert was generated.
  • %SchemaName% — The name of the schema on which the alert was generated.
  • %PackageName% — The name of the package on which the alert was generated.
  • %DatabaseObjectName% — The name of the database object, like table name or function name, on which the alert was generated.
  • %Parameters% — The list of custom parameters used to generate the alert.
  • %AlertInfo% — The detailed database object-level information of the alert.

• Select Test Connection to test notification delivery to the mentioned endpoint.

Defining webhook alert levels

Use the Notifications tab to specify an alert level for webhook endpoints:

• Set All alerts to Yes to enable all alert levels to send notifications.
• To send a notification when a specific alert level is reached, set the slider next to an alert level to Yes. You must set All alerts to No to configure an individual alert level.

Example: sending notifications to slack

In Slack, follow the instructions in Getting started with incoming webhooks to:

• Create a Slack app.
• Activate incoming webhooks for that app.
• Add a webhook that posts to a channel or user of your choice.

The new webhook has a unique URL similar to https://hooks.slack.com/services/x/y/z. You can now configure PEM to send notifications to this URL.

In PEM, create a new webhook, give it a descriptive name, and copy the URL you obtained earlier to the URL field. Ensure that Request method is set to POST and Enable? is set to Yes. Set all the sliders under Alert Notifications to Yes.

Add a header under HTTP headers with the key Content-Type and the value application/json.

Under Payload, delete the default template and specify a template with text as the top-level key as in the following example:

{"text": "%AlertName% on %ObjectType% %ObjectName% is now %CurrentState%"}

You can now test the connection. If it succeeds, PEM issues a notification, and the template you specified appears in your Slack channel as a message.

Save the webhook and continue using PEM as usual. PEM now sends all the alerts to your Slack channel.

Deleting a webhook

To mark a webhook for deletion, in the Webhooks table, select the webhook name and select Delete to the left of the name. The alert remains in the list but in strikethrough font.

Delete is a toggle. You can undo the deletion by selecting Delete a second time. Select Save to permanently delete the webhook definition.

Authenticating webhook requests using OAuth

When OAuth is configured for a webhook endpoint:

1. Before sending a notification, PEM requests an access token from your OAuth provider
2. The token is automatically cached and reused until it expires
3. The access token is included as a Bearer token in the request
4. If authentication fails, PEM automatically refreshes the token and retries

Prerequisites

Before configuring OAuth for a webhook, you need:

1. An existing webhook endpoint in PEM
2. OAuth client credentials from your identity provider:
   • Client ID
   • Client Secret
   • Token URL (authorization server endpoint)
3. Optional: Scopes and Audience values (depending on your provider)
4. Access to a PEM host to run the pemEncryptor utility
5. Access to the PEM backend database as a user with pem_admin role, or a superuser.

Encrypting credentials

The client_id and client_secret must be encrypted before storing in the database. Use the pemEncryptor utility located at /usr/edb/pem/bin/ on the PEM server.

Step 1: Encrypt the Client ID

export PEM_ENC_PAYLOAD='your-client-id'
/usr/edb/pem/bin/pemEncryptor

Store the encrypted output for use in the next section.

Step 2: Encrypt the Client Secret

export PEM_ENC_PAYLOAD='your-client-secret'
/usr/edb/pem/bin/pemEncryptor

Store the encrypted output for use in the next section.

Adding OAuth configuration

Use SQL to configure OAuth for an existing webhook. Replace <encrypted_client_id> and <encrypted_client_secret> with the values obtained from pemEncryptor:

INSERT INTO pem.webhook_oauth_config (
    webhook_id,
    token_url,
    client_id,
    client_secret,
    scope,
    audience,
    auth_method
)
VALUES (
    <webhook_id>,
    '<token_url>',
    '<encrypted_client_id>',
    '<encrypted_client_secret>',
    '<scopes>',                  -- Optional, can be NULL
    '<audience>',                -- Optional, can be NULL
    'client_secret_basic'        -- Optional, defaults to 'client_secret_basic'
);

Configuration parameters

Authentication methods (RFC 6749)

PEM supports two standard client authentication methods defined in RFC 6749 Section 2.3:

Updating configuration

-- Update scopes
UPDATE pem.webhook_oauth_config
SET scope = 'api:read api:write',
    modified_on = CURRENT_TIMESTAMP
WHERE webhook_id = 1;

-- Update token URL
UPDATE pem.webhook_oauth_config
SET token_url = 'https://new-auth.example.com/oauth/token',
    modified_on = CURRENT_TIMESTAMP
WHERE webhook_id = 1;

Removing OAuth configuration

To disable OAuth authentication for a webhook:

DELETE FROM pem.webhook_oauth_config WHERE webhook_id = 1;