Managing a PEM agent v8

Managing job notifications

In the PEM console, you can configure the settings for sending the SMTP trap on success or failure of a system-generated job listed under scheduled tasks or a custom agent job. You can configure these email notification settings at the following three levels to send email notifications to the specified user group. These levels are shown in order of precedence.

  • Job level
  • Agent level
  • PEM server level (default)

Configuring job notifications at job level

You can configure email notification settings at job level only for a custom agent job in one of the following ways:

  • For a new agent job, you can configure the email notification settings in the Notification tab of the Create Agent Job wizard while creating the job.
  • For an existing custom job, you can edit the properties of the job and configure the notification settings.

Use the Notifications tab to configure the email notification settings on job level:

  • Use the Send the notifications field to specify when you want to send the email notifications.
  • Use the Email group field to specify the email group to send the email notification to.

Configuring job notifications at agent level

Select the agent in the tree view, right-click, and select Properties. In the Properties dialog box, select the Job Notifications tab.

Use the Job notifications tab to configure the email notification settings at agent level:

  • Use the Override default configuration? switch to specify if you want the agent level job notification settings to override the default job notification settings. Select Yes to enable the rest of the settings on this dialog box to define when and to whom to send the job notifications.
  • Use the Email on job completion? switch to specify whether to send the job notification when the job completes successfully.
  • Use the Email on a job failure? switch to specify whether to send the job notification when the job fails.
  • Use the Email group field to specify the email group to send the job notification to.

Configuring job notifications at server level

You can use the Server Configuration dialog box to provide information about your email notification configuration at PEM server level. To open the Server Configuration dialog box, select Management > Server Configuration.

Four server configuration parameters specify information about your job notification preferences at PEM server level:

  • Use the job_failure_notification switch to specify if you want to send an email notification after each job failure.
  • Use the job_notification_email_group parameter to specify the email group to send the email notification to.
  • Use the job_retention_time parameter to specify the number of days to retain nonrecurring scheduled tasks in the system.
  • Use the job_status_change_notification switch to specify if you want to send an email notification after each job status change, that is, failure, success, or interrupted.

Managing PEM scheduled jobs

You can create a PEM scheduled job to perform a set of steps you define in the specified sequence. These steps can contain SQL code or a batch/shell script that you can run on a server that's bound with the agent. You can schedule these jobs to suit your business requirements. For example, you can create a job for taking a backup of a particular database server and schedule it to run on a specific date and time of every month.

To create or manage a PEM scheduled job, use the PEM tree to browse to the PEM agent for which you want to create the job. The tree displays a Jobs node, under which currently defined jobs are listed. To add a job, right-click the Jobs node and select Create Job from the context menu.

Use the tabs on the Create - Agent Job dialog box to define the steps and schedule that make up a PEM scheduled job.

Use the General tab to provide general information about a job:

  • Provide a name for the job in the Name field.
  • Set the Enabled switch to Yes to enable a job. Set it to No to disable a job.
  • Use the Comment field to store notes about the job.

Use the Steps tab to define and manage the steps that the job performs. Select Add (+) to add a step. Then, select the compose icon, located at the left side of the header, to open the Step Definition dialog box.

Use the Step Definition dialog box to define the step:

  • Provide a name for the step in the Name field. Steps are performed in alphanumeric order by name.

  • Use the Enabled switch to include the step when executing the job (True) or to disable the step (False).

  • Use the Kind switch to indicate if the job step invokes SQL code (SQL) or a batch script (Batch).

    • If you select SQL, use the Code tab to provide SQL code for the step.
    • If you select Batch, use the Code tab to provide the batch script to execute during the step.
  • Use the On error list to specify the behavior of pgAgent if it encounters an error while executing the step. Select from:

    • Fail Stop the job if you encounter an error while processing this step.
    • Success Mark the step as completing successfully and continue.
    • Ignore Ignore the error and continue.
  • If you selected SQL as your input for the Kind switch:

    • Use the Server field to specify the server that's bound with the agent for which you are creating the PEM scheduled job.
    • Use the Database field to specify the database that's associated with the server that you selected.
  • Use the Comment field to provide a comment about the step.

  • Use the context-sensitive field on the Step Definition dialog box Code tab to provide the SQL code or batch script to execute during the step:

    • If the step invokes SQL code, provide one or more SQL statements in the SQL query field.

    • If the step invokes a batch script, provide the script in the Code field. If you're running on a Windows server, use standard batch file syntax. On a Linux server, you can use any shell script, provided that you specify a suitable interpreter on the first line (such as #!/bin/sh). Along with the defined inline code, you can also provide the path of any batch script, shell script, or SQL file on the filesystem.

      • To invoke a script on a Linux system, you must modify the entry for the batch_script_user parameter in the agent.cfg file and specify the user who runs the script. You can specify either a nonroot user or root for this parameter. If you don't specify a user or the specified user doesn't exist, then the script doesn't execute. Restart the agent after modifying the file.

      • To invoke a script on a Windows system, set the registry entry for AllowBatchJobSteps to true and restart the PEM agent. PEM registry entries are located in HKEY_LOCAL_MACHINE\Software\EnterpriseDB\PEM\agent.

After providing all the information required by the step, select Save.

Select Add (+) to add each step, or select the Schedules tab to define the job schedule.

On the Schedules tab, select Add (+) to add a schedule for the job. Then select the compose icon located at the left side of the header to open the Schedule Definition dialog box.

Use the Schedules definition tab to specify the days and times for the job to execute.

  • Provide a name for the schedule in the Name field.
  • Use the Enabled switch to indicate for pgAgent to use the schedule (Yes) or to disable the schedule (No).
  • Use the calendar selector in the Start field to specify the starting date and time for the schedule.
  • Use the calendar selector in the End field to specify the ending date and time for the schedule.
  • Use the Comment field to provide a comment about the schedule.

Select the Repeat tab to define the days when the schedule executes in a cron-style format. The job executes on each date or time element selected on the Repeat tab.

In each field, select a value to add it to the list of selected values for the field. To clear the values from a field, select the X located at the right-side of the field.

  • Use the fields in the Days box to specify the days when the job executes:
    • Use the Week Days field to select the days when the job executes.
    • Use the Month Days field to select the numeric days when the job executes. Select Last Day to perform the job on the last day of the month, regardless of the date.
    • Use the Months field to select the months when the job executes.
  • Use the fields in the Times box to specify the times when the job executes in hours and minutes.

Select the Exceptions tab to specify any days when you don't want the schedule to execute. For example, you might not want to run jobs on holidays.

Select Add (+) to add a row to the exception table. Then:

  • In the Date column, open a calendar selector and select a date when you don't want the job to execute. Specify **<Any>** to indicate that you don't want the job to execute on any day at the time selected.
  • In the Time column, open a time selector and specify a time when you don't want the job to execute. Specify **<Any>** to indicate that you don't want the job to execute at any time on the day selected.

Select the Notifications tab to configure the email notification settings on job level:

  • Use the Send the notifications field to specify when you want to send the email notifications.
  • Use the Email group field to specify the email group to send the email notification to.

When you finish defining the schedule, you can use the SQL tab to review the code that creates or modifies your job.

Select Save to save the job definition.

After you save a job, the job is listed under the Jobs node of the PEM tree of the server on which it was defined. The Properties tab in the PEM console displays a high-level overview of the selected job, and the Statistics tab shows the details of each run of the job. To modify an existing job or to review detailed information about a job, right-click a job name, and select Properties from the context menu.

Agent privileges

By default, the PEM agent is installed with root privileges for the operating system host and superuser privileges for the database server. These privileges allow the PEM agent to invoke unrestricted probes on the monitored host and database server about system usage, retrieving and returning the information to the PEM server.

PEM functionality lessens as the privileges of the PEM agent decrease. For complete functionality, run the PEM agent as root. If the PEM agent is run under the database server's service account, PEM probes don't have complete access to the statistical information used to generate reports, and functionality is limited to the capabilities of that account. If the PEM agent is run under another lesser-privileged account, functionality is limited even further.

If you limit the operating system privileges of the PEM agent, some of the PEM probes don't return information, and the following functionality might be affected.

Note

The list isn't comprehensive but provides an overview of the type of functionality that's limited.

Probe or actionOperating systemPEM functionality affected
Data And Logfile AnalysisLinux/ WindowsThe Postgres Expert can't access complete information.
Session InformationLinuxThe per-process statistics are incomplete.
PG HBALinux/ WindowsThe Postgres Expert can't access complete information.
Service restart functionalityLinux/ WindowsThe Audit Log Manager, Server Log Manager Log Analysis Expert, and PEM might not be able to apply requested modifications.
Package DeploymentLinux/ WindowsPEM can't run downloaded installation modules.
Batch TaskWindowsPEM can't run scheduled batch jobs in Windows.
Collect data from server (root access required)Linux/ WindowsColumns such as swap usage, CPU usage, IO read, IO write appear as 0 in the session activity dashboard.

If you restrict the database privileges of the PEM agent, the following PEM functionality might be affected:

ProbeOperating systemPEM functionality affected
Audit Log CollectionLinux/WindowsPEM receives empty data from the PEM database.
Server Log CollectionLinux/WindowsPEM can't collect server log information.
Database StatisticsLinux/WindowsThe Database/Server Analysis dashboards contain incomplete information.
Session Waits/System WaitsLinux/WindowsThe Session/System Waits dashboards contain incomplete information.
Locks InformationLinux/WindowsThe Database/Server Analysis dashboards contain incomplete information.
Streaming ReplicationLinux/WindowsThe Streaming Replication dashboard doesn't display information.
Slony ReplicationLinux/WindowsSlony-related charts on the Database Analysis dashboard don't display information.
Tablespace SizeLinux/WindowsThe Server Analysis dashboard doesn't display complete information.
xDB ReplicationLinux/WindowsPEM can't send xDB alerts and traps.

If the probe is querying the operating system without enough privileges, the probe might return a permission denied error. If the probe is querying the database without enough privileges, the probe might return a permission denied error or display the returned data in a PEM chart or graph as an empty value.

When a probe fails, an entry is written to the log file that contains the name of the probe, the reason the probe failed, and a hint that helps you resolve the problem.

You can view probe-related errors that occurred on the server in the Probe Log dashboard or review error messages in the PEM worker log files. On Linux, the default location of the log file is:

/var/log/pem/worker.log

On Windows, log information is available on the Event Viewer.

Agent configuration

A number of configurable parameters and registry entries control the behavior of the PEM agent. You might need to modify the PEM agent's parameter settings to enable some PEM functionality. After modifying values in the PEM agent configuration file, restart the PEM agent to apply any changes.

With the exception of the PEM_MAXCONN parameter, we strongly recommend against modifying any of these configuration parameters or registry entries without first consulting EDB support experts unless you need the modifications to enable PEM functionality.

On Linux systems, PEM configuration options are stored in the agent.cfg file, located in /usr/edb/pem/agent/etc. The agent.cfg file contains the following entries.

Parameter nameDescriptionDefault value
pem_hostThe IP address or hostname of the PEM server.127.0.0.1.
pem_portThe database server port to which the agent connects to communicate with the PEM server.Port 5432.
pem_agentA unique identifier assigned to the PEM agent.The first agent is '1', the second agent is '2', and so on.
agent_ssl_keyThe complete path to the PEM agent's key file./root/.pem/agent.key
agent_ssl_crtThe complete path to the PEM agent's certificate file./root/.pem/agent.crt
agent_flag_dirUsed for HA support. Specifies the directory path checked for requests to take over monitoring another server. Requests are made in the form of a file in the specified flag directory.Not set by default.
log_levelLog level specifies the type of event to write to the PEM log files, one of debug2, debug, info, warning, error. These are in descending order of logging verbosity; debug2 logs everything possible, and error only logs errors.warning
log_locationSpecifies the location of the PEM worker log file.127.0.0.1.
agent_log_locationSpecifies the location of the PEM agent log file./var/log/pem/agent.log
long_waitThe maximum length of time (in seconds) for the PEM agent to wait before attempting to connect to the PEM server if an initial connection attempt fails.30 seconds
short_waitThe minimum length of time (in seconds) for the PEM agent to wait before checking which probes are next in the queue waiting to run.10 seconds
alert_threadsThe number of alert threads to be spawned by the agent. For more information, see About alert threads.Set to 1 for the agent that resides on the host of the PEM server, 0 for all other agents.
enable_smtpWhen set to true for multiple PEM Agents (7.13 or earlier) it might send more duplicate emails. Whereas for PEM Agents (7.14 or higher) it might send fewer duplicate emails.true for PEM server host, false for all others.
enable_snmpWhen set to true for multiple PEM Agents (7.13 or earlier) it might send more duplicate traps. Whereas for PEM Agents (7.14 or higher) it might send fewer duplicate traps.true for PEM server host, false for all others.
enable_nagiosWhen set to true, Nagios alerting is enabled.true for PEM server host, false for all others.
enable_webhookWhen set to true, Webhook alerting is enabled.true for PEM server host, false for all others.
max_webhook_retriesSet maximum number of times pemAgent retries to call webhooks on failure.Default 3.
connect_timeoutThe max time in seconds (a decimal integer string) for the agent to wait for a connection.Not set by default. Set to 0 to indicate for the agent to wait indefinitely.
allow_server_restartIf set to TRUE, the agent can restart the database server that it monitors. Some PEM features might be enabled/disabled, depending on the value of this parameter.False
max_connectionsThe maximum number of probe connections used by the connection throttler.0 (an unlimited number)
connection_lifetimeUse ConnectionLifetime (or connection_lifetime) to specify the minimum number of seconds an open but idle connection is retained. This parameter is ignored if the value specified in MaxConnections is reached and a new connection to a different database is required to satisfy a waiting request.By default, set to 0 (a connection is dropped when the connection is idle after the agent's processing loop).
allow_batch_probesIf set to TRUE, the user can create batch probes using the custom probes feature.false
heartbeat_connectionWhen set to TRUE, a dedicated connection is used for sending the heartbeats.false
batch_script_dirProvide the path where script file (for alerting) is stored./tmp
connection_custom_setupUse to provide SQL code to invoke when a new connection with a monitored server is made.Not set by default.
ca_fileProvide the path where the CA certificate resides.Not set by default.
batch_script_userProvide the name of the user to use for executing the batch/shell scripts.None
webhook_ssl_keyThe complete path to the webhook's SSL client key file.
webhook_ssl_crtThe complete path to the webhook's SSL client certificate file.
webhook_ssl_crlThe complete path of the CRL file to validate webhook server certificate.
webhook_ssl_ca_crtThe complete path to the webhook's SSL ca certificate file.
allow_insecure_webhooksWhen set to true, allow webhooks to call with insecure flag.false

On Windows systems, PEM registry entries are located in:

HKEY_LOCAL_MACHINE\Software\EnterpriseDB\PEM\agent

The registry contains the entries shown in the table.

Parameter nameDescriptionDefault value
PEM_HOSTThe IP address or hostname of the PEM server.127.0.0.1.
PEM_PORTThe database server port to which the agent connects to communicate with the PEM server.Port 5432.
AgentIDA unique identifier assigned to the PEM agent.The first agent is '1', the second agent is '2', and so on.
AgentKeyPathThe complete path to the PEM agent's key file.%APPDATA%\Roaming\pem\ agent.key.
AgentCrtPathThe complete path to the PEM agent's certificate file.%APPDATA%\Roaming\pem\ agent.crt
AgentFlagDirUsed for HA support. Specifies the directory path checked for requests to take over monitoring another server. Requests are made in the form of a file in the specified flag directory.Not set by default.
LogLevelSpecifies the type of event to write to the PEM log files, one of debug2, debug, info, warning, error. These are in descending order of logging verbosity; debug2 logs everything possible, and error only logs errors.warning
LongWaitThe maximum length of time (in seconds) that the PEM agent waits before attempting to connect to the PEM server if an initial connection attempt fails.30 seconds
shortWaitThe minimum length of time in seconds that the PEM agent waits before checking which probes are next in the queue (waiting to run).10 seconds
AlertThreadsThe number of alert threads for the agent to spawn. For more information, see About alert threads.Set to 1 for the agent that resides on the host of the PEM server, 0 for all other agents.
EnableSMTPWhen set to true, the SMTP email feature is enabled.true for PEM server host, false for all others.
EnableSNMPWhen set to true, the SNMP trap feature is enabled.true for PEM server host, false for all others.
EnableWebhookWhen set to true, Webhook alerting is enabled.true for PEM server host, false for all others.
MaxWebhookRetriesSet maximum number of times for pemAgent to retry to call webhooks on failure.Default 3.
ConnectTimeoutThe max time in seconds (a decimal integer string) that the agent waits for a connection.Not set by default. If set to 0, the agent waits indefinitely.
AllowServerRestartIf set to TRUE, the agent can restart the database server that it monitors. Some PEM features might be enabled/disabled, depending on the value of this parameter.true
MaxConnectionsThe maximum number of probe connections used by the connection throttler.0 (an unlimited number)
ConnectionLifetimeUse ConnectionLifetime (or connection_lifetime) to specify the minimum number of seconds an open but idle connection is retained. This parameter is ignored if the value specified in MaxConnections is reached and a new connection to a different database is required to satisfy a waiting request.By default, set to 0 (a connection is dropped when the connection is idle after the agent's processing loop).
AllowBatchProbesIf set to TRUE, the user can create batch probes using the custom probes feature.false
HeartbeatConnectionWhen set to TRUE, a dedicated connection is used for sending the heartbeats.false
BatchScriptDirProvide the path to store the script file for alerting./tmp
ConnectionCustomSetupUse to provide SQL code to invoke when a new connection with a monitored server is made.Not set by default.
ca_fileProvide the path where the CA certificate resides.Not set by default.
AllowBatchJobStepsIf set to true,the batch/shell scripts are executed using Administrator user account.None
WebhookSSLKeyThe complete path to the webhook's SSL client key file.
WebhookSSLCrtThe complete path to the webhook's SSL client certificate file.
WebhookSSLCrlThe complete path of the CRL file to validate webhook server certificate.
WebhookSSLCaCrtThe complete path to the webhook's SSL ca certificate file.
AllowInsecureWebhooksWhen set to true, allow webhooks to call with insecure flag.false

About alert threads

The number of alert threads spawned by an agent is determined by the alert_threads or AlertThreads parameter. In general it is recommended to set this parameter to 1 on the agent which resides on the PEM server and 0 for all other agents. However on PEM server instances with very large numbers of alerts (caused by many monitored servers, many enabled alerts, or high alert frequency) it may be necessary to increase this parameter if alerts are not being evaluated at the configured frequency. In this situation, it is recommended to set this parameter to around 8 on the agent which resides on the PEM server and 0 for all other agents and tune up or down accordingly.

When tuning this parameter it is important to understand that any agent can process any alert, so in general it is unnecessary to have a non-zero number of alert threads on more than one agent. The capacity of the PEM instance to process alerts is determined by the total number of alert thread across all agents. Increasing the number of threads on a specific agent does not give any additional performance for alerts pertaining to servers monitored by that agent.

Each alert thread opens a connection to the PEM server backend, so allocating more threads than necessary does result in additional memory and CPU usage on the PEM server.

Agent properties

The PEM Agent Properties dialog box provides information about the PEM agent from which the dialog box was opened. To open the dialog box, right-click an agent name in the PEM client tree and select Properties from the context menu.

Use the PEM Agent Properties dialog box to review or modify information about the PEM agent:

  • The Description field displays a modifiable description of the PEM agent. This description is displayed in the tree of the PEM client.

  • You can use groups to organize your servers and agents in the PEM client tree. Use the Group list to select the group in which the agent is displayed.

  • Use the Team field to specify the name of the group role that can access servers monitored by the agent. The servers monitored by this agent are visible in the PEM client tree to connected team members. This is a convenience feature. The Team field doesn't provide true isolation. Don't use it for security purposes.

  • The Heartbeat interval fields display the length of time that elapses between reports from the PEM agent to the PEM server. Use the selectors next to the Minutes or Seconds fields to modify the interval.

    Use the Job Notifications tab to configure the email notification settings on agent level:

    • Use the Override default configuration? switch to specify if you want the agent level job notification settings to override the default job notification settings. Select Yes to enable the rest of the settings on this dialog box to define when and to whom to send the job notifications.
    • Use the Email on job completion? switch to specify whether to send the job notification when the job completes successfully.
    • Use the Email on a job failure? switch to specify whether to send the job notification when the job fails.
    • Use the Email group field to specify the email group to send the job notification to.

    The Agent Configurations tab displays all the current configurations and capabilities of an agent.

    • The Parameter column displays a list of parameters.
    • The Value column displays the current value of the corresponding parameter.
    • The Category column displays the category of the corresponding parameter. It can be either configuration or capability.