Understanding job templates v10.6

A job template is a job definition you write once and push to many servers or agents at the same time.

Without templates, running the same maintenance job across many servers means creating the same job by hand on each one, and editing it again on each one when it changes. With a template, you define the steps and schedules once, pick the targets, and PEM creates and maintains one job per target for you.

Typical uses include:

• Running the same VACUUM or ANALYZE health check on every production server, on a regular schedule.
• Generating the same recurring report for every server in a group.
• Running the same log-rotation script on every agent host.

A template isn't a new kind of job. PEM materializes the template into ordinary PEM jobs, one per target, which the agents run exactly like any other job. What the template adds is creating them all at once and keeping them in sync.

Key concepts

One job per target

Each target gets its own job, with its own run history. If one server is down, the others are unaffected — you get a per-target result, not one combined pass/fail.

Jobs multiply per server, not per agent. If one agent monitors five targeted servers, that agent ends up running five jobs from the template, one per server.

Materialized jobs are named for the template they came from:

Prerequisites

• You need the Scheduled Tasks privilege (pem_manage_schedule_task). Without it, the Job Templates menu item doesn't appear.
• Every server you want to target must be actively bound to an agent. A server with no active agent binding can't be a target — the target picker marks it with an info icon explaining why.
• For Report steps, the report templates you want to use must already exist.

Creating a template

To open the Job Templates workspace, select Management > Configurations > Job Templates. Select Add in the Job Templates list. The editor has four tabs: General, Steps, Schedules, and Notifications.

General

The template's type determines what steps you can add and what you can target:

If you choose the wrong type, create a new template — the type can't be changed later.

Steps

A template needs at least one enabled step. The editor and the database both reject a template without one.

Steps run in order, with no parallel execution within a job. See Renaming, deleting, and reordering steps before you rely on reordering them.

Success and Ignore both let the job continue after a failed step; the difference is what the run history shows afterward. Use Ignore when you want the failure to stay visible.

The database you set on a step is one value for every target. If you type a database name and a target server doesn't have it, the step fails on that server with a connection error — other targets are unaffected. Leaving the field blank, so each target uses its own maintenance database, avoids this.

Schedules

A template can carry several schedules; the job runs at the earliest matching time of any enabled one.

The schedule fires when the current time matches every field you set. Leaving a field blank means it matches every value of that field:

The last row is a special case, not a wildcard. Leaving one field blank while setting another (for example, Minutes only) still matches every value of the blank field, and the schedule keeps recurring. Only when every Days and Times field is empty does PEM treat the schedule as a single run instead of a recurring one.

Schedules are interpreted in the PEM server's time zone, not the target server's. The editor doesn't expose a per-schedule time zone field — every schedule you create in the UI inherits the PEM server setting. On a geographically spread fleet, a schedule set for 02:00 runs at 02:00 PEM-server time on every target, not 02:00 local time at each one.

Notifications

These settings are copied onto every materialized job, so all targets notify the same way.

Deploying to targets

Saving a template does nothing on its own. A template with no targets creates no jobs.

From the Job Templates list, select the Manage targets & history icon on the template's row, then Add targets.

The dialog shows what your template type can use:

Selecting a group header selects all its servers, including those in nested groups. Selecting a group is a shortcut, not a subscription: the dialog expands your group selection into an explicit list of servers when you add them. A server added to that group later isn't deployed automatically — add it as a target when you want it included.

When you deploy:

1. Each chosen server is resolved to the one active agent that monitors it. A server with no active agent binding is skipped.
2. One job is created per target.
3. Steps and schedules are copied in, with the server and database filled in concretely for each target.

Deploying a template that would create no jobs at all is rejected, rather than appearing to succeed and doing nothing.

Managing a deployment

The Manage page for a template has a stat strip summarizing target and run status, and two tabs.

Targets tab

Available actions:

Removing a target is sticky: a removed target isn't brought back by later template edits or resyncs. To bring it back, add it as a target again explicitly.

You can permanently delete a removed target's run history only after the target has been removed. If you try before that, PEM refuses with a message telling you to remove the target first.

History tab

A run feed across all targets, showing the log ID, target, agent, status, start time, and duration for each run. Expand a row to see each step's status, result, duration, output, and log details.

You can search by target or agent name, filter to removed targets only, or delete selected history rows.

Editing a live template

Editing a template immediately updates every job it has already deployed. There's no separate "redeploy" step.

When you save an edit, PEM re-materializes every target's job: steps are reconciled, schedules are replaced, and the job's next run time is recalculated — even when the schedule itself didn't change.

What's preserved across an edit:

• Per-target Enabled? settings and removals.
• Run history, including the history of steps you delete.

Renaming, deleting, and reordering steps

Where the deployed jobs appear

Materialized jobs show up in the normal places — the browser tree under their agent, and the Scheduled Tasks view. They're managed by the template and read-only there:

This is intentional: a local edit would be silently overwritten the next time the template re-materializes.

Understanding lifecycle events

A server or agent can move or disappear through ordinary fleet changes, including a high availability (HA) failover or agent takeover. Here's what happens to a template's targets and jobs in each case:

Soft-removal is sticky, however it happened. Whether you removed the target yourself or PEM removed it because its server or agent became unusable, a later template edit or resync won't bring it back. Only explicitly adding it again under Add targets does.

Deleting an agent or server in PEM happens in stages: PEM first marks it inactive, which soft-removes the template's targets and disables their jobs without destroying anything. The object itself, and an agent's materialized jobs and their run history, are removed later, when PEM's deleted-object purge processes it.

Deleting a template deletes the jobs it deployed. The delete confirmation states this explicitly, warning that it will also delete all jobs already deployed from the template, along with their run history, and that this can't be undone.

Auditing template changes

Every change you make through the Job Templates workspace is recorded in pem.event_history with component = 'job_template'. Nothing is written for read-only actions such as opening a template or browsing history.

There's no dedicated screen for this yet. Query the table directly:

SELECT recorded_time, user_name, operation, message
  FROM pem.event_history
 WHERE component = 'job_template'
 ORDER BY recorded_time DESC
 LIMIT 50;

Operations

Both the list's Enabled? switch and a full edit record operation = 'update'. There's no separate disable operation — check details -> 'Changes' -> 'Metadata' -> 'Enabled' to see whether an update changed the enabled state.

A delete row carries the full before-image of the template: every step (with its code) and every schedule (with its recurrence and exceptions). Once the template is gone, this row is the only record of what the fleet was running.

An update row carries a Changes object with only what actually differed, including Metadata (name, description, enabled, notify, and email group, each with before and after values), StepsAdded/StepsRemoved, StepsUpdated, and SchedulesAdded/SchedulesRemoved/SchedulesUpdated. Saving a template without changing anything still records the row, without a Changes object, because the save still re-materializes every deployed job.

Retention and access

Rows are kept for 30 days by default, controlled by the event_history_retention_time configuration parameter and purged by the built-in Event history table cleanup system job. Raise the value if you need a longer audit window.

Any PEM user can read the whole pem.event_history table. There's no row-level filtering on it, and details includes step code — the SQL and shell text your templates run. Keep this in mind when deciding what to put in a step.

Importing and exporting templates

Templates can be moved between PEM servers as .pemjt files.

• Export: Select the templates you want, then select Export.
• Import: Choose a .pemjt file. The Skip existing? option decides what happens to templates whose names already exist on this server: when it's on, those entries are skipped and reported as skipped; when it's off, they're overwritten.

The definition travels; the deployment doesn't. Targets, deployments, and run history are never exported. An imported template arrives with no targets and creates no jobs until you deploy it.

References inside the file, such as report templates and email groups, travel by name and are re-resolved on the destination server. If a name doesn't exist there, the import reports it for that entry rather than failing the whole file.

A file written by a newer PEM than the one importing it is rejected, with a message naming both format versions.

Troubleshooting

Limitations

• One deployment per template. A template has a single target set, not several independent ones.
• Type is immutable. Database and Host templates can't be converted.
• Targeting is per server, not per database. A Database template runs the same step set against one database per target server.
• Group targeting is a selection shortcut, not a live subscription to group membership.
• Schedule time zone isn't editable in the template editor.
• Step execution order can't be changed for steps that are already deployed.
• Detaching a materialized job from its template isn't available in the UI.
• The audit trail has no dedicated screen. Query pem.event_history directly, and note that any PEM user can read it.