Notifications

Overview

• Proxmox Backup Server emits Notification Events in case of noteworthy events in the system. These events are processed based on the global notification settings. Each notification event includes metadata, such as a timestamp, severity level, type, and additional event-specific fields.

• Notification Matchers route a notification event to one or more notification targets. A matcher can have match rules to selectively route based on the metadata of a notification event.

• Notification Targets are a destination to which a notification event is routed to by a matcher. There are multiple types of target, mail-based (Sendmail and SMTP) and Gotify.

Global notification settings can be configured in the GUI under Configuration → Notifications. The configuration is stored in notifications.cfg and notifications-priv.cfg - the latter contains sensitive configuration options such as passwords or authentication tokens for notification targets and can only be read by root.

Notification Targets

Proxmox Backup Server offers multiple types of notification targets.

Sendmail

The sendmail binary is a program commonly found on Unix-like operating systems that handles the sending of email messages. It is a command-line utility that allows users and applications to send emails directly from the command line or from within scripts.

The sendmail notification target uses the sendmail binary to send emails to a list of configured users or email addresses. If a user is selected as a recipient, the email address configured in user's settings will be used. For the root@pam user, this is the email address entered during installation. A user's email address can be configured in Configuration → Access Control → User Management. If a user has no associated email address, no email will be sent.

Note

In standard Proxmox Backup Server installations, the sendmail binary is provided by Postfix. It may be necessary to configure Postfix so that it can deliver mails correctly - for example by setting an external mail relay (smart host). In case of failed delivery, check the system logs for messages logged by the Postfix daemon.

See notifications.cfg for all configuration options.

SMTP

SMTP notification targets can send emails directly to an SMTP mail relay. This target does not use the system's MTA to deliver emails. Similar to sendmail targets, if a user is selected as a recipient, the user's configured email address will be used.

Note

Unlike sendmail targets, SMTP targets do not have any queuing/retry mechanism in case of a failed mail delivery.

See notifications.cfg for all configuration options.

Gotify

Gotify is an open-source self-hosted notification server that allows you to send push notifications to various devices and applications. It provides a simple API and web interface, making it easy to integrate with different platforms and services.

Note

Gotify targets will respect the HTTP proxy settings from Configuration → Other → HTTP proxy

See notifications.cfg for all configuration options.

Webhook

Webhook notification targets perform HTTP requests to a configurable URL.

The following configuration options are available:

• url: The URL to which to perform the HTTP requests. Supports templating to inject message contents, metadata and secrets.

• method: HTTP Method to use (POST/PUT/GET)

• header: Array of HTTP headers that should be set for the request. Supports templating to inject message contents, metadata and secrets.

• body: HTTP body that should be sent. Supports templating to inject message contents, metadata and secrets.

• secret: Array of secret key-value pairs. These will be stored in a protected configuration file only readable by root. Secrets can be accessed in body/header/URL templates via the secrets namespace.

• comment: Comment for this target.

For configuration options that support templating, the Handlebars syntax can be used to access the following properties:

• {{ title }}: The rendered notification title

• {{ message }}: The rendered notification body

• {{ severity }}: The severity of the notification (info, notice, warning, error, unknown)

• {{ timestamp }}: The notification's timestamp as a UNIX epoch (in seconds).

• {{ fields.<name> }}: Sub-namespace for any metadata fields of the notification. For instance, fields.type contains the notification type - for all available fields refer to Notification Events.

• {{ secrets.<name> }}: Sub-namespace for secrets. For instance, a secret named token is accessible via secrets.token.

For convenience, the following helpers are available:

• {{ url-encode <value/property> }}: URL-encode a property/literal.

• {{ escape <value/property> }}: Escape any control characters that cannot be safely represented as a JSON string.

• {{ json <value/property> }}: Render a value as JSON. This can be useful to pass a whole sub-namespace (e.g. fields) as a part of a JSON payload (e.g. {{ json fields }}).

Note

Webhook targets will respect the HTTP proxy settings from Configuration → Other → HTTP proxy

Example - ntfy.sh

• Method: POST

• URL: https://ntfy.sh/{{ secrets.channel }}

• Headers:

  • Markdown: Yes

• Body:

  ```
  {{ message }}
  ```
  

• Secrets:

  • channel: <your ntfy.sh channel>

Example - Discord

• Method: POST

• URL: https://discord.com/api/webhooks/{{ secrets.token }}

• Headers:

  • Content-Type: application/json

• Body:

  {
    "content": "``` {{ escape message }}```"
  }
  

• Secrets:

  • token: <token>

Example - Slack

• Method: POST

• URL: https://hooks.slack.com/services/{{ secrets.token }}

• Headers:

  • Content-Type: application/json

• Body:

  {
    "text": "``` {{escape message}}```",
    "type": "mrkdwn"
  }
  

• Secrets:

  • token: <token>

Notification Matchers

Notification matchers route notifications to notification targets based on their matching rules. These rules can match certain properties of a notification, such as the timestamp, the severity of the notification or metadata fields, as well as arbitrary combinations of these. If a notification is matched by a matcher, all targets configured for the matcher will receive the notification.

An arbitrary number of matchers can be created, each with their own match rules and targets to notify. Every target is notified at most once for every notification, even if the target is used in multiple matchers.

Match Rules

Notification matchers can be configured in the GUI under Configuration → Notifications. Matchers can be created, modified and deleted from this view.

When editing a matcher, match rules can be edited under the Match Rules tab. Rules are represented as a tree. Each tree node can either match on properties of a notification (fields, severity or timestamp) or combine the results of child nodes (e.g. All sub-rules match). Combinator nodes can be nested, which makes it possible to compose arbitrarily complex matching behavior.

The following node types are available:

• Always match: Match any notification.

• All sub-rules match: All direct children of this node must match.

• At least one sub-rule matches: At least one direct child of this node must match.

• Exactly one sub-rule matches: Exactly one direct child of this node must match.

• Match Field: Match a metadata field of a notification. This node can either match if the field value matches one of the values provided, or, alternatively, if it matches a regular expression. Available fields are listed in Notification Events.

• Match Calendar: Check if the timestamp of a notification is within a configured time range, based on a selection of weekdays and one contiguous time range within each day (e.g. Monday to Friday, 09:00 to 17:00). If multiple time ranges within a day must be matched, use an At least one sub-rule matches node with one Match Calendar child per time range.

• Match Severity: Match the severity of a notification. The following severity levels are available: info, notice, warning, error and unknown. Each Match Severity node can specify multiple severity levels it matches on, for instance info and notice.

Each node in the tree can be inverted by checking Invert Rule. For instance, for a Match Severity node matching on info, checking Invert Rule lets the node match on notice, warning, error and unknown.

Under the Targets to Notify tab, one or more notification targets can be selected. All selected targets are notified if a notification is matched.

Matcher Configuration Options

This section gives an overview of the options used by matchers in the configuration file and in the API.

The following options are used in configuration entries for matchers:

• target: Determine which target should be notified if the matcher matches. Can be used multiple times to notify multiple targets.

• disable: Don't evaluate this matcher.

• comment: Comment for this matcher.

• The matching behavior is defined either by the expression option or by the deprecated options below; the two cannot be combined.

  • Expression-based matchers (preferred):

    • expression: Ruleset for this matcher as a single JSON-serialized object (Match Expression Reference).

  • Deprecated matcher options:

    • invert-match: Inverts the result of the whole matcher.

    • mode: Determines how the individual match rules are evaluated to compute the result for the whole matcher. If set to all, all matching rules must match. If set to any, at least one rule must match. Defaults to all.

    • match-calendar: Match the notification's timestamp against a schedule. Days can be specified as an inclusive range (e.g. mon..wed) or enumerated (e.g. mon,wed,sat). If no days are specified, the time range applies to every day of the week (e.g. 08:00-12:00). Examples:

      • match-calendar mon..wed 9-12

      • match-calendar mon,wed,sat 08:15-12:35

      • match-calendar 04:15-16:35

    • match-field: Match a notification's metadata field, either against a list of matched values, or as a regular expression. If the field does not exist, the notification will not be matched. Examples:

      • match-field exact:type=gc

      • match-field exact:type=prune,verify

      • match-field regex:datastore=^backup-.*$

    • match-severity: Match the notification's severity. Examples:

      • match-severity error

      • match-severity warning,error,unknown

A matcher without any matching rules always matches; the configured targets will always be notified.

Starting with Proxmox Backup Server 4.3, match rules created and edited via the GUI are stored as a single option expression in the respective matcher configuration entry. The previously used options (match-field, match-calendar, match-severity, mode and invert-match) will continue to be supported at least until the next major release.

Note

Any existing matcher will be migrated to use the newer expression option when edited through the GUI.

See notifications.cfg for all configuration options.

Match Expression Reference

This section describes the representation used to encode rules in the expression parameter.

The expression option contains a JSON-formatted tree structure of rules, with leaf nodes matching on the properties of a notification (field, severity, calendar) or representing constants (constant). Intermediate nodes and the root node encode the relationship between them (all-of, any-of, one-of, not).

Node Type	Description	Leaf Node	Example
all-of	Match if all child nodes match	No	{"all-of": [<expr>]}
any-of	Match if any child node matches	No	{"any-of": [<expr>]}
one-of	Match if exactly one child node matches	No	{"one-of": [<expr>]}
not	Invert the result of a single child node	No	{"not": <expr>}
constant	Always or never matches, depending on the boolean value	Yes	{"constant": true|false}
match	Match on a property of a notification	Yes	{"match": <match-expr>}

match leaf nodes can have the following variants (<match-expr>):

Match Type	Description	Example
calendar	Check if the notification's timestamp is within a configured time range	{"type": "calendar", "schedule": "mon..fri 9-17"}
field	Check a notification's fields against a regex or a list of permitted values	{"type": "field", "field": "job-id", "values": ["a", "b"]} or {"type": "field", "field": "datastore", "regex": "^backup-.*$"}
severity	Check a notification's severity against a list of permitted values	{"type": "severity", "severities": ["info", "notice"]}

A pretty-printed, full example of a non-trivial expression is shown below. It encodes an expression matching on garbage collection jobs (gc), but only within a given time range (mon..fri 09:00-17:00) or if it has a severity of warning or error. In the API and configuration file, the JSON is expected to be stored as a single line, as the value of the expression option; the line breaks and indentation are only for demonstration purposes.

{
  "all-of": [
    {
      "any-of": [
        {
          "match": {
            "type": "calendar",
            "schedule": "mon..fri 09:00-17:00"
          }
        },
        {
          "match": {
            "type": "severity",
            "severities": [
              "warning",
              "error"
            ]
          }
        }
      ]
    },
    {
      "match": {
        "type": "field",
        "field": "type",
        "values": [
          "gc"
        ]
      }
    }
  ]
}

Notification Events

The following table contains a list of all notification events in Proxmox Backup Server, their type, severity and additional metadata fields. type as well as any other metadata field may be used in match-field match rules.

Event	type	Severity	Metadata fields (in addition to type)
ACME certificate renewal failed	acme	error	hostname
Garbage collection failure	gc	error	datastore, hostname
Garbage collection success	gc	info	datastore, hostname
Package updates available	package-updates	info	hostname
Prune job failure	prune	error	datastore, hostname, job-id
Prune job success	prune	info	datastore, hostname, job-id
Remote sync failure	sync	error	datastore, hostname, job-id
Remote sync success	sync	info	datastore, hostname, job-id
Tape backup job failure	tape-backup	error	datastore, hostname, media-pool, job-id
Tape backup job success	tape-backup	info	datastore, hostname, media-pool, job-id
Tape loading request	tape-load	notice	hostname
Verification job failure	verification	error	datastore, hostname, job-id
Verification job success	verification	info	datastore, hostname, job-id

The following table contains a description of all use metadata fields. All of these can be used in match-field match rules.

Metadata field	Description
datastore	The name of the datastore
hostname	The hostname of the backup server
job-id	Job ID
media-pool	The name of the tape media pool
type	Notification event type

Note

The daily task checking for any available system updates only sends notifications if the node has an active subscription.

System Mail Forwarding

Certain local system daemons, such as smartd, send notification emails to the local root user. These mails are converted into notification events with the type system-mail and with a severity of unknown.

When the email is forwarded to a sendmail target, the mail's content and headers are forwarded as-is. For all other targets, the system tries to extract both a subject line and the main text body from the email content. In instances where emails solely consist of HTML content, they will be transformed into plain text format during this process.

Permissions

In order to modify/view the configuration for notification targets, the Sys.Modify/Sys.Audit permissions are required for the /system/notifications ACL node.

Notification Mode

Datastores and tape backup/restore job configuration have a notification-mode option which can have one of two values:

• Send notifications based on the global notification settings (notification-system).

• Send notification emails via the system's sendmail command (legacy-sendmail). Any targets or matchers from the global notification settings are ignored. This mode is equivalent to the notification behavior for Proxmox Backup Server versions before 3.2. It might be removed in a later release of Proxmox Backup Server.

Settings for legacy-sendmail notification mode

If notification-mode is set to legacy-sendmail, Proxmox Backup Server will send notification emails via the system's sendmail command to the email address configured for the user set in the notify-user option (falling back to root@pam if not set).

For datastores, you can also change the level of notifications received per task type via the notify option.

• Always: send a notification for any scheduled task, independent of the outcome

• Errors: send a notification for any scheduled task that results in an error

• Never: do not send any notification at all

The notify-user and notify options are ignored when using the global notification settings (notification-mode is set to notification-system).

Overriding Notification Templates

Proxmox Backup Server uses Handlebars templates to render notifications. The original templates provided by Proxmox Backup Server are stored in /usr/share/proxmox-backup/templates/default/.

Notification templates can be overridden by providing a custom template file in the override directory at /etc/proxmox-backup/notification-templates/default/. When rendering a notification of a given type, Proxmox Backup Server will first attempt to load a template from the override directory. If this one does not exist or fails to render, the original template will be used.

The template files follow the naming convention of <type>-<body|subject>.txt.hbs. For instance, the file gc-err-body.txt.hbs contains the template for rendering notifications for garbage collection errors, while package-updates-subject.txt.hbs is used to render the subject line of notifications for available package updates.

Notification Thresholds and Reset Schedule (S3 Datastores Only)

Datastores of type S3 keep track of the number of requests being send to the corresponding S3 endpoint and the amount of data being send. Proxmox Backup Server allows to configure threshold for these request and traffic counters to send out notifications if one of the set threshold values is exceeded.

The notification threshold value can be set individually by request method or traffic volume. Notifications will be send out only once per threshold when the threshold has been exceeded. A counter reset is required to bring it below the threshold again in order to get further notifications. Therefore, it is possible to define a threshold reset schedule so request and traffic counters get periodically reset.

Per datastore notification thresholds and their reset schedule are configurable in the Datastore Options for S3 backed datastores.

The following counters thresholds are available for configuration:

Counter Threshold	Description and Usage
s3-get	Number of GET requests: Mainly used for download of data and metadata among the following operations: restore, verification, S3 refresh, garbage collection.
s3-put	Number of PUT requests: Mainly used for upload of data and metadata among the following operations: backup, syncs, metadata changes, content moves.
s3-post	Number of POST requests: Mainly used for modification of data and metadata among the following operations: bulk deletion during garbage collection, content moves.
s3-head	Number of HEAD requests: Mainly used to check for access and existence among which the following operations: checking bucket access, fetching of metadata.
s3-delete	Number of DELETE requests: Mainly used for deleting single objects by the following operations: garbage collection, content moves.
s3-upload	Amount of bytes uploaded to the S3 endpoint, independent of request method.
s3-download	Amount of bytes downloaded from the S3 endpoint, independent of request method.