Skip to content

Alarms and incidents

Use alarm rules to watch a tag and open an incident when its value crosses a threshold or its reading goes bad.

An alarm rule belongs to one tag of one asset on one Edge. When the rule's condition holds long enough, Watchtower opens an incident on the Edge's Incidents tab and emails subscribed users. When the value returns to normal, Watchtower closes the incident.

Task Who can do it
Create a rule Owner, Admin
Acknowledge an incident Owner, Admin, Operator
View incidents Everyone with access to the Edge

Create an alarm rule

  1. Open the Edge, go to the Assets tab and click the asset.
  2. Find the card of the tag you want to watch. The rule form is under its chart.

  3. Fill in the form:

    Field What to enter
    Alarm name 1 to 200 characters. Shown in the incidents list and in email subjects.
    Severity info, warning (default) or critical.
    Condition >, >=, <, <=, == or quality bad.
    Value The threshold that raises the alarm. Disabled for quality bad.
    Clear Optional. The value at which the incident closes.
    For seconds How long the condition must hold before the incident opens, and before it closes. 0 acts on the first reading.

  4. Click Add rule.

Watchtower starts evaluating a new rule within 30 seconds. You can add several rules to the same tag, for example a warning and a critical threshold.

How conditions behave

  • Threshold conditions compare the tag's number. A bool tag counts as 1 (true) or 0 (false). string tags cannot raise threshold alarms.
  • quality bad raises when a reading arrives with bad quality, for example when the device stops answering, and clears when readings are good or uncertain again.
  • Clear stops an alarm from flapping around one value. For > and >=, set Clear below Value; for < and <=, set it above. Between the two values, an open incident stays open. Without Clear, the incident closes as soon as the condition stops holding for For seconds.

Example

Condition >, Value 80, Clear 70, For seconds 60: the incident opens after the tag stays above 80 for one minute, and closes after it stays below 70 for one minute.

Change or remove a rule

The portal creates rules but does not list, edit, disable or delete them. Use the public API with an API token (see Settings and API tokens):

Action Request
List a tag's rules GET /api/v1/metrics/<metric-id>/rules (tag ids: GET /api/v1/assets/<asset-id>/metrics; the asset id ends the asset page address)
Change or disable PATCH /api/v1/rules/<rule-id> with any of name, severity, condition, enabled
Delete DELETE /api/v1/rules/<rule-id>. Refused once the rule has incidents; disable it instead. 
curl -s -X PATCH -H "Authorization: Bearer $WT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}' \
  https://watchtower.example.com/api/v1/rules/<rule-id>

A condition looks like {"op": ">", "value": 85, "clear": 75, "for_s": 60}. op is >, >=, <, <=, == or quality_bad.

Work with incidents

Open the Edge and select the Incidents tab. It lists the latest 200 incidents, newest first.

Column What it shows
Alarm The rule name.
Severity info, warning or critical.
Asset, Metric Where the alarm comes from.
Opened When the incident opened.
Closed When it closed. — while it is still open.
Ack When it was acknowledged, or an Ack button.
State How to recognise it What happens next
Open Closed is — and Ack shows a button. Investigate, then click Ack.
Acknowledged Closed is — and Ack shows a time. Stays open until the value returns to normal.
Closed Closed shows a time. Nothing. The next alarm on this rule opens a new incident.

Click Ack to record that someone is handling the incident. Acknowledging does not close it. There is no manual resolve: Watchtower closes the incident when the clear condition holds. A rule has at most one open incident at a time.

Alarm email

Watchtower emails users whose notification subscription covers the Edge (the whole organization, the Edge's site, its group or the Edge itself) and whose minimum severity is at or below the alarm's severity.

Email Subject
Incident opened <severity>: <alarm name>
Incident closed resolved: <alarm name>
Many alarms at once <N> alarms opened
  • The email gives the alarm name, severity, asset and a link to the Edge. It does not include the value.
  • Each rule sends at most one email every 15 minutes, counting both opened and resolved email.
  • When more than 10 incidents open within 5 minutes, recipients get one summary listing up to 50 alarms instead, at most once every 15 minutes.
  • Every alarm email has an unsubscribe link. It turns off all alarm email for that user. Sign-in and invitation email is not affected.

Subscriptions are set up by Privitty support

You cannot manage who receives alarm email in the portal or through the public API. Without a subscription, alarms still open incidents in the portal but send no email. To add or change subscriptions, contact Privitty support through the channel listed in your Privitty agreement.