Setup a watcher in Kibana to send email notifications

In this blog I showed how you can hook up you SOA Suite stack to ElasticSearch and create dasboards to monitor and report. Another nice feature is that you can set a watcher to monitor the data for you and send emails or post something on Slack when the event occurs. For example I want to be notified by email when more then 25 errors occur in a minute. Lets see how this works.

KibanaWatcher

Setup the watcher

Login to you Kibana cloud instance and go to Management. There click Watcher.
Watcher

Here you see all the configured watchers. We want to create our own custom watch based on JSON click the dropdown and select Advanced Watch.

AdvancedWatch

As you can see, you already get a preconfigured JSON which you can edit to your own liking. Our requirement are:

  • Trigger every minute
  • Triggers when more then 25 errors occured
  • Send email

So lets translate this to the JSON. Dont forget to enter a Name and an ID for the watch.

First part, trigger every minute:

  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },

Second part, trigger when more then 25 errors occure within a minute. As you might have seen in my previous blog, we log the HTTP response code of every call so we want to check on this if it is a 500. So in the search, select the right index. In my case servicedata*. Next in the query, filter on time range from now-1minute. And as a last, match on httpResponseCode 500.

  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "servicedata*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "datum": {
                      "gte": "now-1m"
                    }
                  }
                },
                {
                  "match": {
                    "httpResponseCode": 500
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 25
      }
    }
  },

As the last part, send an email. Input the To value, the Subject and the Body. Be aware though that you have to white list the email address you want to send the email to. Go to https://cloud.elastic.co and log in. Then go to Profile and whitelist the email address.

WhitelistEmail. An email for approval is send to the email address. Just open the email and Click Confirm Email Whitelisting.

ConfirmEmailWhitelisting

So now that we are whitelisted, we should be able to receive email. As the email text we just put a simple message in there.

"actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "'Functioneel Beheer <hugo@redrock-it.nl>'"
        ],
        "subject": "OFM Watcher Notification",
        "body": {
          "text": "{{ctx.payload.hits.total}} errors found in the log the past minute!!!"
        }
      }
    }
  }

Ok now lets try it out. I will just call a service 26 times which shoudl create an error and lets see what it does. My Dashboard sees the errors.

ErrorDashboard

And when we look at the watch, we can see it fired.

WatchFiring

And we got an email.

WatchEmailNotification

As you can see it is quite simple to create notification based on certain search criteria. Great to keep an eye out for certain occurrences.

References:

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.