Reports and Alerts in SPLUNK

Overview

Alerts in Splunk are used to create notification or inform someone when a particular situation is met. It basically is used to notify the support team when any service degradation happens so that they can work and fix the issue.

The difference between dashboards and alerts is that dashboards needs to be manually monitored to understand the service behavior where as alert can automatically trigger when the thresholds are met and notify you over different mediums.

There are basically two types of Alerts:

• Actionable alerts/Real-time alerts – Alerts those needs some action once triggered. It certainly points out some degradation of a service.

•Informative alerts/Scheduled alerts – Alerts those are non-actionable and is triggered at a particular interval giving you some information. These are also known as Reports.

Alert Creation

Steps to create alert:

1. Gather information like threshold, frequency
2. Write down the query capturing the fields as required
3. Run the query for the specific timeframe and save it as alert
4. Edit the alert and provide all relevant information like;
   * Alert Name
   * Permission
   * Real-time/Scheduled
   * If Real-time then provide the cron.
   * Triggered actions
5. Save the alert once again.


Note : The alert name cannot be changes once created.

CRON

CRON is a data string of 5 values which is used to define the alert scheduling and timeframe when it should run.

It is denoted as five *’s where each * has a meaning as below (left to right)

1st * = minutes (0-59)
2nd * = hour (0-23)
3rd * = day of the month (0-31)
4th * = month of the year (0-12)
5th * = day of the week (0-6) where 0 denotes Sunday, 1is Monday and so on.

For example:
To make the alert run at 21.30 on Sunday of every day of the month and every month of the year

cron : (30 21 * * 0)

Actions

Actions as the meaning suggest are the steps that should be taken post the alert is triggered. It is a way to notify teams that this alert is triggered which means some thresholds have met and eventually a service degradation has happened.

There are quite a lot of actions possible as Splunk can contact lot many external tools and create different actions as required.

The most common ones are as below:
1. Email action
2. Incident through any ticketing tool
3. Run a script

Scheduling PDF

Scheduling PDF is an option in Splunk which can be used to email the output of the dashboard in a PDF format.

It is sent as an attachment in an email.