SCOM and Orchestrator Voice Notification Solution with Twilio and Automys.
Cherry Picking SCOM alerts…
Problem:
Issue # 1 : Spam (and I don’t mean the canned pork meat)
Alert spam has been a challenge for anyone that uses SCOM. Finding the perfect balance between notifying IT folks on important issues and avoiding overwhelming them with alerts has proven difficult.
However, when a large number of alerts are generated, generally means a big problem is brewing in your network.
Issue # 2 : Timing
Email and SMS Text notifications can be very effective during working hours. After hours is a different issue all together since most people put their devices down (as they should) and a large number of alerts or a network outage could be missed.
That is where Voice notification comes into play.
However, we wouldn’t want to wake up our IT Admin at 3:00 am for just any issue. Totally fine if you’re testing or if you dislike the person, you know the one always complaining about the amount of alerts they get or when issues do arise, they complaint why they didn’t get more alerts and when.. (ahem), excuse me, back to the subject.
Solution: Voice notification, appropriately named 'Alert Cherry Picker'
The solution provided below is a combination of SCOM, Orchestrator and third party solutions Twillio and Automys.
Logic:
Orchestrator side:
- An SCORCH Runbook monitors SCOM alerts and writes the alert’s details to a Database.
- A PowerShell scripts converts the alert’s UTC time to local time.
- An SQL query is later run to verify if the same alert has been written to the Database in the last x number of minutes. (Here you can customize it to monitor only critical alerts, alerts from specific group, etc.)
- If the alert meets the time and frequency criteria, an event is written to the Orchestrator server event log.
SCOM side:
- A SCOM rule reads the Orchestrator log file and generates another alert. (Voice Notification Alert Trigger)
- A SCOM Command notification configured to listen for the above alert is executed which places the phone call.
This is where the third party comes in.
Noah Stahl has done an amazing job integrating SCOM to communication solution Twilio.
Follow the steps in the guide below to integrate SCOM with twilio.
https://automys.com/library/asset/sms-voice-notifications-with-powershell-system-center-operations-manager-and-t
My contribution:
My contribution here is in the form of the Orchestrator Runbook and the custom SCOM Management Pack that perform the steps described in the logic section above.
Runbooks
The Trimmer Runbook
Activities:
Monitor Alert:
As part of the Operations Manager Integration Pack this activity monitor all SCOM alerts.
‘The Times They Are a-Changin’ script:
Simple script to change the alert’s UTC time to local time:
Write to Database:
This activity writes the alerts details to the custom database. You can find the AlertCherryPickerTable.sql file in this solution to create the needed SQL table that exactly matches the activity parameters.
To the Picker Invoke Runbook Control:
Passes alert’s details to the Picker Runbook, which determines if our thresholds are met.The Picker Runbook
Activities:
Query Database if Alert Exists:
Feel free to modify the details. Here we query SQL to find another alert with the same name in the last 5 minutes.
SQL Query Result Greater Than or equals to X Link:
This link determine if the results from the query are pass to the next activity, which creates an event
Add the minimum number of rows in the Query Result as a condition to pass to the next activity
Create an Event:
An event is created in a Custom EventLog on the Orchestrator server.
SCOM Management Pack:
Management Pack ‘Mundo SCOM SCORH Runbook Event Log Monitoring’ contains the rule “Voice Notification Alert Trigger”.
The alert is disabled. Enable it for the Orchestrator server via an override
Command Channel
Follow Noah’s steps on how to set up a channel. Below is what the end product looks like. Note the message type is Voice.
Alert Subscription
This is what triggers the command channel, which executes the script (Noah’s solution), which makes the call. That’s a lot…
Very important to include the dummy subscriber, and I don’t mean your IT Admin… ba-dum-pump chsh!
Downloads AlertCherryPicker.zip
