Category Archives: Monitoring

More Pushover Configuration

Introduction

After successfully configuring Pushover with PRTG, I moved on to configuring my Synology DiskStation (easy) and VMware vCenter (harder) installation to also notify me via Pushover.

 

Register An Application

In both cases, I registered a new application within the Pushover console.  I did this so that each alert would come though with their own specific icon and description.  See my previous blog posting on how to do this.

I used the following icons for each application :

  • synology Synology
  • vmware VMware

 

Synology DiskStation Notifications

syno-notifications-1This is was the easiest change to make, simply…

  1. Login to your DiskStation,
  2. Open the Control Panel,
  3. Select the Notification option,
  4. Choose either (or both) of the following tabs…
    • Email
    • Push Service
  5. Enter your Pushover API email address
    • [user-key]+a=[api-key]@api.pushover.net

 

VMware vCenter Alerts

Setting up these alerts were a lot harder.  It took a lot of trial and error.  Hopefully this will save you the same hassle.

Before you start, make sure you have the Microsoft dotNet framework version 4 or later installed, as well as PowerShell 4 or later.  Next, we’ll create a folder and two files on your vCenter server…

  1. On the C: drive, create a folder called Scripts,
  2. Create two plain text files and enter the following code…

send-alert.bat

"c:\windows\system32\windowspowershell\v1.0\powershell.exe" c:\scripts\send-alert.ps1 -title '%1' -message '%2'

send-alert.ps1

param(
    [string]$title,
    [string]$message
)

$parameters = @{
    token    = "[api-key]"
    user     = "[user-key]"
    priority = "0"
    title    = $title
    message  = $message
}

$parameters | Invoke-RestMethod -Uri "https://api.pushover.net/1/messages.json" -Method Post

Remember to enter your own api-key and user-key into the .ps1 script file.

 

vmware-definitionsvCenter Configuration

Next we need to configure VMware vCenter to run the batch file whenever an alert condition is met.  At last count there are 68 different alert definitions within vCenter.  You could change each on manually, but that would take all day.  If you have VMware PowerCLI installed (and you really should have) you can script this.

Manual Steps…

  1. vmware-alarm-4Open VMware vSphere Client, and login to your vCenter server,
  2. Select the very top level entry in the tree view on the left,
  3. Choose the Alarms tab along the top,
  4. Click the Definitions button,
  5. The list of 68+ definitions are list.  Double-click on one of them.  I’ll choose Datastore usage on disk,
  6. Select the Actions tab,
  7. Click Add,
  8. From the new entry that appears,
    • change “Send a notification email
    • to “Run a command
  9. Enter the following code into the Configuration column…
c:\scripts\send-alert.bat "{targetName}  >  {eventDescription}" "{triggeringSummary}"
  1. Click OK
  2. Rinse and repeat for the remaining definitions, or…

Automatic Script…

  1. Open a PowerCLI command window, and connect to your vCenter server
  2. Copy and paste the following code, and let it run…
$al = Get-AlarmDefinition
ForEach ($a in $al) {
    Get-AlarmDefinition -Name $a | New-AlarmAction -Script -ScriptPath 'c:\scripts\send-alert.bat "{targetName}  >  {eventDescription}" "{triggeringSummary}"'
}

Make sure you get the single quotes ( ‘ ) and double-quotes ( ” ) correct

For a list of variables you can use, check the VMware documentation.

 

Results

Once you have alerts working, you should get alerts through looking something like this…

space-alert

Push Notifications With PRTG And Pushover

PushoverIntroduction

I was looking for a way for my PRTG installation to alert me whenever I was away from home.  I initially thought of using an “email to SMS message” service, and started looking into this option.  I was then pointed to an API push service called Pushover.

Setting this up to work with PRTG is amazingly simple.  The instructions and scripts I use are taken from this blog, just cleaned up a bit and with more screenshots.

 

Prerequisites

A Pushover account,

Install Python for Windows onto your PRTG server.  Choose Python 2, not 3.

An Android or Apple device with the Pushover app installed,

store-google   store-apple

 

Pushover Instructions

Once you have created your account and verified your email address, you need to create an application in-order to get an API key.

  1. pushover - 1From your main home page,
  2. Next to Your Applications, click Register An Application,
  3. Fill out the displayed form, my entries were…
    • PRTG
    • Application
    • (blank)
    • This image (click to enlarge) – PRTG-75x75
  4. Tick the Terms Of Service check box and click Create Application.

Once the application has been created, you will be presented with an API Key.  You will need this later, along with your User Key.

 

Device Setup

Depending on your device, download the appropriate app and install it.  Log in to the app, and check the settings to see if you need to change anything, specifically the Edit Quiet Hours.

 

Python Scripts

Log on yo your PRTG server, and install Python.  This is a very straight forward installation…

python install - 1   python install - 2   python install - 3   python install - 4   python install - 5

Once done, we need to create two plain text files into the following folder…

C:\Program Files (x86)\PRTG Network Monitor\Notifications\EXE

 

File 1 – pushover.bat

Create a file called pushover.bat and enter the following code…

C:\Python27\Python "C:\Program Files (x86)\PRTG Network Monitor\Notifications\EXE\pushover.py" %*

Change the listed paths if required.

 

File 2 – pushover.py

Create a file called pushover.py and enter the following code…

import httplib, urllib, sys
 
apiKey = sys.argv[1]
userKey = sys.argv[2]
priority = sys.argv[3]
title = sys.argv[4]
msg = ''
 
for count, arg in enumerate(sys.argv):
    if count > 4:
        msg += arg + '\n'
 
conn = httplib.HTTPSConnection("api.pushover.net:443")
conn.request("POST", "/1/messages.json",
    urllib.urlencode({
        "token": apiKey,
        "user": userKey,
        "message": msg,
        "title": title,
        "priority": priority,
        "retry":"30",
        "expire":"3600"
    }),
    { "Content-type": "application/x-www-form-urlencoded" })
response = conn.getresponse()

 

PRTG Setup

Part one

  1. From your PRTG console,
  2. Click Setup > Account Settings > Notifications,
  3. Click the ‘Add New Notification‘ button on the bottom left,
  4. Add a name for your notification – PRTG – ALERT,
  5. Scroll down and check the Execute Program box,
  6. Choose pushover.bat from the dropdown,
  7. Enter the parameters as follows:

[api-key] [user-key] [priority] “%device – %name : %status” “ALERT/ERROR” “Msg: %message”

Example:  abcd1234efgh5678 ab12cd34ef56gh78 0 “%device – %name : %status” “ALERT/ERROR” “Msg: %message”

(For a list of priority values, check out the Pushover FAQs)

  1. Click Save,
  2. Click Test, and you should receive an alert on your device almost immediately.

 pushover - alarmtest

I have three alerts set up..

  • ALERT/ERROR
  • WARNING
  • ALL OK

 

Part two

Now that we have an alert or three setup, we need to make sure your devices start using them.  For my small setup, I set my notifications at the top level, at the Local Probe…

  1. From the PRTG console, select the Local Probe device (or other device if you want more control),
  2. Select the Notifications tab along the top,
  3. Click Add State Trigger,
  4. From the new trigger then appears, choose the options that bests suits you and your environment.  The no notification drop down should list the alerts you setup above.

prtg notifications - 1

The values I have are shown below…

prtg notifications - 2

 

Errors / Issues / Troubleshooting

When I initially set this up the alerts were not coming though.  In order to troubleshoot this issue, I ran the pushover.py script manually.  Do do this…

  1. Open a command prompt on your PRTG server,
  2. Type the following command into the window…
C:\Python27\Python "C:\Program Files (x86)\PRTG Network Monitor\Notifications\EXE\pushover.py" [api-key] [user-key] 0 "Manual Test" "Ignore This Message"

…making sure to enter your API Key and User Key in the correct locations

  1. You should see no output if everything works OK.

In my case, I got SSL handshake errors.  To fix this, I just visited the Pushover API site (https://api.pushover.net) in order for my server to download the SSL certificate.

 

More Pushover Configuration

I have added another post about using Pushover notifications with my Synology DiskStation and VMware vCenter Alerts.  Read about it here.

NTP Time Service

Introduction

Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems…  NTP is intended to synchronize all participating computers to within a few milliseconds of Coordinated Universal Time (UTC).

Taken from Wikipedia

Making sure you have a reliable and accurate time source can help with mitigating issues on your network.

 

Problem

One of my ESXi hosts was replaced recently, and using my automatic build script I had the new replacement up and running quite quickly.  What my script doesn’t do however, is configure the NTP settings with the host and because of this, it’s internal clock was about 6 hours ahead of the rest of the network.

My PRTG Network Monitoring system is a VM that, via DRS, can move freely amongst my three hosts depending on the current workload.  At 5:00 am the other morning, my monitoring server was moved to the offending host.  This screwed up its monitoring results.

As the VM was migrated to the new host, the time suddenly jumped forward by about 6 hours.  The OS then took over sometime after and jumped back to the correct time.  This time flip-flop caused large gaps in the monitoring results, and left me scratching my head for a while figuring out why.!

 

How to fix the above issues

There were two issues at play here, the host NTP settings were not correct, and the guest VM was set to synchronise it’s time with the host.

 

Configure Host NTP Settings

To enable configure each hosts time setting…

  1. Select your host (if you have more than one, you will need to do this on each one in turn),
  2. Select the Configuration tab along the top of the main section,
  3. Down the left hand menu, near the middle, select Time Configuration,
  4. Click Properties at the top right,
  5. From the first window that appears, make sure the NTP Client Enabled option is selected,
  6. Click Options,
  7. From the General item, choose the middle option, choose Start and stop with host,
  8. Select the NTP Settings item, click Add, and enter the IP address or host name of your chosen NTP host,
  9. Tick the option Restart NTP service to apply changes.

NTP-1

NTP-2   NTP-3

If you don’t have an internal NTP time source, the best option is to use 0.pool.ntp.org1.pool.ntp.org2.pool.ntp.org, and 3.pool.ntp.org.  See the link below for more information, or for geographically local sources…

http://support.ntp.org/bin/view/Servers/NTPPoolServers

 

Disable ‘synchronise guest time with host’

I had my home-lab domain configured to use a specific time source, and to push that out to all the Windows machines.  This is a good idea to configure.  What I had done wrong, was allow the virtual machines guests to have their clocks synchronised with the hosts.  This is generally a bad idea.

  1. Right-click a virtual machine, choose Edit Settings…,
  2. Choose the Options tab along the top,
  3. Select the VMware Tools settings,
  4. Un-tick the Synchronize guest time with host option.

While you are there, it’s also a good idea to tick the other option : Check and upgrade Tools during power cycling.

 

If you have a lot of hosts to change, this is going to be a very tedious task.  Thankfully, there is a script that will handle this for you.

The script below will set both options for you:

  • Line 12: Check and upgrade Tools during power cycling,
  • Line 13: Synchronize guest time with host.

Save the script as a .PS1 file, and execute it within PowerCLI.  Remember to change the first line to enter your vCenter server name or IP address.

Connect-VIServer "ENTER VCENTER SERVER HERE"

# Query for the VM guests
$VMGuests = Get-VM

# Loop through your VM guests, set the VM Tools upgrade checkbox and the Sync Time checkbox to true
ForEach ($VMGuest in $VMGuests) {
    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec
    $spec.changeVersion = $VMGuest.ExtensionData.Config.ChangeVersion
    $spec.tools = New-Object VMWare.Vim.ToolsConfigInfo
    
    $spec.tools.toolsUpgradePolicy = "upgradeAtPowerCycle"
    $spec.tools.syncTimeWithHost = $false

    # Apply the changes
    $MyVM = Get-View -Id $VMGuest.Id
    $MyVM.ReconfigVM_Task($spec)
}

So, there you have it.  Make sure all your VM hosts and guest are using the correct time, and are synchronised with a reliable time source.

PRTG Network Monitor

Paessler PRTG Network MonitorIntroduction

If you want to monitor your home lab, and the devices connected to it, printers, switches, routers, etc, you can’t go wrong with the monitoring solution from Paessler, called PRTG Network Monitor.

 

Monitoring

The PRTG software has a few licencing levels, one of which is a free 3o sensor licence.  Not a free time limited demo, a free forever licence.

30 sensors might sound like a small amount when you find out what a sesnor is…

The licensing options are based on the number of sensors (not on the number of devices or IPs). We define one (1) sensor as any particular, individual monitoring entity. One sensor monitors one network service, one URL, one network connection, one port of a switch, one NetFlow export stream, one CPU load, one disk drive, etc. Please have a look at the list of supported sensor types. Monitoring one item from this list counts as one sensor.

Taken From http://www.paessler.com/support/faqs#e1913

…however, I find that 30 is enough for what I want to monitor on my network.  There are some sensors that “bundle” more that one metric into them.  I could use more if I had them though  🙂

Network Port Monitoring

For example: monitoring a single network port counts as one sensor, but that port could return several metrics.  The HP switch I use returns the following information: Downtime, Transmitted, Packets, Broadcast Packets, Multicast Packets, CRC Errors, Undersize Packets, Oversize Packets, Fragments, Jabbers, Collisions, Packets <= 64 Byte, Packets 65 – 127 Bytes, Packets 128 – 255 Bytes, Packets 256 – 511 Bytes, Packets 512 – 1023 Bytes, Packets 1024 – 1518 Bytes, Drop Events.

That’s a lot of information from one sensor, but expected for a network port.

ESX Performance Monitoring

Another type of “bundled” sensor is the ESX Performance Monitoring one.  This gives much more useful and varied information: CPU usage, Datastore total ReadLatency, Datastore total WriteLatency, Disk read, Disk usage, Disk write, disk.deviceLatency, disk.kernelLatency, Downtime, Memory active, Memory consumed, Memory consumed, Memory swap used, Network received, Network transmitted, Network usage, Power

This is a varied collection of useful information, all collected in one sensor.  It covers the four main metrics: CPU, Memory, Disk, Network.  All of these metrics are available in graph form too, so you can see the history of this information over various time periods, from as little as the last two hours, up to a year ago.

You don’t even need to see all these metrics either.  If you are not interested in the “Power” metric for example, a few clicks later and it’s gone from all your graphs.  The data is still being collected, just not shown to you.

 

My Home Network Monitoring

I use it on my home network to monitor and graph the throughput of my internet router, my wireless router, and my Synology Diskstation 1813+ NAS.  I also have monitors for my three NUC ESXi hosts.

With all the data, this tool collects, you might think it’s hard to find what you are looking for, or just want a way of displaying the data to make it easier to look at.  For this, PRTG has Network Maps.  Below is my network map for my home network…

PRTGNetworkMap

The data shown in the graphs are updated every 30 seconds or so (depending on your settings).  You can see that one of my ESXi hosts is offline at the moment, and that some of my critical virtual machines are listed.

The network map layout can be in almost any layout you can think of.  It doesn’t have to be as pretty as mine.

 

SCOM Maintenance Mode

Introduction

Putting servers into maintenance mode within the Microsoft SCOM console is a long and tedious task, especially if you have lots of servers that need to be done all at once.

In very large environments, open the SCOM management console can take a long time, you then have to search for one of your many servers, right-click, enter maintenance mode, fill out the details required (length, reason), click OK.  Rinse and repeat for your list of servers.

My Solution

I have created an application that allows administrators to import lists of servers, and put them all into maintenance mode, with just a few clicks.  My application makes it quick and easy to search for and find the exact servers you need, add them to a list and export that list for future use.

It was written out of the need of putting large numbers of servers into maintenance mode every week for scheduled and staggered Windows patching.  Looking after 300+ servers (a small deployment), it was a pain to find the required servers and put them in to maintenance every week.

The application starts simply enough, just waiting for a list of servers to be entered.  Either by manually typing, copy/paste a list, or drag and drop and existing text file.

s2007mm-1

Main Screen

If you just want to search SCOM, you can do that too.  Click the “Search SCOM” button.  There are several special search options too.  For example, to find all servers currently in an error state, enter !e into the search field.  The full list is shown below…

!e – servers in an error state
!w – servers in a warning state
!o – servers marked as OK
!m – servers in maintenance mode
!n – servers not being monitored

Once you have found the servers you need, select them and click “Add”.  You can also right-click a server and choose Show Maintenance History to open a new window and see the maintenance history for the selected server.

s2007mm-3

Search SCOM

Searching for any Citrix servers that are currently in an error state: “CTX !E

s2007mm-4

Filtering Search Results

Clicking “Next >” from the main screen will take you to the maintenance options page where you can enter the reason for putting the selected servers into maintenance.

These options, can be exported along with the server names for reuse.

The WhatIf.? option allows you to test the maintenance procedure, without making any changes to your infrastructure.

When you are ready, click “Start/Update

s2007mm-5

Maintenance Options

In the screen shot below, I selected two Citrix Data Collector servers, and put them into maintenance mode.

s2007mm-6

Successful Operation

If your environment has any cluster services, this application will handle putting all the cluster resources into maintenance mode too, a difficult and long winded task in the SCOM console.

 

Change Log

Date Description
2015-03 Added: Show Maintenance History option.  Right click a server from the Search SCOM list.
2015-02 Fixed: Scheduled Task GUI
Fixed: Other GUI Issues
Added: Excluded Servers Option
2014-12 Initial Public Release

 

Download

If you are interested in this tool, please download it from the link below…

download-fileSCOM Maintenance Mode.7z