netdata – live server metrics

If your looking for a really good opensource realtime resource monitoring tool check out netdata.io

Very light, looks good and is fast. I have it running as a docker instance on my unraid server (which runs this website, and a couple of other docker instances).

Lots of metrics, check out below… http://sigtar.com:19999

vmware – updating vcenter email alert for monitoring

I’ve used the following powershell script to assist with setting up a consistant email alert on various vmware envionrments.


# Usage ;
# Please manually connect to vCenter, use "Connect-VIServer" -- this promotes usernames and passwords not beings saved with script.
# Update below variable with email(s) use comma as delimita 
$MailtoAddresses= “monitoring@company.com”

#—-These Alarms will send a single email message and not repeat —-
$LowPriorityAlarms=”Timed out starting Secondary VM”,`
“No compatible host for Secondary VM”,`
“Virtual Machine Fault Tolerance vLockStep interval Status Changed”,`
“Migration error”,`
“Exit standby error”,`
“License error”,`
“Virtual machine Fault Tolerance state changed”,`
“VMKernel NIC not configured correctly”,`
“Unmanaged workload detected on SIOC-enabled datastore”,`
“Host IPMI System Event Log status”,`
“Host Baseboard Management Controller status”,`
“License user threshold monitoring”,`
“Datastore capability alarm”,`
“Storage DRS recommendation”,`
“Storage DRS not supported on host”,`
“Datastore is in multiple datacenters”,`
“Insufficient vSphere HA failover resources”,`
“License capacity monitoring”,`
“Pre-4.1 host connected to SIOC-enabled datastore”,`
“Virtual machine cpu usage”,`
“Virtual machine memory usage”,`
“License inventory monitoring”

#—-These Alarms will repeat every 24 hours—-
$MediumPriorityAlarms=`
“Virtual machine error”,`
“Health status changed alarm”,`
“Host cpu usage”,`
“Health status monitoring”,`
“Host memory usage”,`
“Cannot find vSphere HA master agent”,`
“vSphere HA host status”,`
“Host service console swap rates”,`
“vSphere HA virtual machine monitoring action”,`
“vSphere HA virtual machine monitoring error”

#—-These Alarms will repeat every 2 hours—-
$HighPriorityAlarms=`
“Host connection and power state”,`
“Host processor status”,`
“Host memory status”,`
“Host hardware fan status”,`
“Host hardware voltage”,`
“Host hardware temperature status”,`
“Host hardware power status”,`
“Host hardware system board status”,`
“Host battery status”,`
“Status of other host hardware objects”,`
“Host storage status”,`
“Host error”,`
“Host connection failure”,`
“Cannot connect to storage”,`
“Network connectivity lost”,`
“Network uplink redundancy lost”,`
“Network uplink redundancy degraded”,`
“Thin-provisioned LUN capacity exceeded”,`
“Datastore cluster is out of space”,`
“vSphere HA failover in progress”,`
“vSphere HA virtual machine failover failed”,`
“Datastore usage on disk”

#—Set Alarm Action for Low Priority Alarms—
Foreach ($LowPriorityAlarm in $LowPriorityAlarms) {
Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Get-AlarmDefinition -Name “$LowPriorityAlarm” | New-AlarmAction -Email -To @($MailtoAddresses)
# Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Green” -EndStatus “Yellow”
Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Red” # This ActionTrigger is enabled by default.
# Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Red” -EndStatus “Yellow”
# Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Green”
}

#—Set Alarm Action for Medium Priority Alarms—
Foreach ($MediumPriorityAlarm in $MediumPriorityAlarms) {
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Set-AlarmDefinition “$MediumPriorityAlarm” -ActionRepeatMinutes (60 * 24) # 24 Hours
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | New-AlarmAction -Email -To @($MailtoAddresses)
# Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Green” -EndStatus “Yellow”
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | Get-AlarmActionTrigger | Select -First 1 | Remove-AlarmActionTrigger -Confirm:$false
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Red” -Repeat
# Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Red” -EndStatus “Yellow”
# Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Green”
}

#---Set Alarm Action for High Priority Alarms---
Foreach ($HighPriorityAlarm in $HighPriorityAlarms) {
Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Set-AlarmDefinition "$HighPriorityAlarm" -ActionRepeatMinutes (60 * 2) # 2 hours
Get-AlarmDefinition -Name "$HighPriorityAlarm" | New-AlarmAction -Email -To @($MailtoAddresses)
# Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Green" -EndStatus "Yellow"
Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | Get-AlarmActionTrigger | Select -First 1 | Remove-AlarmActionTrigger -Confirm:$false
Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Red" -Repeat
# Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Red" -EndStatus "Yellow"
# Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Green"
}
This is another version i created that just grabs all alarms and sets email trigger. Note that it will delete all the current triggers (so ensure you dont have SNMP traps etc that you need)


# v3 : grab all alarms from vcenter (so should work all versions) and use these for alarm variables
# Usage ;
# Please manually connect to vCenter, use "Connect-VIServer" -- this promotes usernames and passwords not beings saved with script.
# Any alarm that is currently active will send email alert -- please confirm appropriate values for tiggers before running this script.

#Set Notification emails here;
$MailTo= “monitoring@company.com”

#define alarms to be set;
$Alarms = Get-AlarmDefinition | sort Name | select name | ft -HideTableHeaders

foreach ($Alarm in $Alarms)
{
# Delete Trigger;
Get-AlarmDefinition -Name “$Alarm” | Get-AlarmAction | Remove-AlarmAction -Confirm:$false

# Create Trigger;
Get-AlarmDefinition -Name “$Alarm” | New-AlarmAction -Email -To “$MailTo”
}

UPDATE : I generally use the following script now. Less to maintain, and covers any alarms that have not been managed.

# Author : Darren Taylor
# v5 : grab all alarms from vcenter (so should work all versions) and use these for alarm variables
# Usage ;
# Please manually connect to vCenter, use “Connect-VIServer” — this promotes usernames and passwords not being saved with script.
# Any alarm that is currently active will send email alert — please confirm appropriate values for tiggers before running this script.
#
# Note ;
# This script needs to be modified to exclude alarms that are not critical (exclusive rather than inclusive)
# Once Exceptions list is updated, re-run script.

# ——————- VARIABLES ————————-

#Set Notification emails here;
$MailTo= “vcenter@company.com”

# These are the names of the alarms to ignore — i.e. do NOT setup email alert
# THESE ALARMS ARE CONSIDERED NON CRITICAL
$Exceptions= `
“Virtual machine cpu usage”,`
“Virtual machine memory usage”

# ——————- CODE ONLY BELOW ——————-

# TODO:
# Change triggers on some alarms?

#define alarms to be set; (ALL ALARMS)
$Alarms = Get-AlarmDefinition | sort name | select name

foreach ($Alarm in $Alarms)
{

# Test variable in array
Write-Host “Setting Alarm… ” -NoNewLine; Write-Host $Alarm.Name -NoNewLine;

# Delete Trigger; (clears all existing EMAIL triggers)
Get-AlarmDefinition -Name $Alarm.Name | Get-AlarmAction -ActionType:SendEmail | Remove-AlarmAction -Confirm:$false;

# Exceptions to email trigger
$SetAlarm = 1;
foreach ($Exception in $Exceptions) {if($Alarm.Name -eq $Exception){$SetAlarm=0; Write-Host ” Ignored” -foregroundcolor red;}}

# Create Trigger;
if($SetAlarm -eq “1”){Get-AlarmDefinition -Name $Alarm.Name | New-AlarmAction -Email -To $MailTo}

}