Unifi Video Controller NVR for UNRAID

If you run UNRAID at home and you have a UniFi Camera system then check out my latest container….

https://hub.docker.com/r/superd/unifi-nvr/

A dockerised UniFi NVR

Ubiquiti UniFi-Video-Controller (NVR) — Docker Container

Ubuntu 16.04, UniFi-Video-Controller 3.8.3

Setup / Quick Start

docker run \
–net=host \
-v /var/lib/unifi-video/:<YOUR DATA DIR> \
-v /var/log/unifi-video/:<YOUR LOG DIR> \

superd/unifi-nvr

Troubleshooting

UNRAID – If you have issue with MongoDB continually restarting please check your data mapping. I have seen issues where user shares do not work correctly. Please try mapping direct to a single disk or to cache drive to ensure smooth operation.

i.e. instead of /mnt/user/usershare/nvr/data

use either /mnt/cache/usershare/nvr/data or /mnt/disk1/usershare/nvr/data

my first few docker containers

This is my first dabbling in creating my OWN git hub (for the code) and docker hub (for the orchestration / build)

https://hub.docker.com/u/superd/

I have created a container for Unifi-Video NVR and storj. Yet to update documentation on Storj container.

I’m currently working on building containers for a news indexer, either newsnab or nzedb. There is another docker project called pynab which was an interesting idea, but seems to have gone stale over time. It used to be almost hands free indexer that ran reasonably efficiently. I’m hoping to re-create something similar soon.

https://github.com/Murodese/pynab

netdata – live server metrics

If your looking for a really good opensource realtime resource monitoring tool check out netdata.io

Very light, looks good and is fast. I have it running as a docker instance on my unraid server (which runs this website, and a couple of other docker instances).

Lots of metrics, check out the website here – https://github.com/firehol/netdata

Docker – Running Ubiquiti NVR and Plex

downloadBye bye virtual machines and their inherent OS bloat. Docker and containerization is here…

The trick to containerization is picking the right workload (as with most things). Think about data, its state and where it lives and whether there are any benefits to running as a container.

Both Ubiquiti’s NVR and Plex’s media server software run’s some base application, this app within its own container then maps to data (which can exist outside the instance) that is consistent.

The fun continues when you can update a container (updating the running application), but keeping the data intact at another location. This can really help with version control etc where you can sometimes just point the new container at the data and turn off the old instance. Rollback? easy. Turn off new container and roll back to old.

Of course things are easier if you are running applications that do not change the data.  Both NVR and Plex only index and capture new data (in consistant format), which makes moving between application versions much easier.

The nature of containerization means that the full power of the host is taken into regard. This is different to regular visualization where each guest is limited to the virtual hardware it is assigned. There are of course challenges where resource is congested, but this can also happen in the latter (cpu scheduling, under / over allocation of resources).

Availability also has to be built with containers in mind, with load balances and instances across multiple hosts.

Update : this site has now now been migrated from a VM to 2 x docker containers…. One for MySQL Backend and one for WordPress FrontEnd. Containers can be linked – so the WordPress container can access MySQL container via its own local port. Very cool.

setup vmware environment

vmwareOne of the scripts i use to standardise our vmware host deployments;


# Author : Darren T
# v1 : Setup vcenter parameters
# Usage ;
# Please manually connect to vCenter, use "Connect-VIServer" -- this promotes usernames and passwords not being saved with script.
# Modify Variables below before running.
# ------------------- VARIABLES -------------------------
# Database Retention Policy
$Days_Retension = "30"
# Set SMTP Server & Sender Account
$SMTP_Server = "192.168.9.10"
$Sender_Account = "administrator@company.co.nz"
# ------------------- CODE ONLY BELOW -------------------
# Database Retention Policy -- Enable limits
# Get-AdvancedSetting -Entity $defaultVIServer | where {$_.Name -match "^task.|^event."}
Write-Host "Setting Database Retension... " -NoNewLine;
Get-AdvancedSetting -Entity $defaultVIServer -Name "event.maxAge" | Set-AdvancedSetting -Value $Days_Retension -Confirm:$false
Get-AdvancedSetting -Entity $defaultVIServer -Name "event.maxAgeEnabled" | Set-AdvancedSetting -Value $true -Confirm:$false
Get-AdvancedSetting -Entity $defaultVIServer -Name "task.maxAge" | Set-AdvancedSetting -Value $Days_Retension -Confirm:$false
Get-AdvancedSetting -Entity $defaultVIServer -Name "task.maxAgeEnabled" | Set-AdvancedSetting -Value $true -Confirm:$false
# Set SMTP Server & Sender Account
Write-Host "Setting SMTP Server & Sender Account... " -NoNewLine;
Get-AdvancedSetting -Entity $defaultVIServer -Name "mail.smtp.server" | Set-AdvancedSetting -Value $SMTP_Server -Confirm:$false
Get-AdvancedSetting -Entity $defaultVIServer -Name "mail.sender" | Set-AdvancedSetting -Value $Sender_Account -Confirm:$false

nested hypervisor (on ESXi 5.1)

Sometimes you might want to run a hypervisor on a hypervisor for testing purposes…. this is how you pass through the required CPU extensions in ESXi 5.1

Remember you will also need to enable promiscuous mode on the networking side also.

How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1

There are some changes with Nested Virtualization in vSphere 5.1 also officially known as VHV (Virtual Hardware-Assisted Virtualization). If you are using vSphere 5.0 to run Nested ESXi or other nested Hypervisors, then please take a look at the instructions in this article. With vSphere 5.1, there have been a few minor changes to enable VHV.

  1. The new Virtual Hardware 9 compatibility will be required when creating your nested ESXi VM, Virtual Hardware 8 will not work if you are running ESXi 5.1 on your physical host. You will still need to enable promiscuous mode on the portgroup that will be used for your nested ESXi VM for network connectivity.
  2. vhv.allow = “true” is no longer valid for ESXi 5.1 to enable VHV. A new parameter has been introduced called vhv.enable = “true” that is now defined on a per VM basis to provide finer granularity of VHV support. This also allows for better portability between VMware’s hosted products such as VMware Fusion and Workstation as they also support the vhv.enable parameter.
  3. You can now enable VHV on a per VM basis and using the new vSphere Web Client which basically adds the vhv.enable = “true” parameter to the VM’s .VMX configuration file.

 

vmware – powercli enable remote scripts

Neccesary commands to get vmware vsphere power cli scripts runnign in power cli;

(post installation of vsphere power cli extensions)

Set-ExecutionPolicy RemoteSigned
Set-PowerCLIConfiguration -InvalidCertificateAction “Ignore” -Confirm:$false

http://blogs.vmware.com/vipowershell/2011/06/back-to-basics-part-1-installing-powercli.html

HP Gen 8 servers and networking issues – TG3 driver

There is a bug in the tg3 driver on the ESXi hosts (1gbit broadcom cards in the new hosts). If the network card is put under load and netqueue is enabled it will sometimes decide to drop all traffic. Essentially i’ve disabled netqueue and the problems have gone away…. as per this vm kb :

http://kb.vmware.com/kb/2035701

The isues will present themselves as log entires like so;

2012-11-19T18:58:52.137Z cpu17:4155)<6>tg3 : vmnic8: RX NetQ allocated on 1
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic8: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic7: RX NetQ allocated on 1
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic7: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:59:12.139Z cpu21:4155)<6>tg3 : vmnic4: NetQ remove RX filter: 1
2012-11-19T18:59:12.139Z cpu21:4155)<6>tg3 : vmnic4: Free NetQ RX Queue: 1
2012-11-19T18:59:22.137Z cpu24:4155)<6>tg3 : vmnic4: RX NetQ allocated on 1
2012-11-19T18:59:22.138Z cpu24:4155)<6>tg3 : vmnic4: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:59:42.138Z cpu21:4155)<6>tg3 : vmnic7: NetQ remove RX filter: 1
2012-11-19T18:59:42.138Z cpu21:4155)<6>tg3 : vmnic7: Free NetQ RX Queue: 1
2012-11-19T18:59:42.140Z cpu21:4155)<6>tg3 : vmnic4: NetQ remove RX filter: 1
2012-11-19T18:59:42.140Z cpu21:4155)<6>tg3 : vmnic4: Free NetQ RX Queue: 1
2012-11-19T19:00:02.139Z cpu28:4155)<6>tg3 : vmnic8: NetQ remove RX filter: 1

vmware – copy networking port groups to another host

You might be able to do this via “host profiles” but if you do not have the licensing for it, this is the alternative. Very handy if you have over 50 or so port groups. It can be re-run to add to additional hosts as needed.

Install powercli, run the following to obtain your current list of virtual port groups off existing host;

Get-VirtualPortGroup -VirtualSwitch vSwitch0 -VMHost esx-01

Grab the output and place into CSV file or copy straght into Excel…

Following the formatting of the following “esx_switching-generic.csv” import the required data from aboves output. Note : the top line is the headers and should always be at the top of the CSV file.

Type,HostName,SwitchName,NIC,PortGroupName,VLAN,IP,Subnet,KernelGW
Portgroup,esx-02,vSwitch0,,Test_Network1,510,,,
Portgroup,esx-02,vSwitch0,,Test_Network2,511,,,

 

Place this file in a location that is called by this script —

#This script is designed to allow you to configure switches on multiple hosts by
#importing information from a prepopulated .csv file. vMotion switch created based
#on Mike Laverick's posting http://www.rtfm-ed.co.uk/?p=1514
#!!!!!!!Values passed for Type are Case sensitive since comparisons are being made.!!!!!!!

$getinfo = Import-Csv "D:\esx_switching-generic.csv" #need to input locatin of .CSV file

#Connect-VIServer -Server #Need to input appropriate vCenter Server

$getinfo | % {
$Type = $_.Type #!!!! Case Sensitive !!!!!!
$gethost = Get-VMHost -Name $_.HostName
$SwitchName = $_.SwitchName
$PortGroup = $_.PortGroupName
$Nic = $_.NIC
$VLAN = $_.VLAN
$IP = $_.IP
$Subnet = $_.Subnet
$kernelGW = $_.KernelGW

If ($Type -eq "Switch") {
$gethost | New-VirtualSwitch -Name $SwitchName -Nic $Nic
}

#Gets Switch object based on the value for SwitchName (required for several cmd-lets that do not accept Strings)
#'If' statement is used since a vMotion type does not already have a switch configured which will throw up an error.
If ($Type -ne "vMotion") {
$getswitch = Get-VirtualSwitch -VMHost $gethost -Name $SwitchName
}

#Add additional NIC to vSwitch to create a Team
If ($Type -eq "Team"){
$getswitch | Set-VirtualSwitch -Nic $Nic
}

#Add Portgroup to existing switch with VLAN
IF ($Type -eq "Portgroup") {
$getswitch | New-VirtualPortGroup $PortGroup -VLanId $VLAN
}

#Creates vMotion switch and configures vmkernel gateway (located under DNS and Routing in configuration tab)
IF ($Type -eq "vMotion") {

$newvswitch = New-VirtualSwitch -VMHost $gethost -Name $SwitchName -Nic $Nic
$vmotion = New-VirtualPortGroup -VirtualSwitch $newvswitch -Name $PortGroup
New-VMHostNetworkAdapter -VMHost $gethost -PortGroup $PortGroup -VirtualSwitch $newvswitch -IP $IP -SubnetMask $subnet -VMotionEnabled: $true

$vmhostnetwork = get-vmhostnetwork $gethost
set-vmhostnetwork -network $vmhostnetwork -vmkernelgateway $kernelGW
}
}

vmware – updating vcenter email alert for monitoring

I’ve used the following powershell script to assist with setting up a consistant email alert on various vmware envionrments.


# Usage ;
# Please manually connect to vCenter, use "Connect-VIServer" -- this promotes usernames and passwords not beings saved with script.
# Update below variable with email(s) use comma as delimita 
$MailtoAddresses= “monitoring@company.com”

#—-These Alarms will send a single email message and not repeat —-
$LowPriorityAlarms=”Timed out starting Secondary VM”,`
“No compatible host for Secondary VM”,`
“Virtual Machine Fault Tolerance vLockStep interval Status Changed”,`
“Migration error”,`
“Exit standby error”,`
“License error”,`
“Virtual machine Fault Tolerance state changed”,`
“VMKernel NIC not configured correctly”,`
“Unmanaged workload detected on SIOC-enabled datastore”,`
“Host IPMI System Event Log status”,`
“Host Baseboard Management Controller status”,`
“License user threshold monitoring”,`
“Datastore capability alarm”,`
“Storage DRS recommendation”,`
“Storage DRS not supported on host”,`
“Datastore is in multiple datacenters”,`
“Insufficient vSphere HA failover resources”,`
“License capacity monitoring”,`
“Pre-4.1 host connected to SIOC-enabled datastore”,`
“Virtual machine cpu usage”,`
“Virtual machine memory usage”,`
“License inventory monitoring”

#—-These Alarms will repeat every 24 hours—-
$MediumPriorityAlarms=`
“Virtual machine error”,`
“Health status changed alarm”,`
“Host cpu usage”,`
“Health status monitoring”,`
“Host memory usage”,`
“Cannot find vSphere HA master agent”,`
“vSphere HA host status”,`
“Host service console swap rates”,`
“vSphere HA virtual machine monitoring action”,`
“vSphere HA virtual machine monitoring error”

#—-These Alarms will repeat every 2 hours—-
$HighPriorityAlarms=`
“Host connection and power state”,`
“Host processor status”,`
“Host memory status”,`
“Host hardware fan status”,`
“Host hardware voltage”,`
“Host hardware temperature status”,`
“Host hardware power status”,`
“Host hardware system board status”,`
“Host battery status”,`
“Status of other host hardware objects”,`
“Host storage status”,`
“Host error”,`
“Host connection failure”,`
“Cannot connect to storage”,`
“Network connectivity lost”,`
“Network uplink redundancy lost”,`
“Network uplink redundancy degraded”,`
“Thin-provisioned LUN capacity exceeded”,`
“Datastore cluster is out of space”,`
“vSphere HA failover in progress”,`
“vSphere HA virtual machine failover failed”,`
“Datastore usage on disk”

#—Set Alarm Action for Low Priority Alarms—
Foreach ($LowPriorityAlarm in $LowPriorityAlarms) {
Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Get-AlarmDefinition -Name “$LowPriorityAlarm” | New-AlarmAction -Email -To @($MailtoAddresses)
# Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Green” -EndStatus “Yellow”
Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Red” # This ActionTrigger is enabled by default.
# Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Red” -EndStatus “Yellow”
# Get-AlarmDefinition -Name “$LowPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Green”
}

#—Set Alarm Action for Medium Priority Alarms—
Foreach ($MediumPriorityAlarm in $MediumPriorityAlarms) {
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Set-AlarmDefinition “$MediumPriorityAlarm” -ActionRepeatMinutes (60 * 24) # 24 Hours
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | New-AlarmAction -Email -To @($MailtoAddresses)
# Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Green” -EndStatus “Yellow”
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | Get-AlarmActionTrigger | Select -First 1 | Remove-AlarmActionTrigger -Confirm:$false
Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Red” -Repeat
# Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Red” -EndStatus “Yellow”
# Get-AlarmDefinition -Name “$MediumPriorityAlarm” | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus “Yellow” -EndStatus “Green”
}

#---Set Alarm Action for High Priority Alarms---
Foreach ($HighPriorityAlarm in $HighPriorityAlarms) {
Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail| Remove-AlarmAction -Confirm:$false
Set-AlarmDefinition "$HighPriorityAlarm" -ActionRepeatMinutes (60 * 2) # 2 hours
Get-AlarmDefinition -Name "$HighPriorityAlarm" | New-AlarmAction -Email -To @($MailtoAddresses)
# Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Green" -EndStatus "Yellow"
Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | Get-AlarmActionTrigger | Select -First 1 | Remove-AlarmActionTrigger -Confirm:$false
Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Red" -Repeat
# Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Red" -EndStatus "Yellow"
# Get-AlarmDefinition -Name "$HighPriorityAlarm" | Get-AlarmAction -ActionType SendEmail | New-AlarmActionTrigger -StartStatus "Yellow" -EndStatus "Green"
}
This is another version i created that just grabs all alarms and sets email trigger. Note that it will delete all the current triggers (so ensure you dont have SNMP traps etc that you need)


# v3 : grab all alarms from vcenter (so should work all versions) and use these for alarm variables
# Usage ;
# Please manually connect to vCenter, use "Connect-VIServer" -- this promotes usernames and passwords not beings saved with script.
# Any alarm that is currently active will send email alert -- please confirm appropriate values for tiggers before running this script.

#Set Notification emails here;
$MailTo= “monitoring@company.com”

#define alarms to be set;
$Alarms = Get-AlarmDefinition | sort Name | select name | ft -HideTableHeaders

foreach ($Alarm in $Alarms)
{
# Delete Trigger;
Get-AlarmDefinition -Name “$Alarm” | Get-AlarmAction | Remove-AlarmAction -Confirm:$false

# Create Trigger;
Get-AlarmDefinition -Name “$Alarm” | New-AlarmAction -Email -To “$MailTo”
}

UPDATE : I generally use the following script now. Less to maintain, and covers any alarms that have not been managed.

# Author : Darren Taylor
# v5 : grab all alarms from vcenter (so should work all versions) and use these for alarm variables
# Usage ;
# Please manually connect to vCenter, use “Connect-VIServer” — this promotes usernames and passwords not being saved with script.
# Any alarm that is currently active will send email alert — please confirm appropriate values for tiggers before running this script.
#
# Note ;
# This script needs to be modified to exclude alarms that are not critical (exclusive rather than inclusive)
# Once Exceptions list is updated, re-run script.

# ——————- VARIABLES ————————-

#Set Notification emails here;
$MailTo= “vcenter@company.com”

# These are the names of the alarms to ignore — i.e. do NOT setup email alert
# THESE ALARMS ARE CONSIDERED NON CRITICAL
$Exceptions= `
“Virtual machine cpu usage”,`
“Virtual machine memory usage”

# ——————- CODE ONLY BELOW ——————-

# TODO:
# Change triggers on some alarms?

#define alarms to be set; (ALL ALARMS)
$Alarms = Get-AlarmDefinition | sort name | select name

foreach ($Alarm in $Alarms)
{

# Test variable in array
Write-Host “Setting Alarm… ” -NoNewLine; Write-Host $Alarm.Name -NoNewLine;

# Delete Trigger; (clears all existing EMAIL triggers)
Get-AlarmDefinition -Name $Alarm.Name | Get-AlarmAction -ActionType:SendEmail | Remove-AlarmAction -Confirm:$false;

# Exceptions to email trigger
$SetAlarm = 1;
foreach ($Exception in $Exceptions) {if($Alarm.Name -eq $Exception){$SetAlarm=0; Write-Host ” Ignored” -foregroundcolor red;}}

# Create Trigger;
if($SetAlarm -eq “1”){Get-AlarmDefinition -Name $Alarm.Name | New-AlarmAction -Email -To $MailTo}

}