HP Gen 8 servers and networking issues – TG3 driver

There is a bug in the tg3 driver on the ESXi hosts (1gbit broadcom cards in the new hosts). If the network card is put under load and netqueue is enabled it will sometimes decide to drop all traffic. Essentially i’ve disabled netqueue and the problems have gone away…. as per this vm kb :

http://kb.vmware.com/kb/2035701

The isues will present themselves as log entires like so;

2012-11-19T18:58:52.137Z cpu17:4155)<6>tg3 : vmnic8: RX NetQ allocated on 1
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic8: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic7: RX NetQ allocated on 1
2012-11-19T18:58:52.138Z cpu17:4155)<6>tg3 : vmnic7: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:59:12.139Z cpu21:4155)<6>tg3 : vmnic4: NetQ remove RX filter: 1
2012-11-19T18:59:12.139Z cpu21:4155)<6>tg3 : vmnic4: Free NetQ RX Queue: 1
2012-11-19T18:59:22.137Z cpu24:4155)<6>tg3 : vmnic4: RX NetQ allocated on 1
2012-11-19T18:59:22.138Z cpu24:4155)<6>tg3 : vmnic4: NetQ set RX Filter: 1 [00:50:56:71:46:87 0]
2012-11-19T18:59:42.138Z cpu21:4155)<6>tg3 : vmnic7: NetQ remove RX filter: 1
2012-11-19T18:59:42.138Z cpu21:4155)<6>tg3 : vmnic7: Free NetQ RX Queue: 1
2012-11-19T18:59:42.140Z cpu21:4155)<6>tg3 : vmnic4: NetQ remove RX filter: 1
2012-11-19T18:59:42.140Z cpu21:4155)<6>tg3 : vmnic4: Free NetQ RX Queue: 1
2012-11-19T19:00:02.139Z cpu28:4155)<6>tg3 : vmnic8: NetQ remove RX filter: 1

When Is A Trunk Not A Trunk?

Very good article here – http://networkingnerd.net/2011/02/02/when-is-a-trunk-not-a-trunk/

Particuly like setting of Native vLAN and tagged networks over an HP trunk;

So, if HP refers to an uplink carrying multiple VLANs are a tagged port, then does HP have a “trunk”? In fact they do. In HPvania, a trunk is a logical construct that aggregates multiple ports into one logical link. For those of you that might be out there scratching your heads about this one, this means that when you “trunk” a group of ports on an HP switch, you are creating one LACP link from up to four individual ports. This kind of configuration should look like this:

Switch(config)#trunk 19-24
Switch(config)#trk1
Switch(config-trk)#lacp
Switch(config-trk)#vlan 1
Swtich(config-vlan)#untagged trk1
Swtich(config-vlan)#vlan 10
Swtich(config-vlan)#tagged trk1
Swtich(config-vlan)#vlan 99
Swtich(config-vlan)#tagged trk1

Those of you that are fans of irony will appreciate that the above config sets up this LACP port aggregation to pass multiple VLANs to another switch. In other words, we are configuring a Cisco “trunk” on top of an HP “trunk”.

Cisco VTP on HP Switches?

 

Want to propergate vLan information across multiple switches?

Easy in the Cisco world…  Just use VTP – configure a server switch and  your away… (it also has its risks if you introduce another server switch with a lower ID)

What does HP have?  — GVRP (MVRP newer iteration).

http://en.wikipedia.org/wiki/Multiple_Registration_Protocol

Multiple VLAN Registration Protocol (MVRP), which replaced GARP VLAN Registration Protocol (GVRP), is a standards-based Layer 2 network protocol, for automatic configuration of VLAN information on switches. It was defined in the 802.1ak amendment to 802.1Q-2005.

MSTP – nice blog on its inner workings here – http://blog.ine.com/2010/02/22/understanding-mstp/

 


HP Blade Switch – enable crossover

HP GbE2c Ethernet Blade Switch for HP c-Class BladeSystem User Guide
Click here to view the “HP GbE2c Ethernet Blade Switch for HP c-Class BladeSystem User Guide” (http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c00701541/c00701541.pdf)
HP GbE2c Layer 2/3 Ethernet Blade Switch for c-Class BladeSystem User Guide
Click here to go to the “HP GbE2c Layer 2/3 Ethernet Blade Switch for c-Class BladeSystem User Guide” (http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c00865010/c00865010.pdf)

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00700745/c00700745.pdf

Crosslinks

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00816246/c00816246.pdf

HP ProCurve 5400zl Switch Series – How to Enable Jumbo Frames?

I couldn’t find how to do this via the web GUI (hate the new 5400 web console – what happened to the performance graphs??) — anyhow this is the commands to do it via commandline…

5406zl-A# show vlan 200
Status and Counters – VLAN Information – Ports – VLAN 200   VLAN ID : 200  Name : VLAN200   Status : Port-based   Voice : No   Jumbo : No

5406zl-A# configure
5406zl-A(config)# vlan 200 jumbo
5406zl-A(config)# end
5406zl-A# sho vlan 200

Status and Counters – VLAN Information – Ports – VLAN 200  VLAN ID : 200   Name : VLAN200   Status : Port-based   Voice : No  Jumbo : Yes

From a windows box…

ping 10.10.9.1 -f -l 8972

vmware – hp procurve lacp / trunk

Cisco’s Etherchannel and HP’s LACP are very similar – probably why I assumed both are supported by vmware. But as per below – it is not the case.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048

From a procurve perspective the differences between a “trunk” and “static lacp” trunk is the total bandwidth per connection. A typical end point on a “trunk” trunk can transmit various connections down both pipes, but only receive down one – sometimes refered to as TLB (transmit load balancing). In vmwares case when you are using a “trunk” trunk you will have ip hash set as a load balancer, effectively meaning a different nic will be used on the vmware side for each connection a virtual machine makes.

This most probably explains why we hit a 1GBit limit per connection over vmware since a switch “trunk” can only receive back down a single interface. Where as LACP the interface is the team itself (i.e. it is considered as a larger single interface) – and can load balance. Matches up with what i’ve seen on the live switch statistics.

HP’s LACP – should be used where possible – between switches and servers that support it. LACP is a protocol that is wrapped around packets (why both ends need to support it).

“trunk” trunks  – have to be used with vmware at the moment – limits each connections bandwidth to a single interface. (i.e. you will never get more than 1gbit per connection if your nics are all 1gbit).

vMA – Using HP Power Protector agent to shutdown virtual hosts

Goal was to create a vMA which would send a shutdown signal to all the virtual hosts it knows about. I’ve assumed that ESX and management agents that live within the service console will eventually be phased out.

The steps…

  1. Download and install the vMA from vmware – fire up the vm and setup the basic networking required
  2. Create trusts within the vMA to each of the vmware hosts you wish to manage (sudo vifp addserver <server>)
  3. Install HP Power Protector (linux agent) – you could substitute for your UPS client software
  4. Modify the shutdown script (may differ per vendor) to shutdown the esxboxs- “SDScript” (hostops.pl)

The 1st step is easy enough. Find the download at vmware and install (Deploy OVF Template…)

Once up and running you need to create the trust between the vMA and your ESX hosts. Logon using vi-admin and your password. then;

sudo vifp addserver esxbox1 (you will be prompted for the hosts root password)

Do this for each box you wish the vMA to manage.

Once you have installed your UPS agent into the vMA (linux client should work without issue) the next step is to modify the shutdown script to do the work. Within the script you will need the following in it…

vifpinit esxbox1

This sets the context of vMA to this host. Note: the server should have been added as one of the managed

/usr/lib/vmware-vcli/apps/host/hostops.pl –target_host esxbox1 –operation enter_maintenance –url https:///sdk/vimService.wsdl

Note: you need to use the actual name of the host and not its IP. You can get the exact name of the

The HP power protector agent script is located at /usr/local/DevMan/SDScript

If you want to ensure all your guests shutdown cleanly, enable “shutdown virtual machines on host shutdown”. Also note that if you shutdown the host that the vMA is running on it will kill the script. So shutdown the host that the vMA is running on last (remove vm from DRS)

Update (08/04/2010): I have found the above vMA to be quite fiddly. And have had much better luck with the PowerCLI code found on this page : http://www.virtu-al.net/2010/01/06/powercli-shutdown-your-virtual-infrastructure/

I have made some slight mods but essentaillly…

$VCENTER = "vcenter"
Connect-VIServer $VCENTER

# Get All the ESX Hosts
$ESXSRV = Get-VMHost

# For each of the VMs on the ESX hosts (excluding virtual center box)
Foreach ($VM in ($ESXSRV | Get-VM)){
    # Shutdown the guest cleanly
    if ($VM -match $VCENTER){}
    else {$VM | Shutdown-VMGuest -Confirm:$false}
}

# Set the amount of time to wait before assuming the remaining powered on guests are stuck

$WaitTime = 120 #Seconds

do {
    # Wait for the VMs to be Shutdown cleanly
    sleep 1.0
    $WaitTime = $WaitTime - 1
    $numvms = ($ESXSRV | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count
    Write "Waiting for shutdown of $numvms VMs or $WaitTime seconds"
   
    } until ((@($ESXSRV | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count) -eq 0 -or $WaitTime -eq 0)

# Shutdown the ESX Hosts - and remaining virtual center box (if virtual)
$ESXSRV | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}

Write-Host "Shutdown Complete"

# If virtual center box is physical and still alive it will need to be shutdown...

Write-Host "Shutting down virtual center"
shutdown -s -f -t 1