Networking

I couldn’t find how to do this via the web GUI (hate the new 5400 web console - what happened to the performance graphs??) – anyhow this is the commands to do it via commandline…

5406zl-A# show vlan 200 Status and Counters - VLAN Information - Ports - VLAN 200   VLAN ID : 200  Name : VLAN200   Status : Port-based   Voice : No   Jumbo : No

5406zl-A# configure 5406zl-A(config)# vlan 200 jumbo 5406zl-A(config)# end 5406zl-A# sho vlan 200

http://support.microsoft.com/kb/323350

telnet 25 helo me mail from:[email protected] rcpt to:[email protected] data

Type Subject:This is a test, and then press ENTER two times. Type Testing, and then press ENTER. Press ENTER, type a period (.), and then press ENTER.

quit

Thought I’d get a bit of clarification around this one. VMWare states –

VMware recommends that you use multicast mode, because unicast mode forces the physical switches on the LAN to broadcast all Network Load Balancing traffic to every machine on the LAN.

There are some issues which come to the surface if you are using unicast. Where possible we should always try to use a Multicast NLB as Unicast can cause some complications around vSwitch configurations. (requirement to change switch notifications to “No” – and seemingly break port id teaming). Detail below;

I’ve been stuck with this error recently on a Celerra manager device : DpRequest_Max_VS_SuspendedCheckpointsReached

This can sometimes happen when a filesystem is held during a replication task. To check what filesystems are currently being replcated run the following command;

nas_replicate -info -all

Either wait until the replication has completed - check status. Or temporary stop the replication task (stop it from the source side if possible).

go here http://blog.scottlowe.org/2010/01/31/emc-celerra-optimizations-for-vmware-on-nfs/

Turn on the uncached write mechanism for NFS file systems used as VMware datastores. This can have a significant performance improvement for VMDKs on NFS but isn’t the default setting. From the Control Station, you can use this command to turn on the uncached write mechanism: server_mount <data mover name> -option <options>,**uncached** <file system name> <mount point> Be sure to review pages 99 through 101 of the VMware on Celerra best practices document for more information on the uncached write mechanism and any considerations for its use.

Straight from the post below, this is the best way (currently) to load balance your NFS datastores…  No MPIO magic here unfortunately.

http://communities.vmware.com/message/1466595#1466595

Basically you can setup IP alias on the NFS side and then setup multiple connections (using a unique IP) per datastore  on each ESX host. This works well if you are using a team of nics running IP-hash load balancing…

Static EtherChannel.

My setup is as follows:

ESXi 4.0 U1, Cisco 3750 Switches, and NetApp NFS on the storage side.

I have a total of 8 nics. I divided the nics into 3 groups.

2 nics on vSwitch0 for Mgmt & vMotion 3 nics on vSwitch1 for VM’s (Multiple port groups (3 VLANS)) 3 nics on vSwitch2 for IP Storage (Mostly NFS, a little iSCSI) (One vSwitch3 I also have a VM port group for iSCSI access from within the VM)

Since I have 3 nics on my IPStorage port group I needed a way to be able to utilize all three nics and not have the server just use one for ingress and egress traffic. This was done by:

Setting up a static EtherChannel on the cisco switch (Port Channel). Configuring the cisco switch to IP Hash Configure the vSwitch to “Route based on IP Hash” as well.

The next part is to create multiple datastores on the NFS device. Each of my NFS datastores is about 500GB in size. Reason for this is that my larger luns are iSCSI and are access directly from the VM using the MS iSCSI initiator from the VM itself. My NetApp NAS has an address of, let say, 192.168.1.50. So all my data stores are accessible by utilizing the address of “\\192.168.1.50\NFS-Store#”. This will not be useful as the esx box and the cisco switch will always use the same nic/port to access the nas device. This is due to the algorithm (IP HASH) to decide what link it’ll go over. So to resolve the issue, I added IP aliases to the NFS box. NetApp allows to have multiple Ip addresses pointing to the same NFS export, I suspect EMC would do the same. So, I added 2 aliases 51 & 52. Now my NFS datastores are accessible by using Ip address 192.168.1.50,.51, & .52.

So I went ahead and added the datastores to the ESX box using the multiple IP addresses:

Datastore1 = \\192.168.1.50\NFS-Store1 Datastore2 = \\192.168.1.51\NFS-Store2 Datastore3 = \\192.168.1.52\NFS-Store3

If you have more datastores it’ll just repeat: Datastore4 = \\192.168.1.50\NFS-Store4 and so on…

Since having multiple datastores and address to each, the 3 nics on the ESX box dedicated to IP Storage get utilized. It does not aggregate the bandwidth but it does use all three to send and recieve packets. So the fastest speed you will get is 1Gbit, theoretically, each way for traffic but, it is better than trying to cram all the traffic over 1 nic.

I also enabled Jumbo Frames on the vSwitch as well as the vmNic for IP-Stroage. (need the best performance!) I should mention that your NFS storage device should have EtherChannel setup on it as well. Otherwise, you’ll be on the same boat just on the other end of it.

Hope it helps!

Larry B.

I should mention that you should not use different addresses to access the same NFS share (datastore). It is not supported and may cause you issues.

Cisco’s Etherchannel and HP’s LACP are very similar – probably why I assumed both are supported by vmware. But as per below – it is not the case.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048

From a procurve perspective the differences between a “trunk” and “static lacp” trunk is the total bandwidth per connection. A typical end point on a “trunk” trunk can transmit various connections down both pipes, but only receive down one – sometimes refered to as TLB (transmit load balancing). In vmwares case when you are using a “trunk” trunk you will have ip hash set as a load balancer, effectively meaning a different nic will be used on the vmware side for each connection a virtual machine makes.

Check this link for a great script to monitor changes made to a virtual machine via the vmware CBT API. This is perfect for finding culprit machines that are generating a lot of replication traffic if you are replicating over a WAN.

http://www.vmguru.com/index.php/articles-mainmenu-62/scripting/105-using-powershell-to-track-block-change-sizes-over-time

If you have some disks on a virtual machine you don’t want this script to capture then just set them as independent disks (so no snapshots can take place). This is handy if you have your windows page file on a separate disk that you don’t want to be measured as a part of the CBT changes.

I’ve been working through a SRM setup and have been looking at ways to optimize the amount of traffic that is sent over the WAN. The first obvious move is to move your vmware swap files off the replicated LUNS.

Another way is to reduce the sync window. i.e. how often is yoru replication technology trying to keep the source and destination in sync? – Increasing this window can sometimes help you out. But that all depends on your delta’s.

Some very nice examples by Gene on how to setup vyatta as a router on this page;

http://www.sonoracomm.com/support/19-inet-support/233-vyatta-cable

Masquerade Natting;

set service nat rule 10 source address 192.168.77.0/24 set service nat rule 10 outbound-interface eth0 set service nat rule 10 type masquerade

set service nat rule 20 source address 10.200.88.0/24 set service nat rule 20 outbound-interface eth0 set service nat rule 20 type masquerade

Essentially every thing leaving from these two networks will be natted via the eth0 interface. i.e. they will be using the same IP as eth0 for their requests.

type the following to check your default routes…

netstat -rn

Continuing on from this /2010/02/04/vsphere-and-multipathing-iscsi/

you may want to implement jumbo frames to your iSCSI backend…

Enable jumbo frames on your iSCSI target and switches then complete the following on the ESXi hosts (iSCSI initiators)…

This to list your current switch details and port group names…

esxcfg-vswitch -l

The following to allow jumbo frames onto your vswtich (insert your vSwich in following)

esxcfg-vswtich -m 9000 vSwitch0

Then create your kernel port groups that you want to use for iscsi, my commands looked like this… (repeat for each iscsi kernel port you have) – note : DO NOT ADD ANY VMKERNEL(s) from the GUI, instead use the steps below

This is just a quick reference to create a multiplathing iSCSI setup…

Create two virtual kernel switches, one called “iSCSI-1” and the other called “iSCSI-2” (and so on if you have more nics)

Then per kernel portgroup ensure that only one of the nics is active. For the “iSCSI-1” portgroup configure it to override the virtual switch settings and move nic 0 to active and nic 1 to unused. For the “iSCSI-2” portgroup configure it to override the virtual switch settings and move nic 1 to active and nic 0 to unused.

Want iSCSI in opensolaris?

Grab SUNWiscsitgt via package manager.

enable the server via svcadm;

svcadm enable iscsitgt

create your zfs iscsi pool;  (this command will limit iscsi drive to 500GB in size)

zfs create -V 500G tank/iscsi

set isci on via zfs command;

zfs set shareiscsi=on tank/iscsi

check that target is up and running;

iscsitadm list target -v

Done. Should be able to connect via ip from another machine. I have not covered CHAP or any client side configuration. Assumed isolated LAN.

If you’ve setup samba (rather than the builtin cifs/smb) chances are that you can only browse to your samba shares directly on IP only.

If you want your box “network browseable” you’ll need to load nmbd.

You can find a really good write up on modifiying the SMF for samba to autostart the required service here : http://wikis.sun.com/display/BigAdmin/Enabling+Browsing+with+Samba+in+Solaris+10+Update+4

import this file samba.xml

run these commands;

# svcadm disable samba
# svccfg import samba.xml
# svccfg delete svc:/network/samba:default
# svcadm refresh samba:smbd
# svcadm refresh samba:nmbd
# svcadm enable samba:smbd
# svcadm enable samba:nmbd

One of the best articles i have found on this subject is here : http://blog.scottlowe.org/2008/07/16/understanding-nic-utilization-in-vmware-esx/

There is some additional information here on setting up an etherchannel on the cisco side : http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

This can be handy if you need a single VM to use both physical nics in a load-balanced manner - both outbound and inbound. Of course its not really that simple though. This will really only add a benefit if the VM is communicating to multiple destinations (using ip hash - a single destination from a single VM with one IP will always be limited to the same physical nic).

Since im using ZFS as storage via NFS for my some of my vmware environments i need to ensure that latency on my disk is reduced where ever possible.

There is alot of talk about ZFS compression being “faster” than a non-compressed pool due to less physical data being pulled off the drives. This of course depends on the system powering ZFS, but i wanted to run some tests specifically on latency. Throughput is fine in some situations, but latency is a killer when it comes to lots of small reads and writes (in the case of hosting virtual machines)

Otherwise known as trunking or link aggregation. I believe it is the best way to get that additional boost out of your network server while providing a bit of redundancy on link failure. here is how to do it…

Official docs on the process here… http://docs.sun.com/app/docs/doc/819-6990/gdysn?a=view and some good bits here http://blogs.sun.com/nickyv/entry/link_aggregation_jumpstart_post_install

dladm (data link admin) is the tool for the job. List the links you currently have…

dladm show-link

First shut down the links you are currently using..  (you will have to do this on the console)

If you are having problems starting apache on a non-standard port you might find that the problem is related to selinux.

Type this command to check to see what http ports are currently allowed;  (remove filter to show all rules)

semanage port -l|grep http

To add another port type the following (with the port you wish to add etc);

semanage port -a -t http_port_t -p tcp 81

If you want to disable selinux completely then go into /etc/selinux/config and set selinux=disabled. Save then reboot.

Hmm… i’ve been having problems since the 2009.06 (snv_111b) update with cifs.

Cant pin it exactly as it could be “load” related… hmmm.

found this ? http://opensolaris.org/jive/thread.jspa?threadID=107681 this also may be a clue.. http://opensolaris.org/jive/thread.jspa?threadID=92472&tstart=75

imapd ?  might have to go back to 2008.11

You might get better performance if you enable oplocks but there are known issues with it but you can do it just to see if you see any difference:

svccfg -s smb/server setprop smbd/oplock_enable=boolean: true