Daz's bits and bobs …bytes bits
I’ve been stuck with this error recently on a Celerra manager device : DpRequest_Max_VS_SuspendedCheckpointsReached
This can sometimes happen when a filesystem is held during a replication task. To check what filesystems are currently being replcated run the following command;
nas_replicate -info -all
Either wait until the replication has completed – check status. Or temporary stop the replication task (stop it from the source side if possible).
go here http://blog.scottlowe.org/2010/01/31/emc-celerra-optimizations-for-vmware-on-nfs/
Turn on the uncached write mechanism for NFS file systems used as VMware datastores. This can have a significant performance improvement for VMDKs on NFS but isn’t the default setting. From the Control Station, you can use this command to turn on the uncached write mechanism:
server_mount <data mover name> -option <options>,uncached <file system name> <mount point>
Be sure to review pages 99 through 101 of the VMware on Celerra best practices document for more information on the uncached write mechanism and any considerations for its use.
Straight from the post below, this is the best way (currently) to load balance your NFS datastores… No MPIO magic here unfortunately.
http://communities.vmware.com/message/1466595#1466595
Basically you can setup IP alias on the NFS side and then setup multiple connections (using a unique IP) per datastore on each ESX host. This works well if you are using a team of nics running IP-hash load balancing…
Static EtherChannel.
My setup is as follows:
ESXi 4.0 U1, Cisco 3750 Switches, and NetApp NFS on the storage side.
I have a total of 8 nics. I divided the nics into 3 groups.
2 nics on vSwitch0 for Mgmt & vMotion
3 nics on vSwitch1 for VM’s (Multiple port groups (3 VLANS))
3 nics on vSwitch2 for IP Storage (Mostly NFS, a little iSCSI)
(One vSwitch3 I also have a VM port group for iSCSI access from within the VM)Since I have 3 nics on my IPStorage port group I needed a way to be able to utilize all three nics and not have the server just use one for ingress and egress traffic. This was done by:
Setting up a static EtherChannel on the cisco switch (Port Channel).
Configuring the cisco switch to IP Hash
Configure the vSwitch to “Route based on IP Hash” as well.The next part is to create multiple datastores on the NFS device. Each of my NFS datastores is about 500GB in size. Reason for this is that my larger luns are iSCSI and are access directly from the VM using the MS iSCSI initiator from the VM itself.
My NetApp NAS has an address of, let say, 192.168.1.50. So all my data stores are accessible by utilizing the address of “\\192.168.1.50\NFS-Store#”. This will not be useful as the esx box and the cisco switch will always use the same nic/port to access the nas device. This is due to the algorithm (IP HASH) to decide what link it’ll go over. So to resolve the issue, I added IP aliases to the NFS box. NetApp allows to have multiple Ip addresses pointing to the same NFS export, I suspect EMC would do the same. So, I added 2 aliases 51 & 52. Now my NFS datastores are accessible by using Ip address 192.168.1.50,.51, & .52.So I went ahead and added the datastores to the ESX box using the multiple IP addresses:
Datastore1 = \\192.168.1.50\NFS-Store1
Datastore2 = \\192.168.1.51\NFS-Store2
Datastore3 = \\192.168.1.52\NFS-Store3If you have more datastores it’ll just repeat: Datastore4 = \\192.168.1.50\NFS-Store4 and so on…
Since having multiple datastores and address to each, the 3 nics on the ESX box dedicated to IP Storage get utilized. It does not aggregate the bandwidth but it does use all three to send and recieve packets. So the fastest speed you will get is 1Gbit, theoretically, each way for traffic but, it is better than trying to cram all the traffic over 1 nic.
I also enabled Jumbo Frames on the vSwitch as well as the vmNic for IP-Stroage. (need the best performance!)
I should mention that your NFS storage device should have EtherChannel setup on it as well. Otherwise, you’ll be on the same boat just on the other end of it.Hope it helps!
Larry B.
I should mention that you should not use different addresses to access the same NFS share (datastore). It is not supported and may cause you issues.
I’ve been working through a SRM setup and have been looking at ways to optimize the amount of traffic that is sent over the WAN. The first obvious move is to move your vmware swap files off the replicated LUNS.
Another way is to reduce the sync window. i.e. how often is yoru replication technology trying to keep the source and destination in sync? — Increasing this window can sometimes help you out. But that all depends on your delta’s.
For example – In the case of a windows page file on an active server (SQL etc) it could “potentially” change the whole file within an hour. If your replication was set to every hour and the page file was 4gb then you’d be sending at least 4gb every hour. Changing the sync on your replicated LUN to 8hrs instead would mean you’d only send the 4gb of “delta” (i.e. blocks that have changed since original snap)
Problem is that you would typically want to sync your virtual machines on a more frequent schedule than 8hrs. So this is where you need to move your windows page files onto a separate LUN (also replicated), but on a larger sync window (perhaps only once if your servers are static).
I hit this as a problem recently. One of my disks died in my raidz so i ran down to the store and grabbed me a replacement WD10EARS (Western Digital 1Tb Green) drive.
BUT…
The one thing the store didn’t mention to me is the new 4K cluster sizing on the drive. I guess they assume most people run windows (though the issues are also present in XP). See these posts…
http://blog.temeletry.co.uk/2010/05/wd-green-wd10ears/
Unfortunately they really don’t work as well as you’d like in a server
- They come with a 5 second head spin down setting that causes them to park their heads if they have been left idle for more than 5 seconds. As it takes a second or two to spin back up this can result in a very laggy experience during interactive sessions.
- They do not have NCQ or any form of command queing/optimisation. This means that (on FreeBSD at least) you are stuck in the LOOK elevator. In particular this was noticed when doing sequential read & write (think dump|restore tar|untar etc) and interactive tasks simultaneously
- They really suck with FreeBSD and ZFS…
While the other 512-byte sector HDDs were reading/writing at 30MB/s sustained, this EARS model did not exceeded the 1MB/s barrier.
I know for sure that this is related to the 512-byte sector firmware emulation, because the disk works perfectly well if I partition it in a 4k-sector alignment.
The thing is that even in that way, using it in a ZFS RAIDZ configuration the performance is very poor because RAIDZ uses a dynamic stripe size.
The bottom line here is that folks like me, that use different versions of Unix, need the firmware to present the disk as a 4K-sector disk to unleash the full potential of the technology. The OS is already prepared to support that sector size, no need for emulation here.
http://opensolaris.org/jive/thread.jspa?threadID=125702
Some preliminary testing that I have done…the WD20EARS (2TB advanced format drives) actually presents emulated 512byte sectors to the host o/s.
The drive documentation indicates that jumpers 7-8 should be enabled if the o/s does not support advanced format drives – the drive still present 512 bytes sectors.
I have attempted to raise a support ticket querying this, and how one can disable 512byte sector emulation in the drive (perhaps through a firmware upgrade) but I have not received any response to date.
Hopefully is enough people raise support tickets, WD may release firmware that allows the drive to natively present 4k blocks. Other doco indicates several other jumper combinations – all do not seem to make the drive present 4k byte blocks.
Perhaps someone internal to sun that has a relationship with WD may be able to shed some light on this? It would be fantastic to find out that I was just doing something wrong -> then I can get the drives to be seen on 32bit systems (ie – our embedded kit for osol, velitium)
Tested using b133 (64bit intel).
Try to avoid the green drives in ZFS for now. Remember to do your research before you buy a bunch of disks. I was caught off guard by this small change (works fine in win7 etc) which kills performance in ZFS. Ouch.
