Vmware

Sometimes you might want to run a hypervisor on a hypervisor for testing purposes…. this is how you pass through the required CPU extensions in ESXi 5.1

Remember you will also need to enable promiscuous mode on the networking side also.

http://www.virtuallyghetto.com/2012/08/how-to-enable-nested-esxi-other.html

There are some changes with Nested Virtualization in vSphere 5.1 also officially known as VHV (Virtual Hardware-Assisted Virtualization). If you are using vSphere 5.0 to run Nested ESXi or other nested Hypervisors, then please take a look at the instructions in this article. With vSphere 5.1, there have been a few minor changes to enable VHV.

1. The new Virtual Hardware 9 compatibility will be required when creating your nested ESXi VM, Virtual Hardware 8 will not work if you are running ESXi 5.1 on your physical host. You will still need to enable promiscuous mode on the portgroup that will be used for your nested ESXi VM for network connectivity.
2. vhv.allow = “true” is no longer valid for ESXi 5.1 to enable VHV. A new parameter has been introduced called vhv.enable = “true” that is now defined on a per VM basis to provide finer granularity of VHV support. This also allows for better portability between VMware’s hosted products such as VMware Fusion and Workstation as they also support the vhv.enable parameter.
3. You can now enable VHV on a per VM basis and using the new vSphere Web Client which basically adds the vhv.enable = “true” parameter to the VM’s .VMX configuration file.

I’ve used the following powershell script to assist with setting up a consistant email alert on various vmware envionrments.

# Usage ; # Please manually connect to vCenter, use "Connect-VIServer" -- this promotes usernames and passwords not beings saved with script. # Update below variable with email(s) use comma as delimita $MailtoAddresses= “[email protected]”

#—-These Alarms will send a single email message and not repeat —- $LowPriorityAlarms=“Timed out starting Secondary VM”,` “No compatible host for Secondary VM”,` “Virtual Machine Fault Tolerance vLockStep interval Status Changed”,` “Migration error”,` “Exit standby error”,` “License error”,` “Virtual machine Fault Tolerance state changed”,` “VMKernel NIC not configured correctly”,` “Unmanaged workload detected on SIOC-enabled datastore”,` “Host IPMI System Event Log status”,` “Host Baseboard Management Controller status”,` “License user threshold monitoring”,` “Datastore capability alarm”,` “Storage DRS recommendation”,` “Storage DRS not supported on host”,` “Datastore is in multiple datacenters”,` “Insufficient vSphere HA failover resources”,` “License capacity monitoring”,` “Pre-4.1 host connected to SIOC-enabled datastore”,` “Virtual machine cpu usage”,` “Virtual machine memory usage”,` “License inventory monitoring”

keeping things simple – below are the numbers to look for when your investigating cpu scheduling issues withing a vmware environment…

Look for the cpu “ready” counter which is a percentage of time that the virtual machine was ready, but could not get scheduled to run on the physical CPU. CPU ready time is dependent on the number of virtual machines on the host and their CPU loads.

These are the peak / average figures to watch for in vmware performance graphs. Anything above 5% cpu ready time is worth fixing. (Note : the sampling rate is different depending on which view you are looking at)

Thought I’d get a bit of clarification around this one. VMWare states –

VMware recommends that you use multicast mode, because unicast mode forces the physical switches on the LAN to broadcast all Network Load Balancing traffic to every machine on the LAN.

There are some issues which come to the surface if you are using unicast. Where possible we should always try to use a Multicast NLB as Unicast can cause some complications around vSwitch configurations. (requirement to change switch notifications to “No” – and seemingly break port id teaming). Detail below;

go here http://blog.scottlowe.org/2010/01/31/emc-celerra-optimizations-for-vmware-on-nfs/

Turn on the uncached write mechanism for NFS file systems used as VMware datastores. This can have a significant performance improvement for VMDKs on NFS but isn’t the default setting. From the Control Station, you can use this command to turn on the uncached write mechanism: server_mount <data mover name> -option <options>,**uncached** <file system name> <mount point> Be sure to review pages 99 through 101 of the VMware on Celerra best practices document for more information on the uncached write mechanism and any considerations for its use.

Straight from the post below, this is the best way (currently) to load balance your NFS datastores…  No MPIO magic here unfortunately.

http://communities.vmware.com/message/1466595#1466595

Basically you can setup IP alias on the NFS side and then setup multiple connections (using a unique IP) per datastore  on each ESX host. This works well if you are using a team of nics running IP-hash load balancing…

Static EtherChannel.

My setup is as follows:

ESXi 4.0 U1, Cisco 3750 Switches, and NetApp NFS on the storage side.

I have a total of 8 nics. I divided the nics into 3 groups.

2 nics on vSwitch0 for Mgmt & vMotion 3 nics on vSwitch1 for VM’s (Multiple port groups (3 VLANS)) 3 nics on vSwitch2 for IP Storage (Mostly NFS, a little iSCSI) (One vSwitch3 I also have a VM port group for iSCSI access from within the VM)

Since I have 3 nics on my IPStorage port group I needed a way to be able to utilize all three nics and not have the server just use one for ingress and egress traffic. This was done by:

Setting up a static EtherChannel on the cisco switch (Port Channel). Configuring the cisco switch to IP Hash Configure the vSwitch to “Route based on IP Hash” as well.

The next part is to create multiple datastores on the NFS device. Each of my NFS datastores is about 500GB in size. Reason for this is that my larger luns are iSCSI and are access directly from the VM using the MS iSCSI initiator from the VM itself. My NetApp NAS has an address of, let say, 192.168.1.50. So all my data stores are accessible by utilizing the address of “\\192.168.1.50\NFS-Store#”. This will not be useful as the esx box and the cisco switch will always use the same nic/port to access the nas device. This is due to the algorithm (IP HASH) to decide what link it’ll go over. So to resolve the issue, I added IP aliases to the NFS box. NetApp allows to have multiple Ip addresses pointing to the same NFS export, I suspect EMC would do the same. So, I added 2 aliases 51 & 52. Now my NFS datastores are accessible by using Ip address 192.168.1.50,.51, & .52.

So I went ahead and added the datastores to the ESX box using the multiple IP addresses:

Datastore1 = \\192.168.1.50\NFS-Store1 Datastore2 = \\192.168.1.51\NFS-Store2 Datastore3 = \\192.168.1.52\NFS-Store3

If you have more datastores it’ll just repeat: Datastore4 = \\192.168.1.50\NFS-Store4 and so on…

Since having multiple datastores and address to each, the 3 nics on the ESX box dedicated to IP Storage get utilized. It does not aggregate the bandwidth but it does use all three to send and recieve packets. So the fastest speed you will get is 1Gbit, theoretically, each way for traffic but, it is better than trying to cram all the traffic over 1 nic.

I also enabled Jumbo Frames on the vSwitch as well as the vmNic for IP-Stroage. (need the best performance!) I should mention that your NFS storage device should have EtherChannel setup on it as well. Otherwise, you’ll be on the same boat just on the other end of it.

Hope it helps!

Larry B.

I should mention that you should not use different addresses to access the same NFS share (datastore). It is not supported and may cause you issues.

Cisco’s Etherchannel and HP’s LACP are very similar – probably why I assumed both are supported by vmware. But as per below – it is not the case.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004048

From a procurve perspective the differences between a “trunk” and “static lacp” trunk is the total bandwidth per connection. A typical end point on a “trunk” trunk can transmit various connections down both pipes, but only receive down one – sometimes refered to as TLB (transmit load balancing). In vmwares case when you are using a “trunk” trunk you will have ip hash set as a load balancer, effectively meaning a different nic will be used on the vmware side for each connection a virtual machine makes.

fire up the vm, then run the following after initiating a vmware tools install…

mount /dev/cdrom /mnt/cdrom cd /tmp tar zxf /mnt/cdrom/VMwareTools-x.x.x.gz cd vmware-tools-disstrib ./vmware-install.pl

Then just follow the prompts through to the end.

If your running fedora or similar make sure your’ve got gcc and kernel headers…. (you’ll probably have to update kernel too)

yum update shutdown -r now yum install -y gcc make kernel-devel perl

Ubuntu 12.x

apt-get install open-vm-tools

Check this link for a great script to monitor changes made to a virtual machine via the vmware CBT API. This is perfect for finding culprit machines that are generating a lot of replication traffic if you are replicating over a WAN.

http://www.vmguru.com/index.php/articles-mainmenu-62/scripting/105-using-powershell-to-track-block-change-sizes-over-time

If you have some disks on a virtual machine you don’t want this script to capture then just set them as independent disks (so no snapshots can take place). This is handy if you have your windows page file on a separate disk that you don’t want to be measured as a part of the CBT changes.

I’ve been working through a SRM setup and have been looking at ways to optimize the amount of traffic that is sent over the WAN. The first obvious move is to move your vmware swap files off the replicated LUNS.

Another way is to reduce the sync window. i.e. how often is yoru replication technology trying to keep the source and destination in sync? – Increasing this window can sometimes help you out. But that all depends on your delta’s.

This page is hard to find, but it has it all…  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008705

To generate and import an SSL certificate on the VDM server:

Note: Certificates are only required for client‐facing systems (that is, standard, replica, or security VDM servers).

1. All the commands related to certificates need to be run in System Context. From a command prompt, run:at (In 24 hour format) /interactive cmd.exeFor example:at 21:45 /interactive cmd.exeThis opens a new command window in System Context.

2. On the VDM or security server, change the directory to C:\Program Files\VMware\VMware VDM\Server\jre\bin.

3. Run the following command to create a 1024bit self-signed SSL certificate:keytool -genkey -keyalg “RSA” -keystore keys.p12 -storetype pkcs12 -validity 360To generate 2048bit certificate, run the command:keytool -genkey -keyalg “RSA” -keysize 2048 -keystore keys.p12 -storetype pkcs12 -validity 360Note: The keys.p12 file is created in the current directory.

4. Run the following command to create a certificate signing request (CSR):keytool -certreq -keyalg “RSA” -file certificate.csr -keystore keys.p12 -storetype pkcs12 -storepass , w here is the keystore passwordNote: The certificate.csr file is created in the same location.

5. Send the CSR file (certificate.csr) to a certificate authority and request a certificate in PKCS7 format.Note: If the certificate authority does not offer PKCS7 as a format, see Exporting into PKCS7 format in this article for information about exporting the certificate data in the appropriate format.

6. Copy the contents of the file into a text editor and save it as certificate.p7. The content appears similar to:—–BEGIN PKCS7—– MIIF+AYJKoZIhvcNAQcCoIIF6TCCBeUCAQExADALBgkqhkiG9w0BBwGgggXNMIID LDCCApWgAwIBAgIQTpY7DsV1n1HeMGgMjMR2PzANBgkqhkiG9w0BAQUFADCBhzEL i7coVx71/lCBOlFmx66NyKlZK5mObgvd2dlnsAP+nnStyhVHFIpKy3nsDO4JqrIg EhCsdpikSpbtdo18jUubV6z1kQ71CrRQtbi/WtdqxQEEtgZCJO2lPoIWMQA= —–END PKCS7—–

   Note: For more information about importing certificates issued by an OpenSSL CA, see Importing certificates issued by an OpenSSL CA (1007390).

7. Run the following command to import certificates issued by CA:keytool -import -keystore keys.p12 -storetype pkcs12 -storepass -keyalg “RSA” -trustcacerts -file certificate.p7Where is the keystore password (the password you used when exporting the certificate).

8. To configure the View Connection Server to use the new certificate, place the new certificate and copy the keys.p12 file (the keystore file) in C:\Program Files\VMware\View Manager\Server\sslgateway\conf.

9. Modify or create the following file:C:\ProgramFiles\VMware\View Manager\Server\sslgateway\conf\locked.properties

10. Add the following properties:keyfile=keys.p12 keypass=

    Where is the keystore password (the password you used when exporting the certificate).

11. Restart the View Connection Server service.

Exporting into the PKCS7 format

To export into the PKXS7 format:

1. Open the certificate file.

2. Click Details > Copy to File.

3. Click PKCS7, then select Include Details of Certificate.

   The file is saved with a .P7B extension.

This happens when the virtualcenter service tries to start before some of its dependencies are up and running. Typically due to SQL Server not being in a running state when it tries to startup.

If your in windows 2008 you can choose the service to start automatically (delayed), and most of the time this will resolve the problem. If you want to do it properly, and in a more clean way…. you’ll need to add SQL as a dependency.

I picked this trick up off vmware support. If you’ve got your iscsi all setup you can drop to the shell (either ssh or console) and do this to measure your average write throughput.

time vmkfstools -c 10G /vmfs/volumes/san_vmfs/my_vm/fat_disk.vmdk -d eagerzeroedthick

Try larger a larger disk if this is too quick (free space permitting)

Essentially this will initiate the host to create a fat disk in the location above. You will then get a time recorded on how long it takes to execute this command. Then you can use your maths skill to work out the transfer rate…

I’ve run into some issues lately with XP guests that are running off a IDE controller. If you want any kind of performance you must build your machine with the LSI drivers (parallel).

Here is a flp image with the required LSI drivers (Press F6 etc on xp boot)

http://communities.vmware.com/message/1499140#1499140

V2V cold clone process for SBS / exchange / sql / DC’s etc…..    (works in vSphere)

Steps to convert from hyper-v guest to vmware guest

• Remove hyper-v integrated services while hyper-v guest is running. (if possible)
• Note down NIC networking details first
• Clean shutdown hyper-v guest
• Use StarWind Converter (free) to convert VHD to “dynamically growing VMware” image and use “IDE” as type. Note “SCSI” does NOT work if you have not got the LSI / BUS drivers in the source image, just use the default options – dynamic vmware and IDE.
• Create VMware guest shell (virtual machine equivalent to hyper-v specs but without the disks) – remove all hard drives, iscsi controller should also disappear.
• Upload the VMDK to same data store as virtual machine and attach to the virtual guest in the right order. They should be detected as “IDE”
• Start new vmware guest
• If you could not remove integrated services (i.e. hot clone), then you MUST disable hyper-v services immediately.
• Install vmware tools – reboot
• Remove hidden NIC and other orphaned devices, and reconfigure network card as per original.
• Windows may need to be re-activated - do this.

Note : converting a machine to use an IDE controller will limit its performance!

Here is a good presentation about vmwares DVS technology. Essentially a new way to create a vSwitch over a cluster and just add the required hosts into it.

http://download3.vmware.com/vdcos/demos/DVS_Demo_800x600.html

Goal was to create a vMA which would send a shutdown signal to all the virtual hosts it knows about. I’ve assumed that ESX and management agents that live within the service console will eventually be phased out.

The steps…

1. Download and install the vMA from vmware - fire up the vm and setup the basic networking required
2. Create trusts within the vMA to each of the vmware hosts you wish to manage (sudo vifp addserver )
3. Install HP Power Protector (linux agent) - you could substitute for your UPS client software
4. Modify the shutdown script (may differ per vendor) to shutdown the esxboxs- “SDScript” (hostops.pl)

The 1st step is easy enough. Find the download at vmware and install (Deploy OVF Template…)

Most of the time your HA issues are going to be DNS related. So ensure that your vcenter can ping all your hosts by FQDN without issue.  In some cases though a stubborn server may not want to play the game even when everything is configured properly.

This method is considered a “last effort” as you’ll need to run some CLI commands on the ESX box. But i have found it useful in a few situations.

This bug has hit me. Looks like users with roles like vm user / power user cannot send “ctrl-alt-del” via the console even though they have the correct permissions. Our users cannot use ctrl-alt-ins as they are connected via RDP to a machine that has the console installed.

Found this : http://communities.vmware.com/thread/220683;jsessionid=480C8A2C9B9EACA9FF2BB4E1BECA2D53?start=15&tstart=0

Looks like its a known bug and will be fixed in the upcoming VC4.0 update 1 sometime Q3 2009 :(

Since im using ZFS as storage via NFS for my some of my vmware environments i need to ensure that latency on my disk is reduced where ever possible.

There is alot of talk about ZFS compression being “faster” than a non-compressed pool due to less physical data being pulled off the drives. This of course depends on the system powering ZFS, but i wanted to run some tests specifically on latency. Throughput is fine in some situations, but latency is a killer when it comes to lots of small reads and writes (in the case of hosting virtual machines)