vmware – replacing security server ssl cert

This page is hard to find, but it has it all…  http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008705

To generate and import an SSL certificate on the VDM server:

Note: Certificates are only required for client‐facing systems (that is, standard, replica, or security VDM servers).
  1. All the commands related to certificates need to be run in System Context. From a command prompt, run:at <time> (In 24 hour format) /interactive cmd.exeFor example:at 21:45 /interactive cmd.exeThis opens a new command window in System Context.
  2. On the VDM or security server, change the directory to C:\Program Files\VMware\VMware VDM\Server\jre\bin.
  3. Run the following command to create a 1024bit self-signed SSL certificate:keytool -genkey -keyalg “RSA” -keystore keys.p12 -storetype pkcs12 -validity 360To generate 2048bit certificate, run the command:keytool -genkey -keyalg “RSA” -keysize 2048 -keystore keys.p12 -storetype pkcs12 -validity 360Note: The keys.p12 file is created in the current directory.
  4. Run the following command to create a certificate signing request (CSR):keytool -certreq -keyalg “RSA” -file certificate.csr -keystore keys.p12 -storetype pkcs12 -storepass <secret> , w here <secret> is the keystore passwordNote: The certificate.csr file is created in the same location.
  5. Send the CSR file (certificate.csr) to a certificate authority and request a certificate in PKCS7 format.Note: If the certificate authority does not offer PKCS7 as a format, see Exporting into PKCS7 format in this article for information about exporting the certificate data in the appropriate format.
  6. Copy the contents of the file into a text editor and save it as certificate.p7. The content appears similar to:—–BEGIN PKCS7—–
    —–END PKCS7—–

    Note: For more information about importing certificates issued by an OpenSSL CA, see Importing certificates issued by an OpenSSL CA (1007390).

  7. Run the following command to import certificates issued by CA:keytool -import -keystore keys.p12 -storetype pkcs12 -storepass <secret> -keyalg “RSA” -trustcacerts -file certificate.p7Where <secret> is the keystore password (the password you used when exporting the certificate).
  8. To configure the View Connection Server to use the new certificate, place the new certificate and copy the keys.p12 file (the keystore file) in C:\Program Files\VMware\View Manager\Server\sslgateway\conf.
  9. Modify or create the following file:C:\ProgramFiles\VMware\View Manager\Server\sslgateway\conf\locked.properties
  10. Add the following properties:keyfile=keys.p12

    Where <secret> is the keystore password (the password you used when exporting the certificate).

  11. Restart the View Connection Server service.

Exporting into the PKCS7 format

To export into the PKXS7 format:
  1. Open the certificate file.
  2. Click Details > Copy to File.
  3. Click PKCS7, then select Include Details of Certificate.

    The file is saved with a .P7B extension.

vmware – virtualcenter server service fails to start on boot

This happens when the virtualcenter service tries to start before some of its dependencies are up and running. Typically due to SQL Server not being in a running state when it tries to startup.

If your in windows 2008 you can choose the service to start automatically (delayed), and most of the time this will resolve the problem. If you want to do it properly, and in a more clean way…. you’ll need to add SQL as a dependency.

Open regedit and go to the following key;    (find the sevice name that virtual cetner is running under – vpxd in this case)


Open and modify the “Depend on Service” Multi-String and append the following (specific to my environment, but Adam for vmware and sqlserver)


So my DependOnService Multi-String now reads;



On next reboot the virtual center service will wait for its dependencies to start before it tries to start itself.

vmware – measuring iscsi write performance

I picked this trick up off vmware support. If you’ve got your iscsi all setup you can drop to the shell (either ssh or console) and do this to measure your average write throughput.

time vmkfstools -c 10G /vmfs/volumes/san_vmfs/my_vm/fat_disk.vmdk -d eagerzeroedthick

Try larger a larger disk if this is too quick (free space permitting)

Essentially this will initiate the host to create a fat disk in the location above. You will then get a time recorded on how long it takes to execute this command. Then you can use your maths skill to work out the transfer rate…

While this is happening you can open another SSH type esxtop then press “d” and watch the (d)isk throughput on the console. Pressing “v” will show you stats per (v)irtual machine.

hyper-v to esx conversion

V2V cold clone process for SBS / exchange / sql / DC’s etc…..    (works in vSphere)

Steps to convert from hyper-v guest to vmware guest

  • Remove hyper-v integrated services while hyper-v guest is running. (if possible)
  • Note down NIC networking details first
  • Clean shutdown hyper-v guest
  • Use StarWind Converter (free) to convert VHD to “dynamically growing VMware” image and use “IDE” as type. Note “SCSI” does NOT work if you have not got the LSI / BUS drivers in the source image, just use the default options – dynamic vmware and IDE.
  • Create VMware guest shell (virtual machine equivalent to hyper-v specs but without the disks) – remove all hard drives, iscsi controller should also disappear.
  • Upload the VMDK to same data store as virtual machine and attach to the virtual guest in the right order. They should be detected as “IDE”
  • Start new vmware guest
  • If you could not remove integrated services (i.e. hot clone), then you MUST disable hyper-v services immediately.
  • Install vmware tools – reboot
  • Remove hidden NIC and other orphaned devices, and reconfigure network card as per original.
  • Windows may need to be re-activated – do this.

Note : converting a machine to use an IDE controller will limit its performance!

vMA – Using HP Power Protector agent to shutdown virtual hosts

Goal was to create a vMA which would send a shutdown signal to all the virtual hosts it knows about. I’ve assumed that ESX and management agents that live within the service console will eventually be phased out.

The steps…

  1. Download and install the vMA from vmware – fire up the vm and setup the basic networking required
  2. Create trusts within the vMA to each of the vmware hosts you wish to manage (sudo vifp addserver <server>)
  3. Install HP Power Protector (linux agent) – you could substitute for your UPS client software
  4. Modify the shutdown script (may differ per vendor) to shutdown the esxboxs- “SDScript” (hostops.pl)

The 1st step is easy enough. Find the download at vmware and install (Deploy OVF Template…)

Once up and running you need to create the trust between the vMA and your ESX hosts. Logon using vi-admin and your password. then;

sudo vifp addserver esxbox1 (you will be prompted for the hosts root password)

Do this for each box you wish the vMA to manage.

Once you have installed your UPS agent into the vMA (linux client should work without issue) the next step is to modify the shutdown script to do the work. Within the script you will need the following in it…

vifpinit esxbox1

This sets the context of vMA to this host. Note: the server should have been added as one of the managed

/usr/lib/vmware-vcli/apps/host/hostops.pl –target_host esxbox1 –operation enter_maintenance –url https:///sdk/vimService.wsdl

Note: you need to use the actual name of the host and not its IP. You can get the exact name of the

The HP power protector agent script is located at /usr/local/DevMan/SDScript

If you want to ensure all your guests shutdown cleanly, enable “shutdown virtual machines on host shutdown”. Also note that if you shutdown the host that the vMA is running on it will kill the script. So shutdown the host that the vMA is running on last (remove vm from DRS)

Update (08/04/2010): I have found the above vMA to be quite fiddly. And have had much better luck with the PowerCLI code found on this page : http://www.virtu-al.net/2010/01/06/powercli-shutdown-your-virtual-infrastructure/

I have made some slight mods but essentaillly…

$VCENTER = "vcenter"
Connect-VIServer $VCENTER

# Get All the ESX Hosts
$ESXSRV = Get-VMHost

# For each of the VMs on the ESX hosts (excluding virtual center box)
Foreach ($VM in ($ESXSRV | Get-VM)){
    # Shutdown the guest cleanly
    if ($VM -match $VCENTER){}
    else {$VM | Shutdown-VMGuest -Confirm:$false}

# Set the amount of time to wait before assuming the remaining powered on guests are stuck

$WaitTime = 120 #Seconds

do {
    # Wait for the VMs to be Shutdown cleanly
    sleep 1.0
    $WaitTime = $WaitTime - 1
    $numvms = ($ESXSRV | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count
    Write "Waiting for shutdown of $numvms VMs or $WaitTime seconds"
    } until ((@($ESXSRV | Get-VM | Where { $_.PowerState -eq "poweredOn" }).Count) -eq 0 -or $WaitTime -eq 0)

# Shutdown the ESX Hosts - and remaining virtual center box (if virtual)
$ESXSRV | Foreach {Get-View $_.ID} | Foreach {$_.ShutdownHost_Task($TRUE)}

Write-Host "Shutdown Complete"

# If virtual center box is physical and still alive it will need to be shutdown...

Write-Host "Shutting down virtual center"
shutdown -s -f -t 1

vmware – HA issues

Most of the time your HA issues are going to be DNS related. So ensure that your vcenter can ping all your hosts by FQDN without issue.  In some cases though a stubborn server may not want to play the game even when everything is configured properly.

This method is considered a “last effort” as you’ll need to run some CLI commands on the ESX box. But i have found it useful in a few situations.

This page has a great write up on which files HA uses and how to temporary stop the HA service. http://itknowledgeexchange.techtarget.com/virtualization-pro/vmware-ha-failure-got-you-down/

Remember to get to the console on ESXi you logon to the console press Alt-F1 then type “unsupported” (note: you cannot see what you are typing), then enter the root password.

The main bits are as follows;

Stop the HA service

service vmware-aam stop

Check that HA has stopped (if not then use kill command to kill them)

ps ax | grep aam | grep -v grep

Move the current HA config files to a backup directory (before restarting HA)

cd /etc/opt/vmware/aam

mkdir .old

mv * .old

mv .[a-z]* .old

Then back to your vcenter and select Reconfigure for VMware HA on the effected host. Fingers crossed that it starts up and reconfigures without any issues.

vSphere – ctrl-alt-del greyed out

This bug has hit me. Looks like users with roles like vm user / power user cannot send “ctrl-alt-del” via the console even though they have the correct permissions. Our users cannot use ctrl-alt-ins as they are connected via RDP to a machine that has the console installed.

Found this : http://communities.vmware.com/thread/220683;jsessionid=480C8A2C9B9EACA9FF2BB4E1BECA2D53?start=15&tstart=0

Looks like its a known bug and will be fixed in the upcoming VC4.0 update 1 sometime Q3 2009 :(

Luckily vSphere was setup in our pre-production environment – the machines i have running in production are still 3.5 with VC2.5.

zfs compression and latency

Since im using ZFS as storage via NFS for my some of my vmware environments i need to ensure that latency on my disk is reduced where ever possible.

There is alot of talk about ZFS compression being “faster” than a non-compressed pool due to less physical data being pulled off the drives. This of course depends on the system powering ZFS, but i wanted to run some tests specifically on latency. Throughput is fine in some situations, but latency is a killer when it comes to lots of small reads and writes (in the case of hosting virtual machines)

I recently completed some basic tests focusing on the differences in latency when ZFS compression (lzjb) is enabled or disabled. IOMeter was my tool of choice and i hit my ZFS box via a mapped drive.

I’m not concerned with the actual figures, but the difference between the figures

I have run the test multiple times (to eliminate caching as a factor) and can validate that compression (on my system anyhow) increases latency

Basic Results from a “All in one” test suite… (similar results across all my tests)

ZFS uncompressed:

IOps : 2376.68
Read MBps : 15.14
Write MBps : 15.36
Average Response Time : 0.42
Average Read Response Time : 0.42
Average Write Response Time : 0.43
Average Transaction Time : 0.42

ZFS compressed: (lzjb)

IOps : 1901.82
Read MBps : 12.09
Write MBps : 12.28
Average Response Time : 0.53
Average Read Response Time : 0.44
Average Write Response Time : 0.61
Average Transaction Time : 0.53

As you can see from the results, the AWRT especially is much higher due to compression. I wouldn’t recommend using zfs compression where latency is a large factor (virtual machines)

Note: Under all the tests performed the CPU (dual core) on the zfs box was never 100% – eliminating that as a bottleneck.