solaris – storage server

Setup iSCSI target using comstar —
pkg install -v storage-server
pkg install pkg:/SUNWiscsit
svcadm enable stmf
svcadm enable svc:/network/iscsi/target:default
itadm create-target

reboot — -r

Create the iSCSI store —
zfs create -V 100G DataPool/TestDatastore1
sbdadm create-lu /dev/zvol/rdsk/DataPool/TestDatastore1
stmfadm create-hg ESX4-group
stmfadm add-hg-member -g ESX4-group wwn.2100001b329711bd wwn.2101001b32b711bd
stmfadm add-view -h ESX4-group -n 0 600144F03EBEC50000004BA86E460001
stmfadm list-lu -v

P4000 / Lefthand and Windows DSM (MPIO)

Steps to setup MPIO round robin and DSM with P4000 / Lefthand nodes

This example is windows 2008 with 2 x 1gbit nics

  1. Install HP DSM driver onto windows 2008  (this should also install MPIO feature on windows 2008)
  2. Setup 2 x IP’s on windows host – storage subnet (used with MPIO)
    1. Confirm you can ping target from both (enable one at a time – ensure there is no routing on your other nics)
  3. Provision storage and allow server read / write access to windows server (via initiator name)
  4. Open iSCSI on windows 2008
  5. Put in IP of target in discovery tab
  6. On first tab confirm that iSCSI drive is presented to host
  7. Click “connect” – check both auto-connect and use MPIO
    1. click advanced – chose MS iSCSI initiator, choose first IP and target IP. Click o.k. / o.k
    2. click advanced – chose MS iSCSI initiator, choose second IP and target IP. Click o.k. / o.k
    3. Repeat above per additional Nic
  8. Confirm via “devices” that there is x (as many targets as nics) targets per disk
    1. Within devices choose MPIO – change from vendor specific to “round robin”
      1. Note : read / write access to LUN is required when using “round robin” MPIO opposed to the default “vendor specific” which works with read only access. Else you will get an error – “not supported”
    2. Repeat above per “device”
  9. Confirm on Lefthand / P4000 CMC that the LUN has one connection per initiator Nic, and that each connection also has its DSM children (visible in CMC if working).

Confirm that the connection are as expected…

Run some disk benchmark utilities (iometer) and check that traffic is travelling over all the nics you have setup above. You can just use windows builtin task manager to do this.

Check that the right amount of connections are on the CMC for that particular LUN and initiator.  So if you had 3 nodes via 2 initiator nics you would actually have 8 active connections in total (1 per nic (2) and an additional for every nic to each nodes (6))

Note : There are some reported issues with DMS and data corruption. Although i have not seen this myself please be diligent when it comes to data backup esp when production data is involved.

vmware – measuring iscsi write performance

I picked this trick up off vmware support. If you’ve got your iscsi all setup you can drop to the shell (either ssh or console) and do this to measure your average write throughput.

time vmkfstools -c 10G /vmfs/volumes/san_vmfs/my_vm/fat_disk.vmdk -d eagerzeroedthick

Try larger a larger disk if this is too quick (free space permitting)

Essentially this will initiate the host to create a fat disk in the location above. You will then get a time recorded on how long it takes to execute this command. Then you can use your maths skill to work out the transfer rate…

While this is happening you can open another SSH type esxtop then press “d” and watch the (d)isk throughput on the console. Pressing “v” will show you stats per (v)irtual machine.

vSphere and Multipathing iSCSI

This is just a quick reference to create a multiplathing iSCSI setup…

Create two virtual kernel switches, one called “iSCSI-1” and the other called “iSCSI-2” (and so on if you have more nics)

Then per kernel portgroup ensure that only one of the nics is active. For the “iSCSI-1” portgroup configure it to override the virtual switch settings and move nic 0 to active and nic 1 to unused. For the “iSCSI-2” portgroup configure it to override the virtual switch settings and move nic 1 to active and nic 0 to unused.

Now you have to run some esxcli commands to gel things together…. alt-f1 on the esxi console and type “unsupported” followed by your root password. The following is the command i have to run to get both my portgroups to work together on the iscsi hba….  (you can check your vmk number from the networking config screen)

esxcli swiscsi nic add -n vmk1 -d vmhba33
esxcli swiscsi nic add -n vmk2 -d vmhba33

For each target then change the path selection method to “round-robin”.

Go back to storage adapters and click “rescan”

If you want all future iSCSI targets to automatically use round-robin you must also run the following from commandline…   (this is for our HP Lefthand, your “storage array type” may be different. Its listed under your target details). Basically sets round robin as a default for this type of array. In general you should do this first before presenting any LUNs etc, else you may have to bounce your box.

esxcli nmp satp setdefaultpsp –satp VMW_SATP_DEFAULT_AA –psp VMW_PSP_RR

Update : Hardware iSCSI nics

The process for hardware iSCSI initiators is similar to above, but you assign a single kernel port per nic. To find which nic belongs to which iSCSI initiator you must run this command from the CLI;

esxcli swiscsi vmnic list -d vmhba#

vmhba# is the name of the iSCSI adapter.

OpenSolaris – iSCSI

Want iSCSI in opensolaris?

Grab SUNWiscsitgt via package manager.

enable the server via svcadm;

svcadm enable iscsitgt

create your zfs iscsi pool;  (this command will limit iscsi drive to 500GB in size)

zfs create -V 500G tank/iscsi

set isci on via zfs command;

zfs set shareiscsi=on tank/iscsi

check that target is up and running;

iscsitadm list target -v

Done. Should be able to connect via ip from another machine. I have not covered CHAP or any client side configuration. Assumed isolated LAN.


ESX – network utilization

One of the best articles i have found on this subject is here :

There is some additional information here on setting up an etherchannel on the cisco side :

This can be handy if you need a single VM to use both physical nics in a load-balanced manner – both outbound and inbound. Of course its not really that simple though. This will really only add a benefit if the VM is communicating to multiple destinations (using ip hash – a single destination from a single VM with one IP will always be limited to the same physical nic).

switch(config)#int port-channel 1
switch(config-if)#description NIC team for ESX server
switch(config-if)#int gi0/1
switch(config-if)#channel-group 1 mode on
switch(config-if)#int gi0/2
switch(config-if)#channel-group 1 mode on

As per the article ensure you are using the same etherchannel method. The first command shows your current load-blance method, the 2nd command changes it to ip hash.

show etherchannel load-balance
port-channel load-balance src-dst-ip

Another solution is to use multiple iSCSI paths. This is newly supported within vSphere, see this post on setting up multiple paths :

Here is another good article on iSCSI within vSphere :“Multivendor+Post”+on+using+iSCSI+with+VMware+vSphere

Some important points on using EMC Clariion with vSphere :