BUT…

The one thing the store didn’t mention to me is the new 4K cluster sizing on the drive. I guess they assume most people run windows (though the issues are also present in XP). See these posts…

http://blog.temeletry.co.uk/2010/05/wd-green-wd10ears/

Unfortunately they really don’t work as well as you’d like in a server 

• They come with a 5 second head spin down setting that causes them to park their heads if they have been left idle for more than 5 seconds. As it takes a second or two to spin back up this can result in a very laggy experience during interactive sessions.
• They do not have NCQ or any form of command queing/optimisation. This means that (on FreeBSD at least) you are stuck in the LOOK elevator. In particular this was noticed when doing sequential read & write (think dump|restore tar|untar etc) and interactive tasks simultaneously
• They really suck with FreeBSD and ZFS…

http://community.wdc.com/t5/Desktop/Poor-performace-in-OpenSolaris-with-4K-sector-drive-WD10EARS-in/m-p/21132

While the other 512-byte sector HDDs were reading/writing at 30MB/s sustained, this EARS model did not exceeded the 1MB/s barrier.

I know for sure that this is related to the 512-byte sector firmware emulation, because the disk works perfectly well if I partition it in a 4k-sector alignment.

The thing is that even in that way, using it in a ZFS RAIDZ configuration the performance is very poor because RAIDZ uses a dynamic stripe size.

The bottom line here is that folks like me, that use different versions of Unix, need the firmware to present the disk as a 4K-sector disk to unleash the full potential of the technology. The OS is already prepared to support that sector size, no need for emulation here.

http://opensolaris.org/jive/thread.jspa?threadID=125702

Some preliminary testing that I have done…the WD20EARS (2TB advanced format drives) actually presents emulated 512byte sectors to the host o/s.

The drive documentation indicates that jumpers 7-8 should be enabled if the o/s does not support advanced format drives – the drive still present 512 bytes sectors.

I have attempted to raise a support ticket querying this, and how one can disable 512byte sector emulation in the drive (perhaps through a firmware upgrade) but I have not received any response to date.

Hopefully is enough people raise support tickets, WD may release firmware that allows the drive to natively present 4k blocks. Other doco indicates several other jumper combinations – all do not seem to make the drive present 4k byte blocks.

Perhaps someone internal to sun that has a relationship with WD may be able to shed some light on this? It would be fantastic to find out that I was just doing something wrong -> then I can get the drives to be seen on 32bit systems (ie – our embedded kit for osol, velitium)

Tested using b133 (64bit intel).

Try to avoid the green drives in ZFS for now. Remember to do your research before you buy a bunch of disks. I was caught off guard by this small change (works fine in win7 etc) which kills performance in ZFS. Ouch.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

How easy is it to turn on? – very!

Once you have upgraded your zpool to 21 or above you can run the following command at the pool level and deduplication will be over all your data from that point onwards.

zpool set dedup=on tank

Done

Note : Watch your performance, it will drop like a rock if you do not have enough ram for your dedup tables. Do some tests after enabling this feature.

Related posts:

1. Opensolaris – ZFS recovery after kernel panic Recently i hit what i thought was a huge disaster...
2. OpenSolaris – iSCSI Want iSCSI in opensolaris? Grab SUNWiscsitgt via package manager. enable...
3. zfs compression and latency Since im using ZFS as storage via NFS for my...

Related posts brought to you by Yet Another Related Posts Plugin.

Grab SUNWiscsitgt via package manager.

enable the server via svcadm;

svcadm enable iscsitgt

create your zfs iscsi pool;  (this command will limit iscsi drive to 500GB in size)

zfs create -V 500G tank/iscsi

set isci on via zfs command;

zfs set shareiscsi=on tank/iscsi

check that target is up and running;

iscsitadm list target -v

Done. Should be able to connect via ip from another machine. I have not covered CHAP or any client side configuration. Assumed isolated LAN.

Related posts:

1. vSphere and Multipathing iSCSI This is just a quick reference to create a multiplathing...
2. Jumbo Frames on your vSphere ESXi box Continuing on from this http://sigtar.com/2010/02/04/vsphere-and-multipathing-iscsi/ you may want to implement...
3. vmware – measuring iscsi write performance I picked this trick up off vmware support. If you’ve...

Related posts brought to you by Yet Another Related Posts Plugin.

Check it out here;

http://www.freenas.org/

I’m still using opensolaris though as i like running a few virtualbox machines on the same box.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

When i tried to import with zpool import -f tank the machine would lockup and reboot (panic).

The kernel panic;  (key line)

> genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776 size=22528)

Nothing i could do would fix it… tried both of these options in the system file with no success;

set zfs:zfs_recover=1
set aok=1

After a quick email from a Sun Engineer (kudos to Victor), the zdb command line that fixed it;

zdb -e -bcsvL <poolname>

zdb is a read only diagnostic tool, but seemed to read through the sectors that had the corrupt data and fix things??  (not sure how a read only tool does that) – the run took well over 15hrs.

Updated: 20/10/2009

Apparently if you have set zfs:zfs_recover=1 in your system file the zdb command will operate in a different manner fixing the issues it encounters.

Remember to run a zpool scrub <poolname> if you are lucky enough to get it back online.

Related posts:

1. zfs – now has dedup! Cool. zfs as of version 21 has deduplication built in....
2. OpenSolaris – Samba server Time to share your newly created ZFS volume via samba...

Related posts brought to you by Yet Another Related Posts Plugin.

At the moment i’m looking at grabbing one of these UIO cards;

http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm

From what iv’e been reading these cards will work fine in a PCI-e slot (8x / 16x) after a bit of modding and display the drives straight to opensolaris without any additional drivers etc. (same chipset used in various sun servers)

The backplate on a UIO card is essentially on backwards, when you remove the backplate and put the card into the PCI-e slot all the components will appear on the other side to normal. It is possible if you have a spare PCI-e backplate to attach to this card (just unscrew the current backplate and replace).

And the required mini SAS to SATA cables from extreme deal;

http://www.dealextreme.com/details.dx/sku.18023

Done.

Updated : 02/09/2009

Put this card in and bingo no problems. Had to export and re-import the zpool as it had problems with the drives being on a different controller? (hadn’t seen that before), but after that everything was working very well as expected. Cool!

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

There is alot of talk about ZFS compression being “faster” than a non-compressed pool due to less physical data being pulled off the drives. This of course depends on the system powering ZFS, but i wanted to run some tests specifically on latency. Throughput is fine in some situations, but latency is a killer when it comes to lots of small reads and writes (in the case of hosting virtual machines)

I recently completed some basic tests focusing on the differences in latency when ZFS compression (lzjb) is enabled or disabled. IOMeter was my tool of choice and i hit my ZFS box via a mapped drive.

I’m not concerned with the actual figures, but the difference between the figures

I have run the test multiple times (to eliminate caching as a factor) and can validate that compression (on my system anyhow) increases latency

Basic Results from a “All in one” test suite… (similar results across all my tests)

ZFS uncompressed:

IOps : 2376.68
Read MBps : 15.14
Write MBps : 15.36
Average Response Time : 0.42
Average Read Response Time : 0.42
Average Write Response Time : 0.43
Average Transaction Time : 0.42

ZFS compressed: (lzjb)

IOps : 1901.82
Read MBps : 12.09
Write MBps : 12.28
Average Response Time : 0.53
Average Read Response Time : 0.44
Average Write Response Time : 0.61
Average Transaction Time : 0.53

As you can see from the results, the AWRT especially is much higher due to compression. I wouldn’t recommend using zfs compression where latency is a large factor (virtual machines)

Note: Under all the tests performed the CPU (dual core) on the zfs box was never 100% – eliminating that as a bottleneck.

Related posts:

1. iometer testing The following is some notes i have around the parameters...
2. vMA – Using HP Power Protector agent to shutdown virtual hosts Goal was to create a vMA which would send a...
3. vmware – measuring iscsi write performance I picked this trick up off vmware support. If you’ve...

Related posts brought to you by Yet Another Related Posts Plugin.

Fire up add software – click filesystems – enable filter for “smb”, there are three packages generally. I get all three, but you only need the kernel update and the server package. The other is the SMB client.

Once installed make sure you enable the server in servicesgui.

Ensure the filesystem does not have any permission issues. I usually run chmod -R 777 /share just to ensure everyone can access the files without issue.

Add some users into smb password file (U need to create the users and sync the passwords). I usually create a guest user profile

useradd guest

smbpasswd -a guest - it should prompt for password twice (this is the password you use from windows). Press enter twice to leave the password blank.

The configuration can be done via /etc/sfw/smb.conf or via the shared folders admin gui.

I prefer doing the admin via the /etc/sfw/smb.conffile as it tends to let you have more control than the basic options available to you via the GUI. The contents of the file are as follows;  (note: i have included alot of the setting as an example which may contridict other settings)

[global] – global settings, the following are obvious

workgroup = workgroup

server string = opensolaris

wins support = yes - lets your server act as a WINS box

[share] – share name

path = /raidz1/share – share path

available = yes - enabled?

browseable = yes

public = yes

valid users = user1, user2 - only these users can access the share

writable = yes - equivalent to read / write in windows share properties

read only = yes – sets the default permissions to read only

write list = user1, user2 - these users can write to the share. Overrides above “read only” setting.

There are some good examples within /etc/sfw/smb.conf-example. Look there for some tips.

You also have an option of managing samba via the web – SWAT (samba web admin t). To get this up an running enable the swat service svc:/network/swat:default then browse to http://server:901

Optimizing SMB

I’ve found that adding this to /etc/sfw/smb.conf helps throughput in some cases. Try for yourself;  (it tends to put a higher load on cpu)

[global]

aio read size = 1
aio write size = 1

Further to this entry i have discovered that the built in CIFS / SMB service is much more efficient since it is included as part of the kernel. See my other posts on setting up cifs

Updated : 9/08/2009

I’ve swapped back to samba due to the issues i’ve had with cifs in the later releases. Remember if you wish to swap back to samba yo uneed to remove the sharesmb properties from each of your zfs shares – else on reboot zfs will re-enable the server/smb service.

There are some additional settings to ensure that your file server is the master browser for your workgroup. Put these under your [global]

[global]
domain master = Yes
local master = Yes
preferred master = Yes
os level = 35

Apparently on windows the os level reaches only 32 – so setting this to 35 ensures that your file server remains the master browser when an election is performed.

Related posts:

1. Opensolaris – Samba and Network browsing If you’ve setup samba (rather than the builtin cifs/smb) chances...
2. OpenSolaris – iSCSI Want iSCSI in opensolaris? Grab SUNWiscsitgt via package manager. enable...
3. opensolaris v134 – CIFS has gone walkies Errr, I cant find the CIFS service in the 134...

Related posts brought to you by Yet Another Related Posts Plugin.

zfs set sharenfs=rw,nosuid,root=esxhost tank/nfs

Note : You MUST have a resolvable name from the opensolaris box. i.e. you should be able to ping it. I have tried with ip’s only and it will fail. I have edited the /etc/hosts file to include the following line for my config;

# Copyright 2007 Sun Microsystems, Inc. All rights reserved.
# Use is subject to license terms.
#
# ident “%Z%%M% %I% %E% SMI”
#
# Internet host table
#
192.168.9.120 esxhost

This also requires that you are using both DNS and Files in your /etc/nsswitch.conf file. You should have a line like so;

# You must also set up the /etc/resolv.conf file for DNS name
# server lookup. See resolv.conf(4). For lookup via mdns
# svc:/network/dns/multicast:default must also be enabled. See mdnsd(1M)
hosts: files dns mdns

# Note that IPv4 addresses are searched for in all of the ipnodes databases
# before searching the hosts databases.
ipnodes: files dns mdns

i’ve also run this before hand; (to allow full access)

chmod -R 777 /tank/nfs

Update : check this guide http://blog.laspina.ca/ubiquitous/running-zfs-over-nfs-as-a-vmware-store

Related posts:

1. OpenSolaris – iSCSI Want iSCSI in opensolaris? Grab SUNWiscsitgt via package manager. enable...
2. Opensolaris – ZFS recovery after kernel panic Recently i hit what i thought was a huge disaster...
3. OpenSolaris – Samba server Time to share your newly created ZFS volume via samba...

Related posts brought to you by Yet Another Related Posts Plugin.

Check out this build (very nice clear guide)   http://www.stringliterals.com/?p=77

Awesome piece of work.

Related posts:

1. opensolaris v134 – CIFS has gone walkies Errr, I cant find the CIFS service in the 134...

Related posts brought to you by Yet Another Related Posts Plugin.