4k sector hard drives and zfs

I hit this as a problem recently. One of my disks died in my raidz so i ran down to the store and grabbed me a replacement WD10EARS (Western Digital 1Tb Green) drive.

BUT…

The one thing the store didn’t mention to me is the new 4K cluster sizing on the drive. I guess they assume most people run windows (though the issues are also present in XP). See these posts…

http://blog.temeletry.co.uk/2010/05/wd-green-wd10ears/

Unfortunately they really don’t work as well as you’d like in a server :(

  • They come with a 5 second head spin down setting that causes them to park their heads if they have been left idle for more than 5 seconds. As it takes a second or two to spin back up this can result in a very laggy experience during interactive sessions.
  • They do not have NCQ or any form of command queing/optimisation. This means that (on FreeBSD at least) you are stuck in the LOOK elevator. In particular this was noticed when doing sequential read & write (think dump|restore tar|untar etc) and interactive tasks simultaneously
  • They really suck with FreeBSD and ZFS…

http://community.wdc.com/t5/Desktop/Poor-performace-in-OpenSolaris-with-4K-sector-drive-WD10EARS-in/m-p/21132

While the other 512-byte sector HDDs were reading/writing at 30MB/s sustained, this EARS model did not exceeded the 1MB/s barrier.

I know for sure that this is related to the 512-byte sector firmware emulation, because the disk works perfectly well if I partition it in a 4k-sector alignment.

The thing is that even in that way, using it in a ZFS RAIDZ configuration the performance is very poor because RAIDZ uses a dynamic stripe size.

The bottom line here is that folks like me, that use different versions of Unix, need the firmware to present the disk as a 4K-sector disk to unleash the full potential of the technology. The OS is already prepared to support that sector size, no need for emulation here.

http://opensolaris.org/jive/thread.jspa?threadID=125702

Some preliminary testing that I have done…the WD20EARS (2TB advanced format drives) actually presents emulated 512byte sectors to the host o/s.

The drive documentation indicates that jumpers 7-8 should be enabled if the o/s does not support advanced format drives – the drive still present 512 bytes sectors.

I have attempted to raise a support ticket querying this, and how one can disable 512byte sector emulation in the drive (perhaps through a firmware upgrade) but I have not received any response to date.

Hopefully is enough people raise support tickets, WD may release firmware that allows the drive to natively present 4k blocks. Other doco indicates several other jumper combinations – all do not seem to make the drive present 4k byte blocks.

Perhaps someone internal to sun that has a relationship with WD may be able to shed some light on this? It would be fantastic to find out that I was just doing something wrong -> then I can get the drives to be seen on 32bit systems (ie – our embedded kit for osol, velitium)

Tested using b133 (64bit intel).

Try to avoid the green drives in ZFS for now. Remember to do your research before you buy a bunch of disks. I was caught off guard by this small change (works fine in win7 etc) which kills performance in ZFS. Ouch.

VirtualBox – crashing / freezing

I’ve had some problems since my upgrade to virtualbox 2.2.0 on OpenSolaris. After some time all of my linux boxes seem to just die. The virtual machine just stops responding. Strangely there was no problem with my windows vms after the update.

From what i can tell it looks like the upgrade turned off “IO APIC” – this is the bit that seemed to cause the problem. Re-enabling this on all of my linux boxes seems to have fixed the problem. I’ll continue testing for another week and update this post if any problems re-occur.

Updated : 01/09/2009

Here is a bit more on IO APIC from the virtualbox wiki…  (from a windows perspective)
http://www.virtualbox.org/wiki/Migrate_Windows

The hardware dependent portion of the Windows kernel is dubbed “Hardware Abstraction Layer” (HAL). While hardware vendor specific HALs have become very rare, there are still a number of HALs shipped by Microsoft. Here are the most common HALs (for more information, refer to this article: http://support.microsoft.com/kb/309283):

Hal.dll (Standard PC)
Halacpi.dll (ACPI HAL)
Halaacpi.dll (ACPI HAL with IO APIC)

If you perform a Windows installation with default settings in VirtualBox, Halacpi.dll will be chosen as VirtualBox enables ACPI by default but disables the IO APIC by default. A standard installation on a modern physical PC or VMware will usually result in Halaacpi.dll being chosen as most systems nowadays have an IO APIC and VMware chose to virtualize it by default (VirtualBox disables the IO APIC because it is more expensive to virtualize than a standard PIC). So as a first step, you either have to enable IO APIC support in VirtualBox or replace the HAL. Replacing the HAL can be done by booting the VM from the Windows CD and performing a repair installation.

Updated : 5/09/2009

I’ve had even more problems with opensolaris crashing completely after upgrading to the newer versions of virtualbox (3.0.4), and have since reverted back to 2.2.0 which has fixed alot of the hanging issues i have encountered

selinux – opening additional ports / or disabling

If you are having problems starting apache on a non-standard port you might find that the problem is related to selinux.

Type this command to check to see what http ports are currently allowed;  (remove filter to show all rules)

semanage port -l|grep http

To add another port type the following (with the port you wish to add etc);

semanage port -a -t http_port_t -p tcp 81

If you want to disable selinux completely then go into /etc/selinux/config and set selinux=disabled. Save then reboot.

fedora 11 – LAMP install

Setting up LAMP on fedora 11… First install the required modules.

yum install -y httpd.i586 mysql.i586 mysql-server.i586 php.i586 php-mysql.i586 php-gd.i586

Next i’m copying my web source from another linux machine. I’ve run these commands from the destination machine…

cd /var/www/html

rsync -ave ssh 192.168.9.10:/var/www/html/ .

Now my mysql data…

cd /var/lib/mysql

rsync -ave ssh 192.168.9.10:/var/lib/mysql/ .

Accept the certificate and enter the source root password. Next fire up the services…  I usually just go into “setup” then “system services” and enable both httpd and mysqld

Reboot. Done

squid – your transparent proxy friend

Still my favourite light and fast web proxy is Squid. Its very easy to setup and get running on almost all flavours of linux. You can find it here http://www.squid-cache.org/ but will probably find it in your local package manager…

This is for more my own reference… as i don’t think too many people have the need for a transparent proxy. You only need one NIC configured on your VM / machine as its most probably on the same subnet as your dsl etc…

1. Setup squid working as a proxy first!

Set the default gateway on the NIC to your router. I also added a default route 0.0.0.0 sub 0.0.0.0 to the router also just in case the default route didn’t work. But seemed to!

Setup DNS! remember you should be able to resolve a name to an IP without issues (can cause most of the problems) Point your default DNS to your router (most have built in DNS forwarders), else you can just use your ISP’s DNS’s

2. Change the squid.config to have the necessary changes to act as a transparent proxy;

nano /etc/squid/squid.conf

http_port 3128 transparent

3. Put this into the startup script; (usually in rc.d under etc) rc.local

iptables -t nat -A PREROUTING -i eth0 -p tcp –dport 80 -j REDIRECT –to-port 3128

This redirects standard tcp requests to 3128 (squid’s port)… transparent to user

4. Ensure that IP forwarding is ENABLED;

change ip_forward from 0 to 1 (usually a txt file somewhere with 0 in it), do a search and you will find it… most probably in the ipv4 directory. This way all 443 (https and other apps) can forward to the web without issue…

echo 1 > /proc/sys/net/ipv4/ip_forward

Should be it…. else good luck!

Update: I have done an updated post about configuring open solaris as a transparent proxy here : http://sigtar.com/2009/04/22/opensolaris-configuring-squid-as-transparent-proxy/

Windows / Linux – File Migration Tools

In Windows i use robocopy

This is the syntax to copy from one location to another…  (i like how robocopy uses UNC paths)

robocopy \\192.168.9.70\videos \\192.168.9.101\videos /e /r:2 /w:10 /sec
robocopy \\192.168.9.70\software \\192.168.9.101\software /e /r:2 /w:10 /sec

Note: i always copy instead of the move. Run multiple times before switch over to shorten time for final sync. Always use the /SEC to copy over the NTFS permissions. I have had problems with the non X010 version of robocopy not copying NTFS permissions across. Check NTFS permissions before removing the source files. You can update files NTFS permissions with /SECFIX switch if required.

/e moves all directories including empty ones.

In Linux i use rsync

i typically mount a smb share on a windows host…. (you must have cifs client on linux first)

mount -t cifs -n //192.168.9.70/backup /mnt/backup -o username=user,password=mypassword

then use rsync for the copy work…

rsync -rcav /etc/ /mnt/backup/etc/

if your after linux to linux copying then rsync is also the best way to do it (via ssh), logon to your destination box via ssh and run this from your distination diretory..

rsync -ave ssh 192.168.9.10:/var/lib/mysql .

Linux – routing examples

set ip_forward to 1 to enable routing between nics.
type ‘route‘ to insure both networks are routed as expected (connected route added with nic)

Setup a script with the following (CHMOD 0755 the script so it executes)

iptables -F (clears previous iptables stuff)
iptables -P INPUT DROP (will set default policy to DROP all INPUT packets – Incoming to local NICs)
iptables -P OUTPUT DROP (will set default policy to DROP all OUTPUT packets – Outgoing to local NICs)
iptables -P FORWARD DROP (will set default policy to DROP all FORWARDed packets – Routed via local NICs)

allow rules;
iptables -A FORWARD -s 172.23.23.1 -d 172.23.23.2 -p tcp –dport 80 -j ACCEPT (append forward rules – self explanatory)

Add more as required.

Want to use NAT outbound?

wlan0 => external network – route to Internet
eth0 => internal network

#setup masquerading
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -A FORWARD -m state –state ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

A very handy tool and the current best way i’ve found to test firewall rules is to fire up tcpdump. Check man for details but here is an example of monitoring a specific destination port and host on a specific NIC;

tcpdump -i eth1 dst port 22 and dst host 192.168.4.34

If you can see the traffic leaving the interface outbound it has made it past the firewall, if not then you need to have a look at your firewall rules or check your routing table. Type route to see current routing table

Done

Linux – Add DHCP and DNS

How to install DHCP and DNS on Fedora / Centos / Redhat box;

yum -y install dhcp.i386 bind.i386

Set services to auto start using ntsysv — services are called “dhcpd” and “named”.

How to configure dhcp;

nano /etc/dhcpd.conf — configure as per sample provided. The sample is usually located in the same directory.
service dhcpd restart – should start without problem if done correctly

Note: dhcp range has to be on the same network as your adapter. You can run dhcpd from the command line to diagnose any issues.

How to configure bind (DNS);

I usually just forward DNS requests to my internal router. To do this just edit /etc/resolve.conf and place the following line in the file;

nameserver 192.168.9.1

Use your ip above then all should be good. Test by pinging a domain like www.yahoo.com etc… You should get an ip back even if you don’t get successful pings (due to firewall etc). nslookup is also another quick and easy way to check DNS requests.

Linux – Wireless networking

if your card is natively supported type ;

iwconfig

iwlist scanning — should also show you the wireless networks that are in range (will work if driver is loaded)

you should see wlan0 if all is good; (grab your wireless details)

iwconfig wlan0 mode managed (should already be in this mode)

there are some cool options here like “secondary” mode which the node acts as a backup master/repeater. “Repeater” – the node forwards packets between other wire-less nodes

iwconfig wlan0 channel 6 – sets wireless to channel to 6
iwconfig wlan0 essid dwireless – sets ssid to “dwireless”

K – now it gets interesting… iwconfig doesn’t support wpa2 psk out of the bag.
http://hostap.epitest.fi/wpa_supplicant/ – you want to get your hands on wpa_supplicant or similar depending on your distro. This is a good tutorial on getting it all up and running (for ubuntu, but works on other distros) http://ph.ubuntuforums.com/showthread.php?t=571188 similar one here http://www.varesano.net/blog/fabio/wpap … no+ipw2100

If you have the Asus wireless card like i did it is best to use the ndis wrapper. The native drivers do NOT work, this is even though the native driver will seem to work – detects card, and can browse the current local wireless networks. Unfortunately it cannot actually join any network (even the unencrypted ones)

do NOT just type dhclient unless you want all adapters to renew their IP’s…. ensure that you also specify the adapter.

Corporate Firewall – SSH and port 443 goodness

Things you’ll need; (basic instructions)
SSH – either Linux or Open SSH for windows (set this up first and ensure its working!). I have only used Linux, so i do not know the details of setting up a Open SSH box on windows (good luck).

Router with pinhole or port forwarding abilities. — forward your external 443 port to internal 22 on your SSH box

Putty.exe – use this to test your SSH connection locally (you should be able to connect on port 22 locally or 443 externally)

Your home IP – setup a dyndns account on one of the free services available on the web (either use your router if it has it or get a dyndns client that runs on your ssh box) — i use dyndns.org

Putty is the client side device that you will use when you are not local to your network. If you are connecting to your SSH box from external (i.e. at work) then you’ll need to configure a SSH connection profile to connect on port 443. If you have a proxy at work (which you will most probably have) you need to ensure you have filled out your proxy address and authentication details — select HTTP then enter your username / password. Then try to connect.

When you can successfully connect to your SSH box from work via 443 you can then create SSH tunnels. Under SSH on putty there is a tunnels option. This is where you can forward local ports to your remote ports… for example – 127.0.0.1:82 –> 192.168.0.10:3389. Then if you fire up RDC you can connect to your remote machine via 127.0.0.1:82

Simple? — if i haven’t made a specific part clear please ask any questions and i’ll try to make it easier to follow