Category Archives: Solaris Express

zfs – checking your zpool throughput

Posted on by — No Comments ↓

This is quite a good diagnostic for checking your disk throughput. Try copying data to and from your zpool while your running this command on the host…

zpool iostat -v unprotected 2

capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
unprotected  1.39T   668G     18      7  1.35M   161K
c7d0       696G   403M      1      2  55.1K  21.3K
c9d0       584G   112G      8      2   631K  69.3K
c7d1       141G   555G      8      2   697K  70.0K
----------  -----  -----  -----  -----  -----  -----

The above command will keep displaying the above output every 2 seconds (average during that time). I’ve used it a few times to ensure that all disks are being used (in write operations) where needed. Of course read op’s may not be typically across all disks as it will depend where the data is…

As you can see in the output from my “unprotected” zpool, my disk “c7d0” is near full so less write operations will be on this disk. In my scenario most of my reads also come from this disk, this was due me copying most of the data into this zpool when there was only this single disk.

I’ve heard rumor of a zfs feature in future that will re-balance the data across all the disks (unsure if its live or on a set schedule)

Another way to show some disk throughput figures is to run the iostat command like so…

iostat -exn 10

extended device statistics
device    r/s    w/s   kr/s   kw/s wait actv  svc_t  %w  %b
cmdk17    1.0    0.0   71.5    0.0  0.0  0.0   10.9   0   1
cmdk18    0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
cmdk19    0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0
cmdk20    0.8    0.0   33.5    0.0  0.0  0.0   13.5   0   1
cmdk21    0.4    0.0    0.5    0.0  0.0  0.0   15.5   0   1
cmdk22    0.8    0.0   66.3    0.0  0.0  0.0    9.0   0   1
cmdk23    0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0

cmdk24    0.0    0.0    0.0    0.0  0.0  0.0    0.0   0   0

extended device statistics       —- errors —

                     extended device statistics       ---- errors --- 



r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot device


0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c11d1


0.0    7.7    0.0   25.8  0.0  0.0    2.3    4.9   0   3   0   0   0   0 c8d0


0.0   17.6    0.0  238.0  0.0  0.0    0.0    0.3   0   0   0   0   0   0 c9d0


0.0    1.0    0.0    0.8  0.0  0.0    0.0    0.3   0   0   0   0   0   0 c7t0d0


0.0    1.0    0.0    0.8  0.0  0.0    0.0    0.2   0   0   0   0   0   0 c7t2d0


0.0    1.0    0.0    0.8  0.0  0.0    0.0    0.3   0   0   0   0   0   0 c7t3d0


0.7   21.1   29.9  315.0  0.0  0.0    0.0    1.1   0   1   0   0   0   0 c7t4d0


0.7   20.9   29.8  314.9  0.0  0.0    0.0    1.7   0   2   0   0   0   0 c7t5d0


0.8   21.0   34.1  315.0  0.0  0.0    0.0    1.2   0   1   0   0   0   0 c7t6d0


0.5   20.8   21.3  314.8  0.0  0.0    0.0    1.1   0   1   0   0   0   0 c7t7d0

This should show you all your disks and update on a 5 second interval. Copying data back and forth to your drives will show various stats.

OpenSolaris cifs/smb server – configuring ACL’s on shares – Part 2

Posted on by — No Comments ↓

I have done most of the explaining in another post on how the permissions work with a zfs cifs share – link here.

In this post I’m going to show an example of how to add multiple groups / users to a specific folder. Which then recursively adds its ACL”s to any child objects. Its essentially like the other commands i used, but with an extra line per user or group you want to configure; 

chmod -R A=\

owner@:full_set:d:allow,\

owner@:full_set:f:allow,\

user:lisa:full_set:d:allow,\

user:lisa:full_set:f:allow,\

everyone@:rxaARWcs:d:allow,\

everyone@:raARWcs:f:allow \

/protected/

The above command gives the owner and the user “lisa” full permissions, the everybody else group has read only access. If you want to add a group then just use group: instead of user: at the start.

This gives full access to myself and list, but deny’s access to everyone else; 

chmod -R A=\

owner@:full_set:d:allow,\

owner@:full_set:f:allow,\

user:daz:full_set:d:allow,\

user:daz:full_set:f:allow,\

user:lisa:full_set:d:allow,\

user:lisa:full_set:f:allow,\

everyone@:full_set:d:deny,\

everyone@:full_set:f:deny \

/protected/photos/

Done.

OpenSolaris cifs/smb server – configuring ACL’s on shares

Posted on by — No Comments ↓

O.k. from this  point I’m assuming you have created your zfs shares via the zfs command and renamed appropriately like so…..   (if you havent setup cifs yet then check read this)

zfs set sharesmb=on protected/backup

zfs set sharesmb=name=backup protected/backup

Now to check your current shares type…

sharemgr show

zfs

zfs/protected/backup
backup=/protected/backup

zfs/protected/photos
photos=/protected/photos

Now its time to look at NFSv4 ACL’s. This page has a great explanation of ACL’s and how to set things up. http://cuddletech.com/blog/pivot/entry.php?id=939

The ZFS manual here has the compact access and inheritance codes for chmod : http://opensolaris.org/os/community/zfs/docs/zfsadmin.pdf

Here are the current compact codes for access control;

add_file w , add_subdirectory p , delete d , delete_child D , execute x , list_directory r , read_acl c , read_attributes a , read_data r , read_xattr R , write_xattr W , write_data w , write_attributes A , write_acl C , write_owner o

Here are the current compact codes for inheritance control;

file_inherit f , dir_inherit d , inherit_only i , no_propagate n

So… The below command (referencing above compact codes) will give me (the owner) full permissions to files / directories, and read only access to everyone else. I have enabled inheritance so newly created files should also maintain their parents ACL without windows creating its own. Note: i used chown on the root of the my share first  i.e. chown -R daz /protected

then…

chmod -R A=\

owner@:wACpdDo:d:allow,\

owner@:wACpdDo:f:allow,\

everyone@:rxaARWcs:d:allow,\

everyone@:raARWcs:f:allow \

/protected/

Remember to test that this has provided what you want. Connect to your share as guest test the permissions, then connect as the owner and test permissions again. The fun thing about this particular ACL system is that you are not restricted to just one owner and one group. You can add additional lines as required using user: and group: attributes. I’ve done a multi-user ACL post here.

There is alot of flexibility – i’d say even more than the samba server options by a long shot. Its probably a little bit more fiddly getting your commands right, but once up and running you can have alot more control.

There are also “ACL sets” which combine the above attributes into groups….. i.e .you can have just the word “full_set” (full permissions) or “read_set”  (supposedly gives you read, but i couldn’t see any child files after using this) between the first set of colons in the above command… Using ACL sets the above command could be changed to…

chmod -R A=\

owner@:full_set:d:allow,\

owner@:full_set:f:allow,\

everyone@:rxaARWcs:d:allow,\

everyone@:raARWcs:f:allow \

/protected/

This should still give the owner full permissions (previously the owner just inherited the everyone groups permissions it didnt have as well)

These are the possible choices for ACL sets: full_set , read_set , modify_set , write_set

see here for more examples : https://sigtar.com/2009/02/15/opensolaris-cifssmb-server-configuring-acl’s-on-shares-part-2/

—– workgroup authentication —–

Assuming you are NOT using a domain:
Did you enable the password database settings?
user@solaris:~# cat >> /etc/pam.conf

# Seem to need this line for smb / cifs:
other password required pam_smb_passwd.so.1 nowarn

(Control-D to end)

user@solaris:~# passwd
passwd: Changing password for user
New Password:
Re-enter new Password:
passwd: password successfully changed for user

OpenSolaris – Migrating from samba to cifs

Posted on by — No Comments ↓

OpenSolaris 2008.11 has a cifs/smb server built into the kernel. From what i have been reading (and discovering) this is a much more efficient and faster way to get your files out into the network.

Previously i have run a dedicated samba service to share my files as I’ve come from various linux distros which require it. Seems i should be doing things differently on OpenSolaris.

The dedicated samba service is called;

svc:/network/samba:default

The new kernel embedded solaris smb/cifs service is called;

svc:/network/smb/server:default

If you do not have it running or installed you’ll need to add the following packages (use package manager); 

SUNWsmbs

SUNWsmbskr

Update : both these packages are called SUNWsmbfs in newer builds

Enable the service via “services” gui else use the svcadm command; 

svcadm enable svc:/network/smb/server:default

This is a great page on how to do a basic cifs setup http://blogs.sun.com/timthomas/entry/solaris_cifs_in_workgroup_mode

I just need to find a bit more detail on how to configure access to the shares (update: see below). In another post i made previously using samba it was easy enough to restrict guests to read only while giving some users rw access… I’m also interested in whether its possible to backup or manually configure the cifs server via any config file? Still looking into this one.

First step is to disable the samba service and enable the solaris smb/cifs service;

svcadm disable network/samba

svcadm enable -r smb/server – the “r” switch also enables all services that smb requires.

The next step is to add the following line to your /etc/pam.conf file. This is so future changes to passwords  also update the smb password file;

other   password required       pam_smb_passwd.so.1     nowarn

now set the name of the workgroup…

smbadm join -w workgroup

now setup your users (you have to reset your current passwords to update the password file above). I usually create and use a “Guest” account to keep some windows boxes happy. I set the guest account password to blank… 

useradd guest

passwd guest – then enter twice to enter a blank password

Next its time to setup the actual shares. If you have a zfs pool setup (you most likey do if you are running opensolaris) then you can share your data via the zfs command like so….   (i have a “backups” zfs file system within a “unprotected” zpool)

zfs set sharesmb=on unprotected/backups

Unfortunately this creates a share called “unprotected_backups” which can be seen via the sharemgr show -vp command. To rename the share to something more appropriate do the following…

zfs set sharesmb=name=backups unprotected/backups

All going well the share should now be called “backups”, if you have both the old and new name listed restart the smb server : svcadm restart smb/server

From your windows machine you should be able to browse to your share via \\servername\backups, if you get a prompt then type “guest” and press enter. Here is an example of a quick and easy way to mount drives via an XP commandline…

net use * \\serverip\backups /user:guest

Update: i have figured out the ACLs for cifs — See this Post

OpenSolaris – RTL8111/8168B issues

Posted on by — No Comments ↓

I’ve got an integrated RTL8111 nic which seemed to work fine under opensolaris 2008.11. But if the nic was put under load for a various length of time it seemed to just drop off the network.

At first i thought it was my SMB service dying, but after a quick ping i relised i had lost the entire TCP/IP stack on that particular card. Hmm…

It does come back online if you are patient and wait for about 5mins or so.

The web shows that there is some known issues with some cards dropping under load. Most places recommend to get a certified pci-e intel nic and your problems will go away. I’m considering this the last possible option, as i don’t particularly want to spend any more money.

The driver that seems to be at fault is the rge native driver… I have found this bug link that “could” be the issue, but might be specifically for the Realtek 8111C. Add the following at the end of the /etc/system file;

set ip:dohwcksum = 0

This setting is short for “do hardware checksum”. From what i have read setting this to zero moves the checksum calculations from the network card to your cpu (it doesn’t open your system to less error checking etc)

Update: This seemed to initially fix the problem for me, but the issue still occurred again after some time.

This forum thread also pointed to a similar issue… http://opensolaris.org/jive/thread.jspa?threadID=91282

Another solution may be found at the sun HLC site.  From the HLC i have found the home of the driver for the  RTL8111/8168B.  http://homepage2.nifty.com/mrym3/taiyodo/eng/

This driver is called gani Driver link here

This page has a good bit on moving from rge to gani driver…. http://schlaepfer.nine.ch/twiki/bin/view/Schlaepfer/SelfMadeNas2. The only problem is that creating the gani driver doest seem to be straight forward.

1. In /etc/driver_aliases find rge “pci10ec,8168” and exchange it with gani “pci10ec,8168”
2. Move /etc/hostname.rge0 to /etc/hostname.gani0
3. Reboot the system

Update2: I thought i had this driver working properly, but after a reboot everything stopped. I couldn’t even ping an ip on the same subnet. Driver was still loaded as i could ping my own ip.

Now i’m looking into the parameters on the rge driver. To get a list of the variables the device has to modify type…

ndd -get /dev/rge0 \?

of the parameters that are listed only the read and write ones can be changed. adv_pause_cap relates to duplex settings and adv_1000fdx_cap relates to speed. If you disable either of these parameters then they are not negotiated with your switch. Probably not worth touching these ones unless you want to run a gb card at 100 half duplex or something.

I’m experimenting with disabling adv_asym_pause_cap at the moment to see if that helps. By default this is enabled. This can be disabled via..

ndd -set /dev/rge0 adv_asym_pause_cap 0

Update3: so far so good? — the above seems to have removed the issue. I still have the  ip:dohwcksum = 0 setting in the /etc/system file. I might try removing that.

Update4: removing ip:dohwcksum = 0 did not re-create the issue – so leaving it off.

Update5: problem came back (but took much longer to appear). Hmmm…..

Now I’ve got another problem with the rge driver. My CIFS write speed has dropped right back to about 2MB/s. There doesn’t seem to be any issues with the read speed which still pulls through about 70MB/s.

It doesn’t seem to be a CPU bottleneck, so again I’m blaming the rge drivers…  The adv_asym_pause_cap parameter did not seem to make any difference to this particular issue – so I’m not blaming that. I’m currently stuck on this one. hmmm…

Looks like i’m going to have to give up on this one and get a Intel pci-e card. I’ll update this post if a new card fixes all the above problems (therefore pointing at the rge driver as the culprit)

Update6: I’ve got the Intel card, and the problems have not re-appeared as of yet. I’ll update if the problem does show itself, but i believe the problem was the rge driver. Hopefully the rge driver is fixed / updated in future releases of opensolaris.

Please leave a message if anyone has made any progress with the rge driver. Cheer.