Opensolaris – ZFS recovery after kernel panic

Recently i hit what i thought was a huge disaster with my ZFS array. Essentially i was unable to import my zpool without causing the kernel to panic and reboot. Still unsure of the exact reason, but it didn’t seem to be due to a hardware fault. (zpool import showed all disks as ONLINE)

When i tried to import with zpool import -f tank the machine would lockup and reboot (panic).

The kernel panic;  (key line)

> genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776 size=22528)

Nothing i could do would fix it… tried both of these options in the system file with no success;

set zfs:zfs_recover=1
set aok=1

After a quick email from a Sun Engineer (kudos to Victor), the zdb command line that fixed it;

zdb -e -bcsvL <poolname>

zdb is a read only diagnostic tool, but seemed to read through the sectors that had the corrupt data and fix things??  (not sure how a read only tool does that) – the run took well over 15hrs.

Updated: 20/10/2009

Apparently if you have set zfs:zfs_recover=1 in your system file the zdb command will operate in a different manner fixing the issues it encounters.

Remember to run a zpool scrub <poolname> if you are lucky enough to get it back online.

This thread has some additional info…

Update 31/05/2012

This command has also helped me when i cant mount a pool in RW mode

zpool import -F -f -o readonly=on -R /mnt/temp zpool2

13 Replies to “Opensolaris – ZFS recovery after kernel panic”

  1. Thanks! This post saved me from rebuilding a zpool that had about 6TB worth of data!

  2. After a power outgage, my freenas 7 box always returned a kernel panic when tried to import the pool. The OS disk died, so i needed to install a new environment.
    Tried different things:

    – Freenas 7 -> Always panic. Old ZFS version with less options. No “readonly” for example.
    – Nas4Free 9 -> Graid detected the disks as a Nvidia Raid (¿?). Maybe old metadata on disks…
    – Opensolaris 11 -> Didnt find the disks correctly, so unable to import due to vdev error.
    – Freenas 8.3 -> I ran the “zpool import -F -f -o readonly=on -R /mnt/temp zpool2” command and it worked. So im going to backup all the data and re-create the pool.


  3. the readonly mode was very helpful. Next time this happens i’ll try the zdb command.

    I was really worried I’d loose an entire array. It happened at 2am without any warning and no power outage or anything. I still have faith in zfs, but this is very strange.

  4. I just want to mention I’m very new to blogging and truly enjoyed this web blog. Very likely I’m want to bookmark your blog . You really come with fantastic well written articles. Regards for sharing with us your web-site.

  5. TgpFJb Perfect work you have done, this site is really cool with good information.

  6. I just want to mention I am new to weblog and definitely liked this blog. Likely I’m planning to bookmark your blog . You surely have awesome articles and reviews. Regards for revealing your web-site.

  7. I and my buddies appeared to be checking the great guides found on your web site while instantly I got a horrible suspicion I had not thanked the web site owner for those techniques. The ladies came consequently warmed to read through all of them and already have pretty much been using them. Many thanks for really being well considerate and also for making a decision on these kinds of notable guides most people are really needing to know about. Our own honest apologies for not expressing appreciation to you earlier.

  8. I loved as much as you’ll receive carried out right here. The sketch is attractive, your authored subject matter stylish. nonetheless, you command get got an edginess over that you wish be delivering the following. unwell unquestionably come further formerly again since exactly the same nearly very often inside case you shield this increase.

  9. Someone essentially assist to make critically articles I’d state. This is the very first time I frequented your web page and up to now? I surprised with the research you made to create this particular put up extraordinary. Excellent task!

  10. I have been exploring for a bit for any high quality articles or blog posts on this kind of space . Exploring in Yahoo I finally stumbled upon this web site. Reading this info So i¡¦m satisfied to express that I have a very excellent uncanny feeling I discovered just what I needed. I such a lot surely will make sure to do not forget this web site and provides it a glance regularly.

  11. Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you could do with a few pics to drive the message home a bit, but other than that, this is wonderful blog. A fantastic read. I’ll certainly be back.