Opensolaris – ZFS recovery after kernel panic

Recently i hit what i thought was a huge disaster with my ZFS array. Essentially i was unable to import my zpool without causing the kernel to panic and reboot. Still unsure of the exact reason, but it didn’t seem to be due to a hardware fault. (zpool import showed all disks as ONLINE)

When i tried to import with zpool import -f tank the machine would lockup and reboot (panic).

The kernel panic;  (key line)

> genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776 size=22528)

Nothing i could do would fix it… tried both of these options in the system file with no success;

set zfs:zfs_recover=1
set aok=1

After a quick email from a Sun Engineer (kudos to Victor), the zdb command line that fixed it;

zdb -e -bcsvL <poolname>

zdb is a read only diagnostic tool, but seemed to read through the sectors that had the corrupt data and fix things??  (not sure how a read only tool does that) – the run took well over 15hrs.

Updated: 20/10/2009

Apparently if you have set zfs:zfs_recover=1 in your system file the zdb command will operate in a different manner fixing the issues it encounters.

Remember to run a zpool scrub <poolname> if you are lucky enough to get it back online.

This thread has some additional info…

Update 31/05/2012

This command has also helped me when i cant mount a pool in RW mode

zpool import -F -f -o readonly=on -R /mnt/temp zpool2

3 Replies to “Opensolaris – ZFS recovery after kernel panic”

  1. After a power outgage, my freenas 7 box always returned a kernel panic when tried to import the pool. The OS disk died, so i needed to install a new environment.
    Tried different things:

    – Freenas 7 -> Always panic. Old ZFS version with less options. No “readonly” for example.
    – Nas4Free 9 -> Graid detected the disks as a Nvidia Raid (¿?). Maybe old metadata on disks…
    – Opensolaris 11 -> Didnt find the disks correctly, so unable to import due to vdev error.
    – Freenas 8.3 -> I ran the “zpool import -F -f -o readonly=on -R /mnt/temp zpool2” command and it worked. So im going to backup all the data and re-create the pool.


  2. the readonly mode was very helpful. Next time this happens i’ll try the zdb command.

    I was really worried I’d loose an entire array. It happened at 2am without any warning and no power outage or anything. I still have faith in zfs, but this is very strange.

Leave a Reply

Your email address will not be published.