Opensolaris – ZFS recovery after kernel panic
Recently i hit what i thought was a huge disaster with my ZFS array. Essentially i was unable to import my zpool without causing the kernel to panic and reboot. Still unsure of the exact reason, but it didn’t seem to be due to a hardware fault. (zpool import showed all disks as ONLINE)
When i tried to import with zpool import -f tank the machine would lockup and reboot (panic).
The kernel panic; (key line)
> genunix: [ID 361072 kern.notice] zfs: freeing free segment (offset=3540185931776 size=22528)
Nothing i could do would fix it… tried both of these options in the system file with no success;
set zfs:zfs_recover=1
set aok=1
After a quick email from a Sun Engineer (kudos to Victor), the zdb command line that fixed it;
zdb -e -bcsvL <poolname>
zdb is a read only diagnostic tool, but seemed to read through the sectors that had the corrupt data and fix things?? (not sure how a read only tool does that) – the run took well over 15hrs.
Updated: 20/10/2009
Apparently if you have set zfs:zfs_recover=1 in your system file the zdb command will operate in a different manner fixing the issues it encounters.
Remember to run a zpool scrub <poolname> if you are lucky enough to get it back online.
This thread has some additional info…
http://opensolaris.org/jive/message.jspa?messageID=479553
Update 31/05/2012
This command has also helped me when i cant mount a pool in RW mode
zpool import -F -f -o readonly=on -R /mnt/temp zpool2
Thanks! This post saved me from rebuilding a zpool that had about 6TB worth of data!
After a power outgage, my freenas 7 box always returned a kernel panic when tried to import the pool. The OS disk died, so i needed to install a new environment.
Tried different things:
- Freenas 7 -> Always panic. Old ZFS version with less options. No “readonly” for example.
- Nas4Free 9 -> Graid detected the disks as a Nvidia Raid (¿?). Maybe old metadata on disks…
- Opensolaris 11 -> Didnt find the disks correctly, so unable to import due to vdev error.
- Freenas 8.3 -> I ran the “zpool import -F -f -o readonly=on -R /mnt/temp zpool2″ command and it worked. So im going to backup all the data and re-create the pool.
Thansk!!