windows 2012 – dedup

If you have some archive data that you would like to copy onto a  win 2012 server, but dont know how to estimate the end volume size (if VM etc). Start small – about 1/2 of the total capacity of the drive. Robocopy the data onto the drive until nearly full, manually run dedupe, wait until complete, continue robocopy, repeat and rinse. There will be occasions where you will run out of disk space at that point its time to extend your disk.

You will need to enable deduplication on the file server, either hit the GUI or use the following to enable the module;

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication

The powershell commands that you’ll use;

Enable-DedupVolume E:
(enables E drive for deduplication)

Set-DedupVolume E: -MinimumFileAgeDays 0
(default is 5 days – which may be sufficient for achival type files)

Start-DedupJob E: –Type Optimization
(manually starts dedup on E drive)

(shows status of currently running dedup and scheduled dedup jobs)

I have used the following powershell script to continually run the dedup process on my archive drive (during migration work). Checks if dedup processes are running – if not execute dedup on archive drive.

$x = 1
while ($x = 1)
$dedupcheck = Get-Dedupjob | Out-String;
if ($dedupcheck -eq "") {
echo "starting dedup process on F drive";
Start-DedupJob F: -type Optimization;
echo "dedup jobs running - sleeping for 2 min";
sleep 120;
echo "";
echo "";

zfs – now has dedup!

Cool. zfs as of version 21 has deduplication built in. And thats the good dedup – synchronous dedup. i.e. deduped on the fly!

How easy is it to turn on? – very!

Once you have upgraded your zpool to 21 or above you can run the following command at the pool level and deduplication will be over all your data from that point onwards.

zpool set dedup=on tank


Note : Watch your performance, it will drop like a rock if you do not have enough ram for your dedup tables. Do some tests after enabling this feature.