windows 2012 – dedup

If you have some archive data that you would like to copy onto a  win 2012 server, but dont know how to estimate the end volume size (if VM etc). Start small – about 1/2 of the total capacity of the drive. Robocopy the data onto the drive until nearly full, manually run dedupe, wait until complete, continue robocopy, repeat and rinse. There will be occasions where you will run out of disk space at that point its time to extend your disk.

You will need to enable deduplication on the file server, either hit the GUI or use the following to enable the module;

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication

The powershell commands that you’ll use;

Enable-DedupVolume E:
(enables E drive for deduplication)

Set-DedupVolume E: -MinimumFileAgeDays 0
(default is 5 days – which may be sufficient for achival type files)

Start-DedupJob E: –Type Optimization
(manually starts dedup on E drive)

Get-DedupJob
(shows status of currently running dedup and scheduled dedup jobs)

I have used the following powershell script to continually run the dedup process on my archive drive (during migration work). Checks if dedup processes are running – if not execute dedup on archive drive.


cls
$x = 1
while ($x = 1)
{
$dedupcheck = Get-Dedupjob | Out-String;
if ($dedupcheck -eq "") {
echo "starting dedup process on F drive";
Start-DedupJob F: -type Optimization;
};
echo "dedup jobs running - sleeping for 2 min";
sleep 120;
echo "";
get-dedupstatus;
echo "";
}

Windows / Linux – File Migration Tools

In Windows i use robocopy

This is the syntax to copy from one location to another…  (i like how robocopy uses UNC paths)

robocopy \\192.168.9.70\videos \\192.168.9.101\videos /e /r:2 /w:10 /sec
robocopy \\192.168.9.70\software \\192.168.9.101\software /e /r:2 /w:10 /sec

Note: i always copy instead of the move. Run multiple times before switch over to shorten time for final sync. Always use the /SEC to copy over the NTFS permissions. I have had problems with the non X010 version of robocopy not copying NTFS permissions across. Check NTFS permissions before removing the source files. You can update files NTFS permissions with /SECFIX switch if required.

/e moves all directories including empty ones.

In Linux i use rsync

i typically mount a smb share on a windows host…. (you must have cifs client on linux first)

mount -t cifs -n //192.168.9.70/backup /mnt/backup -o username=user,password=mypassword

then use rsync for the copy work…

rsync -rcav /etc/ /mnt/backup/etc/

if your after linux to linux copying then rsync is also the best way to do it (via ssh), logon to your destination box via ssh and run this from your distination diretory..

rsync -ave ssh 192.168.9.10:/var/lib/mysql .