Using Robocopy to create a Windows backup

I wanted to create a regular incremental backup of my local files on my Windows box, and copy them to a NAS.

I ended up using robocopy, a batch file, and the Windows scheduler to achieve this.

Read on to see how.

There is information all over the internet for doing this, but I found it in dribs and drabs in various places, so in the spirit of DataHamster I have collated it here into one post.

This post is a bit of a work in progress, so I will be updating and expanding it periodically.

Overview

I’ve wanted a regular backup ability on my Windows box for a while now, but without having to buy extra software.

The built in Windows 7 backup has proved to be unsuitable as a) it is really unreliable and often fails on me, and b) it can’t cope with drives that are bigger than 2TB.

Criteria

My criteria for a backup solution:

Must be free.
Must do actual file copies rather than putting everything into a proprietary binary file.
Must be able to back up to a network share.
Must create a sensible directory structure on the destination.
Must be able to only copy files that have changed.
Must remove files on the destination that have been deleted on the source, and add files that have been added. In other words, a simply mirror will be sufficient – I don’t need a full versioned historical backup. My NAS can do periodic snapshots so the primary goal is to get the files onto the NAS.
Must be easy to maintain – add and remove folders without editing the script, so reading the folders from a text file for example.
Must be able to be scheduled.
Ability copy open files using Shadow Copy is desirable.

Solution

Robocopy

I decided that robocopy, in conjunction with a Windows batch file, would achieve my goals. Robocopy is kind of like xcopy on steroids, although it is directory-based rather than file-based. But that’s exactly what I want.

However, robocopy doesn’t support Shadow Copy. This could be rectified with something like ShadowSpawn or using the VSS SDK directly.

File list

The files will be read from a file called robobackup_dirs.txt which will reside in the same folder as the batch file (see next).

To populate the list, you could go into Windows Explorer and Ctrl-LeftClick all the folders you want to include, then Shift-RightClick and select Copy as Path and paste it into the file. The folders will be enclosed in double quotes, but that’s ok as our batch file is going to be able to cope with that.

Our script will also be able to cope with blank lines. And you can add comments, or comment out a line, by starting a line with a semi-colon.

The batch file

This is where we iterate over the folders in robobackup_files.txt and call robocopy on each.

robobackup.bat

@echo off
setlocal DisableDelayedExpansion
set "destpath=\\GEORGE\backup\%COMPUTERNAME%"
set "logfile=%destpath%\backup.log"
	
if not exist %destpath% mkdir %destpath%
if exist %logfile% copy nul %logfile%

for /F "delims=" %%i in (robobackup_dirs.txt) do (
	if not "%%~di"=="\\" call :process "%%~i" "%%~di" "%%~pni"
)
echo.
echo Done!
exit /b
goto :EOF

:upper
	for %%a in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do call set "%1=%%%1:%%a=%%a%%%"
goto :EOF

:process
	setlocal EnableDelayedExpansion
	set source=%~1
	set di=%~2
	set pni=%~3
	if "%pni:~-1%"=="\" set pni=%pni:~0,-1%
	set drive=!di:~0,1!
	call :upper drive
	set "dest=%destpath%\!drive!!pni!"

	call robocopy "!source!" "!dest!" /MIR /W:10 /R:1 /XA:SH /MT:32 /XJD /XJF /FFT /DST /Z /dcopy:T /a+:a /NP /TEE /log+:%logfile%
	endlocal
goto :EOF

Let’s break that down. If you’re not interested and just want to use it “as is” then feel free to skip to the next section.

@echo off

If you need this line explaining then you really should skip to the next section. 🙂

setlocal DisableDelayedExpansion

The FOR loop has been shown to have issues with Delayed Expansion set, so we need to turn it off.

set "destpath=\\GEORGE\backup\%COMPUTERNAME%"
set "logfile=%destpath%\backup.log"
	
if not exist %destpath% mkdir %destpath%
if exist %logfile% copy nul %logfile%

I have chosen to hardcode the destination path in the variable destpath which will obviously be different for you.

My NAS is called GEORGE and it has a Windows share called \\GEORGE\backup
%COMPUTERNAME% is a system environment variable that yields the name of the computer on which the batch file is running.

So if your computer is called Hamster, then the backup destination is going to be \\GEORGE\backup\Hamster

We then make sure that this folder exists by creating it if it doesn’t, and we also define the name of the logfile and then, if the file exists from a previous run, we clear it.

for /F "delims=" %%i in (robobackup_dirs.txt) do (
	if not "%%~di"=="\\" call :process "%%~i" "%%~di" "%%~pni"
)

We now iterate over all lines in robobackup_dirs.txt. Blank lines and comment lines will be skipped. The rest of the lines are going to be interpreted as paths.

%%~di is the drive letter part of the path. For non-UNC paths, this will be the drive letter (eg. C:), whilst for UNC paths this will be “\\”. We skip those. But for non-UNC paths we call a function called process, which I will come to shortly.

%%~i is the whole line, but with quotes removed.

%%~pni is the path and filename. This is necessary as something like d:\documents will be interpreted as a filename but d:\documents\ will be interpreted as a path. But %%~pni will grab both.

:upper
	for %%a in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do call set "%1=%%%1:%%a=%%a%%%"
goto :EOF

This is just a little helper function that makes the contents of a variable upper case

:process
	setlocal EnableDelayedExpansion
	set source=%~1
	set di=%~2
	set pni=%~3
	if "%pni:~-1%"=="\" set pni=%pni:~0,-1%
	set drive=!di:~0,1!
	call :upper drive
	set "dest=%destpath%\!drive!!pni!"

	call robocopy "!source!" "!dest!" /MIR /W:10 /R:3 /XA:SH /MT:32 /XJD /XJF /FFT /DST /B /Z /dcopy:T /a+:a /NP /TEE /log+:%logfile%
	endlocal
goto :EOF

This function is the meat of the batch file.

First we store the parameters, stripping off any quotes (hence the ~).

If the path ends in a back slash then we remove it. We also remove the colon from the drive letter and make it upper case.

Then we assemble the destination path. So if you were backing up c:\My Files from a computer called Hamster, then the destination path would be \\GEORGE\backup\Hamster\C\My Files

We then call robocopy. We do the best we can without shadow copy (so some retries for locked files, but not waiting for ages), use network restartable copying, do a mirror copy (as per requirements) skipping system and hidden files, we skip junction points to prevent infinite recursion issues, correct for DST and other timestamp issues, and use multi-threading.

If you wish more detail then read the command in conjunction with the robocopy documentation, and that will explain all.

Creating a task schedule

I don’t intend to go into this in great detail as this is very well documented.

At it’s simplest, this involves launching the Windows Task Scheduler and selecting Create Basic Task, and then getting it to run robobackup.bat on a periodic basis.

It’s worth noting that robocopy can be instructed to schedule a recurring task, and even check for changes in real time, but I did not explore this option.

TODO

Use Shadow Copy. This will involve creating a shadow drive for each source drive. For greater efficiency, I’ll make sure I only mount a new shadow drive if the drive letter has changed. Users could be encouraged to sort their list of folders by drive letter for greatest efficiency although it won’t be a prerequisite.
Provide a way of specifying a file which holds a list of file names and folder names to exclude, like the xcopy parameter /EXCLUDE:file
Provide an ability to specify which folders should be recursed into and which should not. Currently folders are always recursed. This may need to be implemented as a list of folders to exclude, since that’s the way robocopy works in /MIR mode.