Using xcopy for backups

9–10 and 20–22 December 2012

Save often, save early

“Save often, save early”, says the old adage. “Better safe than sorry” is another one.

Programs like MS Word have an option for automatically saving the edited document periodically. I don’t like that and don’t use it, because it always happens at the wrong moment. I prefer pressing ctrl-s myself, when it logically makes sense, that is, after finishing a full sentence or paragraph.

It’s the same with making backups in a wider sense, backups of new and changed files: I want to be in control.

Xcopy

Most of DOS is rather unsophisticated. Unix command line tools (sometimes called user land programs) are much better.

Xcopy however is an exception to this rule. This is a very powerful and useful program. I use it for keeping my backups up-to-date.

Xcopy is somewhat reminiscent of rsync, which is available on some (or all?) Unix systems, although rsync is much more powerful, in that it also has capabilities for remote synchronisation of different computers. Xcopy can also do that, but only if the remote storage has been made make accessible as a network drive.

Complicated

Like so many powerful utilities, xcopy has a large number of rather complicated command line option (also called command line switches). I can never remember them.

So for convenience, I created an MSDOS batch file – well, in fact it is a script for a 32 bits Windows command line. More on that in my previous article. There I also described how I achieved that I can use the command conveniently, without specifying, or even remembering, where I stored it.

I gave my batch file (script, command file) the name xcopybat.bat . If you want to take my advice and use something similar, you can of course call it anything you like. Just don’t give it a name that already exists as a command in the system, because then you won’t be able to access that older command (unless you use the full path).

How to use `xcopybat`

Practical

Before going into some more details and backgrounds, I’ll give you an example of how I use my self-made xcopybat command to make backups.

In case you want to try my xcopybat yourself, this is what it contains:

xcopy /s/e/i/d/r/k/y/z *.* %*

Remember, in order to make it usable, you may need to take some measures first, as explained here.

Storing files on disk

For storing my files, I always use directories (also known as folders) to keep things organised. And I create these directories and subdirectories myself on drives like c: or d:, not from locations that Windows systems propose, such as ‘My Documents’ – which in fact, my user name being RuudH, is c:\Users\RuudH\Documents .

Of course you can use anything you like, included those predefined disk locations. You can (and should, in my opinion) create subfolders there too.

Anyway, my photograph files, for example, I keep under d:\Fotos. Below that are subfolders naming some persons who sometimes send me photographs I didn’t take myself. Pictures I did take myself (or my wife, usually) I store in subfolders, the name of which start with the date in the format yyyymmdd, or just the month: yyyymm. Then follows a hyphen (-), followed in turn by a textual description of what it is about (without any spaces, because these cause inconveniences when using them with command line tools).

So pictures of a holiday in Portugal in 2001, for example would be kept in d:\Fotos\200108-Vakantie-in-Portugal. (‘Vakantie’, as you might have guessed, is the Dutch word for holiday.) Other files than those containing photos, for work or fun (well, most of the fun is also a form of work, or might one day turn into that), I organise in a similar fashion.

Making and updating backups

I make backups on portable USB disk drives, USB sticks and disk drives on other computers, via the network. If you have more than one disk, or several partitions in your computer, you might also make backups from d: to c: or from c: to d:, etc.

Now suppose I want to backup my photos from d: to a USB disk, which when plugging it in, is automatically mounted as f: .

First I have to make the source directory (folder) the current directory. So in the command line, I issue the commands:
d:⏎
cd \fotos⏎
each followed by return (⏎, on some keyboards marked ‘Enter’). The command line prompt (i.e., what is displayed at the left, at the beginning of the line) reflects this, by becoming d:\Fotos> .

Now I run my xcopybat command to create the backup, as follows:
xcopybat f:\Fotos⏎

This creates (if it wasn’t already there) a folder called Fotos in the root directory of the backup drive F:. It also creates any subfolders, and subfolders of those, as needed. In other words, the complete directory structure (folder structure), that I decided to create on my computer disks in order to keep my files organised, is automatically reflected on the backup disk.

In the appropriate folders, copies of the files are stored. What’s more, next time I do that, only new files, and old files that have meanwhile changed, are written to the backup disk. This speeds up the backup process considerably.

Multiple copies

Hard disk storage is cheap nowadays. That includes portable USB disks. I own three of those, of 250, 500 and 350 gigabytes (GB) in size, bought over the years, in that order. An abundance of space. Of data I consider important, they allow me to keep as many as 23 copies on each of those three USB disks. Every time I make a backup, I use the disk I haven’t used the longest, and on that, I update the oldest backup.

It is important to have both recent and older backups. That’s because if you have ruined a document without noticing, recent backups tend to contain that broken file too. You need older backups of several dates, in order to find one that still, and already, contains the lost pieces of data.

To see which backups are recent and which are not, I use an implementation of the well-known Unix ‘list sorted’ program (ls). For example, I keep a local working copy of my website under d:\rudhar.com, and to check the ‘whatsnew’ file of the backup copies on drive f:, I issue the command:

ls -lt ru*/in*/w*w/whatsnew.htm.

The -l means ‘long listing’ -t tells ls to sort the files by date and time. This results in (somewhat simplified):

 2786 Dec 21 12:12 rudha1.com/index/whatsnew/whatsnew.htm
 2786 Dec 21 12:12 rudha9.com/index/whatsnew/whatsnew.htm
 2786 Dec 21 12:12 rudha8.com/index/whatsnew/whatsnew.htm
 2786 Dec 21 12:12 rudhar.com/index/whatsnew/whatsnew.htm
55473 Dec 20 14:20 rudha11.com/index/whatsnew/whatsnew.htm
55473 Dec 20 14:20 rudha10.com/index/whatsnew/whatsnew.htm
54119 Dec 12 11:44 rudha4.com/index/whatsnew/whatsnew.htm
54119 Dec 12 11:44 rudha5.com/index/whatsnew/whatsnew.htm
53944 Dec 11 23:26 rudha6.com/index/whatsnew/whatsnew.htm
53944 Dec 11 23:26 rudha7.com/index/whatsnew/whatsnew.htm
53698 Dec  9 15:36 rudha3.com/index/whatsnew/whatsnew.htm
53698 Dec  9 15:36 rudha2.com/index/whatsnew/whatsnew.htm

This tells me that backups number 2 and 3, being the oldest one, are eligible for the next round.

The MSDOS command dir also has an option (/od) to show results sorted by modification time, and it supports wild cards (*), but not as well as Unix tools (actually: shells): in those, the wild cards can also be used for directory names on all levels, not just for file names.

A similar Unix command tells me which is my oldest e-mail backup on the current USB disk or memory stick:

ls -lt f:/in*/eu*/in.mbx

In.mbx is Eudora’s inbox and d:\internet\eudora is where I keep both the program and its data, that is, the mailboxes and their index files (*.toc), in several folders.

This way, I can keep some 15 copies (by varying the path name element internet to interne1, interne2 etc.), of some 18 years of e-mails, currently in 98 folders which contain a total of 3031 thematically organised mailboxes. Yet, backing them up takes only a few minutes.

I you might be trying something similar: do not forget to unmount the USB drive the proper way first, using the green icon in Windows’ lower right bar, before pulling the USB plug. With large amounts of data being transferred, it may take a while before everything is settled on the disk. If you pull the plug too soon, the data will get corrupted.

Command line options used

This is background info only. You can use the command xcopybat as is, without knowing why it works the way it does.

But if you are curious, here is some info. As said, the command I put in my batch file is: xcopy /s/e/i/d/r/k/y/z *.* %*

You can obtain descriptions of the command line switches (the letters after the slashes) by running xcopy /? | more.

/E specifies that xcopy “copies directories and subdirectories, including empty ones”. /S does the same, but without the empty ones. So I could have omitted /S, but it doesn’t hurt to mention that one as well.

/I is used so the target directory is created if it wasn’t already there.

The purpose of command line switch /D is that only those files are copied that are newer the corresponding file in the backup path.

/R makes xcopy overwrite read-only files: if such a file has been changed in my working copy (under the current directory), despite being write-protected (that is, read-only), I do want it to be backed up. /K makes the backup files read-only if the original file was, and /Y means that xcopy should overwrite existing files (i.e., older backup files, backed up earlier) without asking me all the time. I often back up a lot of files at a time, so I don’t want all those questions.

The last parameter is /Z, which makes sure that partially copied files will not be left behind and not rewritten in a following attempt, in case the backup is done over a network that fails in the middle of the command. Or some other smart algorithm I may not fully understand and that is none of my business as long as it works well.

After the command line switches (parameters), I specified *.* to mean ‘all files’. (On a Unix system, * would be used for this, and *.* would be interpreted literally, i.e. only files with at least one dot in the name would match.)

Finally, there is a %* to achieve that any command line parameters that are given to xcopybat, will be passed on to xcopy. In practice, there is only one: the target directory. But this could probably be used to pass extra command line switches to xcopy by specifying them as such to xcopybat. I never do that and I didn’t test it.

Note: directory trees

ICT people, or perhaps only dinosaurs like me, have the strange habit of depicting the tree structure, in which directories and files are stored on a disk, with the root on top. As you know, biological trees grow into the air so they have their roots below, in the soil.

To make it worse, in diagrams and path name strings, the root is usually supposed to be to the left, so “lower” directories (also called subdirectories or subfolders) appear more to the right.

Example: if a photo file is stored as
d:\Fotos\200208-Vakantie-Portugal\PonteDoLima\DSC0012.jpg ,
200208-Vakantie-Portugal is a subdirectory (lower level folder) of Fotos . Likewise, PonteDoLima is said to be “below” 200208-Vakantie-Portugal , which is under Fotos , which in turn is under the root directory ‘\’ of drive D: .

This may be confusing if you’re not used to it, so I thought I’d explain.