Server Backup Puzzle

Here's the puzzle: while hard disks have gotten larger and cheaper, the day is still 24 hours long. Backup software seems to process files at 20-25 GB/hour, so if you have more than 500 GB on a server to backup each night, you cannot finish one job before needing to start the next. This may lead to problems.

Why is backup software so slow? For comparison, a straight file-copy job in Windows or the command line goes at the rate of about 100 GB/hr.

• It doesn't matter if you are backing up to a USB-attached external hard disk, a NAT device or an eSATA hard disk; the rate stays the same (20-25 GB/hr).
  
• It doesn't seem to matter what the software is either - Backup Exec, NTBackup, et al.
  
• Curiously, backing up a single file, like an Exchange data store, runs more quickly - 40 GB/hr.

My solution, if you can call it that, is to:

1. Write a batch file to copy files that are not open/locked by running services. Then schedule the batch file to run every night.
   
2. Use backup software to backup files that are open/locked.

Kan Yabumoto explains the issue of open/locked files, "As in any multi-tasking environment (including networked environments), when a file is opened by a process (a running program), an attempt to access the file by another process (such as the XXCOPY program) is totally dependent on how the program which opened the file in the first place wants to share it. Often times, the first program is not very cooperative, or just self-centered and won't give any access to anyone else." http://tech.groups.yahoo.com/group/xxcopy/message/1426

Backup software uses the Volume Shadow Copy Service (VSS) to make/use shadow copies of files to get around the access/sharing problem.

There are at least two problems with my solution:

1. Some if not all of the batch-file/command-line copy commands do not handle long file names gracefully.
2. It is messy and not always obvious which files need to be backed up versus those which can be copied.

If there is a better way, I'd appreciate it if someone would let me know!