VSS Crash-Consistent vs. Application-Consistent VSS Backups (post 2 of 2)

When is Application-Consistent Backup Vital? Not all situations require an application-consistent backup. Things such as file and print servers will be fine with crash-consistent and possibly inconsistent backups. If your application doesn’t provide a VSS writer, there might not even be a way to get an application-consistent backup of it while its containing machine is live. The most common need for application-consistent backups is the usage of database-backed applications.

NOTE: This is the second blog post in a 2-series post.  You can read the first post in this series here.

When is Application-Consistent Backup Vital?

Not all situations require an application-consistent backup. Things such as file and print servers will be fine with crash-consistent and possibly inconsistent backups. If your application doesn’t provide a VSS writer, there might not even be a way to get an application-consistent backup of it while its containing machine is live. The most common need for application-consistent backups is the usage of database-backed applications. In particular, Microsoft Exchange and SQL Server are fairly dependent upon application-consistent backups. In their default configurations (non-circular logging and full recovery modes, respectively), they do not normally write directly to their database files but instead rely upon log files. When their VSS writer is triggered, they flush all pending operations from memory and commit the contents of their log files into the main database files. This behavior ensures that a crash during normal operations will pose a risk to the active log file, not the primary database. If application-consistent backups are not taken on a regular basis and these applications are left in their default configurations, the log files will eventually consume all available space. Even if transaction quantity is low enough and disk space ample enough to prevent consumption from being a major concern, restoring either system from a crash-consistent state is much more painful and risky than restoring from an application-consistent state.  Also, if these applications are left in their default configuration and backed up using a non-application-consistent method, even if it is a full image-level backup, they will not commit their logs.

Backup Operations in Hyper-V

Looking at the simplistic file-based layout of a virtual machine, it is tempting to assume you can use any available file-based backup or even copying software. Unless you are shutting down or saving the virtual machines, avoid falling into this trap. If you are saving the VMs, ensure you are including the VSV and BIN files in your copies. Even if you follow all these steps, Hyper-V takes a particular ownership of its virtual machines and won’t like it if you try to simply copy in the files of a virtual machine that were running on a different host. If you intend to back up the virtual machines without saving them, ensure that whatever method you utilize triggers VSS. There are a handful of Internet posts suggesting that you can simply run copying tools like Robocopy against your VHD files and everything will be fine. Without calling on VSS to pause I/O first, this technique can easily result in a VHD that is unusable. If you intend to use nothing but freely available tools and scripts, research things like Diskshadow and implement your scripts properly.

Hyper-V has its own VSS writer. This deceptively simple sentence means that by deploying on Hyper-V, you automatically have the ability to get application-consistent backups of virtual machines from the host level. There are some conditions to be met in order to ensure this works as expected, but the basic operation is that when Hyper-V’s VSS is triggered, it notifies VSS within the virtual machine, which in turn notifies all of its registered VSS writers that a backup is taking place.

No VSS Writer Available?

In some cases, you need an application-consistent backup but there is no VSS writer available. One example of this is MySQL. Hyper-V backups of virtual machines containing MySQL will always result in either a crash-consistent or an image-level backup. For MySQL, the latter is probably acceptable as MySQL doesn’t perpetually expand the log file. However, if you’re using MySQL within a VSS-aware VM, then a Hyper-V-based backup tool is going to take a crash-consistent backup. MySQL (like any other database system) isn’t always recoverable from a crash-consistent backup; even when recovery is possible, it may be painful. MySQL is just one example; any number of line-of-business applications could tell a similar tale. In the case of MySQL, one solution is to find a guest-level backup application that is MySQL-aware and can back it up properly. For applications for which no backup application has a plug-in, you may need to have pre- and post-backup scripts that stop services or close applications. If brief downtime is acceptable, you can disable the Backup item in Hyper-V Integration Services, thereby forcing Hyper-V to save the state of the VM during backup. This technique results in an image-level backup and can be used on any application that doesn’t have a VSS writer.

General Requirements for Application-Consistent Backups in Hyper-V

Note that some backup software contains its own methods to ensure application consistency and may be able to circumvent some or all of these rules. Please consult with your application vendor for further details.

• The VM must have a current version of Hyper-V Integration Services installed and the Backup component must be enabled.
• The VM must have a compliant implementation of VSS (Vista Enterprise or later for desktop operations and Windows Server 2003 SP2 or later for servers)
• All of a VM’s VHDs must reside on the same volume (whether it is local or CSV)
• All of the VHDs must be formatted with NTFS. This is a limitation even in a physical environment as VSS does not interact with FAT volumes.
• None of the VHDs can contain Dynamic volumes; they must all be Simple (not to be confused with Dynamic vs. Fixed VHD, either of which can be used)
• The volume holding the VHDs must have enough free space for the VSS snapshot; volumes contained by the VHDs must also have enough space to hold their VSS snapshots. The exact need will vary based on several conditions but the safe line is at 20%.

Special Considerations for Cluster Shared Volumes

With VSS operating at the volume level, it might seem that a Cluster Shared Volume is automatically covered, but this isn’t precisely true. VSS does operate on volumes, but it is constrained to the host it is running on. A CSV is simultaneously connected to multiple hosts. In normal mode, only the metadata for a CSV is handled by a single host; all other virtual machine data can be directly accessed by the host that owns it. Therefore, when VSS quiesces a CSV and takes a snapshot, no other host in the cluster is aware of it. In this instance, the volume-level nature of VSS can be a threat. Even if you’re specifically calling VSS on a host to back up only a VM owned by that host, VSS is snapping the entire CSV. No other host is aware of the VSS operation and will carry on with its I/O operations, so the entire status of the VSS snapshot is questionable. Furthermore, and much more dangerous, there is nothing preventing another host from initiating a VSS snapshot while the first is still being used. Partly to address this situation, Microsoft implemented “Redirected Access” mode. In this state, all I/O for a given CSV is handled by the coordinator node. If any other host in the cluster wishes to access that CSV, its I/O is encapsulated in SMB blocks and funneled through the coordinator node. This ensures that the VSS snapshot is not disturbed and that it is not trampled on by another VSS operation.

If you are using CSVs, your backup solution absolutely must be able to turn on redirected access mode. There is no guaranteed safe way to copy virtual machine data off of a CSV without turning on redirect access unless the VSS snapshot is not used and the VM is saved or turned off.

NOTE: This is the second blog post in a 2-series post.  You can read the first post in this series here.

Have any questions?

Leave a comment below!

Backing up Hyper-V

If you’d like to make backing up your Hyper-V VMs easy, fast and reliable, check out Altaro Hyper-V Backup v4. It’s free for up to 2 VMs and supports Hyper-V Server 2012 R2! Need more? Download a 30-day trial of our Unlimited Edition here: http://www.altaro.com/hyper-v-backup/.

(Don’t worry, we hate spam too!)