While the road to deployment can be long and arduous, it’s not the final destination. Hyper-V has a solid record of dependable stability, but ongoing regular maintenance is a necessity. With a bit of planning, administrative effort of this upkeep can be minimized. Your goal is to keep Hyper-V and its virtual machines running as well as possible and be aware of any conditions that require attention, such as critical patches and regular cleanups.
Hyper-V and Operating System Patching
At this time, Microsoft releases patches each month on the second Tuesday. This cycle is commonly called “patch Tuesday”. Infrequently, patches for serious security flaws are released as quickly as Microsoft can clear them for production use. You’ll want to do what you can to keep your Hyper-V systems reasonably up-to-date. Leaving systems unpatched can result in sub-optimal performance and exposure to attacks. Patching on release day risks running afoul of an update with more problems than solutions. You’ll have to find a comfortable place between these two extremes.
For all but the smallest organizations, it’s wise to consider a multi-layered patching strategy. Systems have different operational needs and will often benefit from tailored patching.
Effects of Host Patching on Guests
Before you begin planning your patching schedule, it’s important to understand host-guest system interaction. Some patches, such as those for the .Net Framework, can cause services and applications to be taken offline while the patch is being applied, not just during a reboot. Hyper-V itself is not one of these systems. If a patch affects Hyper-V, it will be applied during the reboot portion only. The same is not necessarily true for management and monitoring applications.
Patching operations in the management operating system might possibly affect virtual machines because it does require more resources than normal operations. Hyper-V will always do its best to maximize resource allocation for guests, but some things, such as heightened disk activity in the management operating system, might have noticeable impact. In practical terms, the likelihood of noticeable performance degradation during patching is directly correlated with the degree of density and resource utilization.
When a Hyper-V host shuts down gracefully, it sends a notification first to the cluster service, if it is present. The cluster service takes responsibility for highly available guests. Then, Hyper-V deals with non-cluster guests. In a 2012 R2 cluster, highly available guests are Live Migrated when the host is rebooted. This process can only be stopped by host-locking guests through the Preferred Owners setting, but any HA guest that cannot be moved will always be placed in a saved state on host reboot.
Guests that are not highly available are handled by the local system in accordance with their Automatic Shut Down action. Of these options, Save is the safest. The graceful shut down option does not wait forever; you may find that some guests are forcefully turned off when the host is taken offline. When the host is fully online after the reboot, HA virtual machines are returned to their prior state and non-HA machines follow their Automatic Start Action.
General Patching Strategies
The most aggressive patching strategy is to simply set up each system to patch automatically. They’ll check roughly once per day for updates and apply them at a particular time. Since reboots should occur within an hour of the start of patching, this process could be predictable enough for a small environment, especially those that do not have round-the-clock operations. This scenario leaves you with the lowest possible risk of stability or security breaches due to out-of-date systems, but places you at the highest risk of damage from faulty patches.
The least aggressive patching method involves disabling all automatic patching and handling them in a fully manual fashion. This grants the highest level of control to the administrator, but also demands the greatest amount of effort and places the systems at the highest amount of risk due to the fallibility of human operators.
A balanced approach uses a delay of some sort. If possible, a test environment can be set up that replicates the production environment. Patches can be deployed there immediately and tested for impact. Once cleared, they can be installed on production systems. In order to facilitate this method, you can employ Windows Server Update Services (WSUS). This is a built-in component of Windows Server from 2012 onward and can provide a single management point for patching most Microsoft systems. You can learn more about this feature on TechNet.
If you can’t create a test environment, an alternative option is to only update a subset of your environment at a time. This can be handled through manual patching or by controlling update approvals using WSUS.
Hyper-V Host Patching Strategies
Patching the guests is fairly straightforward. They can be handled like any physical system. It’s the hosts that need some extra attention. Any downtime for the host will affect its guests in some fashion.
It would make sense to coordinate host and guest patching in some fashion. Since the guests will be taken offline when the host goes down, patching both together would require only a single maintenance window. Unfortunately, at this time, there is no freely-provided or simple automated way to synchronize host operations with the guests.
If all of your virtual machines have been set to Save as their Automatic Shut Down action, then you can set patching schedules in any fashion that you like. Even if a guest is in the middle of a patch cycle when the host is taken offline, it will cleanly resume when the host starts back up and brings its guests online.
If you have some guests set to Shut Down, then the safest approach is to schedule them to update at different times than the host so that they don’t overlap. Spreading the schedules of hosts and guests across different days can minimize the possibility of a host shutdown interfering with guest operations. However, it also means that guests will be offline twice in any patch cycle that causes a host restart.
The best way you can mitigate this effect in a single-host environment is to use Hyper-V Server as the management operating system as opposed to Windows Server. Hyper-V Server has fewer patches in general and, partially as a result, fewer reboots. Whatever you use, do not employ any more roles and services than are absolutely necessary.
Anything that is not directly involved with operating or maintaining virtual machines should be run inside a guest. The fewer components that are running in the management operating system, the fewer patches the management operating system will require, which translates directly into fewer opportunities for an update that requires interruption to guest services.
Cluster Aware Updating
In a cluster environment, you have access to a powerful tool called Cluster Aware Updating (CAU).This is a tool that automates the synchronization of downtime across nodes. Only one will be taken down at a time, and its guests will be automatically Live Migrated to other nodes before it is patched. The process will repeat until all nodes in the cluster are up-to-date. With (CAU), you can configure the hosts to patch at any time that you like, as they will not interfere with the guests in any way except for the Live Migration operation. You can read more about this feature and how to configure it here.
Virtual Machine File Cleanup
Over time, changes in virtual machines will likely result in an overuse of disk resources from overgrown and orphaned files. You have a few options that you can periodically undertake to reclaim this space.
Dynamically Expanding Disk Files
The default mode for new virtual hard disks is the dynamically expanding type. Despite a great deal of prejudice against it from the performance characteristics of prior versions, it is the best choice for most applications. You can read more about it in the storage section. Over time, these files will grow. For some situations, such as operating system disks, they will eventually normalize to a functional maximum. For others, they may yield some meaningful savings if they are compacted. This operation reads the allocation table inside the guest, looking for blocks that were used in the past but are now marked as empty. Those blocks are removed from the file and the file is shrunk accordingly.
While a compact operation is typically harmless, there is a danger. Any inconsistencies between the allocation table and the actual arrangement of data and free space can result in damage. It’s recommended that you run a chkdsk and get a backup of each disk before compacting it. This is especially true for non-Windows systems that might react to the compact operation in unexpected ways. To carry out a compact operation, refer to the storage section.
How to Compact Dynamically Expanding Hard Disk Files
To maximize the efficacy of compacting a dynamically expanding VHDX file, there are a few advance steps that you can take. First, use a tool to zero out any unused blocks on the disk from inside the guest operating system. Windows does not have this functionality natively, but there are a number of free and commercial applications available that can perform this task. Microsoft provides one through its Sysinternals site called SDelete. Linux has several utilities and methods as well; your best option is to perform an Internet search for how to zero free space on the distribution that you are using.
Defragmenting the virtual hard disk can also have some positive effect on the compact operation, although the extent is generally minimal. A defragment operation can sometimes consolidate numerous partially used sections into a more optimal layout.
The Virtual Hard Disk heading of section 3.4 describes how to perform a compact operation using Hyper-V Manager. For PowerShell, use the Optimize-VHD cmdlet. Pay special attention to the detailed description of the Mode parameter. It states that the most thorough mode of Full requires the virtual hard disk to be mounted in read-only state (as do the Quick and Retrim options). The following is an example of how this would be carried out:
Mount-VHD 'D:\Virtual Hard Disks\compactme.vhdx' –Readonly
Optimize-VHD -Path 'D:\Virtual Hard Disks\compactme.vhdx' -Mode Full
Dismount-VHD 'D:\Virtual Hard Disks\compactme.vhdx'
Orphaned Virtual Machine Files
System Center Virtual Machine Manager and other tools will perform a complete delete of a virtual machine. Hyper-V Manager and PowerShell will remove all of the virtual machine’s files except the virtual hard disks. These are easy enough to clean up.
The problem is that there are other ways for virtual machines to become orphaned. Storage Live Migration is intended to delete the source file, but that doesn’t always happen for various reasons. Improper merges or other problems could cause files to be left behind. Virtual machines could be exported or manually copied and these copies, or their sources, could be forgotten. There is no built-in automated method to clear these files, but a script does exist that can scan for them.
Standard System Maintenance and Monitoring
The management operating system is Windows Server (or, in the case of Hyper-V Server, a derivative). As such, it requires the same sort of routine care. This means watching event logs, keeping firmware and drivers up to date, etc. Fortunately, most driver installers will work just as well for a GUI-less management operating system as they will for the full Windows Server. For those few that won’t, you can use pnputil as explained in the section on installing Hyper-V in non-graphical environments.
Most traditional Windows monitoring systems can work with Hyper-V. You can use free systems like Server Manager and Performance Monitor to watch and alert on problems. You can use commercial solutions such as Microsoft’s System Center Operations Manager and third party tools for enhanced monitoring that requires less administrative overhead. The newly released Azure Operational Insights tool can be used to provide some similar services, even for your on-premises hosts. For many smaller customers, it is even free. More on that here.
If you’re looking for something more robust but are constrained by budget, a viable solution is to use Nagios (http://www.nagios.org). This is not a trivial undertaking for initial configuration, but it is not overly difficult to maintain once deployed. Refer to this article to get started: How to Monitor Hyper-V with Nagios
Management and Monitoring tools
Did you know we’ve collected a big list of 101 Free Hyper-V Management & Monitoring Tools and Resources on this blog? Worth browsing through and picking up something new.
