The Actual Performance Impact of Spectre/Meltdown Hyper-V Updates

The Spectre and Meltdown vulnerabilities have brought a fair amount of panic and havoc to the IT industry, and for good reason. Hardware manufacturers and operating system authors have been issuing microcode updates and patches in a hurry. As administrators, we need to concern ourselves with three things: the risks of running unpatched systems, the performance hit from patching, and quality control problems with the patches. If you’re skimming, then please pay attention to the section on ensuring that you get the update — not everyone will automatically receive the patches. Below I’ll run down what you need to know to ensure you’re protected plus a benchmark analysis of the performance impact of the recently released update patches.

What are Spectre and Meltdown?

Both Spectre and Meltdown are hardware attacks. Your operating system or hypervisor choice does not affect your vulnerability. It might affect fix availability.

Spectre is a category of assault that exploits a CPU’s “branch prediction” optimizations. A CPU looks at an instruction series that contains a decision point (if condition x then continue along path a, else jump to b) and guesses in advance whether it will follow the “else” code branch or continue along without deviation. The Wikipedia article that I linked contains further links to more details for those interested. Because the Spectre vulnerability encompasses multiple attack vectors, the predictor has more than one vulnerability. All of the ones that I’m aware of involve fooling the CPU into retrieving data from a memory location other than the code’s intended target. Spectre affects almost every computing device in existence. Fixes will affect CPU performance out of necessity.

Meltdown belongs to the Spectre vulnerability family. It affects Intel and some ARM processors. AMD processors appear to be immune (AMD claims immunity; I do not know of any verified tests with contrary results). As with Spectre, Meltdown exploits target the CPU’s optimization capabilities. You can read the linked Wikipedia article for more information. For a less technical description, Luke posted a great analogy on our MSP blog. Whether you read more or not, you should assume that any running process can read the memory of any other running process. The fix for this problem also includes a performance hit. That hit is potentially worse than Spectre’s — an estimated 5 percent to as much as 30 percent. I’ll revisit that later.

What are the Risks of Running without Patches?

Properly addressing Spectre and Meltdown will require more administrative effort than most other problems. Admins in large and complex environments will need to plan deployments. With unpredictable performance impacts, some will want to wait as long as possible before doing anything. The more avoidant of us will want to just ignore it as much as possible and hope that it just sorts itself out. Eventually, we’ll all have new chips, new operating systems, and redesigned applications that won’t be susceptible to these problems. That won’t be ubiquitous for some time, though. We need more immediate solutions.

In one sense, an unpatched system will be fairly easy to assault. Attacks can be carried out via simple Javascript. Unprivileged user code can access privileged kernel memory. The processor cache is an open book for many of these vulnerabilities. It looks to me as though some attack vectors could be used to read essentially any location in memory, although some other accounts dispute that. These are hardware problems, so in the hypervisor world, they transcend the normal boundaries between virtual machines as well as the management operating system. These threats are serious.

However, they cannot be exploited remotely. Attacking code must be executed directly on the target system. That separates the Spectre and Meltdown vulnerabilities from other nefarious vectors, such as Heartbleed. Furthermore, Spectre-class attacks are a bit of a gamble from the attacker’s side. Even if they could read any location in memory, it will be mostly without context. So, something may look like a password, but a password to what? The clues might be alongside the password or they might not.

So, the risk to your systems is extremely high, but the effort-to-reward ratio for an attacker is also high. I do know of one concern: LSASS keeps the passwords for all logged-on users in memory, in clear text. You should assume that a successful Spectre or Meltdown attack might be able to access those passwords. Also, attacks can come from Javascript in compromised websites. You shouldn’t be browsing from a server at all, so that could help. But what about remote desktop sessions? Do you keep those users off of the general Internet? If not, then any of them could unwittingly risk everyone else on the same host.

I cannot outline your risk profile for you, but I would counsel you to work from the assumption that a compromise of an unpatched system is inevitable.

Should I Patch My Hyper-V Hosts, Guests, or Both?

One of my primary research goals was to determine the effects of only applying OS patches, only applying firmware updates, and doing both. I did not find full, definitive answers. However, it does appear that some kernel protections only require OS updates. Others indicated that they would require firmware updates, but they did not make it clear whether or not a firmware update alone would address the problem or if a combination of firmware and OS updates were necessary. However, there is no question that the best protection only comes from updating your hardware and software.

Also, the guests’ kernels must be made aware of the changes to the physical layer in order to fully utilize their mitigation techniques, which can only be done if both the host and the guest are updated.

Therefore, for the fullest protection, you must:

• Patch the host
• Perform host configuration changes
• Update the firmware on the host
• Patch the guests
• Cold boot the guests

Also, be aware that “host” means any Hyper-V host. Windows Server, Hyper-V Server, or desktop Windows — patch them all.

We’ll go over the how-to in a bit. Let’s knock out some other questions first.

What Happens if a Hyper-V VM Updates Before Its Host?

Well, the host isn’t patched, so that’s a problem. Aside from that, nothing bad will happen (generally speaking — outliers always exist). But the guest will not be aware of the changed capabilities of the host if it is not cold booted after the host is updated. Until all of the above has been done, the guest will not have the fullest protection.

What If No BIOS Update is Available?

If your hardware does not have an available update, then you still need to apply all software patches. That will mitigate several kernel attacks.

If you are running Windows/Hyper-V Server 2012 R2 or earlier as your management operating system, I’m sad to say that you cannot do much else. Practice defense-in-depth: if no attacker can access your systems, then they cannot execute an attack. Remember that the patches are still applicable and will help — but don’t lose sight of the hardware-based nature of the problems. If possible, upgrade to 2016.

For 2016, you can restrict which processors and cores that the management operating system and guests use. That helps to confine attacks. The relevant features are not well-documented. I had planned on performing some research and writing an article on them, but none of that has happened yet. In the meantime, you can read Microsoft’s article on using these features to mitigate Meltdown and Spectre attacks on systems that do not have an available hardware update. It contains links to instructions on using those features.

What About Non-OS Software?

Software plays an enormous role in all of this. Application developers can take precautions to protect their cached information. Interpreters and compilers can implement their own mitigations. The January updates included patches for IE and Edge to help block Spectre and Meltdown attacks via their Javascript interpreter. Microsoft has released an update to their C++ compiler that will help prevent Spectre attacks.

Expect your software vendors to be releasing updates. Remember that the nature of the attacks means that otherwise benign software can now be an attack vector.

Quality Control Problems with Spectre and Meltdown Patches

I am aware of three problems in the deployment pipeline for the Spectre and Meltdown fixes from Microsoft.

1. Older AMD processors (Athlon series) reacted very badly to the initial patch group that contained the Spectre and Meltdown fix. In response, Microsoft stopped offering them while working on a fix. I have a Turion x2 system with Hyper-V that was offered the patch after that, so I believe that Microsoft has now issued a new fix.
2. Many antivirus applications caused blue screen errors after the fixes were applied. To prevent that, Microsoft implemented a technique that will ensure that no system will receive the update until it has been deliberately flagged as being ready. Pay special attention, as that affects every system!
3. Some systems simply will not take the update. I wish that I could give you some solid guidance on what to do. I do not know of any tips or tricks particular to these patches. I had one installation problem that I was able to rectify by disabling the “Give me updates for other Microsoft products when I update Windows” option. However, that particular system seemed to have problems of its own apart from the patches, so I cannot say for certain if it was anything special with this patch. You may need to open a ticket with Microsoft support.

How to Ensure that You Receive Microsoft’s Spectre and Meltdown Patches

These patches were deployed in the January 2018 security patch bundle. My system showed it as “2018-01 Security Monthly Rollup for OS Name” with a follow-up Quality Rollup. That deployment vehicle might lead you to believe that just running Windows Update will fix everything. However, that’s not necessarily the case. In order to avoid blue screens caused by incompatible antivirus applications, the updater looks for a particular registry key. If it doesn’t exist, then the updater assumes that the local antivirus is not prepared and will decline to offer the update. Since Microsoft now issues security patches in cumulative roll-up packages, it will decline every subsequent security update as well.

If Your System Runs Antivirus

In this case, “system” means host or guest. If you run an antivirus application, it is up to the antivirus program to update the registry key once it has received an update known to be compatible with the Spectre/Meltdown fixes. If you have an antivirus package and the updater will not offer the January 2018 security pack, contact your antivirus vendor. Do not attempt to force the update. Switch antivirus vendors if you must. Your antivirus situation needs to be squared away before you can move forward.

If Your System Does Not Run Antivirus

Personally, I prefer to run antivirus on my Hyper-V host. I recommend that everyone else run antivirus on their Hyper-V hosts. I’m aware that this is a debate topic and I’m aware of the counter-arguments. I’ll save that for another day.

If you do not run antivirus on your Hyper-V host (or any Windows system), then you will never receive the January 2018 security roll-up or any security roll-up thereafter without taking manual action. Microsoft has left the task of updating the related registry key to the antivirus vendors. No antivirus, no registry update. No registry update, no security patches.

In your system’s registry, navigate to HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersion. If it doesn’t already exist, create a new key named “QualityCompat”. Inside that key, create a new DWORD key-value pair named “cadca5fe-87d3-4b96-b7fb-a231484277cc”. Leave it at its default value of 0x0. You do not need to reboot. The updates will be offered on the next run of Windows Update (pursuant to any applicable group policy/WSUS approvals).

For more information, you can read Microsoft’s official documentation: https://support.microsoft.com/en-us/help/4072699/january-3-2018-windows-security-updates-and-antivirus-software.

How to Apply, Configure, and Verify the Spectre and Meltdown Fixes for Hyper-V and Windows

Microsoft has already published solid walk-throughs:

• For Hyper-V hosts and non-virtualized systems: https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution
• For Hyper-V guests: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/CVE-2017-5715-and-hyper-v-vms

Things to absolutely not miss in all of that:

• Without the post-install registry changes, your Hyper-V hosts will not have the strongest protection
• Depending on your hypervisor and guest configurations, you may need to configure additional registry settings
• Each virtual machine must be COLD-booted after the host has been updated! A simple restart of the guest OS will not be sufficient.
• Live Migrations between patched/unpatched hosts will likely fail. Keep this in mind when you’re attempting to update clusters.

After you’ve applied all of the patches, you can verify the system’s level of protection. The linked articles include instructions on acquiring and using a special PowerShell module that Microsot built just for that purpose. However, several readers pointed me to a more thorough tool named InSpectre, published by Gibson Research Corporation.

The Performance Impact of the Spectre and Meltdown Patches on a Hyper-V Host

We’ve all seen the estimations that the Meltdown patch might affect performance in the range of 5 to 30 percent. What we haven’t seen is a reliable data set indicating what happens in a real-world environment. The patches are too new for us to have any meaningful levels of historical data to show. I can tell you that my usage trends are not showing any increase, but that is anecdotal information at best. I also know of another installation (using a radically different hardware, hypervisor, and software set) that has experienced crippling slowdowns.

So, for the time being, we’re going to need to take some benchmarks. For anyone who missed the not-so-subtle point: performance bottlenecks found by benchmarks do not automatically translate to performance bottlenecks in production.

Impact on Storage

From what I’ve seen so far, these patches have the greatest detriment on storage performance. Since storage already tends to be the slowest part of our systems, it makes sense that we’d find the greatest impact there.

First up is Storage Space Direct. Ben Thomas has already provided some benchmarks that show a marked slow down. Pay close attention to the differences, as they are substantial. However, most of us would be more than happy with the “slow” number of 1.3 million IOPS. Those numbers are difficult to read in comparison with builds that would be normative in a small or medium-sized business.

I also found an article detailing the storage performance impact on an Ubuntu system using ZFS in a VMware client. This helps reinforce the point that these problems are not caused by or related to your choice of hypervisor or operating system.

I don’t have an S2D environment of my own to do any comparison. However, I will perform some storage benchmarking on a standard Hyper-V build with local storage and remote Storage Spaces (not S2D) storage.

Performance Benchmarks

I ran a number of performance checks to try to help predict the impact of these updates.

DISCLAIMER: The below benchmarks are a result of these stress tests running on this particular set of hardware. The performance impacts discussed below are not consistent across all hardware platforms and your mileage may vary depending on your own situation. The below exercise is an illustration of what the measured performance impact is on this hardware, and acts as a guide for showing you how to do so in your own environments if desired.

The Build Environment and Methodology

I used the build outlined in our eBook on building a Hyper-V cluster for under $2500. To save you a click, the CPUs are Intel Xeon E3-1225 (Sandy Bridge). We should expect to see some of the more “significant” impacts that Microsoft and Intel warn about.

The tests were performed on one of the compute nodes. Some of the storage tests involved the storage system.

The host runs Windows Server 2016. It has multiple guests with multiple operating systems. I chose three representatives: 1 Windows 2012 R2, 1 Windows 10, and 1 Windows Server 2016. All run GUI Windows because Passmark won’t generate a full CPU report under Core.

None of these systems have the patch yet.

I took the following steps:

1. Collected CPU and storage benchmarks from the host, a Windows 10 guest, the WS2012R2 guest, and the WS2016 guest. Tests were run separately.
2. Ran a CPU stress utility on some guests while testing others.
3. Ran a storage stress utility on some guests while testing others.
4. I applied available firmware updates to the host and the OS patch to the management OS and all guests.
5. Repeated steps 1-3 for comparison purposes.

I used these tools:

• Passmark for benchmarking
• Prime95 x64 for CPU load: https://www.mersenne.org/download/#download
• DiskSpd.exe for disk load: https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6cd2f223

Testing Phase 1: The Pre-Patch Raw Results

To begin, I ran benchmarks against the host and some of the guests separately. This establishes a baseline.

Note: The below numbers are simply scores provided by the bench-marking application. Higher numbers mean better performance.

Phase 1 Analysis

The primary purpose of phase 1 was to establish a performance baseline. The numbers themselves mean: “the systems can do this when not under load”.

The host has four physical cores. Each virtual machine has two vCPU. As expected, each VM’s CPU performance comes out to about half of the host’s metrics.

Testing Phase 2: Pre-Patch Under CPU Load

Next, I needed to have some idea of how a load affects performance. This system serves no real-world purpose, so I needed to fake a load.

Hyper-V is architected so that loads in the management operating system take priority over the guests (I didn’t always know that, in case you find something that I wrote to the contrary). Therefore, I didn’t feel that simulating a load in the management operating system would provide any value. It will drown out the guests and that’s that.

To test, I ran Prime95 on four virtual machines. I set the torture test to use small FFTs because we’re interested in CPU burn, not memory:

For whatever reason. that only appeared to run the CPUs up to about 25% actual usage:

The guests all believe that they’re burning full-bore, so this might be due to some optimizations. I thought the load would be higher, but we should not be overly surprised; hypervisors should do a decent job at managing their guests’ CPU usage. No matter what, this generates a load that the host distributed across all physical cores, so it works for what we’re trying to do.

While Prime95 ran in those four VMs, I tested each of the originals’ CPU and disk performance separately:

Testing Phase 3: Pre-Patch Under Disk Load

Phase 3 is like phase 2 except that we’re simulating heavy disk I/O load. It’s a little different though because the guests are spread out between local and remote storage.

I used DiskSpd.exe with the following command line:

Diskspd.exe -b256K -d600 -o8 -t4 -a0,1 #0

That’s the default from the documentation with the exception of -d600 and #0. The -d600 sets the duration. I used a five-minute timeout so that I would have enough time to start the stress tool on all of the guests and run the CPU and disk benchmarks. The #0 selects a disk that the VMs have.

Testing Phase 4: Post-Patch Raw Results

Now we’re simply going to repeat the earlier processes on our newly-updated systems. Phase 4 aligns with Phase 1: a straight benchmark of the host and representative guests.

Phase 4 Analysis

We don’t see that much CPU performance slowdown, but we do see some reduction in storage speeds.

Testing Phase 5: Post-Patch Under CPU Load

This test series compares against phase 2. Several VMs

Testing Phase 6: Post-Patch Under Disk Load

This is the post-patch equivalent of phase 3. We’ll test performance while simulating high disk I/O.

Again, I used DiskSpd.exe on several VMs with the following command line:

Diskspd.exe -b256K -d600 -o8 -t4 -a0,1 #0

Consolidated Charts

These charts reorganize the above data to align each result with its system rather than its condition.

Average Host CPU Performance Reduction is roughly 2%

Average Host Disk Performance Reduction is roughly 17.4%

Average Guest CPU performance reduction is roughly 5%

Average I/O performance reduction is roughly 47.4%

Average CPU performance reduction is roughly 5.3%

Average I/O performance reduction is roughly 6.5%

Average CPU performance reduction is roughly 11.8%

Average I/O performance reduction is roughly 42.6%

Data Analysis

The analysis has a fairly simple conclusion: some CPU functions degraded noticeably, others did not, the impact on I/O varied from very little to significant. The Windows 2016 guest does have a wide disparity in its disk statistics. Based on the other outcomes, I suspect an environmental difference or that I made some error when running the traces more than I believe that the raw speed degraded by 50%.

Conclusion

What we learn from these charts is that the full range of updates causes CPU performance degradation for some functions but leaves others unaffected. Normal storage I/O does not suffer as much. I imagine that the higher impacts that you see in the Storages Spaces Direct benchmarks shared by Ben Thomas are because of their extremely high speed. Since most of us don’t have storage subsystems that can challenge a normal CPU, we aren’t affected as strongly.

Overall, I think that the results show what could be reasonably inferred without benchmarks: high utilization systems will see a greater impact. As far as the exact impact on your systems, that appears to largely depend on what CPU functions they require most. If your pre-patch conditions were not demanding high performance, I suspect that you’ll suffer very little. If you have higher demand workloads, I’d recommend that you try a test environment if possible. If not, see if your software vendors have any knowledge or strategies for keeping their applications running acceptably.

Tell Me Your Experiences

I’ve given a breakdown of the benchmarking from my set-up but I’m curious to know if others have experienced a similar performance impact from the patches. Furthermore, if you still have any lingering doubts about the whole issue I’m ready to answer any questions you have about Meltdown, Spectre or anything else related to these security vulnerabilities and their respective patches. Contact me through the comments section below!