How to Hot Add/Remove Memory in Hyper-V 2016

We all make mistakes. Sometimes big ones, sometimes little ones. What happens when you don’t provision for the correct amount of memory for a Hyper-V virtual machine? The process normally includes scheduling downtime, bringing the VM down, changing the memory allocation, bringing it up, and then performing your post-downtime procedure. A new feature in Hyper-V 2016 substantially eases the pain. With some guest operating systems, you can increase or decrease the amount of allocated memory on the fly.

Requirements for Hyper-V Hot Add/Remove Memory

Most of the requirements for hot add/remove of memory follow common sense expectations:

• Hypervisor must be at least a 2016 version (Windows 10, Windows Server 2016, Hyper-V Server 2016, or any Semi-Annual Channel release)
• Guest operating system must support Hyper-V’s technique for hot add/remove of memory
• Virtual machine must not have Dynamic Memory enabled
• Virtual machine configuration version must be a 2016 level
• You cannot remove memory below 1 gigabyte (1000MB in the tools, not 1024)
• You cannot remove memory that the guest won’t let go of
• You cannot add memory beyond host availability or capacity

Virtual machines can be Generation 1 or Generation 2.

Guest Operating System Support for Hyper-V Hot Add/Remove Memory

Hot add/remove of memory is not a singular technology but an umbrella term for different techniques. I freely admit that I don’t understand them all. What matters is that not every operating system supports every method.

These operating systems will work with Hyper-V 2016’s hot add/remove of memory:

• Windows 10 or later (I only tested with Enterprise and Pro)
• Windows Server 2016 or later
• Windows Server Semi-Annual Channel
• Some Linux guests… I want to be more specific, but I don’t know exactly how to tell, either

For Linux, check the official documentation on supported guests (that’s the home page, the distribution list links are at the left and bottom). Look for Runtime Memory Resize. I believe that’s the correct entry. However, some of the notes attached to it also reference Dynamic Memory.

Hyper-V Dynamic Memory is not Compatible with Hot/Add Remove Fixed Memory

Dynamic Memory and hot add/remove of memory do not work together. If Dynamic Memory is enabled for a virtual machine, this feature is disabled.

Dynamic Memory does employ a memory hot-add technique, but it is not the same. Dynamic Memory does not use hot remove at all.

I do not know how this affects NUMA distribution. NUMA is not propagated into a guest with Dynamic Memory enabled — or at least, it wasn’t on 2012 R2. Fixed memory VMs do get NUMA propagation. When you hot add or remove memory from a Windows VM, it behaves differently than with Dynamic Memory. My assumption then, is that it handles NUMA propagation without any problems. Linux, however, behaves similarly to Dynamic Memory. So, I don’t know the story there.

Virtual Machine Configuration Version Support for Hyper-V Hot Add/Remove Memory

Hyper-V 2016 has been updated a few times since release. Sometimes, that included an update to the virtual machine configuration version. If you create a virtual machine with a current patch level of Hyper-V 2016, you’ll get at least a version 8 VM. 2016 initially released with the hot add/remove memory feature, so any virtual machine created by 2016 should support it. My oldest virtual machine is at version 7 and it works with hot add/remove.

You can see the configuration version of a virtual machine in Hyper-V Manager:

You can also use PowerShell. The Microsoft.HyperV.PowerShell.VirtualMachine object (the output from Get-VM) exposes a Version property. As of 2.0 of the Hyper-V PowerShell module, the default formatting of Get-VM’s output includes a Version column. For any version of Get-VM, you can directly ask for the version:

Get-VM Win10-Management | select Version

If you use Live Migration or import/export to bring a virtual machine over from 2012 R2, it will be at version 5.0. Hot add/remove of memory will not work with that version.

How to Add or Remove Memory from a Running Hyper-V Guest with PowerShell

Probably the best part of this feature is that you don’t need to learn any new tricks. The StartupMemory value continues to represent the memory setting for fixed memory VMs. So, just use Set-VMMemory with the StartupBytes parameter:

Set-VMMemory -VMName sv16g1 -StartupBytes 1gb

If you don’t get any output, the command worked.

How to Add or Remove Memory from a Running Hyper-V Guest with Hyper-V Manager

To change the memory allocation of a running virtual machine using Hyper-V Manager, access its Memory tab on its Settings dialog. Then, just set the value of RAM as desired:

Choosing Between Dynamic Memory and Hyper-V Hot Add/Remove Memory

Personally, I feel like this feature primarily exists to check more boxes on the feature comparison lists against other hypervisors. I’ve run into a few single-issue voters and some “Microsoft is always wrong” types that decided that the absence of this feature was a show-stopper for Hyper-V. Most of the world just didn’t care. Provision your VMs wisely and you’ll rarely use this feature.

From that, I’d say to just decide whether or not you want to use Dynamic Memory. My default recommendation has been to use Dynamic Memory in the absence of a hard block, and that has not changed. Reasons:

• Dynamic Memory works for far more guest operating systems, including most Linux distributions
• You can reduce the minimum and raise the maximum for a running virtual machine using Dynamic Memory (you need to turn it off to change the startup, raise the minimum, or lower the maximum)
• Dynamic Memory helps reduce wasteful allocation

Of course, Dynamic Memory isn’t always the best solution. In those cases, choose fixed. Then, if you ever need it, the hot add/remove memory feature will be there (pursuant to all of those other requirements, of course).

Troubleshooting Hyper-V Hot Add/Remove Memory

Hyper-V Manager often shows a non-descript “Failed to modify device ‘Memory’” error:

It does have some clear messages, and those work well. For instance, if you try to assign more memory than the host has, and the guest operating system supports hot add/remove, you’ll be told the exact maximum amount that you can use. If the guest operating system doesn’t support the feature, then you will get the same non-descript error. Sometimes.

Sometimes, it tells you outright that the guest operating system doesn’t support it:

PowerShell often has more helpful messaging. For instance, let’s say that you just created a virtual machine and as its booting, you realize that you didn’t give it enough memory. So, you change it right away. Unfortunately, the guest hasn’t loaded up all the way and can’t accept the change. Hyper-V Manager will just give you the useless message shown above. PowerShell, on the other hand, outright says: “Please make sure the guest operating system has fully booted, and try again.” That can come in handy when you have a virtual machine that is applying patches but locks its login screen so that you can’t tell what’s going on.

Note: If you’re automating, you might like to know when a guest can be considered “fully booted”. The answer: no one really knows. Hopefully, changing the memory of a running virtual machine is not a thing that people ever feel the need to automate. I suppose if it’s something that you need to do, you can just loop the cmdlet with a Start-Sleep line until it succeeds or throws a different error.

PowerShell will also helpfully remind you that you can’t use this feature alongside Dynamic Memory. Hyper-V Manager doesn’t need to do that since it locks the field:

So, what do you do if you get a non-descript error? Well, you power cycle the guest and try again. Did you say, “But doesn’t that defeat the purpose!?” Why, yes. Yes, it does. However, I was thoroughly convinced that none of my Linux distributions worked with this feature. I was tinkering with other items and testing minimums and maximums, so I shut down all of my Linux guests. To get some screenshots for this article, I tried again with those Linux guests. Every single one of them worked after that!

What You See in the Guest

I took a Windows Server 2016 VM with a startup setting of 1GB. I powered it on. I increased its startup memory to 4GB, then dropped it back down to 1GB. This is what Task Manager looked like:

At first glance, it appears that Task Manager sees a 4GB maximum. Look closer, though. Only the chart has a 4.0GB number. Everywhere else correctly shows the 1.0GB maximum. With Dynamic Memory ballooned from 4GB down to 1GB, most of the items would still show a 4GB maximum. See my Dynamic Memory article for some screenshots and explanations.

On Linux, both top and free continue to show the highest amount of memory that I allocated to the system. They did not reduce their maximums to match a reduction in memory. Instead, they showed a higher amount of used memory. That’s also how Dynamic Memory works, so I’m guessing that Microsoft uses the balloon driver for this feature in Linux guests.