VMware Hot-Add: How and When to Use it

Should I be using Hot-add?

One of my most asked questions I get in terms of day to day management of virtual machines is whether or not you should turn on the CPU and memory hot-add features and why would VMware NOT turn them on by default! Yes, it is a feature that allows you to hot-add memory and CPU to a VM all while they’re powered on. It can definitely make your life as an administrator easier. When you tell users “no downtime” they smile and get really into it. Hot-add seems like a crazy idea to most people because they can always think of a time when it would have been nice to just give the VM a little more resources.

In most cases, only very few types of hardware can be added while the VM is running. Network adapters, USB adapters, hard disks, etc. For all other types of virtual hardware, the VM has to be powered off before they can be added or removed. Another thing to think about, your operating system and applications have to be configured to recognize the virtual hardware you want to add.

What are the requirements for Hot-add/Hot-plug:

• Your virtual machines need to run at minimum hardware version 7.
• Hot-add/Hot-Plug is not compatible with Fault Tolerance
• vSphere Advanced, Enterprise or Enterprise plus.
• Only hot-add is possible. You cannot “hot-remove” RAM or vCPUs.
• Hot-Add/Hot-plug must be supported by the VM operating system!
• Guest-OS licensing limitations need to be monitored and taken into consideration. You are changing the number of vCPUs/RAM!

It definitely can be tempting to turn it on for every VM, but it’s one of those options that you really have to weigh and consider the positives and negatives. It is a setting that you can change inside a template and therefore be changed permanently on any future VMs you deploy from that template. My general opinion is to leave it off unless you have a good reason for it and the information below lists why. Again, before we get too deep into it, the setting only allows adding and not removing resources. If you want to remove, you’re looking at a full power down.

If you’re wondering if the setting is already turned on or off on your VMs, you can check by editing the settings of the VMs and quickly look at the CPU and Memory section. If the VM is powered on and the CPU and memory fields cannot be edited as shown in the picture below, the setting is not enabled.

A few reasons for leaving it disabled

Reason #1: vNUMA is disabled on the VM.

Once the Hot-Add feature is enabled on a VM, vNUMA is automatically disabled. vNUMA is enabled by default on all virtual machines that have 8 or more vCPUs configured because it will improve performance on the larger VMs. Now, not all of your VMs will have large numbers of CPUs but you will still need to weigh up the ability to easily add or remove CPU and memory on the fly vs. improved performance.

NUMA is becoming increasingly more important to ensure workloads, like databases, allocate and consume memory within the same physical NUMA node that the vCPUs are scheduled.  When a virtual machine is sized larger than a single physical NUMA node, a vNUMA topology is created and presented to the guest operating system.  This virtual construct allows a workload within the virtual machine to benefit from physical NUMA, while continuing to support functions like vMotion. In vSphere 6.5, vSphere will now always present the optimal vNUMA topology unless you use advanced settings.

Reason#2: Operating System support.

Not all operating systems support hot-add CPU and memory. I do know Red Hat has support for it and it has worked for me on Ubuntu/Debian machines before but the best thing to do is to check the VMware Compatibility Guide as it’s always updated. Generally, the operating systems listed below support the feature. I’ve had good success with Red Hat, Ubuntu and Debian Linux VMs.

• Linux: Newer Linux kernels have support for CPU hot add. To verify this, check the documentation for your Linux distribution in the documentation directory of the distribution source code. The documentation contains directions for special boot time switches related to CPU Hot Plug. CPU hot-remove is not supported at this time.
• Windows Server 2003 32bit/64bit – (Standard and Enterprise)
• Windows Server 2008 32bit (Standard, Enterprise and Datacenter)
• Windows Server 2008 64bit (Standard, Enterprise and Datacenter)
• Windows Server 2008 R2 (Standard, Enterprise and Datacenter)
• Windows Server 2012 (Standard, Enterprise and Datacenter)
• Windows Server 2012 R2 (Standard, Enterprise and Datacenter)
• Windows Server 2016 Standard, (Standard, Datacenter)

Where do I turn it on?

You can turn the settings on by expanding the CPU and Memory sections of the VM as shown below:

So no hot-remove?

Now that we’ve added CPU and memory on the fly, it would be nice to hot remove it right? Well, as it stands right now, it’s a one-way street. Again, you can add but cannot remove resources without shutting them down first. The operating systems would probably freak out a bit if you pulled the memory out of it. Could you imagine how a SQL Server would handle that? Yeah, it would not be pretty! So sadly, we still have to shut our virtual machines down to remove resources. Maybe someday we’ll get there in the not so distant future!

Wrap up

One thing that you might be thinking now that you have it enabled, how do you properly manage and monitor the virtual machines that have this setting turned on. By default, it’s not easy to find the VMs that have these hot add settings turned on. There are basically no distinguishable things in the inventory that are obvious and the VMs just blend in with the others. Well, that’s where PowerCLI comes in. I have had great success with this script that’s posted out at VMwareArena. You can check it out here. Basically what it does is scan your vSphere environment and churns out a nice CSV file that gives you the status of CPU and memory hot-add. It will show you the virtual machines that have the options enabled.

Your thoughts on Hot-add

I’m always happy to hear what you guys think of the tools I cover in this blog so let me know your experiences with VMware Hot-Add. Finally, if you need any clarification on the above information, I’m here to help. Contact me in the comments section below!

Thanks for reading!

[the_ad id=”4738″][thrive_leads id=’18673′]