Testing VM encryption in your home lab with Docker in vSphere 6.5

VMware made it a point to introduce a number of security enhancements and features in vSphere 6.5 amongst which the ability to encrypt virtual machines and their drives. VM encryption is handled by the hypervisor and is totally transparent to the guest OS meaning you can encrypt a VM regardless of the guest OS running on it. The architecture is KMIP 1.1 and XTS-AES 256 driven.

The process of encrypting a VM is simple enough but the dependency of having a third party Key Management Service (KMS) makes it somewhat hard to test VM encryption in your home lab. A KMS doesn’t exactly come cheap and is something you will not find freely available.

In today’s post, I’d like to share my experience of how I went about testing VM encryption using vSphere 6.5.

Why do I need a KMS?

A KMS, in short, is where encryption keys are created and managed. A vCenter Server will request keys from the KMS when it needs to perform a cryptographic task such as encrypting a VM.

The VM encryption process flow is as follows. The vCenter Server is paired with a KMS of your choice. When a cryptographic call is made, an AES encryption key is requested by vCenter from the KMS. This key, called the Key Encryption Key (KEK), is then forwarded to ESXi by vCenter Server. The vCenter Server does not store any KEKs but instead saves an index or ID mapping to them. ESXi, in turn, generates its own internal keys, DEKs, which are encrypted using a KEK. The architecture is very secure in that decrypted data is never stored to disk.

The KEK encrypted DEK is finally used to encrypt a VM.

Figure 1 – Encryption keys flow and usage (Source: VMware)

An example of a KMS is the one offered by Amazon. I was tempted to avail of Amazon’s free trial but chose not to since my credit card details were asked for. Whenever I do this, I generally end up paying a hefty bill as I often forget to null the agreement before the trial period expires. To be safe, I opted for a wallet-friendly solution; the freely available KMS called PyKMIP.

Setting up a KMS

A KMS can be a bit of a pain to set up but thankfully I came across this great article by William Lam which explains how to deploy a PyKMIP KMS based server using a Docker container. I am not in any way a Docker expert, hardly a user to be honest, so I won’t be going into any great detail other than showing you how to deploy Docker on an existing or newly created Linux VM.

If you have a VM running some flavor of Linux like Centos, just go ahead and run the commands listed below with which you’ll download and install the latest version of Docker, start the service and make sure it starts again after a reboot.

sudo yum check-update
sudo yum install curl
curl -fsSL https://get.docker.com/ | sh
sudo systemctl start docker
sudo systemctl enable docker

To pull and run PyKMIP inside a Docker container, run the following:

docker pull lamw/vmwkmip
docker run --rm -it -p 5696:5696 lamw/vmwkmip

Figure 2 – Retrieving and running a docker container in Centos

You should now have a functional KMS running on port 5696. As per William’s article – and the warning displayed upon starting the service – this method must only be used for testing purposes since the keys are not written to disk and are lost after a reboot.

Configuring vCenter Server

The main requirement is obviously vCenter Server 6.5 since VM encryption is a feature it introduced and one that so far is only supported on 6.5. The next most important task, is setting up the KMS – or registering with an online one – and configuring vCenter to talk to it. Here’s how.

Using the vSphere Web Client, go ahead and select the vCenter Server name from Navigator. Click on the Configure tab and select Key Management Servers. Hit the Add KMS button and enter the details as requested.

Figure 3 – Configuring vCenter to talk to a KMS

Note: Use ifconfig to get hold of the KMS server’s IP address. In an online KMS scenario, the configuration details are given to you by the service provider subscribed to.

Figure 4 – Obtaining the IP address of the KMS server using ifconfig

After the Add KMS window pops up, type in the details as shown in Figure 5 and press OK when done. You should see Connecting to the server … displayed while vCenter pairs up with the KMS.

Figure 5 – Configuring the KMS connection in vCenter

If all went according to plan, you will see a number of established and closed network connections on the KMS console. In Figure 6, I’m running a Docker container with PyKMIP on a Centos 7 VM hosted in VMware Workstation Player.

Figure 6 – Network connections being established to the KMS

In addition, the status for the KMS connection should be Normal as seen in Figure 7. You can also see that I set up two separate KMS connections, the last of which is shown disconnected simply because the VM hosting the KMS was powered off at the time.

Figure 7 – KMS connection details and status in vCenter Server

Creating an Encryption Storage Policy

VM encryption in vSphere 6.5 is policy driven. To understand what this means, click on the Home icon in vSphere Web client and select Policies and Profiles.

Figure 8 – Accessing the Policies and Profiles page in vSphere Web Client

Next, click on VM Storage Policies and click once more on VM Encryption Policy. Change to the Manage tab and click on the Edit button. From here, you’ll be able to set up common rules, rule-sets and verify which datastores are compatible, or not,  with VM encryption.

Figure 9 – Managing a VM Encryption Policy in vSphere Web Client

It’s best practice to clone the existing encryption policy or create a new one. Leave the original one as is. You can then apply the encryption policy to existing VMs or simply select the policy from the New Virtual Machine wizard when creating a new VM.

The following video illustrates how you can create a new encryption policy.

Encrypting a virtual machine

Now that we have all the necessary components in place, it’s time we encrypted a VM.

Before I continue, you should know that to perform cryptographic tasks in vSphere, the account you’re logged in with must be assigned the following privileges:

• Cryptographic Operations
• Global.Diagnostics
• Host.Inventory.Add host to cluster
• Host.Inventory.Add standalone host
• Host.Local operations.Manage user groups

In fact, you will find a predefined admin role called No cryptography administrator which is missing all the above privileges as shown in Figure 10. This role comes in handy where you have large environments managed by multiple admins and you want to restrict access to encrypted resources.

Figure 10 – The No cryptography administrator role in vSphere 6.5

I should also mention that there are two options when it comes to encrypting a VM. A VM consists of a number of files including its virtual disks or VMDKs. When encrypting a VM, you can either encrypt the whole lot (VM home + hard disks) or it constituent files, hard disks excluded (VM home).

To encrypt a VM, carry out the following:

• Highlight and power off the VM you want encrypted
• Right-click on the VM. From the menu, select VM Policies followed by Edit VM Storage Policies.
  
• Select the encryption policy from the VM Storage policy drop-down menu and click on the Apply to all button to encrypt both the VM and drives. If you want to encrypt VM home only, just select the encryption policy from the drop-down menu next to it (Fig. 12).
• To encrypt the VM’s hard disks, the encryption policy must be applied to both VM home and the hard disks (Fig. 11).

Figure 11 – Encrypting a VM and its hard disks

Figure 12 – Excluding a VM’s hard disks from being encrypted

The time taken to encrypt a VMDK will obviously depend on the drive’s size. The larger the disk, the more time is takes to encrypt it. There are several ways by which you can tell if a VM is encrypted or not. Look out for the padlock symbol or check the Encryption field values under VM Hardware and the hard drive settings for the VM (see Fig. 13). You can also use quick filters or add the encryption field to the view when listing VMs in vSphere client.

Figure 13 – How to spot an encrypted VM

To decrypt a VM, you simply apply the Datastore Default policy or one that does not include an encryption components.

There’s one last thing you should be aware of. Encrypting a VM should automatically enable a setting on ESXi called Host Encryption Mode. You’ll find this under the Security Profile from the Configure tab for a given ESXi host. If the setting is disabled, you won’t be able to encrypt VMs. To enable it manually, click on Edit button and set Encryption Mode to Enabled.

You’ll need to repeat the same procedure on any ESXi host on which encrypted VMs will reside. Once enabled, it cannot be disabled unless all encrypted VMs are moved off of it.

Figure 14 – Manually enabling Encryption Mode on ESXi 6.5

Further info

I must stress that, to date, I only tested VM encryption in a lab environment under ideal conditions. Deploying encrypted VMs in live environments is a totally different kettle of fish so you need to assess the risks and be aware of what could go wrong.  For instance, backing up an encrypted VM is done in the clear, meaning that the VM needs to be decrypted when being backed up. This could pose a security risk, if backups are somehow mislaid or not managed properly.

Some trivial things that you cannot do:

• Encrypt a powered on VM.
• Encrypt a VM that has snapshots.
• Include memory in a snapshot.
• Suspend an encrypted VM.
• Export to OVF.

Figure 15 – A few unsupported tasks related to encrypting VMs

For a complete list, refer to this and this. You should also abide by the recommended best practices, the most vital of which, in my opinion, is to never encrypt the VM running vCenter Server. If your processors support it, enable AES-NI to improve performance during cryptographic intensive processes.

Conclusion

The ability to encrypt VMs is an important feature to have if security is your utmost concern. With vSphere 6.5, VM encryption is easy to implement but does require a third party KMS for the provisioning and management of encryption keys. Encryption is handled by the hypervisor, so you can encrypt any VM regardless of what’s on it. Using newly defined privileges, you will be able to restrict who has access to encrypted resources from a vSphere management perspective.

An inbuilt KMS module or service would be the cherry on the cake as this would remove the dependency on external key management service providers. Perhaps, this is something we will see in future releases of vSphere.

There are a couple more aspects relating to VM security I haven’t touched on, these being encrypted vMotion and Secure Boot, something I may cover in the near future.

[the_ad id=”4738″][the_ad id=”4796″]