The Anatomy of a VMware Virtual Machine

Virtual machines are essentially software constructs, their basic constituents being files of varying formats as you might have guessed. In this post, we’ll look at some of these files and highlight the importance of having some insight about what’s delves under the hood. This will give you that extra edge needed when it’s time to fix some mishap after the proverbial sludge hits the fan.

Of Files and Folders

Diving straight in, we explore the file types that are created when a virtual machine is first deployed. I’m using an ESXi 6.0 host with a single datastore created on local storage – i.e. storage that is directly attached to the server running ESXi.

When replicating the steps below, you’ll use a terminal emulator such as putty to connect to ESXi via SSH and the vSphere client as well. This provides you with two different views of the virtual machine files for reasons I’ll explain in just a second.

Step 1 – First, create a VM from the vSphere client. For this example, I chose CentOS as the guest OS but you could just as well leave it bare. I named the VM snoopy. Don’t ask!

Step 2 – From the vSphere client, select snoopy. Locate Storage under the Summary tab, right-click on the datastore name and select “Browse Datastore”. A “Datastore Browser” window should pop up where you’ll find a folder called snoopy. Click on the folder to view its contents (Figure 1).

Note: You’ll also notice that the folder bears the same name as the VM. It’s important to keep in mind that the folder name remains as is even when a VM is later renamed unless it is storage vMotioned. This can present some difficulties locating a VM’s  folder after the virtual machine has been renamed.

Figure 1 – A listing of the virtual machine’s file in Datastore Browser

Step 3 – Next, SSH to the host and locate the VM’s folder. The default path is /vmfs/volumes/<datastore name>/<virtual machine name>. Just in case you’re wondering, the GUID looking directories next to datastore1 in Figure 2 are in fact mounts mapping to system partitions that serve as repositories to varied stuff, including the hypervisor files used to boot ESXi.

Figure 2 – A listing of the virtual machine’s files from the ESXi shell

We can now compare the file listings from steps 2 and 3. Notice how the <vm-name>-flat.vmdk file is missing from the directory browser view. For the time being, the one thing you should be concerned about is that vmdk stands for Virtual Machine Disk which is the file extension used to denote a VMware based virtual disk as well as its descriptor file. More on this later.

Step 4 – Power on the VM and take a snapshot. We do this just to see if there are other files created when executing tasks such as snapshots. And surely enough, a handful of new files are created when a snapshot is taken; see Figs. 3 & 4. Yet again, we notice that some files are only visible from the ESXi shell; Fig.4 – file is boxed in red.

Figure 3 – New files are created when a snapshot of the VM is taken

Figure 4 – Some files can only be viewed from the ESXi shell

Virtual Machine File Types

Now that we have some insight on the basic file types making up a VM as well as the extra ones created during its lifetime, we can go ahead and list each and every one going by the file extension. The table below gives a brief explanation of the function of each.

Making Good Use Of What You Know

Now that you know about the purpose of each file, it’s time to apply this knowledge to some real world examples.

Adding a VM to the inventory

There are times when one or more virtual machines end up missing from the inventory i.e. they are no longer listed anywhere in vSphere Client irrespective if you connect directly to an ESXi host or to vCenter Server. You know the VM is still up and running since you still have access to it, can ping it, ssh to it, etc. So what do you do?

The first sensible fix would be to restart the management agents on the ESXi host or restart the VMware vCenter Server service; see this for versions predating vSphere 6. This will probably fix any inventory related issues you might be experiencing. If not, it could well be the case of someone having mistakenly deleted the VM from the inventory while it was powered off.  There’s a very easy fix to this.  Just locate the VM’s folder (remember that the folder name might not match the VM’s name if the latter has been renamed) on whichever datastore it may reside. When you find it, select the folder and right-click on the vmx file. From the context menu, select “Add to inventory” (Figure 5). This should re-register your VM to the inventory.

Figure 5 – Adding a VM to the inventory

I’ll illustrate the process in the following video.

Recovering from a deleted vmdk descriptor situation

It’s that dreaded Monday morning yet again, and as it often happens, the laws of probability are having a blast conspiring against you. You quickly discover that a critical VM bit the dust during the weekend. You try to power it back up only to be presented with the error shown in Figure 5. Backups are something you really never cared for, so you quickly make the transition from backups are for wussies  mode to one of  panic.

The first thing you should so is invest in a robust backup software solution such as Altaro VMBackup. Other than that, is there anything else you can do to solve the problem? Well, it all depends on the extent of the issue at hand.

If the flat vmdk file is intact, you could simply have a corrupted or deleted descriptor file. If that’s the case, try and replicate the procedure in this VMware KB article.

Figure 5 – Error message when powering up a VM with a missing vmdk descriptor

In this video, I will simulate the disk error and try to recover from it.

The procedure I used is somewhat different than the one described in the KB article. Here are the steps in chronological order;

• While ssh’d to ESXi, delete the descriptor file to simulate a .vmdk was not found error. This is of course just for demo purposes.
• Power on the VM. Verify that the error message is displayed.
• Note down the vmdk size using both the command line and vSphere client. If the disk was removed from the VM’s hardware settings, only a restore can save the day.
• Add an identically sized secondary disk to the VM. In the video I used an alternative datastore as I ran out of disk space on the first one.
• Copy the descriptor file snoopy_1.vmdk from the secondary disk just created to the VM folder. Rename it so it matches the original descriptor, snoopy.vmdk.
• Edit snoopy.vmdk using the vi editor, changing the vmdk flat file reference to the correct one. I also amend the CBT reference as I has previously enabled CBT.
• Lastly, remove or delete the secondary disk. Power on the VM to verify that the procedure worked.

Conclusion

As we’ve seen, knowing your virtual machine files will help you recover from those situations which would otherwise be deemed insurmountable. Of course, this is no substitute to proper backup software to safeguard the availability and integrity of both your virtual machines and data.

[the_ad id=”4738″][the_ad id=”4796″]