How it Works: Understanding vSAN Architecture Components

We all know how cool the vSAN architecture is by now and it is no secret that its use has been growing significantly over the last few years. In fact, it has almost reached the status of commodity and is the foundation of the VCF stack. We will not cover how to set it up but rather how does vSAN work under the hood.

For the uninitiated, vSAN is a storage feature that is integrated into vSphere and pools disk space from multiple ESXi hosts. I always say it makes a non-useful disk, useful. Local disk without vSAN is just that! Local disk. You can’t do much with it! Either way, I’d recommend checking out the Altaro VMware vSAN Webinar, if you’re looking to be brought up to speed.

We are going to discuss vSAN architecture in a bit more depth in this post, as in some of the more in-depth components to make it work behind the scenes. Yes, it’s easy to setup, manage and use. A lot of people have asked me in the past how difficult it can be to troubleshoot it and I’ve outlined some of the tools in the Troubleshooting ebook I wrote for Altaro. You can grab a copy of it using this link. I’ve heard people even go as far as say it’s magical because it’s so easy. Magical, indeed.

“Because it is easy to operate, the vSAN architecture is more probably complicated than you might think”

Just because it’s easy doesn’t give you a free pass on knowing how the vSAN architecture works though. At some point, you’ll likely have to troubleshoot it and you’ll want to know this underlying stuff. Also, it is great to how to easily create a vSAN iSCSI LUN. Understanding how something works make figuring out how to fix it when it goes wrong a thousand times easier. Or you can setup vSAN using a nested environment. However, aside from the practical gain, it’s also just super cool knowing how it works!

vSAN Server Roles

How does vSAN work? Let’s first start with a few specifics about vSAN architecture and the server roles. Roles are applied during cluster discovery when all nodes participating in Virtual vSAN elect a master.

There is one master that is responsible for getting CMMDS (clustering service, more on that later) updates from all nodes and distributing these updates to agents.

If no host can communicate with any other host, each host will be in its own partition and will be designated its own “master” node. In these scenarios when multiple hosts are reporting themselves as “master”, we call this a partition, which is a very similar concept to vSphere High Availability.

This is an important part of the vSAN architecture: a Sphere administrator has no control over roles. It’s not something that you can adjust easily or worry about. If you’re like me, you enjoy knowing how it works, regardless. Here are the roles that make up the vSAN architecture:

Master role

• • A cluster should have only one host with the Master role. More than a single host with the Master role indicates a problem

• • The host with the Master role receives all CMMDS updates from all hosts in the cluster

Backup role

• • The host with the Backup role assumes the Master role if the current Master fails

• • Normally, only one host has the Backup role

Agent role

• • Hosts with the Agent role are members of the cluster

• • Hosts with the Agent role can assume the Backup role or the Master role as circumstances change

• • In clusters of four or more hosts, more than one host has the Agent role

Main vSAN Components

While we are only skimming the surface of how vSAN works, those are pretty advanced concepts and relate directly to how the product works under the hood. As a vSAN customer, chances are you won’t have to dig into this unless required by VMware support. However, it is important to understand which component does what, just like with vSphere and vCenter.

In this chapter, we will focus on the following components of the vSAN architecture. I’ve heard a few people use the “house building” analogy and it’s stuck with me. So I’ll explain the components using those terms. They are also depicted in the VMware vSAN architecture diagram below.

• • Cluster-Level Object Manager (CLOM): The architect

• • Distributed Object Manager (DOM): The contractor

• • Local Log-Structured Object Manager (LSOM): The worker

• • Cluster Membership, Monitoring, and Directory Services (CMMDS): The project manager

• • Reliable Datagram Transport (RDT): The Building Supply Delivery Truck

• • Object Store Filesystem (OSFS): The decorator (sort of)

• • Storage Policy-Based Management (SPBM): The furniture

“The CLOM process runs on every node in the vSAN architecture.”

Cluster-Level Object Manager (CLOM)

The CLOM process is managed with the /etc/init.d/clomd stop/start/restart command. You may encounter vSAN advanced settings referring to CLOMD that enable you to change the behavior of vSAN such as rebuild delays.

• • It validates that objects can be created based on policies and available resources.

• • It is responsible for object compliance.

• • It defines the creation and migration of objects.

• • It distributes loads evenly between the vSAN nodes.

• • It is responsible for proactive and reactive re-balancing.

Distributed Object Manager

DOM is the Contractor, it runs on each ESXi host in the cluster and handles the following functions: Note that the DOM exists in kernel space, meaning there is no daemons to monitor or restart without rebooting the host.

• • DOM receives instructions from the CLOM and other DOMs running on other hosts in the cluster.

• • DOM communicates and tells the LSOM to create local components of an object.

• • DOM services on the hosts of a vSAN cluster communicate to coordinate the creation of components.

• • All DOMs in a vSAN cluster re-synchronize objects during a recovery.

In the vSAN architecture, each object in a vSAN cluster has a DOM owner and a DOM client.

There is one DOM owner that exists per object and it determines which processes are allowed to send I/O to the object.

The DOM client performs the I/O to an object on behalf of a particular virtual machine and runs on every node that contains components.

Local Log-Structured Object Manager

LSOM is the Worker. It creates the local components as instructed by the DOM that was described previously.

• • It provides read and write buffering

• • It performs the encryption process for the vSAN datastore

• • It reports unhealthy storage and network devices

• • It performs I/O retries on failing devices

• • It interacts directly with the solid-state and magnetic devices

• • It performs solid-state drive log recovery when the vSAN node boots up

Cluster Membership, Monitoring, and Directory Services

The vSAN VMware architecture diagram shows the importance of CMMDS. It is the Project Manager. It provides the topology and object configuration information to the CLOM and DOM components of the vSAN architecture. Information about components published into CMMDS can be found from any node using cmmds-tool.

• • It selects the owners of the objects

• • It inventories all items, such as hosts, networks,

and devices

• • It stores object metadata information, such as policy-related information on an in-memory database, other components like DOM/LSOM publish into CMMDS

• • It discovers, maintains, and establishes a cluster of networked node members

• • It defines the cluster roles: Master, Backup, and Agent

• • Production issues will occur if CMMDS cannot communicate or update information

Object Store File System

The OSFS component is a daemon that runs on every participating node in a vSAN cluster. Its purpose is to offer compatibility with a file-system like a construct, even though the vSAN architecture is object-based.

• • Provides compatibility with a filesystem-like structure

• • Creation of the vSAN Datastore. Interestingly, there is no such thing as vSAN directories, those are simply vSAN objects that are formatted with VMFS.

• • Takes care of the “VM namespace” objects, mapping friendly names, and so on. The objects are then made available in the vSAN Datastore

Storage Policy-Based Management

Storage Policy-Based Management or SPBM is the most visible component of the vSAN VMware architecture as it is what the user will leverage to define where and how to place specific objects based on, you guessed it, policies.

• • SPBM are defined at the vCenter level in the storage policies category and leverages the vSphere API for Storage Awareness (VASA) through a provider running on each host.

• • The policies are sent to CLOMD during provisioning to let it know how to place workloads.

• • When applying a policy to a VM or disk, or modifying a policy the changes are pushed in SPBM and forwarded to CLOMD.

• • Lets you specify varying degrees and areas of object storage placement including protection, encryption, compression, and so on.

Reliable Datagram Transport

In the house-building analogy, RDT is the Delivery Truck. It is the network protocol that is optimized to send very large files and that is used for the transmission of vSAN VMware architecture traffic between hosts.

The CMMDS publishes link health status and RDT uses this data to quickly create and delete transport connections with minimum delay incurred by link failures

How do all the Components Interact?

In a nutshell for the vSAN architecture, when the CLOM receives a request to create an object, the CLOM determines whether the object can be created with the selected VM storage policy. If the object can be created, the CLOM instructs the DOM to create the components. The DOM then decides what components are created locally and instructs the LSOM to create the local components. The LSOM interacts at the drive layer and provides persistent storage. The DOM interacts only with the local instance of the LSOM. If components are required on other nodes, then the DOM interacts with the DOM on the remote node.

As you can see, there are far more moving pieces to this than just building a VM and adding disks! Hopefully, this post has helped you understand a few more components of vSAN!

To properly protect your VMware environment, use Altaro VM Backup to securely backup and replicate your virtual machines. We work hard perpetually to give our customers confidence in their VMware backup strategy.

To keep up to date with the latest VMware best practices, become a member of the VMware DOJO now (it’s free).

Wrap up

While operating vSAN is fairly easy from a VI admin perspective, the vSphere client hides well the complexity of the vSAN architecture. It takes a number of server roles and components to offer resilient, low latency, and secure virtualized storage spanning local disks across several servers. Add on top of that vSAN 7 new features such as HCI mesh compute clusters or stretched vSAN architectures and you obtain a state-of-the-art distributed virtualized storage backend. Multiple businesses and SMBs are already benefiting from vSAN.

While it is now a few years old, you can also find extra content in this interview of vSAN experts. Feel free to add your questions below in the comments if you want any more clarification about anything presented here or just on vSAN architecture generally.