Achieve Event-Driven Automation with VMware Event Broker Appliance (VEBA)

Organizations worldwide are in the middle of a paradigm shift in provisioning, managing, monitoring, and configuring their infrastructure. The cloud revolution has prompted a change in the way businesses think about infrastructure, leading to Infrastructure-as-Code and DevOps processes that blur the lines between development and IT operations.

While cloud operations have been driven by DevOps processes for some time now, on-premises tools and techniques have lagged behind the cloud. As a result, VMware admins are on a journey of adopting cloud-native processes on-premises.

Overview of VMware Event Broker Appliance (VEBA)

So, what is the VMware Event Broker Appliance (VEBA)? VEBA is released as a VMware Fling. VMware Flings are unofficial VMware solutions created by VMware software engineers without production support. However, they are powerful and often used in production environments for various use cases.

Popular and helpful VMware Flings often eventually make it to production-supported solutions. A great example is the VMware OS Optimization Tool used for years by Horizon administrators. It has now become officially supported with the release of VMware Horizon 2111. So, case in point, VMware Flings often produce viable solutions that eventually become production-supported.

The VMware Event Broker Appliance (VEBA) is a VMware Community open source project released as a VMware Fling in 2019. It enables customers to create event-driven automation using vCenter Server Events easily. The idea behind the Fling is simple. It brings modern technologies and innovations out of the cloud-native world, like Kubernetes, to help cloud admins build event-driven automation based on vCenter Server events.

Take Advantage of Event-Driven Automation in a VMware vSphere Environment

Event-driven automation with vCenter Server is compelling. Today, vCenter Server ships with over 1800+ events out-of-the-box. These include events that represent state changes like a virtual machine created, a host added, a VM getting powered-on or powered-off. It can also consist of vCenter Server alarms that are more metric-based. Metric-based alarms consist of alarms driven from thresholds. For example, a CPU is running above a certain threshold, a datastore is under a certain percentage of free space.

This information has been in vCenter Server for years now and can drive very basic event-driven automation that can carry out tasks. For example, using the built-in event automation directly in vCenter Server, you can send an email, SNMP trap, or run a command using event-driven automation from vCenter Server event capabilities.

The capabilities provided around vCenter events have not changed much in the past few years. However, organizations today are looking for more robust and modernized automation tools and capabilities to fit in with current cloud-driven automation capabilities.

The VEBA solution takes the traditional events supplied by vCenter Server through the vSphere Client and augments the in-box capabilities with a cloud-native approach. What use cases does the VEBA solution help customers solve?

VMware Event Broker Appliance (VEBA) Benefits and Applications

VEBA targets many different use cases in customer environments. Common use cases that are solved using VEBA include:

Notification:

• • Customers can receive alerts for modern alert and notification platforms, including SMS, Microsoft Teams, Slack, and others with modern API endpoints

• • Receive real-time updates for business-critical objects contained in your vSphere SDDC

• • Provide real-time monitoring of resources that are infinitely customizable to suit the needs of your business

Automation:

• • VEBA can also provide robust automation capabilities based on virtual machine activities in the environment, including applying vSphere tags, security settings, and other configurations to both VMs and hosts.

• • Triggered health monitoring jobs such as checking for long-running processes, including vSphere snapshots

Integration:

• • Automatically trigger API calls to third-party solutions providing remote API endpoints and trigger the API calls based on vSphere infrastructure events

• • VEBA can integrate with ticketing solutions used in the enterprise today, including Jira Service Desk, Pager Duty, ServiceNow, Salesforce-based incidents. These may include a workload or hardware failure.

• • You can integrate VEBA with AWS EventBridge and other public cloud services

Remediation:

• • Proactively perform tasks based on certain types of events in your VMware vSphere environment. For example, request additional capacity if you see threshold metrics peaking.

• • Allows automation teams and site reliability engineers (SREs) to code run books for providing automated resolution

Audit:

• • Track your VMware vSphere assets based on VM creation and deletion events and have these events trigger CMDB database updates

• • Use VEBA as a powerful security auditing tool allowing the forwarding of security-related events (logins and resource access) to in-house security teams for compliance and security investigation if needed

• • Provide an accurate log of configuration changes to assist with troubleshooting and debugging

Analytics:

• • Reduce the load on your vCenter Server as VEBA allows shipping events to an external database that can be queried instead of viewing events directly on vCenter Server

• • Easily identify trends in the vSphere environment, including event duration, specific users who are generating unnecessary events, and visibility to other workflows happening in the environment

In addition to the official use cases listed above, imagination is the only limitation to designing functions that carry out tasks in customer VMware environments. When you look at the VEBA community site, you see functions that can do the following:

• • Attach tags containing desired configuration settings to a VM and have it automatically reconfigure the next time it powers down

• • Automatically tag a VM upon a vCenter event

• • Disable alarm actions on a host when entering maintenance mode and re-enable alarm actions on a host after it has exited maintenance mode.

• • Send an email notification when warning/error threshold is reached for Datastore Usage Alarm in vSphere

• • Use VEBA to trigger vRealize Orchestrator workflow using the vRO REST API

• • Send an email listing VMs restarted due to a host failure in an HA enabled cluster

• • Limit the scope of other functions by allowing filtering of events by vCenter Inventory paths using standard regex

• • Automatically synchronize VM tags to NSX-T

• • Add Custom Attribute to a VM upon a vCenter event

• • Send a Slack notification triggered by a VMware Horizon Event

• • Send a Microsoft Teams notification triggered by a VMware Cloud Notification Gateway SDDC Event

• • Accepts an incoming webhook from the vRealize Network Insight Databus, constructs a CloudEvent, and sends it to the VMware Event Router

VMware Event Broker Appliance (VEBA) Architecture

The VMware Event Broker Appliance has been designed in a way that allows a very modular design. The core components of the VMware Event Broker Appliance include:

• • Kubernetes and containers – Provides robust platform capabilities, including self-healing, secrets and configuration management, resource management, and modularity/extensibility

• • Photon OS – An open-source Linux container host optimized for cloud-native applications, including support for Kubernetes out-of-the-box.

• • VMware Event Router – The event router supports multiple event stream sources, including VMware vCenter, VMware Horizon, and incoming webhooks. It also supports multiple event stream processors, including Knative, OpenFaaS, and AWS EventBridge.

• • Contour – An ingress controller for Kubernetes that deploys Envoy proxy as a reverse proxy and load balancer

The VMware Event Broker Appliance is deployed as a single virtual machine in its current form, with no option for scale-out or high availability. However, it is built on the principles of a microservices architecture running on Kubernetes. Individual services communicate using TCP/IP. Most of the communication happens “inside the box” of the VM.

High availability options such as Kubernetes cluster capabilities appear to be on the roadmap for future VMware Event Broker Appliance releases. However, while this increases the availability of the capabilities VEBA provides, it will also increase the architectural complexity of the solution.

VEBA uses Knative – What is it?

One of the powerful components of VMware Event Broker Appliance is its use of the Knative project. What is Knative? It is a project to provide a more approachable abstraction to Kubernetes. As you know, if you have worked with Kubernetes for any time, it is a robust platform solution. It has now transitioned from a container orchestration tool to a powerful API interface to a cloud-native architecture.

At its best, many feel like working with Kubernetes requires very low-level interactions, including YAML files, deployments, ingress controllers, and other components. Knative helps abstract the lower reaches of Kubernetes and helps keep the focus on the applications.

It allows deploying and managing modern serverless workloads using two core components – Serving (Knative Service) and Eventing (Broker, Channel, etc.).

Deploying the VMware Event Broker Appliance (VEBA)

The process to deploy the VMware Event Broker Appliance (VEBA) is straightforward. The first thing you need to do is download the appliance OVA file from the VMware Flings site. You can do that here:

• • VMware Event Broker Appliance | VMware Flings

• • VMware_Event_Broker_Appliance_v0.7.1.ova – filename at the time of writing

• • File size – 2GB

Download the VMware Event Broker Appliance (VEBA) from the VMware Flings site

The deployment process is the standard OVA deployment process using the vSphere Client.

Point the OVA deployment to the OVA file downloaded from the VMware Fling site

Specify the name of the appliance and the folder in your vSphere inventory.

Configure the name and folder location in vSphere inventory

Select a compute resource for the OVA appliance deployment in your vSphere SDDC.

Select the compute resource to deploy your VEBA OVA

Review the initial details of the OVA deployment for the VMware Event Broker Appliance.

Review the details of the initial OVA deployment

Select the vSphere datastore you want to house your VMware Event Broker Appliance deployment and set the VM Storage policy.

Select the storage for your VEBA OVA deployment

Select the virtual network to connect your VMware Event Broker Appliance.

Select a vSphere virtual network for your VEBA appliance

The Customize template page is the configuration step you want to look over carefully. On this configuration step, you set the following:

• • Hostname

• • IP Address

• • Subnet

• • Gateway

• • DNS

• • NTP

• • vSphere credentials for communicating with vCenter Server

• • vSphere credentials for installing the vSphere Client plugin

• • Pointing to an existing event processor

Customize template configuration deploying the VMware Event Broker Appliance

Another section of the customised template screen showing the vCenter Server and VMware Horizon configuration.

Customize template screen showing the VCSA and VMware Horizon configuration

Review and Finish-out the deployment of the appliance.

Finish the deployment of the VEBA OVA appliance

The VMware Event Broker Appliance deployment allows deploying a vSphere Client plugin that provides seamless integration with the vSphere Client and VEBA.

The VMware Event Broker Plugin successfully deployed requiring a refresh of the browser

After refreshing your browser session, you will see the VMware Event Broker option listed under your Menu options in the vSphere Client. The VMware Event Broker plugin allows deploying functions using the vSphere GUI instead of the VEBA command line.

The VMware Event Broker plugin installed and listed in the vSphere Client

Now that we have the VEBA appliance deployed, we can clone down the official VEBA repo, providing the example code and functions for deploying to a local VEBA instance.

Cloning the VEBA Repository

The official open-source code for the VMware Event Broker Appliance (VEBA) is hosted on Github. To get started with event-driven automation in your local vSphere SDDC, you first need to clone down the repository locally. Doing this allows viewing code examples and using the prebuilt examples as templates to deploy your own event-driven notifications.

To view the official code repository for VEBA, you can navigate to the Github page here:

1. 1. https://github.com/vmware-samples/vcenter-event-broker-appliance

To clone the repo from the command line, enter the following:

2. 2. git clone https://github.com/vmware-samples/vcenter-event-broker-appliance.git

Cloning the official VEBA repo to pull down the examples

Building a Docker Container

The workflow that we will perform to get to the point of testing the function for event-driven automation involves:

1. 1. Cloning the VEBA repo – This is the step above that we have completed
   2. Building the Docker container – Using the included Dockerfile, we need to build the Docker container and upload the container to the Docker registry or a local registry if you have one
   3. Testing it locally – Using the supplied test script in the cloned repo files, we can test the functionality locally before applying the configuration to the Kubernetes cluster running in the VEBA appliance
   4. Applying the configuration to the VMware Event Broker Appliance – Finally, we deploy the configuration to the VEBA appliance

For example, let’s customize and upload a function to the VMware Event Broker Appliance. It will involve creating a Docker container with a function to send an email for the triggered alert. First, let’s look at building the Docker container. In the cloned VEBA repo, you will find the knative > powershell > kn-ps-email folder under the examples folder in the parent vcenter-event-broker-appliance folder. Before customizing the files included in the root of the kn-ps-email folder, we can go ahead and test how the function works using the test folder.

Viewing the Dockerfile for the email function for VEBA

The test folder contains files that we can test the functionality. It is a great way to get familiar with the VEBA process and build containers to deploy the functions. To see how we do this, first, we need to go ahead and build a default container. Again, you don’t have to worry about customizing anything for your email server just yet.

To build the container with Docker, use the following command:

• • Docker build -t <Docker username>/kn-ps-email:1.0 .

We use a Docker Hub user with the command above and name and tag the container. ***Note*** don’t forget the trailing dot. The container builds successfully. At this point, it is still a local container since we have not pushed the container to the Docker registry.

Building a test Docker container to develop VEBA functions locally

After creating the container, we can push the container to the Docker registry. As we will detail later, you may most likely be fine using the default container referenced in the function.yaml file. However, the upload process is needed if you need a custom container. Uploading to Docker Hub will be the option many may want to use if you have no local registry running. To push your container to the Docker registry, use the command:

• • docker push <Docker username>/kn-ps-email:1.0

Now that we have the Docker container built from the Dockerfile and uploaded to the Docker registry, we can use the sample test file to test the notification alert payload sent to the function. Change your focus into the test folder from the command line. You will now want to edit your docker-test-env-variable file to point to a real email server. You can even use something like smtp.gmail.com to test the email delivery.

Update the following variable names within the docker-test-env-variable file

• • SMTP_SERVER – Address of Email Server

• • SMTP_PORT – Port of Email Server

• • SMTP_USERNAME – Username to send email (use empty string for no username)

• • SMTP_PASSWORD – Password to send email (use empty string for no password)

• • EMAIL_SUBJECT – Subject to use for email

• • EMAIL_TO – Recipient email address (comma separated)

• • EMAIL_FROM – Sender email address to use

I am using VS Code as it makes it easy to have two split PowerShell windows to work with. Below, I am running the send-cloudevent-test.ps1 file from one window and start and run the Docker container from the other window. When you run the Docker container, you start the container with the parameter –env-file and pass the docker-test-env-variable file to this parameter.

Testing the Docker container and event-driven automation with the test PowerShell script

You should get a successful test with a StatusCode 200. This result means the email was sent successfully. I received the email to my Gmail account, as you can see below.

Test email is received using the VEBA email event test script

Now, we can finish out the configuration to deploy the function code to your local VEBA.

Deploy the Event-Driven Function to your VEBA Appliance

To interact with your VMware Event Broker Appliance, you need to have connectivity to the VEBA Kubernetes cluster using kubectl. As you may already know from working with other Kubernetes infrastructure, kubectl communicates with the destination cluster using the Kubernetes config file. To get this file, you can connect to your VEBA appliance using your SSH/SCP tool of choice and copy over the Kubernetes config file from the VEBA appliance.

You will copy the config file to your local development workstation to the special .kube folder where kubectl looks for the configuration file. In Windows, this is created under the user profile directory.

Copy the Kubernetes config file from your VEBA appliance to your admin workstation

Once you have this file copied down, you should be able to run normal kubectl commands against your VMware Event Broker Appliance, like you would any other Kubernetes cluster.

Viewing pods on the VEBA appliance

We have already pushed our Docker container to the Docker Hub registry earlier in the walkthrough. This step is now crucial because the VMware Event Broker Appliance function needs to reach out and pull down the container.

Next, you need to update your email_secret.json file with your email configuration and create a Kubernetes secret that can be accessed from the function.

Now, to create the Kubernetes secret, use the following commands:

# Create the secret using your customized email-secret file with your mail server config

• • kubectl -n vmware-functions create secret generic email-secret –from-file=EMAIL_SECRET=email_secret.json

# Update the label for the secret to show up in VEBA UI

• • kubectl -n vmware-functions label secret email-secret app=veba-ui

Next, you need to edit the function.yaml file with the name of the container image if you changed this from the default. As the documentation mentions, the default VMware container image will work for most. However, if you have changed the name and uploaded a custom Docker container to the registry, it must be referenced in the function.yaml file.

By default, the function deployment will filter on the VmRemovedEvent vCenter Server Event. If you wish to change this, update the subject field in the function.yaml to the desired event.

Finally, deploy the function to the VMware Event Broker Appliance (VEBA) using the command:

• • kubectl -n vmware-functions apply -f function.yaml

VEBA Fixes and Other Hints to Functions

The latest version of VEBA has introduced many fixes to the platform as it continues to evolve. The December 2021 release includes:

• • Fix special character handling for VEBA vSphere UI plugin

• • Fix imagePullPolicy for knative-contour in air-gap deployment

• • Improved website documentation

• • More Knative Function Examples

To protect your VMware environment, Altaro offers the ultimate VMware backup service to secure backup quickly and replicate your virtual machines. We work hard perpetually to give our customers confidence in their backup strategy.

Plus, you can visit our VMware blog to keep up with the latest articles and news on VMware.

Wrapping Up

The VMware Event Broker appliance truly allows customers to unlock the hidden potential of events in their vSphere-powered SDDC and create event-driven automation. It includes support for both vCenter Server and VMware Horizon events. As shown, it is easily deployed using the prebuilt OVA appliance, and functions are readily available in the community.

Be sure to check out the official sites for VEBA, including:

• • VMware Event Broker Appliance | VMware Flings

• • VMware Event Broker Appliance (vmweventbroker.io)