Hyper-V Key-Value Pair Data Exchange Part 3: Linux

Some time ago, I discovered uses for Hyper-V Key-Value Pair Data Exchange services and began exploiting them on my Windows guests. Now that I’ve started building Linux guests, I need similar functionality. This article covers the differences in the Linux implementation and includes version 1.0 of a program that allows you to receive, send, and delete KVPs.

For a primer on Hyper-V KVP Exchange, start with this article: Hyper-V Key-Value Pair Data Exchange Part 1: Explanation.

The second part of that series presented PowerShell scripts for interacting with Hyper-V KVP Exchange from both the host and the guest sides. The guest script won’t be as useful in the context of Linux. Even if you install PowerShell on Linux, the script won’t work because it reads and writes registry keys. It might still spark some implementation ideas, I suppose.

What is Hyper-V Key-Value Pair Data Exchange?

To save you a few clicks and other reading, I’ll give a quick summary of Hyper-V KVP Exchange.

Virtual machines are intended to be “walled gardens”. The host and guest should have limited ability to interact with each other. That distance sometimes causes inconvenience, but the stronger the separation, the stronger the security. Hyper-V’s KVP Exchange provides one method for moving data across the wall without introducing a crippling security hazard. Either “side” (host or guest) can “send” a message at any time. The other side can receive it — or ignore it. Essentially, they pass notes by leaving them stuck in slots in the “wall” of the “walled garden”.

KVP stands for “key-value pair”. Each of these messages consists of one text key and one text value. The value can be completely empty.

How is Hyper-V KVP Exchange Different on Linux?

On Windows guests, a service runs (Hyper-V Data Exchange Service) that monitors the “wall”. When the host leaves a message, this service copies the information into the guest’s Windows registry. To send a message to the host, you (or an application) create or modify a KVP within a different key in the Windows registry. The service then places that “note” in the “wall” where the host can pick it up. More details can be found in the first article in this series.

Linux runs a daemon that is the analog to the Windows service. It has slightly different names on different platforms, but I’ve been able to locate it on all of my distributions with sudo service –status-all | grep kvp. It may not always be running; more on that in a bit.

Linux doesn’t have a native analog to the Windows registry. Instead, the daemon maintains a set of files. It receives inbound messages from the host and places them in particular files that you can read (or ignore). You can write to one of the files. The daemon will transfer those messages up to the host.

On Windows, I’m not entirely certain of any special limits on KVP sizes. A registry key can be 16,384 characters and there is no hard-coded limit on value size. I have not tested how KVP Exchange handles these extents on Windows. However, the Linux daemon has much tighter constraints. A key can be no longer than 512 bytes. A value can be no longer than 2,048 bytes.

The keys are case sensitive on the host and on Linux guests. So, key “LinuxKey” is not the same as key “linuxkey”. Windows guests just get confused by that, but Linux handles it easily.

How does Hyper-V KVP Exchange Function on Linux?

As with Windows guests, Data Exchange must be enabled on the virtual machine’s properties:

The daemon must also be installed and running within the guest. Currently-supported versions of the Linux kernel contain the Hyper-V KVP framework natively, so several distributions ship with it enabled. As mentioned in the previous section, the exact name of the daemon varies. You should be able to find it with: sudo service –status-all | grep kvp. If it’s not installed, check your distribution’s instruction page on TechNet.

All of the files that the daemon uses for Hyper-V KVP exchange can be found in the /var/lib/hyperv folder. They are hidden, but you can view them with ls‘s -a parameter:

Anyone can read any of these files. Only the root account has write permissions, but that can be misleading. Writing to any of the files that are intended to carry data from the host to the guest has no real effect. The daemon is always monitoring them and only it can carry information from the host side.

What is the Purpose of Each Hyper-V KVP Exchange File?

Each of the files is used for a different purpose.

• .kvp_pool_0: When an administrative user or an application in the host sends data to the guest, the daemon writes the message to this file. It is the equivalent of HKLMSOFTWAREMicrosoftVirtual MachineExternal on Windows guests. From the host side, the related commands are ModifyKvpItems, AddKvpItems, and RemoveKvpItems. The guest can read this file. Changing it has no useful effect.
• .kvp_pool_1: The root account can write to this file from within the guest. It is the equivalent of HKLMSOFTWAREMicrosoftVirtual MachineGuest on Windows guests. The daemon will transfer messages up to the host. From the host side, its messages can be retrieved from the GuestExchangeItems field of the WMI object.
• .kvp_pool_2: The daemon will automatically write information about the Linux guest into this file. However, you never see any of the information from the guest side. The host can retrieve it through the GuestIntrinsicExchangeItems field of the WMI object. It is the equivalent of the HKLMSOFTWAREMicrosoftVirtual MachineAuto key on Windows guests. You can’t do anything useful with the file on Linux.
• .kvp_pool_3: The host will automatically send information about itself and the virtual machine through this file. You can read the contents of this file, but changing it does nothing useful. It is the equivalent of the HKLMSOFTWAREMicrosoftVirtual MachineGuestParameter key on Windows guests.
• .kvp_pool_4: I have no idea what this file does or what it is for.

What is the Format of the Hyper-V KVP Exchange File on Linux?

Each file uses the same format.

One KVP entry is built like this:

• 512 bytes for the key. The key is a sequence of non-null bytes, typically interpreted as char. According to the documentation, it will be processed as using UTF8 encoding. After the characters for the key, the remainder of the 512 bytes is padded with null characters.
• 2,048 bytes for the value. As with the key, these are non-null bytes typically interpreted as char. After the characters for the value, the remainder of the 2,048 bytes is padded with null characters.

KVP entries are written end-to-end in the file with no spacing, headers, or footers.

For the most part, you’ll treat these as text strings, but that’s not strictly necessary. I’ve been on this rant before, but the difference between “text” data and “binary” data is 100% semantics, no matter how much code we write to enforce artificial distinctions. From now until the point when computers can process something other than low voltage/high voltage (0s and 1s), there will never be anything but binary data and binary files. On the Linux side, you have 512 bytes for the key and 2,048 bytes for the value. Do with them as you see fit. However, on the host side, you’ll still need to get through the WMI processing. I haven’t pushed that very far.

How Do I Use Hyper-V KVP Exchange for Linux?

This is the part where it gets fun. Microsoft only goes so far as to supply the daemon. If you want to push or pull data, that’s all up to you. Or third parties.

But really, all you need to do is read to and/or write from files. The trick is, you need to be able to do it using the binary format that I mentioned above. If you just use a tool that writes simple strings, it will improperly pad the fields, resulting in mangled transfers. So, you’ll need a bit of proficiency in whatever tool you use. The tool itself doesn’t matter, though. Perl, Python, bash scripts,… anything will do. Just remember these guidelines:

• Writing to files _0, _2, _3, and _4 just wastes time. The host will never see it, it will break KVP clients, and the files’ contents will be reset when the daemon restarts.
• You do not need special permission to read from any of the files.
• _1 is the only file that it’s useful to write to. You can, of course, read from it.
  • Deleting the existing contents deletes those KVPs. You probably want to update existing or append new.
  • The host only receives the LAST time that a KVP is set. Meaning that if you write a KVP with key “NewKey” twice in the _1 file, the host will only receive the second one.
  • Delete a KVP by zeroing its fields.
• If the byte lengths are not honored properly, you will damage that KVP and every KVP following.

Source Code for a Hyper-V KVP Exchange Utility on Linux

I’ve built a small utility that can be used to read, write, and delete Hyper-V KVPs on Linux. I wrote it in C++ so that it can be compiled into a single, neat executable.

Long-term, I will only be maintaining this project on my GitHub site. The listing on this article will be forever locked in a 1.0 state.

Compile Instructions

Each file is set so that they all live in the same directory. Use make to build the sources and sudo make install to put the executable into the /bin folder.

Install Instructions

Paste the contents of all of these files into accordingly-named files. File names are in the matching section header and in the code preview bar.

Transfer all of the files to your Linux system. It doesn’t really matter where. They just need to be in the same folder.

Run:

make
sudo make install

Usage Instructions

Get help with:

• hvkvp –help
• hvkvp read –help
• hvkvp write –help
• hvkvp delete –help

Each includes the related keys for that command and some examples.

Code Listing

The file list:

• makefile
• main.cpp
• hvkvp.h
• hvkvp.cpp
• hvkvpfile.h
• hvkvpfile.cpp
• hvkvpreader.h
• hvkvpreader.cpp
• hvkvpremover.h
• hvkvpremover.cpp
• hvkvpwriter.h
• hvkvpwriter.cpp

makefile

OBJS = hvkvp.o hvkvpfile.o hvkvpreader.o hvkvpwriter.o hvkvpremover.o main.o
CC = c++
DEBUG = -g0
CXXFLAGS = -Wall -fomit-frame-pointer -std=c++11 -c -fexceptions -O3
LFLAGS = -Wall
OUT = hvkvp
INSTALLDIR = /bin/

BUILDSTAT = stat $(OUT) 2>/dev/null | grep Modify
INSTALLEDSTAT = stat $(INSTALLDIR)$(OUT) 2>/dev/null | grep Modify

all : clean $(OBJS)
	$(CC) $(LFLAGS) $(OBJS) -o $(OUT)
	
hvkvp.o : hvkvp.cpp
	$(CC) $(CXXFLAGS) hvkvp.cpp

hvkvpfile.o : hvkvpfile.cpp
	$(CC) $(CXXFLAGS) hvkvpfile.cpp

hvkvpreader.o : hvkvpreader.cpp
	$(CC) $(CXXFLAGS) hvkvpreader.cpp

hvkvpwriter.o : hvkvpwriter.cpp
	$(CC) $(CXXFLAGS) hvkvpwriter.cpp
	
hvkvpremover.o : hvkvpremover.cpp
	$(CC) $(CXXFLAGS) hvkvpremover.cpp
	
main.o : main.cpp
	$(CC) $(CXXFLAGS) main.cpp
	
clean :
	rm -f *.o $(OUT)

install : $(OUT)
	@if [ "$(BUILDSTAT)" != "$(INSTALLEDSTAT)" ];
	then
		cp -p $(OUT) $(INSTALLDIR);
	fi

uninstall : $(INSTALLDIR)$(OUT)
	rm $(INSTALLDIR)$(OUT)

main.cpp

/*************************************************************************************
main.cpp (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#include <getopt.h>
#include <iostream>
#include "hvkvpreader.h"
#include "hvkvpremover.h"
#include "hvkvpwriter.h"

using std::cerr;
using std::cout;
using std::endl;
using std::string;

string GeneralHelp =
"Hyper-V KVP Interface      (c) 2017 Eric Sironn"
"Usage: hvkvp COMMAND [OPTIONS]nn"
"Available Commandsn"
"------------------n"
"read                       Read KVP(s)n"
"write                      Write a KVPn"
"delete                     Delete KVP(s)nn"
"Optionsn"
"-------n"
"-k, --key KEY              Key to operate onn"
"-s, --keyset KEYSET        out: Read guest-to-host key(s)n"
"                           in: Read host-to-guest key(s)n"
"                           params: Read auto-set parameter(s)n"
"-v, --value [VALUE]        Value for keyn"
"-i, --ignorecase           Ignore UPPERCASE/lowercase of key(s)n"
"-m, --exactmatch on|off    on: key must match exactly (doesn't override -i)n"
"                           off: partial key match (USE WITH CAUTION!)n"
"-n, --invertmatch          Operate on keys that DO NOT match -k.n"
"                           Can combine with -i and -mn"
"-x, --noout                No output after "write" or "delete"n"
"-h, --help                 Show help. Use with a command to see examplesn"
;

string ReadHelp =
"Hyper-V KVP Interface      (c) 2017 Eric Sironn"
"Usage: hvkvp read [OPTIONS]n"
"Options (Ignores -v/--value and -x/--noout parameters)n"
"-------n"
"-k, --key [KEY]            Key to operate onn"
"-s, --keyset KEYSET        out: Read guest-to-host key(s)n"
" (Default in)              in: Read host-to-guest key(s)n"
"                           params: Read auto-set parameter(s)n"
"-i, --ignorecase           Ignore UPPERCASE/lowercase of key(s)n"
"-m, --exactmatch on|off    on: key must match exactly (doesn't override -i)n"
"                           off: partial key match (default for read)n"
"-n, --invertmatch          Operate on keys that DO NOT match -k.n"
"                           Can combine with -i and -mn"
"-x, --noout                No output after "read" after "write" or "delete"n"
"-h, --help                 Show this help textn"
"Examplesn"
"hvkvp read -s outn"
": Reads all KVPs in the guest-to-host listnn"
"hvkvp read -k VirtualMachineIdn"
": Reads the "VirtualMachineId" field from the hostnn"
"hvkvp read -k virtualmachineid -in"
": Reads the "VirtualMachineId" field from the hostn"
;

string WriteHelp =
"Hyper-V KVP Interface      (c) 2017 Eric Sironn"
"Usage: hvkvp write [OPTIONS]n"
"Options (Ignores -n/--invertmatch and -s/--keyset parameternWorks with guest-to-host only!n"
"-------n"
"-k, --key KEY              Key to operate onn"
"-v, --value [VALUE]        Value for keyn"
"-i, --ignorecase           Ignore UPPERCASE/lowercase of key(s)n"
"-m, --exactmatch on|off    on: key must match exactly (default for write)n"
"                           off: partial key match (USE WITH CAUTION!)n"
"                           Can combine with -i and -mn"
"-x, --noout                No output after "write" or "delete"n"
"-h, --help                 Show this help textn"
"Examplesn"
"hvkvp write -k GuestReport -v "My report"n"
": Creates or overwrites key "GuestReport" with value "My Report"nn"
"hvkvp write -k guestreport -v "My report" -in"
": Creates/overwrites key "GuestReport", ignoring case, with value "My Report"nn"
"hvkvp write -k guestrep -v "My report" -i -m offn"
": Creates/Overwrites any key containing "guestrep", ignoring case,n  with value "My Report"n"
;

string DeleteHelp =
"Hyper-V KVP Interface      (c) 2017 Eric Sironn"
"Usage: hvkvp delete [OPTIONS]n"
"Options (Ignores -s/--keyset parameter; works with guest-to-host only)n"
"-------n"
"-k, --key KEY              Key to operate onn"
"-v, --value [VALUE]        Value for keyn"
"-i, --ignorecase           Ignore UPPERCASE/lowercase of key(s)n"
"-m, --exactmatch on|off    on: key must match exactly (default for remove)n"
"                           off: partial key match (USE WITH CAUTION!)n"
"-n, --invertmatch          Operate on keys that DO NOT match -k.n"
"                           Can combine with -i and -mn"
"-x, --noout                No output after "write" or "delete"n"
"-c, --cleaninvalid         Detects and removes invalid KVPsn"
"-d, --cleanduplicates      Detects and removes duplicate KVPsnn"
"-A, --DELETE_ALL           Deletes all guest-to-host KVPsn"
"-h, --help                 Show this help textn"
"Examplesn"
"hvkvp delete -k GuestReportn"
": Deletes KVP with key "GuestReport"nn"
"hvkvp delete -k guestreport -nin"
": Deletes all KVPs EXCEPT "GuestReport", ignoring casenn"
"hvkvp write -cin"
": Deletes all invalid and duplicate KVPsn"
;

enum opmodes
{
	None,
	Read,
	Write,
	Delete
};

int main(int argc, char **argv)
{
	string Key("");
	string Value("");
	int HelpRequested = 0;
	int CaseInsensitive = 0;
	bool MatchExactOverride = false;
	int MatchExact = 0;
	int InvertMatch = 0;
	int NoOutput = 0;
	int CleanInvalid = 0;
	int CleanDuplicates = 0;
	int DeleteAll = 0;
	KVPReadPools ReadSource = KVPReadPools::HostToGuest;

	static struct option long_options[] = {
		{ "help",				no_argument, &HelpRequested, 1 },
		{ "keyset",				required_argument, 0, 's' },
		{ "key",					required_argument, 0, 'k' },
		{ "value",				required_argument, 0, 'v' },
		{ "ignorecase",		no_argument, &CaseInsensitive, 1 },
		{ "matchexact",		required_argument, 0, 'm' },
		{ "invertmatch",		no_argument, &InvertMatch, 1 },
		{ "noout",				no_argument, &NoOutput, 1 },
		{ "cleaninvalid",		no_argument, &CleanInvalid, 1 },
		{ "cleanduplicates",	no_argument, &CleanDuplicates, 1 },
		{ "DELETE_ALL",		no_argument, &DeleteAll, 1 },
		{0, 0, 0, 0}
	};

	int EnteredOptionCharacter;
	int EnteredOptionIndex;
	opmodes OpMode = None;

	if (argv[1])
	{
		if (!strncmp(argv[1], "read", 4)) { OpMode = Read; }
		else if (!strncmp(argv[1], "write", 5)) { OpMode = Write; }
		else if (!strncmp(argv[1], "delete", 6)) { OpMode = Delete; }
	}

	while (EnteredOptionCharacter != -1)
	{
		EnteredOptionCharacter = getopt_long(argc, argv, "hs:k:v:im:nxcdA", long_options, &EnteredOptionIndex);
		switch (EnteredOptionCharacter)
		{
		case 'k':
			if (optarg) { Key.assign(optarg); }
			break;
		case 's':
			if (optarg)
			{
				switch (optarg[0])
				{
				case 'o':
				case 'O':
					ReadSource = KVPReadPools::GuestToHost;
					break;
				case 'p':
				case 'P':
					ReadSource = KVPReadPools::HostToGuestIntrinsic;
					break;
				default:
					ReadSource = KVPReadPools::HostToGuest;
					break;
				}
			}
			break;
		case 'v':
			Value.assign(optarg);
			break;
		case 'i':
			CaseInsensitive = 1;
			break;
		case 'm':
		{
			MatchExactOverride = true;
			string MatchArg = string(optarg);
			if (MatchArg == "on") { MatchExact = 1; }
			else if (MatchArg == "off") { MatchExact = 0; }
			else { HelpRequested = 1; }
		}
		break;
		case 'n':
			InvertMatch = 1;
			break;
		case 'x':
			NoOutput = 1;
			break;
		case 'c':
			CleanInvalid = 1;
			break;
		case 'd':
			CleanDuplicates = 1;
			break;
		case 'A':
			DeleteAll = 1;
			break;
		case -1:
			break;
		default:
			HelpRequested = true;
			break;
		}
	}

	if (OpMode == Write)
	{
		if (HelpRequested) { cout << WriteHelp; }
		else if (Key.empty())
		{
			cout << "Key name (-k/--key) must be specified for a write operation" << endl;
			return EINVAL;
		}
		else
		{
			HVKVPWriter Writer;
			Writer.CaseInsensitive = CaseInsensitive;
			if (MatchExactOverride) { Writer.MatchExact = MatchExact; }
			try
			{
				Writer.WriteKVP(Key, Value);
			}
			catch (int ErrorCode)
			{
				errno = ErrorCode;
			}
			catch (...)
			{
				cout << "Unknown error occurred during a write operation" << endl;
			}
		}
	}
	else if (OpMode == Delete)
	{
		if (HelpRequested) { cout << DeleteHelp; }
		else if (Key.empty() && !DeleteAll && !CleanDuplicates && !CleanInvalid)
		{
			cout << "Delete must include a key (-k/--key) and/or at least one of the following: n"
				"   -c/--cleaninvalidn   -d/--cleanduplicatesn   -A/--DELETE_ALL" << endl;
			return EINVAL;
		}
		else
		{
			HVKVPRemover Remover;
			if (DeleteAll)
			{
				try
				{
					Remover.RemoveAll();
				}
				catch (int ErrorCode)
				{
					errno = ErrorCode;
				}
				catch (...)
				{
					cout << "Unknown error occurred while attempting to delete all outbound KVPs" << endl;
				}
			}
			else
			{
				Remover.CaseInsensitive = CaseInsensitive;
				Remover.InverseMatch = InvertMatch;
				if (MatchExactOverride) { Remover.MatchExact = MatchExact; }
			}
			try
			{
				Remover.RemoveKVP(Key, CleanDuplicates, CleanInvalid);
			}
			catch (int ErrorCode)
			{
				errno = ErrorCode;
			}
			catch (...)
			{
				cout << "Unknown error occurred during a delete operation" << endl;
			}
		}
	}
	else if (OpMode == None)
	{
		cout << GeneralHelp;
	}

	// other operations must fall through to an opportunity to read
	if (!errno && OpMode != None && !NoOutput)
	{
		HVKVPReader Reader;
		if (HelpRequested)
		{
			cout << ReadHelp;
		}
		else
		{
			if (OpMode == Read)
			{
				Reader.CaseInsensitive = CaseInsensitive;
				Reader.InverseMatch = InvertMatch;
				if (MatchExactOverride) { Reader.MatchExact = MatchExact; }
				Reader.Pool = ReadSource;
			}
			else
			{
				// we only get here if a write or delete operation occurred, and those can only target G2H
				Reader.Pool = KVPReadPools::GuestToHost;
				Key.assign("");
			}
			try
			{
				for (auto KVP : Reader.ReadKVPs(Key))
				{
					cout << KVP << endl;
				}
			}
			catch (int ErrorCode)
			{
				errno = ErrorCode;
			}
			catch (...)
			{
				cout << "Unknown error occurred during a read operation" << endl;
			}
		}
	}

	if (errno) { cerr << strerror(errno) << endl; }
	return errno;
}

hvkvp.h

/*************************************************************************************
hvkvp.h (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#ifndef __HVKVP_H_INCLUDED
#define __HVKVP_H_INCLUDED 1

#include <array>
#include <fstream>
#include <linux/hyperv.h>

#define HV_KVP_EXCHANGE_FIELDSIZE (HV_KVP_EXCHANGE_MAX_KEY_SIZE + HV_KVP_EXCHANGE_MAX_VALUE_SIZE)

#define HVKVP_DEFAULT_KVP_DIRECTORY	"/var/lib/hyperv/"

#define HVKVP_BASEFILENAME			".kvp_pool_"

#define HVKVP_POOL_EXTERNAL_ID			"0"		// host-to-guest pool populated by a user
#define HVKVP_POOL_GUEST_ID				"1"		// guest-to-host pool populated by a user
#define HVKVP_POOL_GUESTPARAMETER_ID	"3"		// host-to-guest pool auto-populated by the Hyper-V host

#define HVKVP_FILE_TOGUEST				HVKVP_DEFAULT_KVP_DIRECTORY HVKVP_BASEFILENAME HVKVP_POOL_EXTERNAL_ID
#define HVKVP_FILE_TOHOST				HVKVP_DEFAULT_KVP_DIRECTORY HVKVP_BASEFILENAME HVKVP_POOL_GUEST_ID
#define HVKVP_FILE_GUESTPARAMETER	HVKVP_DEFAULT_KVP_DIRECTORY HVKVP_BASEFILENAME HVKVP_POOL_GUESTPARAMETER_ID

union hvkvp
{
	struct kvp
	{
		std::array<char, HV_KVP_EXCHANGE_MAX_KEY_SIZE> key;
		std::array<char, HV_KVP_EXCHANGE_MAX_VALUE_SIZE> value;
	} pair;
	std::array<char, HV_KVP_EXCHANGE_FIELDSIZE> raw;
};

class HVKVPRecord
{
private:
	hvkvp KVP;
	std::streampos SourceStreamPosition = -1;

public:
	static const bool hvkvp_isgood(const hvkvp& KVP);
	const bool good() const { return hvkvp_isgood(KVP); }
	const bool bad() const { return !good(); }
	void Clear();
	void ClearKey();
	void ClearValue();
	const char* SerializedData() const { return KVP.raw.data(); }
	static const size_t DataSize() { return HV_KVP_EXCHANGE_FIELDSIZE; }
	const char* SerializedKey() const { return KVP.pair.key.data(); }
	static const size_t KeySize() { return HV_KVP_EXCHANGE_MAX_KEY_SIZE; }
	const char* SerializedValue() const { return KVP.pair.value.data(); }
	static const size_t ValueSize() { return HV_KVP_EXCHANGE_MAX_VALUE_SIZE; }
	void Deserialize(const std::array<char, HV_KVP_EXCHANGE_FIELDSIZE>& FileBytes, const std::streampos FileOffset);
	const std::string Key() const { return std::string(KVP.pair.key.data()); }
	void SetKey(const std::string& val);
	const std::string Value() const { return std::string(KVP.pair.value.data()); }
	void SetValue(const std::string& val);
	friend std::istream& operator>>(std::istream& InputStream, HVKVPRecord& kvpin);
	friend std::ostream& operator<<(std::ostream& OutputStream, const HVKVPRecord& kvpout);
	friend std::fstream& operator<<(std::fstream& OutputFileStream, const HVKVPRecord& kvpout);

	HVKVPRecord(const std::string& NewKey = std::string(), const std::string& NewValue = std::string());
	HVKVPRecord& operator=(const HVKVPRecord&);
	~HVKVPRecord();
};

#endif // __HVKVP_H_INCLUDED

hvkvp.cpp

/*************************************************************************************
hvkvp.cpp (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#include "hvkvp.h"

using std::string;

const bool HVKVPRecord::hvkvp_isgood(const hvkvp& KVP)
{
	/***************************
	1: key and value start with non-null characters and end with null characters; no null characters before non-null
	2: key cannot be all null (at least 1st character must be non-null)
	3: value can be all null
	***************************/
	if (KVP.raw[0] == '') { return false; }

	bool NullFound = false;
	for (int CheckPos = 1; CheckPos < HV_KVP_EXCHANGE_FIELDSIZE; CheckPos++)
	{
		if (CheckPos == HV_KVP_EXCHANGE_MAX_KEY_SIZE) { NullFound = false; }	// Reset null locator because this is the first position of the value
		if (KVP.raw[CheckPos] == '') { NullFound = true; }
		else { if (NullFound) { return false; } }
	}
	return true;
}

void HVKVPRecord::Clear()
{
	ClearKey();
	ClearValue();
}

void HVKVPRecord::ClearKey()
{
	std::fill(KVP.pair.key.begin(), KVP.pair.key.end(), 0);
}

void HVKVPRecord::ClearValue()
{
	std::fill(KVP.pair.value.begin(), KVP.pair.value.end(), 0);
}

void HVKVPRecord::Deserialize(const std::array<char, HV_KVP_EXCHANGE_FIELDSIZE>& FileBytes, const std::streampos FileOffset)
{
	std::copy(FileBytes.begin(), FileBytes.end(), KVP.raw.begin());
	SourceStreamPosition = FileOffset;
}

void HVKVPRecord::SetKey(const string& val)
{
	ClearKey();
	if (val.length() > 0)
	{
		auto endtext = val.end();
		if (val.length() > HV_KVP_EXCHANGE_MAX_KEY_SIZE)
		{
			endtext = val.begin() + HV_KVP_EXCHANGE_MAX_KEY_SIZE;
		}
		copy(val.begin(), endtext, KVP.pair.key.begin());
	}
}

void HVKVPRecord::SetValue(const string& val)
{
	ClearValue();
	if (val.length() > 0)
	{
		auto endtext = val.end();
		if (val.length() > HV_KVP_EXCHANGE_MAX_VALUE_SIZE)
		{
			endtext = val.begin() + HV_KVP_EXCHANGE_MAX_VALUE_SIZE;
		}
		copy(val.begin(), endtext, KVP.pair.value.begin());
	}
}

HVKVPRecord::HVKVPRecord(const string& NewKey, const string& NewValue)
{
	SetKey(NewKey); SetValue(NewValue);
}

HVKVPRecord& HVKVPRecord::operator=(const HVKVPRecord& rhs)
{
	if (this != &rhs)
	{
		SourceStreamPosition = rhs.SourceStreamPosition;
		std::copy(rhs.KVP.raw.begin(), rhs.KVP.raw.end(), KVP.raw.begin());
	}
	return *this;
}

HVKVPRecord::~HVKVPRecord()
{
	Clear();
}

std::istream& operator>>(std::istream& InputStream, HVKVPRecord& kvpin)
{
	HVKVPRecord KVPtemp;
	KVPtemp.SourceStreamPosition = InputStream.tellg();
	for (unsigned long i = 0; i < KVPtemp.KVP.raw.size(); i++)
	{
		if (!InputStream || InputStream.bad()) { return InputStream; }
		InputStream >> std::noskipws >> KVPtemp.KVP.raw[i];
	}

	kvpin = KVPtemp;

	return InputStream;
}

std::ostream& operator<<(std::ostream& OutputStream, const HVKVPRecord& kvpout)
{
	OutputStream << std::noskipws << string(kvpout.KVP.pair.key.data()) << ": " << string(kvpout.KVP.pair.value.data());
	return OutputStream;
}

std::fstream & operator<<(std::fstream& OutputFileStream, const HVKVPRecord& kvpout)
{
	for (size_t i = 0; i < kvpout.DataSize(); i++)
	{
		OutputFileStream.put(kvpout.KVP.raw.data()[i]);
	}
	return OutputFileStream;
}

hvkvpfile.h

/*************************************************************************************
hvkvpfile.h (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#ifndef __HVKVPFILE_H_INCLUDED
#define __HVKVPFILE_H_INCLUDED 1

#include <regex>
#include "hvkvp.h"

class HVKVPFile
{
private:
	std::regex Filter;
protected:
	bool UseFilter;
	void SetFilter(const std::string& FilterString);
	long GetFileSize(std::string FileName);
	HVKVPFile() {};
	const bool KeyMatches(const std::string& TestKey, const bool InvertMatch = false) const;
public:
	bool MatchExact = false;
	bool CaseInsensitive = false;
};

#endif // __HVKVPFILE_H_INCLUDED

hvkvpfile.cpp

/*************************************************************************************
hvkvpfile.cpp (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#include <sys/types.h>
#include <sys/stat.h>
#include "hvkvpfile.h"

// GCC versions < 4.9 ship with a partially-implemented version of regex
#if __GNUC__ < 4 || 
				  (__GNUC__ == 4 && (__GNUC_MINOR__ < 8 || 
											(__GNUC_MINOR__ == 8 && 
											 __GNUC_PATCHLEVEL__ <= 5)))

#define USE_OLD_MATCH_STYLE 1
#endif

using std::string;

long HVKVPFile::GetFileSize(string FileName)
{
	struct stat FileStatistics;
	if (stat(FileName.c_str(), &FileStatistics) == -1)
	{
		throw(errno);
	}
	return FileStatistics.st_size;
}

void HVKVPFile::SetFilter(const string& FilterString)
{
	if (FilterString.length())
	{
		UseFilter = true;
		string NewFilter;

#ifdef USE_OLD_MATCH_STYLE
		// because we can't pawn our case-sensitivity issues off on regex, upper-case the string. works in most cases
		if (CaseInsensitive)
		{
			for (auto ch : FilterString)
			{
				NewFilter.push_back((char)toupper(ch));
			}
		}
		else
		{
			NewFilter.assign(FilterString);
		}
#else
		NewFilter.assign(FilterString);
#endif
		if (!MatchExact) { NewFilter = ".*" + NewFilter + ".*"; } // regex_search is not universally implemented, either. this always works, and is cheap enough for character strings of these sizes
		Filter = std::regex(NewFilter, (CaseInsensitive ? std::regex_constants::icase : std::regex_constants::basic));
	}
	else
	{
		UseFilter = false;
		Filter = std::regex();
	}
}

const bool HVKVPFile::KeyMatches(const string& TestKey, bool InvertMatch) const
{
#ifdef USE_OLD_MATCH_STYLE
	string CasedKey;
	if (CaseInsensitive)
	{
		for (auto ch : TestKey)
		{
			CasedKey.push_back((char)toupper(ch));
		}
	}
	else
	{
		CasedKey = TestKey;
	}
	return InvertMatch xor std::regex_match(CasedKey, Filter);
#else
	return InvertMatch xor std::regex_match(TestKey, Filter);
#endif
}

hvkvpreader.h

/*************************************************************************************
hvkvpreader.h (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#ifndef __HVKVPREADER_H_INCLUDED
#define __HVKVPREADER_H_INCLUDED 1

#include "hvkvpfile.h"

enum class KVPReadPools { HostToGuest, GuestToHost, HostToGuestIntrinsic };

class HVKVPReader : public HVKVPFile
{
public:
	KVPReadPools Pool = KVPReadPools::HostToGuestIntrinsic;
	bool InverseMatch = false;
	std::vector<HVKVPRecord> ReadKVPs(const std::string& KeyNameFilter = std::string());
};

#endif // __HVKVPREADER_H_INCLUDED

hvkvpreader.cpp

/*************************************************************************************
hvkvpreader.cpp (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#include <vector>
#include "hvkvpreader.h"

using std::string;

static string SelectSourceFile(KVPReadPools SourcePool)
{
	string SourceFile;
	switch (SourcePool)
	{
	case KVPReadPools::GuestToHost:
		SourceFile = HVKVP_FILE_TOHOST;
		break;
	case KVPReadPools::HostToGuest:
		SourceFile = HVKVP_FILE_TOGUEST;
		break;
	default:
		SourceFile = HVKVP_FILE_GUESTPARAMETER;
		break;
	}
	return SourceFile;
}

std::vector<HVKVPRecord> HVKVPReader::ReadKVPs(const string& KeyNameFilter)
{
	std::vector<HVKVPRecord> FoundKVPs;
	string SourceFile = SelectSourceFile(Pool);
	std::ifstream KVPSourceFile(SourceFile, std::ios::binary);
	if (KVPSourceFile.is_open())
	{
		SetFilter(KeyNameFilter);
		HVKVPRecord KVPIn;
		while (KVPSourceFile >> KVPIn)
		{
			if (!UseFilter || (KeyMatches(KVPIn.Key(), InverseMatch)))
			{
				FoundKVPs.push_back(KVPIn);
			}
		}
	}
	else
	{
		throw(strerror(errno));
	}
	return FoundKVPs;
}

hvkvpremover.h

/*************************************************************************************
hvkvpremover.h (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#ifndef __HVKVPREMOVER_H_INCLUDED
#define __HVKVPREMOVER_H_INCLUDED 1

class HVKVPRemover : public HVKVPFile
{
private:
	std::string FileName = HVKVP_FILE_TOHOST; // at this time, we can only remove KVPs from the guest to host file
	std::string KeyToRemove;
	bool RemoveDuplicates;
	bool RemoveInvalid;
	std::vector<std::streampos> ValidKVPLocations;
	const bool IsDuplicateKey(std::vector<std::string>& FoundKeys, const std::string& KeyToCheck);
	void LocateValidKVPLocations(std::fstream& KVPFile);
	void RemoveInvalidLocations(std::fstream& KVPFile);
public:
	HVKVPRemover() { MatchExact = true; }
	bool InverseMatch = false;
	void RemoveKVP(const std::string& KeyName, const bool CleanDuplicates = false, const bool CleanInvalid = false);
	void RemoveAll();
};

#endif // __HVKVPREMOVER_H_INCLUDED

hvkvpremover.cpp

/*************************************************************************************
hvkvpremover.cpp (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#include <vector>
#include <unistd.h>
#include "hvkvpfile.h"
#include "hvkvpremover.h"

using std::string;

static int TruncateFile(const string& FileName, std::streampos NewLength)
{
	return truncate(FileName.c_str(), NewLength);
}

const bool HVKVPRemover::IsDuplicateKey(std::vector<string>& FoundKeys, const string& KeyToCheck)
{
	errno = 0;
	std::regex DuplicatesFilter(KeyToCheck, (CaseInsensitive ? std::regex_constants::icase : std::regex_constants::basic));
	for (auto FoundKey : FoundKeys)
	{
		if (regex_match(FoundKey, DuplicatesFilter)) { return true; }
	}
	FoundKeys.push_back(KeyToCheck);
	return false;
}

void HVKVPRemover::LocateValidKVPLocations(std::fstream& KVPFile)
{
	errno = 0;

	// Hyper-V only reads the LAST instance of a key. To facilitate duplicate removal, work backward to gather information
	long RecordLength = HVKVPRecord::DataSize();
	long InitialFileSize = GetFileSize(FileName);

	if ((InitialFileSize % RecordLength) && !RemoveInvalid)
	{
		errno = EUCLEAN;
		return;
	}

	SetFilter(KeyToRemove);
	HVKVPRecord CurrentKVP;
	std::vector<std::string> FoundKeys;
	std::streampos RecordStartingPosition;

	bool Keep;

	KVPFile.seekg(0, std::fstream::end);
	while (KVPFile.seekg(-RecordLength, KVPFile.cur))
	{
		Keep = false;
		RecordStartingPosition = KVPFile.tellg();
		KVPFile >> CurrentKVP;
		if (CurrentKVP.bad()) { Keep = !RemoveInvalid; }
		else
		{
			if (!KeyMatches(CurrentKVP.Key(), InverseMatch))
			{	// xor would be cleaner but would force the duplicate check to be run when unnecessary
				if (!RemoveDuplicates || !IsDuplicateKey(FoundKeys, CurrentKVP.Key())) { Keep = true; }
			}	// else: this is the key that the user asked to remove, so do nothing; this will leave "Keep" unset, effectively marking this KVP for deletion
		}

		if (Keep) { ValidKVPLocations.push_back(RecordStartingPosition); }
		if (!KVPFile.bad()) { KVPFile.clear(); }
		KVPFile.seekg(RecordStartingPosition, KVPFile.beg);
	}
	if (errno == EINVAL) { errno = 0; } // the while loop terminates when we attempt to set seek to a negative position, which causes errno to be set to EINVAL
}

void HVKVPRemover::RemoveInvalidLocations(std::fstream & KVPFile)
{
	// use each valid location as a source to sequentially overwrite file, then truncate the file
	KVPFile.clear();
	HVKVPRecord SourceKVP;
	long RecordLength = HVKVPRecord::DataSize();
	std::vector<std::streampos>::reverse_iterator ValidLocations = ValidKVPLocations.rbegin();
	for (unsigned long i = 0; i < ValidKVPLocations.size(); i++)
	{
		KVPFile.seekg(ValidLocations[i], KVPFile.beg);
		KVPFile >> SourceKVP;
		if (!KVPFile.bad()) { KVPFile.clear(); }
		if (KVPFile.good())
		{
			KVPFile.seekp((i * RecordLength), KVPFile.beg);
			KVPFile << SourceKVP;
		}
	}
	if (!KVPFile.bad()) { KVPFile.clear(); }
	if (KVPFile.good())
	{
		std::streampos FileLength = KVPFile.tellp();
		KVPFile.close();
		TruncateFile(FileName, FileLength);
	}
}

void HVKVPRemover::RemoveKVP(const string& inKeyName, const bool inRemoveDuplicates, const bool inRemoveInvalid)
{
	errno = 0;
	KeyToRemove = inKeyName;
	RemoveDuplicates = inRemoveDuplicates;
	RemoveInvalid = inRemoveInvalid;
	
	std::fstream KVPFile(FileName, std::ios::binary | std::ios::in | std::ios::out);
	if (KVPFile.is_open())
	{
		LocateValidKVPLocations(KVPFile);
		if (errno) { throw(errno); }
		RemoveInvalidLocations(KVPFile);
	}
	if (errno) { throw(errno); }
	return;
}

void HVKVPRemover::RemoveAll()
{
	errno = 0;
	std::ofstream KVPFile(FileName);
}

hvkvpwriter.h

/*************************************************************************************
hvkvpwriter.h (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#ifndef __HVKVPWRITER_H_INCLUDED
#define __HVKVPWRITER_H_INCLUDED 1

#include "hvkvpfile.h"

class HVKVPWriter : public HVKVPFile
{
public:
	HVKVPWriter() { MatchExact = true; }
	void WriteKVP(const std::string& InputKey, const std::string& NewValue);
};

#endif // __HVKVPWRITER_H_INCLUDED

hvkvpwriter.cpp

/*************************************************************************************
hvkvpwriter.cpp (Hyper-V KVP Interface)
Copyright(C) 2017  Eric Siron

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110 - 1301, USA.
*************************************************************************************/

#include "hvkvpwriter.h"

using std::string;

void HVKVPWriter::WriteKVP(const string& InputKey, const string& NewValue)
{
	errno = 0;
	if (InputKey.empty())
	{
		throw(EINVAL);
	}
	string FileName(HVKVP_FILE_TOHOST);
	long InitialFileSize = GetFileSize(FileName);
	if (errno) { return; }
	std::fstream KVPFile(FileName, std::ios::binary | std::ios::in | std::ios::out);
	if (KVPFile.is_open())
	{
		// Hyper-V only reads the LAST value for a key. Overwrite ALL instances in case duplicates exist. If none exist, append.
		HVKVPRecord NewKVP(InputKey, NewValue);
		HVKVPRecord CurrentKVP;
		SetFilter(InputKey);
		bool KVPWasOverwritten = false;
		long CurrentKVPStartingPosition = 0;
		while (KVPFile.good() && CurrentKVPStartingPosition < InitialFileSize)
		{
			KVPFile >> CurrentKVP;
			if(KeyMatches(CurrentKVP.Key()))
			{
				KVPFile.seekp(CurrentKVPStartingPosition + CurrentKVP.KeySize());
				KVPFile.write(NewKVP.SerializedValue(), NewKVP.ValueSize());
				KVPWasOverwritten = true;
			}
			
			CurrentKVPStartingPosition = KVPFile.tellg();
		}
		
		if (!KVPWasOverwritten && !KVPFile.bad()) // eof and probably fail will be set, but bad() is bad
		{
			KVPFile.clear();
			KVPFile.seekg(0, std::fstream::end);
			KVPFile << NewKVP;
		}
	}
	if (errno) { throw errno; }
}

More in this series:

Part 1: Explanation

Part 2: Implementation