“Unable to connect to the MKS” Error – How to Fix it

When was the last time you tried to remote console to a VM only to be greeted with the annoying Unable to connect to the MKS or similar message? Some of the causes leading to this issue range from firewalls blocking the required ports, badly configured DNS, and ESXi services in need of a restart. Today’s post explores a number of ways you can use to remote console to a VMware virtual machine and what you can do when remote connections fail, and you are unable to connect to the MSK.

Speaking of firewalls, adding a persistent firewall rule to ESXi can be daunting. I discuss how to do this in How to create persistent firewall rules on ESXi.

What is MKS?

MKS stands for Mouse, Keyboard and Screen. In vSphere, you can remote console to VMs using the standard vSphere clients and products such as VMware Workstation.

In pre-6.5 vSphere releases, you can use the legacy vSphere client which has now been deprecated for a long time. As shown next, just highlight the virtual machine you want to connect to and click on the Console tab. Alternatively, right-click on the VM and select Open Console. The latter method opens a stand-alone console window for you.

Remote console using the vSphere Client

If you have vCenter installed, the vSphere Web Client provides you with two remote console options, these being VMRC (the standalone VMware Remote Console) and the Web Console.

You can change the default virtual console by clicking on your user in the top right corner > My Preferences > Default Console > You can then select the web or vmrc as your default console.

You also get both options directly underneath the preview of the VM screen in the VM view. VMRC is installed by following the link given in vSphere client or by downloading it from here.

NOTE: Users who have VMware Workstation installed will have noticed that selecting Launch Remote Console brings up Workstation’s VMRC instead of the standalone one. To override this behavior, uninstall the standalone VMRC and reinstall it back.

Back to vSphere Web Client, clicking on the VM’s console screen or selecting Open Console from the VM’s context menu will launch the default remote console.

Choose between the web console and VMRC.

Remote console using the host client (ESXi embedded)

Interestingly, you can configure a virtual machine such that you connect to it using a remote client such as tightVNC. To do this, add the following lines to the VM’s configuration (Edit Settings -> VM Options tab -> Advanced Settings -> Edit Configuration). You must disable the ESXi firewall for this to work or add a rule to allow 5900 through, this being the default VNC port. If memory serves me right, this worked with ESXi 5.x versions but to be honest, I had little time to try it out on 6.x. With ESXi 6.5, I manage to establish a connection but it keeps asking for a password regardless of the one specified in the VM’s configuration.

Configuring a VM to accept remote connections from VNC

What can go wrong and how to fix it

Some of the MKS errors (for instance, vmware unable to connect to the mks internal error) you might see are as follows:

• • Error connecting: Host address lookup for server <SERVER> failed: The requested name is valid and was found in the database, but it does not have the correct associated data being resolved for.

• • Error connecting: cannot connect to host <host>: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because the connected host has failed to respond.

• • Error connecting: You need to execute access in order to connect with the VMware console. Access denied for config file.

• • Unable to connect to MKS: Failed to connect to server IP:903.

1. Firewalls

Depending on the version of ESXi / vCenter Server installed, TCP ports 902, 903, 443 and 9443 need to be reachable when using the web based and standalone remote consoles. With vSphere 6.5, I managed to replicate a common problem where a personal firewall, or one sitting between the client computer and ESXi networks, is blocking one or more ports especially 902. To do this, I enabled the Windows firewall and created an outbound rule blocking 902 as shown.

Managing Microsoft Windows firewall rules

When the firewall rule is enabled, the following is what you get when trying to remote console to a virtual machine. The error message might differ slightly depending at which point the connection is dropped.

Remote console connection errors due to a firewall blocking ports 902 and 9443

Changing the firewall rule to Allow from Block fixes the “unable to connect to the mks” issue. Often, the lost ability to remote console occurs right after someone installs or rolls out security software incorporating a personal firewall. From my experience, other primary suspects include updates to group policies governing the Windows firewall and unannounced changes to ACLs on switches / routers. If you’re not the one administering the latter, talk to your colleagues to try and narrow down the source of the problem.

You can use netstat to verify that connections on port 902 are being established when you remote console to a VM. If you don’t see any entries, the issue is definitely firewall / network related. Another trick you can use is to telnet <ESXi IP> 902 from a command prompt or terminal. If you get a reply, it means that ESXi is listening on the port and ready for connections.

Using netstat to inspect network connections

If you’re not running a personal firewall, SSH to ESXi and temporarily disable the ESXi firewall by running esxcli network firewall set –enabled false. Verify that you can connect. If not, it’s a firewall sitting in between the management station and ESXi that’s dropping the connection.

If the problem persists after having ruled out firewalling as a problem source, move on to the next check.

2. DNS

Connection to remote console can also be impeded if ESXi and vCenter Server hostnames do not resolve correctly. To determine correct DNS resolution, use ping <ESXi hostname> or nslookup to verify that your ESXi and vCenter Server hostnames are resolving to the correct IP addresses and, likewise, to the correct hostnames. If the issues persists, try adding your ESXi hosts and IP addresses to \windows\system32\drivers\etc\hosts, run ipconfig /flushdns and retry.

192.168.10.10 ESX65-a.lab.local

192.168.10.11 ESX65-b.lab.local

192.168.10.12 ESX65-c.lab.local

3. Free Disk Space

If firewalling and DNS are not the culprit, verify that the partition where /var lives has enough free disk space. To do this, run df -h from an SSH session on the ESXi hosting the VM you’re trying to remote console to. Make sure that the partition where /var is located is not full.

Checking for free disk space on ESXi

Another command you could use is vdh -h.

Making sure /var has free disk space using the vdf -h command

4. Permissions

Verify that the permissions set on the VM’s VMX file are set correctly (755). To check the permissions, run the command:

ls -l *.vmx

-rwxr-xr-x 1 root root 3995 Aug 18 10:00 Windows 7.vmx

We can see that the permissions are correctly set for the VMX file of a VM called Window 7.

• • rwx = 111 = 7

• • r-x = 101 = 5

• • r-x = 101 = 5

If the VMX file permissions are not set to 755, run the following command:

chmod 755 ‘Windows 7.vmx’

Note: On ESXi 6.5, setting the permissions to read only had no effect so the permissions issues might only apply to older versions of ESXi.

5. If nothing else works!

If none of the above worked, try doing the following:

• • Check that the time on ESXi is set correctly and in sync with the rest of the infrastructure.

• • vMotion the VM you’re unable to remote console to, to another ESXi host and retry.

• • Restart the management agents on ESXi as per the following KB.

• • Reboot the VM.

• • Reboot ESXi.

Note: Sometimes to fix if vmware remote console is unable to connect to the mks is as trivial as pressing a key while the focus is given to the console window. This was common with older versions of the legacy client and vSphere releases where nothing is displayed in the VM’s console window. Hitting Enter or another key usually brings the console back to life. An alternative fix is to launch the console in its own window by right-clicking on it and selecting Open Console.

Conclusion

Being able to remote console to a virtual machine is, of course, an integral part of any admin’s job. Seeing the dreaded “Unable to connect to the MKS” message and being able to console can be highly frustrating more so when you need to fix something fast. In most cases, the problem lies with a firewall blocking one or more required ports and missing or bad DNS entries. For the rest, it’s usually enough to vMotion a VM to another host or restart the management agent on ESXi.

If you enjoyed reading this post, why not have a look at The Complete List of VMware Articles.