Troubleshooting Adapter Teaming in Hyper-V Server

Native adapter teaming is a hot topic in the world of Hyper-V. It’s certainly nice for Windows Server as well, but the ability to spread out traffic for multiple virtual machines is practically a necessity. Unfortunately, there is a still a lot of misunderstanding out there about the technology and how to get it working correctly.

We’ve written a couple of posts on this subject previously, but one in particular should be considered background material. If you haven’t read it, then start with part four of our networking series, which focused on Link Aggregation and Teaming. We’re also going to talk about load-balancing algorithms in this article, so make sure you’re up to date on part 8 of that same series, which dealt with that subject. We’ll revisit some of the highlights of those articles out of necessity, but it’s not my intent to reproduce them here. Please make sure you’ve read them before tackling this one.

1. Be Able to Recognize Proper Behavior

The reason I want you to read those articles is because, by my observation, most teams are behaving properly; it’s the administrator’s expectations that are broken. I doubt there’s more than a week that goes by without someone complaining that they built a new team with twelve network adapters and tried a file copy and it “only” goes at 120 megabytes per second. For starters, don’t use file copy as a network testing tool. It’s OK for an initial “sniff test”, but don’t bring it to me as proof that your network isn’t performing up to par. File copy is disk-bound at both the source and the destination and that alone is just too many variables to even come close to calling it a proper network test.

I’ve used a number of tools in my time, but lately it’s mostly been Microsoft’s NTttcp. There are other free ones out there that are easier to use, but most say, “Not for commercial use,” so NTttcp it is. First, let me warn you that the included documentation is mostly junk. I think it was written for an older version and subsequently forgotten. Read the material on the download page for more usable instructions. The other thing I don’t like about this tool is that whereas most others have a “server mode” that sits and listens for connections indefinitely, this one does not. You have to flip back and forth between source and destinations to start them somewhat close to each other, and once the test is done, the “server” exits as well. That could rule out usage in some instances. Anyway, when testing network speeds, find a network testing tool.

But, this post is not about network speed. It’s about troubleshooting teaming. These are the things you must know before you even get started:

• Link aggregation is not bandwidth aggregation. Just because you group a bunch of physical adapters together doesn’t mean that all communications are suddenly going to flow at their combined rate.
• Other components matter. Networking, by definition, involves lots of pieces working together. It takes some work to get them all to do so harmoniously. If you only look at one thing, then you may not find the source of the problem.
• Just because you think a transmission should be balanced to another path doesn’t mean the system does. This concept is from the previous two and will be expanded upon throughout the article.
• LACP is not faster than static teaming. This is a really old myth that refuses to die. LACP does two things. First, the switches know which ports should be in the team. If any one of those ports isn’t connected to a remote port that is communicating in the designated team, then the switch stops using that port. Second, the switches will periodically transmit special frames (LACPDU) intended to check the health of each port-to-port connection.
• 10GbE requires a knowledge update. If you just bring everything you know about 1GbE and assume it will all be the same for 10GbE, you’re in for some unpleasant surprises.
• Your results will be inconsistent. One test tells you little. Two tests don’t tell you much more. You need to take some time to perform thorough testing before making any decisions.
• Networking people know more about this stuff than Windows people. When I started working with teaming, I worked with Cisco certified engineers because that’s what made sense to me. Apparently, I was in the vast minority. When you get stuck on a link aggregation concept, it’s a good idea to start by asking someone who is specifically skilled in networking.
• The modes that you pick matter. Dynamic load-balancing with LACP teaming has an entirely different performance profile than the Hyper-V port algorithm with switch independent teaming. Make sure you know what your selected profile is supposed to look like.
• It’s still Windows. Even if networking and teaming aren’t your strongest skills, you still have access to the same basic troubleshooting toolkit that everyone in our industry should have. Check Event Viewer. Use logical processes to isolate problems. Ask someone.

The takeaway from this section is that you must know what is reasonable to expect from a properly configured network team. It may not be what you think. The previously linked articles should help to explain it.

2. Physical Switch Configuration Errors

The switch independent teaming mode does reduce the need to configure physical switches, but it does not eliminate it.

LAG/Port Channel Settings

Cisco calls their aggregated ports “Port Channels”. Most everyone else calls them Link Aggregation Groups (LAG). Whatever you call them, they have to be configured – unless your Hyper-V/Windows team is in “Switch Independent” mode, in which case your switch must absolutely not be configured with a LAG or Port Channel.

You can get some quick and handy, although usually not thorough, troubleshooting information by running Get-NetLbfoTeamMember.

Trunk/Tagged/Untagged Settings

At face value, these have much more to do with the switches in general than with LAG and Port Channel, but they can’t be forgotten. If you build a LAG or Port Channel, you have to ensure that any VLANs you want to use with your Hyper-V systems are allowed on the trunk – or, for non-Cisco systems, are appropriately set as tagged or untagged traffic on the LAG. While you’re at it, make sure that you understand and have properly configured any native VLANs (untagged for non-Cisco) and set the appropriate PVID for non-Cisco switches. See part two of the previously referenced networking series if you need a refresher on these terms.

Port Security

On switch independent teams, port security can be a nightmare. The one I’ve seen most often is MAC address security. As a rule, a single MAC address should appear on one switch port and it should pretty much stay there. If it shows up on another, especially if there is no disconnect of the original port, that could be a MAC spoofing attack. For a switch independent Windows network team, it’s part of normal operations. Symptoms of a port security problem are intermittently working network connections. The physical switch will allow MAC addresses to operate on the first port it sees them on, but if they move to another port for any reason, they’ll be blocked there.

This can still be a problem with LACP and static teams when clusters are involved due to Live Migration. Your best bet is to only use port security for unteamed physical hosts.

Check Cables

No excuses. It doesn’t take long to swap switch ports just to see if the problem follows the cable. Many switches these days have basic cable testing functionality. Real cable testers can be had fairly cheaply.

Match Frame Sizes

If you’re going to use Jumbo Frames, that’s fine. Just make sure that everyone is configured the same way. All switches ship with Jumbo Frames disabled by default. Every switch I’ve ever used required a restart for them to be enabled. We have an older article that explains how to set up Jumbo Frames for Hyper-V Guests. It was written specifically for Hyper-V guests, but the process is the same for any network adapter. Look for a forthcoming article updated for the more recent versions. The basic rundown is that the host’s adapters in the team you want to enable for jumbo frames must have these options set.

3. Understand Ethernet and TCP/IP Pathing

Network pathing was integral to part 8 of the networking series and is worth a quick synopsis. Frames moving across an Ethernet network must follow a very rigid set of rules. The TCP/IP suite we all use on that Ethernet framework was designed with the idea that frames would take multiple paths, some safer than others. It has a great deal of tolerance of out-of-sequence and dropped frames. However, its limits are not infinite and it must adhere to the rules of the underlying layer 2 system (Ethernet in our case), and all of this comes at a cost: TCP/IP is not an overly efficient suite, especially TCP. The header-to-data size ratio is high and the compute resources necessary to deal with out-of-sequence and dropped frames can be significant. This should not be seen as a bad thing; TCP’s designers had to choose between quick and reliable and they went with reliable. For the rest of the protocols in the suite, routability and scalability independent of the layer 2 network was also an important goal and this incurs overhead of its own.

As a result of this reliability and scalability, no single communications transmission from the TCP/IP suite downward will ever take more than a single physical path. There are applications built atop TCP/IP that can simulate multi-pathing behavior, such as MPIO and SMB multi-pathing, but TCP/IP and Ethernet do not have the ability. So, when you’re running some sort of program or file copy across a big team and it only goes as fast as the slowest participating member between the source and the destination, this is why.

Things to remember here:

• In order for the Hyper-V switch, or any other switch, to use a different physical pathway for a given transmission, it must believe it is unique among all other transmissions.
• Just because you believe two transmissions are distinct doesn’t mean that the system does.
• Just because the source system believes two transmissions are distinct doesn’t mean that the target system does.
• The physical switch(es) are factors in and of themselves. They can easily present bottlenecks of their own.
• Load-balancing is round-robin in Hyper-V, so distribution is really luck of the draw. Sometimes, you just really get unlucky.
• Switches have to perform load-balancing too. Just as with Hyper-V, sometimes, you just really get unlucky.

I hope then, you’ll see why your results may be somewhat inconsistent.

4. Modes Still Matter

Remember how I said the modes matter in the final bullet point of #1? Well, that’s because they can have a great deal of impact. Switch independent modes can especially present problems. With the introduction of the new Dynamic load-balancing algorithm, I’ve seen a lot of people start blindly recommending switch independent mode. It’s easier to configure, that’s for sure, but that doesn’t make it better. What happens with switch independent modes is that, at best, all inbound traffic to a given virtual adapter is restricted to a single physical port. That means that there is no load-balancing algorithm that’s going to allow that virtual adapter to exceed the speed of a single physical link for inbound traffic, no matter how many physical inbound pathways there are. If you’re not using Dynamic or Hyper-V Port load-balancing, then all inbound traffic to a teamed Hyper-V switch is restricted to a single physical port. This is just the way that MAC addresses work. It’s not necessarily a bad thing, especially if you’re using 10GbE. It’s just something that you must be aware of when trying to determine why a team isn’t going as quickly as you think it should. Refer to the articles from the first paragraph of this post for details.

Another thing is that switch independent and static teaming modes cannot detect partial link failures. I’m not entirely certain what they do if a link is completely down, although I suspect it varies. I have seen these modes cause severe problems when a single pair of a cable turned out to be broken and making poor connections. LACP is designed to detect problems such as these. It’s not entirely perfect, as I’ve had occasional problems there as well, but it is much better than the other methods. The takeaway here is that the mode you select will interact differently when there are problems in the physical links.

One danger is that not all physical switches are created equally. For most switches built in the last few years, there probably isn’t anything to worry about. However, some of the teaming modes existed for some time in draft standards, so there was some variance in implementations. This was especially true prior to the adoption of 802.3ad, but even that “standard” wasn’t always standard. 802.1ax seems to be enjoying much more success.

5. Hardening Causes Performance Problems

Something I uncovered with the help of a Microsoft engineer is that there are a lot of ways to harden TCP/IP from the past that cause performance problems in 10GbE. The issue is that these hardening techniques impose additional processing. A single CPU core is required to process each unique stream of data, and right now, one core can’t process 10 Gbps of TCP/IP packets. When these hardening techniques are employed, performance really drops. These settings can be found in this Microsoft article. One that makes a particularly heavy impact is “EnablePMTUDiscovery”, as it forces packet payloads to their minimum size. This means the maximum amount of overhead is necessary to transmit and receive each packet, resulting in a significantly large CPU load.

Obviously, these settings exist for a reason. They should not be applied blindly, though, especially on virtual adapters using 10 GbE or greater connections.

6. 10 GbE is Not Just Faster Gigabit

The transition from Fast Ethernet to Gigabit was fairly smooth because client computing hardware outpaced the development of networking hardware. By the time gigabit became more or less ubiquitous, contemporary CPUs were able to handle its packet load processing with relative ease. With 10 GbE, that’s really not the case. The fastest CPUs cannot process 10 gigabits per second of data encapsulated in TCP/IP packets. especially with standard payload sizes (jumbo frames means less processing). Just as with gigabit, most of the traditional offloading techniques either do nothing or make things worse. However, things like RDMA and RSS can really boost performance. However, these are usually only found on higher end cards, especially RDMA. While 10 GbE itself is becoming more affordable all the time, 10 GbE with RDMA will still easily cost several hundred dollars more than an otherwise equivalent non-RDMA card.

Something I try to impress upon people all the time is that if you truly need better performance, the only way to get there is by buying better hardware. There is no software solution that can make cheap hardware perform like high-end hardware. True 10 GbE speeds are extremely expensive to achieve at this time. What this means for teaming is that you will get better performance out of teamed 10 GbE NICs in production than you will in simple testing, depending upon the teaming and load-balancing methods you select. The easier it is for the team to determine that a stream can be balanced to another physical CPU, the better utilization you’ll get. The best way to ensure you get that sort of balancing is by having multiple streams in flight across unique sources, destinations, and applications. This sort of thing can be annoying to build good tests for, but real-world configurations usually meet it easily.

As a bonus, most real-world applications don’t demand nearly as much networking throughput as your tests. The sort of “failed” experiments that administrators set up are often never felt by end users.