Save to My DOJO
I often lament the way that small businesses are treated by the majority of technical writers and a significant number of software companies. You know the ones I’m talking about — those who think of “small business” as a nebulous concept that means “something under a few thousand employees”. They take what they know and just scale it down to some arbitrary minimum that in no way represents a small business.
Fortunately, I’ve spent most of my career working with and for truly small businesses, right down to the loners building up companies out of their basements. I know the struggles of insufficient time and budget shortfalls and lack of expertise. I also know what works and what doesn’t. What I intend to share with you is the distillation of all that experience into the strategy that I would follow if I ever struck out on my own to build a startup. This is only the first of my articles targeting the small business environment. I’ve got plenty for you.
To begin, I want to introduce a set of 8 Hyper-V best practices for small businesses that are important to consider.
1. Backup is Your Highest Priority
In my second full-time IT position, one of my very first responsibilities was assisting small business customers across the country with backup problems. I held that position for five years. During that period, I accumulated stories that would give you nightmares. I’ve had to say the words, “I’m sorry, but your data is unrecoverable,” more than once. The most disheartening thing I see is forum posts from small businesses wanting to know what the “best free backup” program is. “Best” is not where they’re drawing the line; it’s drawn at “free”. Don’t get me wrong; free is good — but only if it meets every single one of your needs. Skimping on backup can very easily mean placing your entire business at risk, unless your business is able to withstand a complete data loss event. Even if it can withstand that today, are you not planning to grow?
I expect you to ask, “How can I know if a particular backup solution meets my needs?” That’s a wonderful question, and I’ll answer it even if you didn’t ask. Here’s how you know:
- Every last piece of data that you care about must be backed up for at least as long as you’ll ever care about it. If you’re in a particular industry, such as finance, make sure that you understand the regulations around data retention that apply to you. Regulations aren’t always the deciding factor; I once had to recover a 7-year-old e-mail to aid a plaintiff’s case. Some free backup applications have a limit on retention. When evaluating an application on these grounds, remember that “retention” only applies to the age of the backup itself; if the data that was backed up last night is 5 years old, then the backup age is less than a day.
- Any backup data in the same geographical location as the source data is forfeit to the whims of chance. You need some way to move it offsite, preferably out of the range of any natural disaster that might destroy your main location. For small businesses, there is a limit to the effectiveness of very long-range offsite storage. If your town is entirely destroyed by a flood and all of your customers are in that town, then how far do you really need to move your data? If your business is an insurance agency, you’ll likely answer that question differently than would someone who owns a landscaping outfit. Of course, your own business is likely insured, so you’ll need to protect any data that would be needed to file any claims in the event of a business-ending disaster. The takeaway of this point is that the backup solution that you use must allow you to take your data offsite and you must follow through.
- The backed-up data must be recoverable. I often hear backup referred to as “an insurance policy”. That analogy is awful. People go out of their way to avoid using insurance policies because filing a claim often leads to premium increases, and in the worst cases, policy cancellations. Practice data restoration often. The primary benefit is that it proves that your backup data is good, your backup application is good, and that your skills are good. I have worked with several applications through the years that take absolutely lovely backups that no one in the world can restore data from. I can’t imagine anything that would be less useless. You won’t know if yours is one of those without trying it out. The second benefit is that practice makes perfect. You don’t want to be learning how to restore data in the middle of a crisis.
- You can’t be the only one that knows. For the most part, this isn’t entirely about the backup application. Working from the assumption that your business is important to more people than you, such as those that your income supports, make sure that you leave some form of documentation that would allow someone else to recover your data in the event that you are… unavailable. This affects your backup application choices in that you don’t want some obscure application that no one else can figure out.
- Encryption matters. You only want to be mentioned in news headlines for good things. “Small Business Corp Loses Customer Records on Unencrypted Data Tapes” does not qualify.
I certainly understand that cost is important, but it is secondary at most. Give up something else before you skimp on backup.
2. Backup is Your Redundancy
Despite the word “backup” in the title, backup is not the target of this point. “Redundancy” is. I know at least a few people that seem to derive great pleasure from terrorizing forum posters over a “single point of failure”. Ignore them.
With Hyper-V being the focal technology of this post, we’ll start with that. In the “all businesses are the same only different sizes” philosophy, you’ll have at least two clustered hosts connected to fully redundant shared storage with every component connected to a minimum of two switches. That sounds good, and believe me, it is. But it’s more than twice as expensive as a single host with internal storage connected to a single switch. “Single point of failure!” scream the people that don’t have to deal with your budgetary issues. Well, I took a course last year from an instructor who continually spoke of the “single point of success.” “Gives it a more positive tone, don’t you think?” he asked when I inquired about it. He’s right. Fewer than 2% of computer problems are caused by hardware failure and fewer parts means less complexity and fewer things to troubleshoot and maintain. If you have a good backup and you know how to use it, take comfort in the knowledge that the odds are heavily in your favor.
3. Quality Matters
A core requirement to making the previous point work is choosing quality at every point. If you save $500 on a shiny new mid-tower to run Hyper-V but have to spend three days in the forums and several hours on the phone or e-mail with technical support because Hyper-V won’t start, that is three days and several hours that you weren’t out drumming up business. If you have trouble getting it to work, you’re probably going to have trouble keeping it working.
Hardware components fail sometimes, and sometimes they’re shipped out new in non-working condition. It’s rare, but unavoidable. What you need to concern yourself with is going with the most likely equipment that will succeed. Start on WindowsServerCatalog.com. Ask around. Perform an Internet search for “<vendorname> Hyper-V problems”.
Oh, and quality isn’t important just for hardware.
4. Get Help
If I were to start a new business, I probably would not hire any technical assistance to start. I’m fairly certain I know what I’m doing. However, if I started putting in a noticeable amount of my time doing technical support, I wouldn’t hesitate to pick up the phone. Neither should you. Even if you’re knowledgeable today, running a business is going to consume a lot of time. You’re eventually going to have to choose between maintaining your top-tier technical status and running a business. At some point, you’re going to have to pass the burden on.
Of course, as a small business, you’re not hiring any full-time engineers. But, there are many small technology services companies out there that cater to the small business. Find them.
In your search, get help with that too. Point #3 applies here just as strongly as it does to hardware. It’s a sad fact that many technology service companies are terrible with technology and owe their entire existence to customers that have even less knowledge. A few pointers:
- If you’re not technically inclined, attend local small business expos and networking sessions where you can meet other local small business owners. They love to share their horror stories and most will gladly make recommendations.
- If you are technically inclined, you can spend a bit of time researching technical subjects, at least enough to keep abreast of the current state of the art. Sometimes, you’ll just find people that will flat-out tell you how to spot frauds. For instance, if you have service provider insisting that you can’t oversubscribe CPU, must turn off Dynamic Memory, and can never use dynamically expanding virtual hard disks, that provider is uneducated and/or trying to sell you hardware that you don’t need. I have one tip for those times when you discover that your provider is clueless and/or unethical: do not attempt to argue with them or help them get better. Fire them immediately and move on. In almost twenty years of being in this industry, I have never seen any evil or willfully ignorant technology provider come to the light.
- Good engineers do not work for cheap. When I left the provider industry to move to internal IT, I was charging a higher rate than anyone else that I knew of. However, I charged individual customers noticeably fewer hours than my competitors, so I needed that higher rate to stay afloat. Do you want the firm that charges $60 per hour and will need a permanent office of their own in your building or the firm that charges $200 per hour and will be done in two days?
5. Most “Best Practices” were Written by People Who Do Not Believe that You Exist
As far as the technical world is concerned, you might as well name your company “The Tooth Fairy” or “Santa Claus”. Two or more domain controllers on separate physical units? Who is going to pay for all of that? Multiple Internet providers to one building? Not a chance. Redundant everything? Maybe if you win the lottery — assuming you can ever afford a ticket.
Most “Best Practices” lists are built on solid reasoning and they are important to understand. With solid understanding, you can also discern where your small business can deviate. Hopefully, you’ll have a good provider or assistant (#4) that can guide you in the proper direction. If your provider is blindly quoting and adhering to a list they printed off of someone else’s website, that’s a bad sign. The word “Why” is the most important tool in your kit in these situations. A provider’s explanation is less important than they way that they answer. If they don’t seem certain, it’s because they aren’t. If they have a canned, practiced answer that in no way involves or references your situation, they do not understand what they are doing.
6. Use Hyper-V in a Way that Saves Money
I used to work with a fairly standard build for small business: one server installation that did everything. Best practice? Absolutely not. But, when clients are working with annual technology budgets that are smaller than a first-class ticket to the nearest airport, you make do. Nowadays, choices are simpler with things like Office 365 and Exchange Online. The recent pricing shake-up that Microsoft enacted with its OneDrive product likely, and deservedly, has small business owners nervous about signing on with any Microsoft cloud service. However, Small Business Server is no longer an option (and frankly, was never a wonderful option) and the cost of acquiring and maintaining some Microsoft servers on-premises is prohibitively expensive.
Considerations for hosting on-premises:
- If you can’t get reliable, affordable Internet access, skip cloud providers. They don’t care if you can’t connect.
- If you’re going to have any servers on-premises at all, Active Directory is a must-have. Only go workgroup if you’ll only be using personal computers.
- SQL Server Express is far cheaper to host in-house than almost any cloud provider can match. However, treat this like #1; don’t buy less than you need and be ready for growth.
- Non-Microsoft software may be cheaper than cloud solutions, especially if you’d have to rent a virtual machine or dedicated host.
Whatever servers you choose to bring in-house, your goal should be to fit them all in Hyper-V. When I had to do one-host installations, I always had to worry about compatibility problems with all of those applications, including Active Directory, sitting on one unit. You don’t have to do that. You may have to purchase more Windows Server licenses, but you do not need to purchase more hosts.
Oh, and even though I already mentioned it: oversubscribe CPU, turn on Dynamic Memory (where appropriate), and use dynamically expanding disks (where appropriate).
The reason I wrote this point is that many people are going to tell you that the purpose of Hyper-V is so you that can buy more hosts and switches and storage to “avoid single point of failure!”. Virtualization for you is a money-saving move. Backup is your redundancy.
7. Small Is OK
I would never buy a Hyper-V host with fewer than 16 cores total or less than 256 gigabytes of memory and any host that I purchase will absolutely connect to my EMC SAN. I also have a campus agreement for ridiculously cheap Windows Server licenses, a history of volume purchases that nets me extremely favorable hardware pricing, and a budget for IT alone that is larger than many towns. Many other writers are either in the same place as I am or have never worked directly with small businesses and just do not understand what you’re going through.
Let’s set some realistic upper and lower boundaries:
- I don’t see many people over-recommending CPU, so you’re probably not in any real danger. Most small business’ Windows Server instances just need to have 2 vCPU available for contention issues. Few will put any meaningful load on them. I would consider keeping the ceiling at 8 total physical cores in each physical host just so you don’t run afoul of per-core licensing coming in 2016. I’d love to try to give you a hard number, but unfortunately, that’s not possible. However, had I ever had the sort of hardware and hypervisors available to me then that I do now, I don’t think any of my 20-or-fewer user clients would have needed anything larger than six total cores in a host.
- Memory is also easy to oversize. I would say that a well-proven average for a Windows Server instance running basic services is 2 GB, including the management operating system. Of course, the more roles and applications that you pack into an instance, the more it is likely to need. The big point is, don’t overdo it and make any provider reason out what they’re recommending within a few gigabytes. Don’t try to drop in 128 GB just because everyone else is doing it. If you’re only going to run two virtual machines, 16GB of physical memory is probably more than you’ll need but gives you breathing room at an affordable price point.
- Disk is where you can save.
- Internal storage is just fine.
- RAID-10 is not a requirement, despite what many claim. RAID-5 and -6 are both fine, but spend extra for a hardware controller. If you’re not certain whether or not the RAID controller advertised for a system is hardware, the give-away is whether or not it can be battery-supported. You do want battery support.
- For local storage, I would skip Storage Spaces for at least one more version iteration. Battery protection isn’t even an option, for one thing. For another, its redundancy features are handled on your CPUs — the same ones I told you not to oversize. In my opinion, Storage Spaces is still better when it’s used as remote, shared storage. I would back that up by pointing out that Microsoft is one of those vendors that are stymied by the small business.
- Rather than buying a few very large disks, try for a few more smaller disks. The performance and resiliency is superior, especially when using RAID-5 or -6.
- 10 GbE is an absurd expenditure for a small business. Ditto RDMA, SR-IOV and other high-end networking features. If you are using internal storage, two to four 1GbE connections are perfect. Add 2 if you’re using iSCSI.
- VDI’s benefits are in features. It is not a money-saver. The typical small (and medium) business should avoid VDI.
- OEM pricing usually works out better than volume pricing for a small business. If you have a decent reseller, they will help you verify that. They should not just give you a pricing sheet and leave it to you to figure out, nor should they give you a 5-minute synopsis of whatever the current state of volume licensing is and expect you to decide on your own. Those are signs that you need to find another provider.
- Your backup solution is part of your server solution, not an afterthought. Price out the software and the supporting hardware at the same time.
- Buy an uninterruptible power supply for your host. I have seen an uncountable number of instances in which thousands of dollars of server-class equipment went to the recyclers and days of transactions were lost due to refusal to purchase a $250 device. If you’re banking on insurance coverage, they also realize the value of a UPS and that will be one of the first questions they ask when you file your claim.
8. This Version is OK
When it comes to technology, don’t try to keep up with the Joneses. I liked Microsoft a lot better when they kept their operating system releases fairly far apart and issued periodic Service Packs. Those days are gone. However, you don’t need to feel pressured to jump to each new version as soon as it comes out. The people that call you on each new release date are not looking out for your best interests. All they can see is “licensing margins” and “engagement fees”.
Upgrade when:
- The version that you have is no longer sufficient for your needs in a way that is addressed by a newer release.
- You are purchasing a replacement host.
- Your current version is nearing the end of its support lifecycle.
What’s Coming
What I’d like to do in future articles is expand #7 from a generic list into some practical build ideas to help guide the perplexed. If you’re a small business or provide service to small businesses, I’d like to hear your stories, suggestions, and questions.
Not a DOJO Member yet?
Join thousands of other IT pros and receive a weekly roundup email with the latest content & updates!
16 thoughts on "8 Hyper-V Best Practices for Small Businesses"
We’re a consultancy provider/MSP, it’s great to see some recognition of the fact that “best practices” lists and sizing guides aimed at large enterprises really don’t say anything much of use to small businesses. Fully agree with pretty much everything in this post and that’s a rarity.
2GB per Windows server instance sounds on the low side to me; is that using Dynamic memory on all the guests?
I’m reluctant to recommend RAID5 these days, partly because of the URE issue that may or may not rear it’s head (numbers suggest it should happen pretty frequently on large rebuilds whereas anecdotal evidence suggests it’s actually pretty rare) and partly the complexity. Disks are fast and cheap these days so I tend to use multiple RAID1 arrays of differing speeds where there’s obvious ways to split IO load and RAID10 where there isn’t. Is the higher storage density/value why you’d choose RAID5 instead?
Really nice to see some pragmatic, practical advice and approaches for SMEs, huge swathes of the industry just don’t seem to understand the environment these companies work in.
I do also do put quite a lot of value on the fact that RAID1 member disks have meaningful and useful data on when considered individually, something that’s not true for any other RAID type.
Hi Jim, and thanks!
Re 2GB being low: I would like to see some work done to aggregate knowledge around the small business so that we can come up with more solid numbers. I chose that number based on the fact that as an industry, we often still size things on the basis of “think of the physical deployment and make a virtual mirror of it” rather than defaulting to the virtual mindset. For instance, 2GB for your average small business’s domain controller or printer server is up to double what those roles need and is on the high side for a file server, but who would ever build a physical host in 2016 with only 2GB of memory? So, I think I’m fairly accurate with my projection. Yes, I did mean with Dynamic Memory, also keeping in mind that you can always increase the amount allotted to any virtual machine’s Dynamic Memory setting without interrupting it. If we have good reason to say that the average expected memory usage should be higher, I’m listening and will be happy to modify my post.
Re RAID-5: I should probably have referred back to some of my earlier work on this, but the URE issue is going to be more of a concern with the gigantic platter sizes we’re working with these days. URE, performance, and price are the reasons I prefer more smaller disks as opposed to fewer big disks. You can get more overall capacity at reasonable performance at a lower price point than a roughly equivalent RAID-10. But, if a customer or situation demands great big platters, then I agree that I would be hesitant to implement RAID-5. This article is a very good, and I would say required, read on the subject: http://benchmarkreviews.com/26100/raid5-ure-rebuild-means-the-sky-is-falling/
Interesting article; I reckon seems to go a bit far in the opposite direction though and is being blasé about the exact behaviour that *has* lost people data. Some large RAID5 arrays really don’t successfully rebuild. It doesn’t really matter how unlikely the rebuild failure was once it’s your array that’s not rebuilding. It’s certainly not a case of “RAID5 is dead” but I would say it’d be questionable to put anything important that belonged to someone else on RAID5 with 3TB spinning metal disks.
On the memory I my instinctive numbers on this are based around the fact that the Windows license is going to have cost at least £400 or so per instance (assuming running Hyper-V as host and using the 2 guest licenses). Most SMBs will thus likely split all their services over 2 guests on one host. This means the 2 guests between then will likely be doing File Serving, Directory, WSUS, Backups, SQL for a LOB application or 2, RRAS and maybe a couple of IIS sites, assuming they’ve pushed the email out to the cloud. To put it another way, I’m not sure I provisioning server software that cost £400 and then giving it £20’s worth of memory to play with constitutes maximizing return on the software license investment costs.
Sorry Jim, I didn’t get notification of this comment for some reason.
I didn’t quite get the same impression from the article that you did. As on-demand service providers, we clearly remember the dozens of array failures, especially the ones that involved fireworks, but we quickly forget the thousands that retired peacefully. This is why I beat the “backup” drum so hard.
Of course, we might not also have the same array size in mind. I’m thinking of those four- and six- spindle configurations. I’ve only had two arrays of those sizes absolutely lose it, and transplanting their disks into a replacement array from the same vendor made all of our problems go away. Even if I’m just lucky enough to never have had it any worse than that, it’s a good sign of what the odds are.
I see what you’re saying re: memory. I’ll clean up the post’s language a bit. What I really wanted to drive home was that most small businesses do not need a 128GB box even though they’re going to hear that number (or higher) recommended as a minimum.
We’re a consultancy provider/MSP, it’s great to see some recognition of the fact that “best practices” lists and sizing guides aimed at large enterprises really don’t say anything much of use to small businesses. Fully agree with pretty much everything in this post and that’s a rarity.
2GB per Windows server instance sounds on the low side to me; is that using Dynamic memory on all the guests?
I’m reluctant to recommend RAID5 these days, partly because of the URE issue that may or may not rear it’s head (numbers suggest it should happen pretty frequently on large rebuilds whereas anecdotal evidence suggests it’s actually pretty rare) and partly the complexity. Disks are fast and cheap these days so I tend to use multiple RAID1 arrays of differing speeds where there’s obvious ways to split IO load and RAID10 where there isn’t. Is the higher storage density/value why you’d choose RAID5 instead?
Really nice to see some pragmatic, practical advice and approaches for SMEs, huge swathes of the industry just don’t seem to understand the environment these companies work in.
I finally figured out the key to avoiding drive failures….bear with me…
A few years back I needed a new server too replace 8 old power-sucking units. I chose to virtualize the old iron (DL360 G3’s and two Quad-Proc 8U IBM units (!!) and a Dell PowerEdge 2650, and two Nortel Switches…
Total electrical bill, per month, for that old mess was about $100 – more if the IBM units were turned on (they had (4) 400 watt power supplies – each). Turning them on as well brought the electrical cost to about $150 per month.
To that end I narrowed the search down to a HP DL380 or a Dell of some sort. The current model (at the time) was the DL380 G7 and it had all sorts of power saving features. Being a sucker for high disk density and the new 2.5″ SAS drives I ordered up a nice 16 drive 2U unit…..
I think I got 3 drives replaced in the first year under warranty (Seagate 10K 6G units). Perhaps two the next year and two after that and then the warranty ran out and I started buying drives and paying for them myself.
I believe it’s on a RAID 6 Array (can’t recall) but I do know I have two simultaneous drive failure capability. Anyway, as you can guess, a drive would die and I’d end up on e-bay digging up a new OE drive for the lowest price with shipping. This happened about 4 times in 1.5 years and became a bit of an annoyance. In fact, I was starting to rue the fact that I had special ordered up the unit with the 16 drive option…..
Anyway, one day I got pissed off and banged out a deal on 6 new replacement drives. I only needed one, but I figured with 5 spares sitting right on top of the server I could just plug ‘n play without getting mad and that this would buy me a few years of peace.
Well guess what? Since that time not one single drive has failed. Nothing. Nada. Zilcho. Zippo. So I have 5 new drives sealed in their bags just sitting there and now the array refuses to eat drives.
For the past 3 or 4 years it’s digested a drive on an intermittent, but regular, basis. Not any more. The only conclusion I can draw is that somehow it ‘knows’ there are ready spares at hand and so it’s given up.
That’s my story, it’s true, and I’m stickin’ to it 🙂 If you want to solve intermittent drive failure issues the answer is simple, buy more spares then you need and as soon as your server sees them it will stop monkeying around on you.
Oh, and by the way, if you recall my electrical bill, per month, was between $100 and $150 ? After virtualizing the machines, and taking advantage of the new power saving features of the new (at the time) DL380 G7 I found that the average power draw, per day, is a whopping 195 watts (!!) That’s with 16 drives, two 6-core processors and two 750 watt power supplies. The virtualization also replaced two physical switches and brought me down to only requiring ONE UPS. Average monthly cost to run? $15 representing a 5 year savings of $5,100 in electricity alone.
Loading up the machine so that all cores are pegged at 100% and the 16 drives are 100% busy? The highest power draw I’ve been able to get out of it was 500 watts…..
I love war stories! Thanks for sharing.
When I was young, my father literally laid hands on a computer and said, “Be healed!” and it never acted up again.
Or the woman that used was just afraid to call in problems on it anymore. I’m going with the first thing.
I don’t doubt it but I have to be honest too, there’s times I’ve approached it from the dark side and just kicked/slapped and/or beat the thing into submission 😉
Nevertheless, I’m so impressed with virtualization, on so many fronts, that I’m extremely pleased. I used to order up used servers any time I needed to add one. Going overkill on the DL380 means I can add VM’s at will as it’s under-used at the moment.
But…I’m completely honest with respect to the drives. They kept on failing until I broke down and bought 5 new cold spares. I leave them on top of the machine in their sealed bags and it’s never dropped a drive since. I think it’s realized that all the fun is over now…..
Have a great weekend and thanks for the enjoyable articles. They are well done and very helpful.