Save to My DOJO
Are there any bumper stickers or t-shirts out there that say, “I was using Hyper-V before using Hyper-V was cool?” If there are, I think I need one.
I built my first production system on Hyper-V 2008 R2 not long after it debuted. So, I missed the early-adopter crowd by a couple of years but I’ve definitely been in this for a while. The only problem with sticking with a tech as it grows from infancy is that sometimes things change and you miss out. One of the things that changed on me was the way that time synchronization works between Hyper-V and its guests. There’s only been a single casualty that I know of, and that is the virtualized domain controller.
I’ve written about this subject before. The earliest is a complete article on configuring time for a Windows domain. Next was the practical article of the virtualized domain controller series. I didn’t talk about virtualized domain controllers at all in the first article, which I have since revised. In the virtualized domain controller article, I recommended that domain controllers have the Hyper-V Time Synchronization service set to partially disabled. I’ve left the original text in that article so you can go there to see what I had to say, if you desire.
To save you the trouble of jumping around and reading a lot of other articles, I’ll give you a bit of necessary backstory.
In a Windows domain configured with defaults, the domain controller that holds the Primary Domain Controller Emulator (PDC Emulator) flexible single master operations (FSMO) role is considered the authoritative time source for the entire domain. One way or another, every single other computer in the entire domain gets its time from that single source. As a result, it’s quite important for the time on that system to be correct. What most people do, and what I explained how to do in the first article that I linked above, is configure the PDC emulator to continually update its own time from a known valid external source.
A typical virtual machine is configured to get its time directly from the Hyper-V host. This eliminates all sorts of concerns, such as guests in different domains or workgroups and lots of needless time synchronization network traffic. What it’s an especially good solution for is the somewhat unique nature of the virtual machine. Using the Save State operation, it is possible to put a virtual machine into a sort of suspended animation for an effectively infinite amount of time. When a virtual machine is resumed from this state, it has absolutely no idea that anything happened at all. As a result, its clock will be set to the exact time that the save operation was conducted. If that was fairly recent, then there shouldn’t be a problem, especially for a modern operating system. It will eventually seek out an authoritative time source. A problem arises when the clock is very different.
Due to security concerns, there is a maximum time differential of only five minutes between a client and server in a Kerberos conversation or the two systems are considered skewed. The upshot of skewed systems varies wildly. A Windows client desktop might be a few days behind and the only consequence is a lot of logged warnings and errors. An Exchange Server system will panic at a very small skew. If a clock is really skewed, then it will have lots of strange issues, especially when dealing with SSL certificates. However, there is a maximum amount of skew beyond which a domain controller will not update a client that asks for a time synchronization. This problem is directly solved for Hyper-V guests via the Hyper-V Time Synchronization Service.
The explanations that you can find about this service really are not entirely accurate, and some things appear to have changed over time. I did a fair bit of testing, and, as of 2012 R2, this is how Hyper-V and the Hyper-V Time Synchronization Service interact with a guest virtual machine:
- Hyper-V is responsible for time operations when the guest is off, the Hyper-V Time Synchronization Service is responsible for time operations while it is on.
- Hyper-V “maintains” the guest’s clock while it is a turned off state. I don’t know the exact mechanics, but if a virtual machine does not have any time synchronization of its own set up (including the Hyper-V Time Synchronization Service), its clock will not lose time while it is off. A previous article that I read said that Hyper-V would inject the management operating system’s time into the virtualized real-time clock. That does not appear to be the case (at least, in this version). If you disable all time services in the guest, you can move the machine into a different time zone and set its clock horribly wrong and it will never be corrected. However, it will not lose time when it is off. This means that Hyper-V is adding any missed time to the real time clock but is not otherwise interfering with it.
- The Hyper-V Time Synchronization Service requires the Windows Time Service (or the Linux equivalent) to do its job. I didn’t do any experimentation with Linux, but it’s safe to assume that operations there will be analogous. In a Windows guest, if the Windows Time Service is stopped, the Hyper-V Time Synchronization Service never does anything except at initial Windows boot-up. So, if you disable the Windows Time service and leave the Hyper-V Time Synchronization Service enabled and then change the clock, it will never be corrected until the guest is restarted.
- Only the Hyper-V Time Synchronization Service is guaranteed to correct the clock after a resume from saved state. If the Hyper-V Time Synchronization service is not running, then the guest’s clock will continue tracking time as though it was never saved. If its Windows Time service is started and has a valid source, it will attempt to update at its next scheduled interval (again, it won’t know that any special amount of time has passed since the last update). Depending on its source’s maximum skew setting, that process might fail. Unlike normal operations, this does not depend on the Windows Time service also being started.
- The “partial disable” technique for the Hyper-V Time Synchronization Service does not work. If both the Hyper-V Time Synchronization Service and the Windows Time service are enabled, then the guest will get its time from the management operating system. The registry hack will cause the Windows Time service to report that its source is something other than the Hyper-V service, but the guest’s time will be synchronized to the management operating system’s clock anyway.
[optin-monster-shortcode id=”u4bw5fa5efzfughm”]
Hyper-V Time Synchronization Service in Practice
In almost all cases, you will want the synchronization service to be enabled for every guest. The reason is simply because there’s not much of a reason not to. The biggest risk is that the host’s time will drift, which will cause all the guests to drift. That’s why it’s very important to ensure that the host is configured to synchronize against an authoritative source regularly. The instructions were linked above, but here’s the link again.
Time Synchronization for Virtualized Domain Controllers
The subject of time synchronization for virtualized domain controllers has always been a controversial issue. Some people were never able to get it to work correctly at all. Others had mixed success. Now, with the partial disable registry entry not working, virtual domain controllers are the one true exception to the “every guest” rule. The Hyper-V Time Synchronization Service must be disabled for the PDC emulator. If not, and its host ever loses time, then the entire domain will eventually shift and won’t recover on its own.
Worse, during my testing, a shift of more than a few minutes caused failures in things such as remote desktop connections until the entire domain had picked up on the new time. While not quite as critical, time synchronization should also be disabled for any other virtualized domain controllers. The final configuration should follow these rules:
- The PDC emulator, whether virtual or physical, should be synchronizing its time from a known valid external source, such as authoritative Internet-based systems or a dedicated hardware clock.
- The PDC emulator, if virtualized, must not be synchronized to the hypervisor.
- While not as critical, other virtualized domain controllers besides the PDC emulator should also not be synchronized to the hypervisor. If left at defaults, they will synchronize directly from the PDC emulator.
- All hypervisors’ management operating systems should synchronize from the domain.
- All other guests should synchronize from the hypervisor.
- As indicated in the linked article, all physical machines should synchronize from the domain.
Time Sync in Hyper-V
The problem with disabling time synchronization for domain controllers is when they go through a Save State operation. Since the time synchronization is disabled, their clocks aren’t synchronized unless they’re rebooted. They will, of course, attempt to update at their next interval, but there are two issues with that. First, if the skew is too great, they will keep their incorrect time. Second, they will hand out the incorrect time to any client that attempts to synchronize with them. That’s why I always kept time synchronization partially disabled up until I learned that it doesn’t work anymore.
It’s easy to say that you’ll never save your domain controllers; I believe you. But, Hyper-V might decide to save them when you reboot their host(s). If the host is clustered and the virtual domain controller isn’t, you can almost guarantee that’s exactly what will happen. This is not meant to scare you or talk you out of virtualizing domain controllers. It’s just something you need to be aware of. If this is all set up correctly and you notice your clocks losing time, you just do what you would have done in the physical world: check the PDC emulator.
Disclaimer
I performed a substantial amount of testing on all of these claims prior to and during the crafting of this article. However, the Windows Time service has a track record of occasionally displaying erratic behavior. It is possible that some of my findings are not entirely accurate. It is also possible that my findings are 100% accurate but that not everyone will be able to duplicate them with 100% precision. If working with any time sensitive servers or applications, always take the time to verify that everything is working as expected.
If you encounter results that are different to mine, let me know by leaving a comment below!
Not a DOJO Member yet?
Join thousands of other IT pros and receive a weekly roundup email with the latest content & updates!
More in this category
About the Author
Eric Siron
23275 thoughts on "How to Get Hyper-V Time Synchronization Right"
In Short.
Time relies on “CPU clicks” in virtual world. This may work, but must not.
Thats why there is still the recommondation of VMWare (KB: 1318) and in deeper looks even at MS Knowledgebase to put the DC NOT into ANY VM but keep it on a small Hardware.
The only workaround i am aware is, to install an ntp client, that synchronises every 5 Minutes. Not the best idea to save bandwith 🙁
best regards
Christian
BTW: i started to run productive servers without R2 on the good old Vista Kernel 🙂
btw. My article was writen at 6:28 GMT 1. Altaro Server is showing 6:34. Maybe a time issue 🙂
Time in a physical Windows deployment is also dependent upon CPU cycles. As soon as Windows starts, it ignores the RTC and uses a low-priority system thread to maintain time. Virtualizing it doesn’t change that.
I don’t see anything in the VMware KB 1318 that talks about keeping a physical DC, but given the difference in speed and multi-threading capabilities between CPUs today and CPUs when that article was written, it would be an irrelevant recommendation anyway. If a virtualized domain controller is losing time on relatively current hardware in 2015, the only possible reasons are that the vDC is not pulling from an appropriate source or that its host’s CPUs are too heavily overcommitted. The solutions are to fix the source and make sure that a virtualized DC is not sitting on a heavily overcommitted host. I would never tell anyone to throw away a physical DC if they have one, but in this day and age, they are an endangered species that will soon become extinct.
Feel free to disagree, but keep in mind that I have been running virtualized DCs of my own and helping others with theirs continuously since 2010 (Hyper-V — back to 2008 if you count ESX) and have still not experienced any time drift issue that wasn’t minor and that could not be readily resolved. I’ll take experience over a KB article any day.
In Short.
Time relies on “CPU clicks” in virtual world. This may work, but must not.
Thats why there is still the recommondation of VMWare (KB: 1318) and in deeper looks even at MS Knowledgebase to put the DC NOT into ANY VM but keep it on a small Hardware.
The only workaround i am aware is, to install an ntp client, that synchronises every 5 Minutes. Not the best idea to save bandwith 🙁
best regards
Christian
BTW: i started to run productive servers without R2 on the good old Vista Kernel 🙂
Time in a physical Windows deployment is also dependent upon CPU cycles. As soon as Windows starts, it ignores the RTC and uses a low-priority system thread to maintain time. Virtualizing it doesn’t change that.
I don’t see anything in the VMware KB 1318 that talks about keeping a physical DC, but given the difference in speed and multi-threading capabilities between CPUs today and CPUs when that article was written, it would be an irrelevant recommendation anyway. If a virtualized domain controller is losing time on relatively current hardware in 2015, the only possible reasons are that the vDC is not pulling from an appropriate source or that its host’s CPUs are too heavily overcommitted. The solutions are to fix the source and make sure that a virtualized DC is not sitting on a heavily overcommitted host. I would never tell anyone to throw away a physical DC if they have one, but in this day and age, they are an endangered species that will soon become extinct.
Feel free to disagree, but keep in mind that I have been running virtualized DCs of my own and helping others with theirs continuously since 2010 (Hyper-V — back to 2008 if you count ESX) and have still not experienced any time drift issue that wasn’t minor and that could not be readily resolved. I’ll take experience over a KB article any day.
In Hyper-V manager under the settings of a VM, under Management > Integration Services you can uncheck “Time synchronization”; is that enough on DC’s or do we still need to go in and disable the Hyper-V Time Synchronization Service inside the VM? Instructions I’ve previously followed only mentioned to make sure that Integration Services option was not enabled for DC’s.
It won’t hurt anything to disable it from within the guest, but unchecking it in Hyper-V Manager should do the trick.
Good article!
Only thing I would make a note is if you have FreeBSD/Linux using Samba to connect to the domain, they have the same rule about being within five minutes!
I tend to deploy one of these NTP Clocks inside the domain and have both the FMSO machine and any unix machines get their NTP from it, that way there’s no skew. Linux keeps more accurate time than Windows does.
http://www.bswusa.com/Site-Control-Broadcast-Tools-NTP-Server-Sentinel-P8700.aspx
There’s some concerns about using https://www.ntppool.org/en/ as those machines are not trusted. If you host your own NTP server, you know that you’re deploying accurate (and secure) time for your domain.
If a hardware clock isn’t an option, syncing to the domain is the next best option.
If you require truly real-time clock operations, then yes, the on-premises hardware clock is the best option. Not many people really need that level of precision. While not mentioned in this article, I typically convert the domain clock from NT5DS to NTP mode and synchronize every device to it. I find that the most important thing is to have all local clocks synchronized, even if they are a few milliseconds off from whatever is considered official.
Good article!
Only thing I would make a note is if you have FreeBSD/Linux using Samba to connect to the domain, they have the same rule about being within five minutes!
I tend to deploy one of these NTP Clocks inside the domain and have both the FMSO machine and any unix machines get their NTP from it, that way there’s no skew. Linux keeps more accurate time than Windows does.
http://www.bswusa.com/Site-Control-Broadcast-Tools-NTP-Server-Sentinel-P8700.aspx
If a hardware clock isn’t an option, syncing to the domain is the next best option.
Eric,
Great post, and quite interesting to learn that the partial disable technique for the Hyper-V Time Synchronization service no longer works. But can you clarify something for me? Why would I not want to instead sync the Hypervisor to the external NTP source (it being a hardware server) and then sync the virtual PDC Emulator to the Hypervisor via the Hyper-V Time Synchronization service? That seems more logical to me, and eliminates the dependency of the Hypervisor to one of its guests. Note that I am running a standalone Hyper-V server hosting virtual-only PDC and servers for a small enterprise configuration. I do not intend to join the Hypervisor server itself to the domain provided by the virtual PDC.
Why in the world would you ever not join the Hyper-V system to the domain? What do you feel guilty for that you are choosing to inflict that of level of pain upon yourself?
Odds are high that it would be OK for the hypervisor to directly contact an external source. The reason that I don’t do it is because time sync issues have a tendency to go unnoticed until they start causing problems. By having all internal systems refer to a single authoritative internal source, you ensure that everyone sinks or swims in lockstep and you know exactly where to go in the event of any synchronization issues. When multiple systems refer directly to an external source, you introduce the potential for internal time differentials. Overall, it’s better for everyone to have the same incorrect time than for internal skew to exist.
The most common reason not to have the Hyper-V Host joined to the domain in an SME (as the questioner detailed was the case here) is that there is only a single Hyper-V Host and it contains the only DC (or DCs). With the Hyper-V Host joined to the domain, when the host is booted, the DC will obviously not be available until it (a guest on this host) has been started, therefore during boot, the Hyper-V Host will be left flapping in the breeze with regards to both Domain authentication as well as time synchronization.
Those who say it cannot be done should not interrupt those who are doing it.
This myth has been repeatedly debunked to the point that I am struggling to believe that anyone still believes it in 2016.
My Hyper-V Host Servers (2) are joined to our domain on the DC VM under our VM Cluster manager and we have had no issue with the Host authenticating after restart. The Hyper-V runs as a service, therefore the machines are re-started after reboot. And as you know, Domain Credentials are usually cached. This is why your Domain connected Laptops work outside the office prior to a VPN connection.
Exactly! Laptops at the coffee shop are my go-to example for debunking most myths around domain membership.
Eric,
Great post, and quite interesting to learn that the partial disable technique for the Hyper-V Time Synchronization service no longer works. But can you clarify something for me? Why would I not want to instead sync the Hypervisor to the external NTP source (it being a hardware server) and then sync the virtual PDC Emulator to the Hypervisor via the Hyper-V Time Synchronization service? That seems more logical to me, and eliminates the dependency of the Hypervisor to one of its guests. Note that I am running a standalone Hyper-V server hosting virtual-only PDC and servers for a small enterprise configuration. I do not intend to join the Hypervisor server itself to the domain provided by the virtual PDC.
When running a single hardware host, I would never dream of joining the host to the guest PDC domain. When running multiple hardware hosts at a site, I do join the host to the guest PDC domain, as each physical host as a DC as a guest and having the hosts in the domain makes their (the host) management MUCH easier.
In this scenario, the HOST is part of the domain, and thus should get its time from the guest PDC. But the guest PDC gets its time from the HOST unless you fully disable the Hyper-V Time Syncronization Service. Sounds like a problem waiting to happen. I can’t believe that partial disabling does not work, sounds like a bug to me.
Thoughts?
“When running a single hardware host, I would never dream of joining the host to the guest PDC domain.” Why? There is zero chance I would ever not join a Hyper-V host to a domain if one is available.
Yes, the host would get its time from the PDC guest if they’re in the same domain. Why would that be a problem? The “host” is just another Windows member server. The fact that it’s the management operating system of a hypervisor is a secondary concern, at most.
I don’t know why partial disabling doesn’t work anymore, but there is no technological reason why the Hyper-V Time Synchronization service would be beholden to the registry key. The Windows’ clock APIs are not dependent upon the Windows Time service so they could easily bypass it. It could be a bug or an oversight or it could be intentional.
When running a single hardware host, I would never dream of joining the host to the guest PDC domain. When running multiple hardware hosts at a site, I do join the host to the guest PDC domain, as each physical host as a DC as a guest and having the hosts in the domain makes their (the host) management MUCH easier.
In this scenario, the HOST is part of the domain, and thus should get its time from the guest PDC. But the guest PDC gets its time from the HOST unless you fully disable the Hyper-V Time Syncronization Service. Sounds like a problem waiting to happen. I can’t believe that partial disabling does not work, sounds like a bug to me.
Thoughts?
I’ve tried just about every combination/permutation of keeping perfect clock sync in hyper-v possible with the best success rate (so far) being a clock issue every 3 or 4 months requiring a stop and re-start of the service combined with the odd w32tm /resync
After re-visiting this issue, periodically, on and off for several years I finally sat down determined to draw up a solution that works for us. I just implemented a new one today, it’s the only one we haven’t tried and it looks like this;
One server which is not a member of the domain (I have nothing against that though other then event log entries on startup – which I hate).
5 (or more) guests, all in their own domain, 3 of which are DC’s
Time integration services enabled on all of them
I reset all guests using w32tm /unregister and then re-registered the service to ensure they are all using default native OOB settings. Re-boots on all to be sure during the appropriate window.
I then went to the host and set it up with no less then 4 of the North American NTP pools and then dialed in the service on the host so that it’s checking every 15 minutes. This will keep the clock accurate on the host compared to accurate external time.
The VM’s are all stock now with respect to time services. So they naturally go to any of the 3 DC’s for their time which in turn get their time from the host.
The theory behind this is that, when confronted with Microsoft’s obvious contradictions about the scenario, it’s better to go back to their engineers expected OOB design parameters.
Ie, that the average person installing Hyper-V and VM’s will accept the defaults and not monkey with NTP at all.
In this way, it’s all dependent on the host. However, even if the host, which uses a RTC, skewes the VM’s will all get the same skewed time. In short, they would be in sync.
I like to think that the host, when using the RTC, will be quite accurate and that the time service itself is just a precision fine-tuning device that adjusts the RTC. However, I could find no docs on specifically how the host uses the RTC other then one person’s view that it is only polled once during Host startup and then ignored. I have no idea if this is true. If so, I think that would be a shame as a modern RTC can be pretty darn accurate.
On the other hand, given that the host is polling 4 external time servers at a rate of every 15 minutes, I like to think that the host itself can keep accurate time even if it ‘misses’ once in a while.
Is this the right solution? I don’t know (yet). Only time will tell (groan)
I do know I’ve tried everything else and have not been able to achieve true set it and forget it. It seems that every 3-6 months I have an issue of clock drift when setting the PDC emulator on the VM to get external time. Usually due to some weird thing where it could not contact the NTP server (ISP issue?) and then somehow got stuck and did not self-correct at a later point. I did see an article that claimed that if the time in Windows got more then 5 seconds ahead the time service would fall over and die. That one came from Microsoft themselves. But, there’s so much stale info out there that this may have been from the past.
Anyway, this host-based time service brings me back to a single point of failure which should be only the hardware host and it’s connectivity to the Internet. Even if it failed for some reason it should still mean that all VM’s (and in our case internal machines) are still synced with identical time until the host itself regains precision from an external source.
The way I track the success of the time service is pretty interesting. I set my PVR to use the DC as the time source. In this way, I find out really fast when the clock is off because my recordings are off.
But, I have to say, all the conflicting docs from MS and the fact that no-one else has posted THE defacto answer tells me that I needed to just come up with it myself. I’d like to see a permanent solution to this and am hopeful about this one but I’ve been burned before.
I’m guessing the only way to truly nail down the Windows Time Service network-wide on anything approaching 100% uptime and accuracy would be to buy a cesium decay rate clock and install it on the network in such a manner that all machines have permanent access to it no matter what anyone does to the network.
Tempus Fugit – at least when it comes to VM clocks 🙂
…speaking of which, I posted the above at 7:56 PM GMT-8 on February 29 yet your site shows it was posted on March 1st (!!!)
Some quick tracing seems to indicate the server is located in Dallas, Texas which would put you in the America/Chicago time zone or, approximately, 10PM on February 29, 2016 (CST)
Hmmmm….methinks your clock might be off….by a lot! 🙂
I’m not privy to all the technical details, but as Altaro Software is based out of Malta, I would assume that the site is programmed around their local time.
Keep in mind that Windows is not a real-time operating system, so time drift is inevitable. Find a happy place where the drift isn’t killing you and be happy!
As far as I know, the reason that Windows doesn’t rely on the RTC isn’t about reliability; it’s about authority. Having everyone agreeing on the wrong time is better than everyone being on their own time. Seems that you’re headed in that direction, so hopefully you find a good place to be.
I did a bit of pricing on hardware-based clocks that can be installed in the local network, and the pricing isn’t too bad. If I were really concerned about accurate time, I think the investment would be worthwhile.
I’m cautiously optimistic on the time front with this solution. I was troubleshooting an issue on an Exchange 2010 VM that required setting the clock back to Jan 1, 2012 while installing a piece of software.
So I went into, changed it, closed the time panel, and…it changed right back to the right time. I thought maybe I hit an update window just right so I changed it again. It instantly changed back. Year, date/month – the whole thing. And is was instant.
For me that was a confirmation with respect to the Hyper-Visor time integration service. It appears to change the clock as soon as it goes out – literally within a second or two. I was quite impressed. Seeing how fast it works means that all the VM’s will be ‘insta-change’ so now I can just focus on the physical server and it’s connection to the external NTP server.
I’d suggest to anyone to try changing the clock on a VM to the wrong time – just for fun and just to see how fast it comes back t the right time when using hyper-v time integration. It’s like it’s made of rubber.
After seeing that I now know, 100% for certain, that the time amongst the Hyper-V vm’s cannot possibly be out more then perhaps for one second – but even that I think is probably not possible.
I’ve tried just about every combination/permutation of keeping perfect clock sync in hyper-v possible with the best success rate (so far) being a clock issue every 3 or 4 months requiring a stop and re-start of the service combined with the odd w32tm /resync
After re-visiting this issue, periodically, on and off for several years I finally sat down determined to draw up a solution that works for us. I just implemented a new one today, it’s the only one we haven’t tried and it looks like this;
One server which is not a member of the domain (I have nothing against that though other then event log entries on startup – which I hate).
5 (or more) guests, all in their own domain, 3 of which are DC’s
Time integration services enabled on all of them
I reset all guests using w32tm /unregister and then re-registered the service to ensure they are all using default native OOB settings. Re-boots on all to be sure during the appropriate window.
I then went to the host and set it up with no less then 4 of the North American NTP pools and then dialed in the service on the host so that it’s checking every 15 minutes. This will keep the clock accurate on the host compared to accurate external time.
The VM’s are all stock now with respect to time services. So they naturally go to any of the 3 DC’s for their time which in turn get their time from the host.
The theory behind this is that, when confronted with Microsoft’s obvious contradictions about the scenario, it’s better to go back to their engineers expected OOB design parameters.
Ie, that the average person installing Hyper-V and VM’s will accept the defaults and not monkey with NTP at all.
In this way, it’s all dependent on the host. However, even if the host, which uses a RTC, skewes the VM’s will all get the same skewed time. In short, they would be in sync.
I like to think that the host, when using the RTC, will be quite accurate and that the time service itself is just a precision fine-tuning device that adjusts the RTC. However, I could find no docs on specifically how the host uses the RTC other then one person’s view that it is only polled once during Host startup and then ignored. I have no idea if this is true. If so, I think that would be a shame as a modern RTC can be pretty darn accurate.
On the other hand, given that the host is polling 4 external time servers at a rate of every 15 minutes, I like to think that the host itself can keep accurate time even if it ‘misses’ once in a while.
Is this the right solution? I don’t know (yet). Only time will tell (groan)
I do know I’ve tried everything else and have not been able to achieve true set it and forget it. It seems that every 3-6 months I have an issue of clock drift when setting the PDC emulator on the VM to get external time. Usually due to some weird thing where it could not contact the NTP server (ISP issue?) and then somehow got stuck and did not self-correct at a later point. I did see an article that claimed that if the time in Windows got more then 5 seconds ahead the time service would fall over and die. That one came from Microsoft themselves. But, there’s so much stale info out there that this may have been from the past.
Anyway, this host-based time service brings me back to a single point of failure which should be only the hardware host and it’s connectivity to the Internet. Even if it failed for some reason it should still mean that all VM’s (and in our case internal machines) are still synced with identical time until the host itself regains precision from an external source.
The way I track the success of the time service is pretty interesting. I set my PVR to use the DC as the time source. In this way, I find out really fast when the clock is off because my recordings are off.
But, I have to say, all the conflicting docs from MS and the fact that no-one else has posted THE defacto answer tells me that I needed to just come up with it myself. I’d like to see a permanent solution to this and am hopeful about this one but I’ve been burned before.
I’m guessing the only way to truly nail down the Windows Time Service network-wide on anything approaching 100% uptime and accuracy would be to buy a cesium decay rate clock and install it on the network in such a manner that all machines have permanent access to it no matter what anyone does to the network.
Tempus Fugit – at least when it comes to VM clocks 🙂
…speaking of which, I posted the above at 7:56 PM GMT-8 on February 29 yet your site shows it was posted on March 1st (!!!)
Some quick tracing seems to indicate the server is located in Dallas, Texas which would put you in the America/Chicago time zone or, approximately, 10PM on February 29, 2016 (CST)
Hmmmm….methinks your clock might be off….by a lot! 🙂
Hi Eric,
I wonder if I can get your opinion on this: We are migrating our production environment: Environment A has all virtual DCs (PDC syncs to external time source, other DCs sync to PDC) and the Hyper-V hosts domain joined to the Virtual DCs. Environment B has a separate Domain for the Hyper-V/Fabric, with Environment A eventually going to exist inside Environment B as it’s own virtual entity.
The problem we are encountering is that a number of guest servers migrated to Environment B are getting their time reset multiple times (3-4 times) around midnight when the Hyper-V Host and the PDC are conflicting. This is causing services to restart and our SCOM instance to report the servers as Down. We have tried running the registry key to turn off Hyper-V integration, as my understanding is that you need the Time Sync for restores/recoveries etc. However this is not working and we are evaluating whether we turn off the Hyper-V Time Sync for all guest servers in Environment B. Can you give your thoughts and any implications we need to be aware of in terms of restore/recovery etc.
As I said in the article, that registry key no longer has any effect. The only way to disable time sync is by actively turning it off, preferably from the virtual machine’s property sheet. You must fully disable time sync for all of your domain controllers. Yes, they’ll have incorrect time if they are resumed from saved state or restored from backup. There is no value in saving the state of a domain controller so don’t do it. If you have to restore, just make sure to manually refresh its clock as soon as possible.
After doing that and giving it all some time to stabilize, your very next step is to ensure that the Hyper-V hosts are staying in sync with the PDC.
Not until those things are done should you even consider turning off time sync for the other virtual machines.
Hi Eric,
I wonder if I can get your opinion on this: We are migrating our production environment: Environment A has all virtual DCs (PDC syncs to external time source, other DCs sync to PDC) and the Hyper-V hosts domain joined to the Virtual DCs. Environment B has a separate Domain for the Hyper-V/Fabric, with Environment A eventually going to exist inside Environment B as it’s own virtual entity.
The problem we are encountering is that a number of guest servers migrated to Environment B are getting their time reset multiple times (3-4 times) around midnight when the Hyper-V Host and the PDC are conflicting. This is causing services to restart and our SCOM instance to report the servers as Down. We have tried running the registry key to turn off Hyper-V integration, as my understanding is that you need the Time Sync for restores/recoveries etc. However this is not working and we are evaluating whether we turn off the Hyper-V Time Sync for all guest servers in Environment B. Can you give your thoughts and any implications we need to be aware of in terms of restore/recovery etc.
Much thanks for this, the first article to explain how it works and let me know why the partial disable wasn’t working on the DC i was trying to get to stop using the Host OS. Bookmarked this for future reference!
Why would you not simply sync the Hyper-V management OS with a reliable, external time source and have the PDC use the host as its external time source? Regardless of the PDC’s source–host or external–it should always have the correct time. The host should be able to sync with an external source even if it’s a member of a domain, correct?
I’d want to see it in action to be certain. Time synchronization is a finicky thing in any case and I’d be worried about the host synchronizing from the domain hierarchy even after being told not to. I think that if you could reliably force the host to use an external source, then really the only thing standing in your way would be tradition. I’d have to think on it some more to be sure there’s not some “oh yeah” show stopper, but I would not criticize anyone that made this configuration work.
So what if I have a domain with only Hyper-V hosts and Physical Domain controllers. If I let the PDC get its time from a good time source, let the Hyper-V hosts get their time from that domain. Can I then let virtual PDC’s hosted on those Hyper-V host sync from their host?
In fair weather, that will probably be fine. But the PDC emulator role is not forcefully locked to any particular DC. If you demote the current PDC emulator and don’t pay strict attention to where the role goes, you could inadvertently wind up with a circular dependency. Even if you remember, you may get promoted out and your successor may not pick up on the configuration. I would personally prevent all DCs from synchronizing from Hyper-V.
Thanks for sharing!
Thanks For sharing
I was recently horrified to discover that my VMs’ clocks were being influenced by the Hyper-V host even though they are domain joined, I disabled the VMIC time provider, and w32tm /query /source did not show VMIC. I’m trying to get to root cause. I believe you are calling this “partial disable”. Can you provide a source for your comment “The registry hack will cause the Windows Time service to report that its source is something other than the Hyper-V service, but the guest’s time will be synchronized to the management operating system’s clock anyway.”?
Thank you
“Horrified” seems a bit dramatic. Surely it’s not that bad?
I could have provided an external source at the time that I wrote the article, but I don’t recall anymore. I saw it on my own systems, though. If your host is far enough off from the domain, then you can see the results for yourself.
I know that:
a) this article is somewhat old, and
b) this comment will be at the bottom of a long list of other comments.
That said, hopefully this helps someone.
In the domains we manage, multiple devices were having time drift, including domain controllers. As I had just taken over this job, I checked to see how time synchronization was set up.
1. All Hyper-V hosts were running Windows Server 2012 or 2012 R2.
2. Guests were a mix of Server 2012, 2012 R2, and 2019
3. All Hyper-V hosts were domain joined
4. All Hyper-V guests were syncing from the hosts
5. All Hyper-V hosts were syncing with Windows to default server. (default settings)
6. Servers were losing/gaining time fairly often, with “bad” results.
We made the following changes 2.5 years ago:
1. Disabled Time synchronization in Hyper-V
2. On PDC emulator machines (7 unrelated domains) set time sync to reliable server. (w32tm.exe /config /manualpeerlist:”0.ca.pool.ntp.org 1.ca.pool.ntp.org 2.ca.pool.ntp.org 3.ca.pool.ntp.org” /syncfromflags:manual /reliable:YES /update)
3. On all other VM guest servers set to sync from PDC emulator (w32tm /config /syncfromflags:DOMHIER /update)
Since that time we have had zero sync issues.
Does Microsoft publish authoritative documentation on time settings for domain controllers hosted by Hyper-V?
Thanks Eric, and Ken, for your great write ups. I remember years ago getting burned when a DC restarted after a long power outage (several days). Because the Hyper-V host had a much different time (from its hardware BIOS clock) than the virtual DC/PDC (resuming from a saved state), the virtual DC failed. Sorry, I did not document the settings or incident, its just a bad memory!
I do not question your advice, but rather wonder: is Microsoft silent on this configuration?