SBS 2011 Backups Failing – VSS Error 0x800423F3 – Event ID 8230 (spfarm, spsearch)

A customer using Altaro Hyper-V Backup on Windows 2008 R2 Enterprise SP 1 contacted us because attempts to back up a virtual machine that had Small Business Server 2011 (SBS) installed were failing due to a VSS error.

We began troubleshooting by collecting the Altaro Error Logs, Windows Application and System Event Logs, and a dump of the VSS Writers and Providers from the Hyper-V host.

From the Altaro Error logs, we saw that the VSS request was failing with the error VSS_E_WRITERERROR_RETRYABLE (0x800423F3L). Normally, the cause of VSS errors can be determined by checking the Windows event logs, but in this case, those contained no clues. That typically means that the underlying problem is within the virtual machine itself.

We accessed the Application and System Event Logs from the SBS 2011 virtual machine itself. The Application event log held many VSS warnings, verifying that the problem stemmed from a condition within the SBS 2011 VM.

All VSS warnings had the same ID – Event ID 8230.

The pivotal item in the event description is the reference to the VssAccessControl registry key. We retrieved a copy of the customer’s “HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesVSSVssAccessControl” registry key and immediately saw that, apart from the “NT AuthorityNetworkService” account, there were 2 other accounts present.

At this point we fired up our test SBS 2011 test server for comparison and found the same accounts in the same location.

Regedit showing the 2 other accounts on our SBS 2011 test server.

As you can see, one of the accounts (spsearch) is the same one referenced in the event 8230. With this information, we started an Internet search for more information about this issue. Everything we found seemed to point to Microsoft Sharepoint, primarily because both the “spsearch” and “spfarm” accounts are Sharepoint accounts. We also found posts by people that were experiencing similar issues with non-Altaro backup software. They were able to get backup working again by deleting the references to the “spsearch” and “spfarm” accounts.

This presented a conundrum. Backups of our own test SBS 2011 VM were not failing which meant that the presence of these accounts alone could not account for the failure; we had to find the difference that would allow us to reproduce the issue that the customer was experiencing. Because the customer’s SBS installation had been upgraded to service pack 1, we updated ours. This made no difference. We queried the customer to see if he knew of any other changes to the system. He told us that in August, he had installed Service Pack 1 for Sharepoint 2010 Foundation. We installed that on our test VM as well and – bingo! We experienced the same issue: backups failed with errors similar to those that the customer experienced.

We tested our finding by modifying the registry as shown (for reproduction purposes only):

1. Open “regedit.exe”
2. Browse to the [HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesVSSVssAccessControl] key
3. Make a backup of that key (these steps are presented so you can reproduce them, not make them permanent)
4. Change the DWORD value of ‘ALTSBSspfarm’ from 1 to 0
5. Change the DWORD value of ‘ALTSBSspsearch from 1 to 0
6. Restart the ‘Microsoft Software Shadow Copy Provider’ service
7. Restart the ‘Volume Shadow Copy’ service
8. Retry Backup

Backups succeeded! To confirm, we reverted the entries and restarted the Volume Shadow Copy Service and tried a backup, which failed.  This let us know that we were on the right track, but Microsoft placed those accounts there for a reason. We continued our research and stumbled upon a blog post by Susan Bradley [SBS Diva]. This article explained that if Sharepoint Service Pack 1 is installed on SBS 2011, an extra step is needed.  The user must run the “psconfig” command to fix Sharepoint and backups. We followed Susan’s instructions as shown in the Solutions section, and the backups worked! This allows us to correct the problem without changing Microsoft’s settings for the SharePoint accounts.

We directed the customer to Ms. Bradley’s blog. His response was: “I ran the PSCONFIG command as you requested and it fixed it!  Thanks for your excellent support!”

Solution

1. Open an Administrative command prompt
2. Change directory to C:Program FilesCommon FilesMicrosoft SharedWeb Server Extensions14BIN
3. Run “PSConfig.exe -cmd upgrade -inplace b2b -force -cmd applicationcontent -install -cmd installfeatures”

Thank you Susan Bradley!