I recently had the joy of upgrading an old server from Windows Server 2012 R2 to 2016. This server was supposed to be decommissioned over a year ago and recent efforts to try and remove this server resulted in more and more pushback from users… So this weekends project was to get this ageing, out of support, out of warranty, spinning rust server into the “modern” age of an OS from 10 years ago.

This server is a local file server that syncs up to Azure using Azure File Sync. The plan was to phase this server out and switch to a fully Azure based setup for this site. The Users complained enough to make us cancel those plans, even though the tests we ran with them were accepted.

Now UAC me, Now you don’t…#

This server has been through a lot in it’s lifetime, upgrades, plenty of windows updates and a few hardware changes. But most of all, it’s had a few disk failures and therefore, RAID rebuilds. So I thought my first check would be to make sure the OS files were all intact and happy…

Of course, this starts with opening up powershell or CMD to run SFC or DISM. Open the start menu, right click powershell, run as administrator… Nothing. No UAC prompt, nothing showing in task manager, no sings of life at all. I tried CMD, same thing…

Both powershell and CMD would open as my user, but not administratively. Even trying to elevate sessions with runas did the same thing, the session freezes and never opens the elevated process.

This is a new one for me, I’ve seen corrupted OS’s before after accidentally pulling the storage out from underneath my early homelab VMs a few times… But the OS works perfectly on this machine, It’s been perfectly stable, survived reboots and updates and has even had recent software installs on it without an issue. What’s going on here?

Okay, maybe there’s something in the event logs to say why it isn’t opening anything as administrator. I open event viewer and see some random events, nothing relavent, in fact… Nothing at all in the last hour or so.

Then, out of nowhere, All my powershell sessions opened. As if openeing event viewer woke something up in the background. The OS suddenly started responding correctly and I could launch applications as administrator again.

Strike 1.

To me, this felt like a storage issue, I’ve seen weird things like this happening when the storage is painfully slow or has underlying issues. Before running the upgrade, i wanted ot make sure the current OS would survive being rug-pulled and upgraded so there’s a stable restore point to roll back to if the upgrade fails.

I open up a fresh powershell session, type in the “This will fix everyting” command.. and nothing.

PS C:\WINDOWS\system32> sfc /scannow

No output, no progress, nothing. Great… Maybe DISM will work?

PS C:\WINDOWS\system32> dism /online /cleanup-image /scanhealth

Same thing… No output, no progress.

Speak to your Administrator#

I decided to bin off the domain account and logon as local admin. This was a good idea as SFC and DISM both worked as the local admin account. +1 to me here!

They both went through and showed no issues. Great, I don’t really believe it, but I’ll go with it.

I downloaded as many drivers as i could find from HPE’s support site and dumped them into the C: drive just in case and grabbed a copy of Server 2016 Standard from a local file server and went to mount the ISO and… You guessed it… Nothing.

Back to the weird storagey issue again. I did some clicking around and checked a few more things and ended up opening device manager. To my surprise there was a Microsoft virtual CD device showing up with an exclamation point next to it, as if there was a missing driver. Before i could check on it, it disappeared and the ISO popped up in explorer and had mounted itself.

Strike 2.

At this point, I decided the best option was to continue with the upgrade. If the 2012 OS is dodgy, replacing it with a “fresh” 2016 version would help things, right?

I did my pre-requisite checks, disabled any excess services that are no longer in use and made sure the Azure File Sync was uninstalled and the file share permissions were removed.

If you’re also doing this sort of in-place upgrade whilst running Azure File Sync, Microsoft recommends uninstalling the sync agent before upgrading, then reinstalling it afterwards. Do not unregister it from the Azure portal!

With my laptop propped up on my desk beside me, I clicked through the setup wizard, and set off the upgrade.

Did I ever tell you what the definition of insanity is?#

Some time later, the ILO console greets me with a familiar logon screen. We have reached the future… 10 years ago.

I decided to continue working with the local admin for now, it’s served me well (And it’s password is easier to type…). I checked the data was all in tact, yep all good. Drivers are all working, we have internet access and it can see the dom- ah. It’s lost it’s domain authentication.

No worries, there’s a powershell command to fix that.

PS C:\WINDOWS\system32> Reset-ComputerMachinePassword -Server TheLocalDC -Credential Domain\MyUser

This time, no output is a good thing… Before I reboot it, I checked to see if nay updates were pending. Might as well kill 2 birds with one stone.

There were a few rollup updates pending, so I left those to finish off and clicked reboot.

Everything looked good, the server showed update progress, rebooted and went past the windows logo. Surely it can’t be this easy?

Yep. It wasn’t that easy.

Failure configuring Windows updates - Reverting changes

I did some sigging and found that i was running a very early release of server 2016. So i was likely in the same hole that some Windows 7 machines ended up in when microsoft decided to update their update system and render old releases unable to update without some… trickery.

I assumed this would be the same case, with that instance, I had a list of specific KB’s to install in a specific order to bring the update system back online. So, looking at the failed update, it was trying to install a 2021-01 rollup package.

I went onto the update catalog and pulled down the rollup package directly, sometimes installing it manually can bypass any weirdness with windows update on vastly outdated machines. The installation took a while, but it showed successful and I went for another reboot.

Strike 3.

Azurely you can’t be serious#

It was getting late, so I decided to try and work on the Azure File Sync Agent. Once this was up and running, the rest could be sorted out over the next few days.

Past me was a genius and pre-downloaded the sync agent. A simple double click of the MSI file and it was installed. Great, all sorted… Right?

The sync agent reported into Azure and showed the server was now running Server 2016. The sync status showed green, and all was happy.

I tested authentication and found that the domain authentication is broken again… I guess the update rollback also rolled back the machine password reset. No worries, I’ll do it again and reboot.

PS C:\WINDOWS\system32> Reset-ComputerMachinePassword -Server TheLocalDC -Credential Domain\MyUser
PS C:\WINDOWS\system32> Restart-Computer

After the reboot, I checked the status of the Azure agent and the service was stopped… I tried to start it again and it immediately stopped. Great, another thing to fix.

Maybe the SMB share is now working locally at least… Nope. The domain auth was still broken.

Turns out, I had missed yet another update rollback, which rolled back the domain join… Again.

Strike 4?

“Insanity is doing the exact same fucking thing over and over again, expecting shit to change”#

I tried a number of potential fixes throughout the day, checking forums, tweaking registry keys, resetting update catalogs… I think this is the definition of insanity.

But then it hit me.

So far, all of this was done through an ILO console, as RDP complained about CredSSP mitigations. I was getting annoyed at using the console and it timing out on me, so I thought fixing RDP would be a good place to get to. Maybe the CredSSP mitigation update would be my stepping stone to more recent updates?

I found it, KB4103720 - The potential needle in the haystack. I set the install going and waited…

It worked. The update installed and I now had RDP access, Great success!

The Night Shift#

At this point, I had RDP, I even got it joined back to the domain properly this time. What next?

The Azure Sync service was still not starting. I can only assume it just needs more updates. So, I ran the installation for the 2021-01 rollup package.

At this point, it was past midnight, so I really didn’t want to diagnose it any further if this update failed. I kept an eye on it until it needed me to click the reboot button and shut my laptop and went to sleep with the hope it will be enough to bring everything back online by the morning.

Well, it paid off! All that effort and annoyance over the weekend resulted in a server that is no longer going to be cut off by Microsoft and users have no idea anything even happened in the background.

Oh yeah, I didn’t even mention why this update was so urgent. The azure file sync service has been EOL for a while on Server 2012 R2. Microsoft set a date in December of January 27th for 2012 agents to stop syncing. Up to this point, the intention had been to test again with users and narrow down the performance issues and solve them. But with so much going on this quarter, this didn’t happen… So this server had to be updated ASAP.

There’s a reason I run a fully Linux based selfhost/homelab setup…