We often hear from people who have had their website built elsewhere and while it met the design brief, the site itself is unstable. Our team are deeply technical so we are often able to step in and help, whether that means making changes to the website or a system that underpins it.
In this article, I’m going to trace one particular issue on a website that we built and have managed in a stable state for numerous years. It all started with an image from a colleague who was unable to deploy a change to the website.
“It looks like the website directory has permission issues, any ideas?”
Any server administrator will recognize that this screenshot is a troubling sight. The line of question marks indicates that the system is unable to identify the contents, or even existance, of the website directory. The directory in question is mounted from an external Microsoft Azure Storage account and has been working without issue since the servers were built in 2023.
Further investigation confirmed that this wasn’t just a permissions issue; effectively that directory didn’t exist because the server could no longer access the Microsoft Azure Storage account.
The same was true across the production and staging environments.
The websites hosted on the servers were still running because they operate from a local cache of the files. This meant the cause may not have happened immediately, which is troubling as the failure had been invisible.
The “mount” command confirmed that the server thought that this was still mounted (aka connected) and available; however any attempt to unmount & re-mount it returned an access denied error.
A trawl through the log files for both the server and Azure Storage account confirmed the date of the first seemingly related warning. This seemed to correlate with a third party’s deployment of updates which was to bring the Microsoft Azure account in line with the latest best practice guidance. As the related documentation states:
“The Azure Landing Zones (Enterprise-Scale) architecture provides prescriptive guidance coupled with Azure best practices, and it follows design principles across the critical design areas for organizations to define their Azure architecture. It will continue to evolve alongside the Azure platform and is ultimately defined by the various design decisions that organizations must make to define their Azure journey.”
Looking more closely there had been a number of changes made together. At this point, we started to reverse these changes one at a time until we discovered the cause of the issue. The seemingly innocuous setting “enableHttpsTrafficOnly” had been enabled on all Storage accounts as part of a policy entitled “Enforce-TLS-SSL-H224”.
This meant that Azure is requiring an SSL/TLS encrypted connection but the server is not. While we could disable the HTTPS requirement that would mean that we would no longer be following Azure’s best practice guides despite the connection being within a protected security group making interception highly unlikely.
With this discovery, we “just” need to make the server encrypt the connection and close the ticket. However, the Network File System (NFS) would need to provide the ability to encrypt the connection in a compatible form but it seemingly didn’t.
Fortunately recently Microsoft had published a new “Helper” whose description includes: “can be used to provide a secure communication channel for NFSv4 traffic. This is achieved by implementing TLS encryption for NFS traffic”
By installing this “Helper” and reconfiguring the necessary scripts to use the new connection type we were finally able to re-mount the storage account.
Having done so the web server can once again connect to the Azure Storage Account and put everything back into sync and the developer can finally deploy their changes that started this whole episode.
This issue is a perfect example of why a website needs more than just great design. It needs a technically capable team behind the scenes to support and maintain it. From diagnosing obscure infrastructure changes to implementing secure, standards-compliant solutions, problems like these can easily derail a business if not handled properly.
