Rackspace hosted Exchange suffered a catastrophic failure beginning December 2, 2022 and is still ongoing as of 12:37 AM December fourth. Initially described as connection and login problems, the guidance was ultimately updated to reveal that they were handling a security event.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the morning hours of December 2, 2022. At first there was no word from Rackspace about what the issue was, much less an ETA of when it would be fixed.
Clients on Buy Twitter Verified reported that Rackspace was not reacting to support emails.
This has been quite the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours or so. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client privately messaged me over social networks on Friday to relate their experience:
“All hosted Exchange clients down over the previous 16 hours.
Not sure the number of business that is, but it’s considerable.
They’re serving a 554 long hold-up bounce so individuals emailing in aren’t knowledgeable about the bounce for numerous hours.”
The main Rackspace status page provided a running upgrade of the blackout however the initial posts had no details besides there was an interruption and it was being investigated.
The very first official update was on December second at 2:49 AM:
“We are investigating an issue that is impacting our Hosted Exchange environments. More details will be published as they appear.”
Thirteen minutes later on Rackspace started calling it a “connectivity problem.”
“We are investigating reports of connection problems to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their e-mail customer(s).”
By 6:36 AM the Rackspace updates described the ongoing issue as “connectivity and login issues” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation stage” of the blackout, still attempting to determine what failed.
And they were still calling it “connection and login problems” in their Cloud Workplace environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later Rackspace described the scenario as a “substantial failure”and began providing their customers free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround up until they comprehended the issue and could bring the system back online.
The main guidance mentioned:
“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further concerns while we continue work to restore service. As we continue to work through the origin of the concern, we have an alternate solution that will re-activate your ability to send and get emails.
At no cost to you, we will be offering you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 up until further notification.”
Rackspace Hosted Exchange Security Incident
It was not up until almost 24 hr later at 1:57 AM on December 3rd that Rackspace formally revealed that their hosted Exchange service was struggling with a security occurrence.
The announcement further revealed that the Rackspace service technicians had powered down and detached the Exchange environment.
“After further analysis, we have actually figured out that this is a security incident.
The recognized effect is isolated to a part of our Hosted Exchange platform. We are taking required actions to evaluate and protect our environments.”
Twelve hours later on that afternoon they updated the status page with more details that their security group and outdoors professionals were still working on fixing the outage.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not released details of the security event.
A security event normally includes a vulnerability and there are two extreme vulnerabilities presently in the wile that were covered in November 2022.
These are the two most present vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack enables a hacker to read and change information on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assaulter has the ability to run destructive code on a server.
An advisory released in October 2022 explained the effect of the vulnerabilities:
“A confirmed remote attacker can carry out SSRF attacks to escalate privileges and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the opponent can potentially get to other resources by means of lateral movement into Exchange and Active Directory site environments.”
The Rackspace interruption updates have actually not indicated what the specific issue was, just that it was a security event.
The most current status update since December 4th stated that the service is still down and consumers are encouraged to migrate to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in dealing with the incident. The accessibility of your service and security of your data is of high significance.
We have committed substantial internal resources and engaged first-rate external competence in our efforts to minimize unfavorable effects to clients.”
It’s possible that the above noted vulnerabilities relate to the security incident affecting the Rackspace Hosted Exchange service.
There has been no announcement of whether consumer details has actually been jeopardized. This event is still ongoing.
Featured image by Best SMM Panel/Orn Rin