Rackspace hosted Exchange suffered a disastrous outage beginning December 2, 2022 and is still ongoing as of 12:37 AM December 4th. Initially referred to as connection and login issues, the assistance was ultimately updated to announce that they were handling a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the problem was, much less an ETA of when it would be resolved.
Consumers on Buy Twitter Verification reported that Rackspace was not reacting to support e-mails.
This has been rather the day with #Rackspace. Every hosted exchange client has actually been down for 14 hours or so. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace customer privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange customers down over the previous 16 hours.
Not sure the number of companies that is, however it’s considerable.
They’re serving a 554 long hold-up bounce so people emailing in aren’t knowledgeable about the bounce for numerous hours.”
The official Rackspace status page used a running upgrade of the blackout but the initial posts had no info other than there was a blackout and it was being investigated.
The very first authorities upgrade was on December second at 2:49 AM:
“We are investigating a concern that is affecting our Hosted Exchange environments. More information will be posted as they become available.”
Thirteen minutes later Rackspace started calling it a “connection concern.”
“We are investigating reports of connection problems to our Exchange environments.
Users might experience an error upon accessing the Outlook Web App (Webmail) and syncing their email customer(s).”
By 6:36 AM the Rackspace updates described the continuous problem as “connectivity and login problems” then later that afternoon at 1:54 PM Rackspace announced they were still in the “examination phase” of the outage, still trying to determine what went wrong.
And they were still calling it “connectivity and login concerns” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later Rackspace referred to the scenario as a “substantial failure”and started offering their consumers free Microsoft Exchange Strategy 1 licenses on Microsoft 365 as a workaround till they comprehended the issue and could bring the system back online.
The official assistance mentioned:
“We experienced a substantial failure in our Hosted Exchange environment. We proactively shut down the environment to prevent any additional problems while we continue work to restore service. As we continue to resolve the root cause of the issue, we have an alternate service that will re-activate your capability to send and get emails.
At no charge to you, we will be offering you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 until more notification.”
Rackspace Hosted Exchange Security Occurrence
It was not till nearly 24 hours later at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was suffering from a security incident.
The statement even more revealed that the Rackspace service technicians had powered down and detached the Exchange environment.
“After further analysis, we have identified that this is a security occurrence.
The recognized impact is isolated to a part of our Hosted Exchange platform. We are taking essential actions to evaluate and safeguard our environments.”
Twelve hours later that afternoon they upgraded the status page with more info that their security team and outdoors experts were still dealing with resolving the interruption.
Was Rackspace Service Affected by a Vulnerability?
Rackspace has not released information of the security occasion.
A security event normally involves a vulnerability and there are 2 extreme vulnerabilities currently in the wile that were patched in November 2022.
These are the 2 most present vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack permits a hacker to read and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assailant has the ability to run malicious code on a server.
An advisory released in October 2022 explained the impact of the vulnerabilities:
“A validated remote aggressor can carry out SSRF attacks to escalate advantages and perform arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted versus Microsoft Exchange Mailbox server, the assaulter can potentially access to other resources through lateral movement into Exchange and Active Directory environments.”
The Rackspace outage updates have not indicated what the specific problem was, just that it was a security incident.
The most present status update as of December fourth specified that the service is still down and customers are encouraged to move to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make progress in dealing with the incident. The availability of your service and security of your information is of high significance.
We have actually dedicated extensive internal resources and engaged first-rate external competence in our efforts to reduce unfavorable effects to consumers.”
It’s possible that the above noted vulnerabilities belong to the security event impacting the Rackspace Hosted Exchange service.
There has been no announcement of whether consumer details has actually been jeopardized. This occasion is still continuous.
Included image by Best SMM Panel/Orn Rin