Today’s network events
To our customers and partners,
As you know, Intermedia experienced a significant network interruption today. As CEO of Intermedia, I want to offer a profound apology.
I also want to explain what happened, what we’ve learned, and what we’re going to change.
First, you should know that our services are now stable. There has been no data loss and there were no security breaches. And all of the emails that were sent to you during this issue will be delivered as the backlog is processed.
Here’s what we know.
Around midnight on Wednesday night, August 28th, we experienced an issue with our core routers. We implemented a fix to this issue, and we believed that it was completely resolved by 3 AM EST on Thursday.
At 7 AM EST on Tuesday, September 3, we observed some further anomalies on our core routing equipment. Our attempts at intervention were unsuccessful, and corruption in our core routing table propagated to the edge routers in all our US datacenters. This created significant packet loss between the edge and core in each datacenter and it prevented delivery of service to our customers. In addition, as our communication systems reside in the same datacenters, our ability to communicate with customers and partners was disrupted.
At 8 AM EST on Tuesday, we started restoring network connectivity by systematically rebooting each of the affected network devices. Our network and services were brought back online throughout the day, with complete network restoration by 3:30 PM EST.
As connectivity was restored to each datacenter, our system administration team focused on resolving any residual server issues caused by the network outage. The final server-related issues were resolved by approximately 6:30 PM EST. Our support team was able to take phone calls beginning around 3 PM EST.
What comes next.
As of now, all our services are back online. We now have three tasks ahead of us.
The first task is to complete our RFO (Reason For Outage) report to fully identify the root causes. We’re working with our network equipment vendor to complete this report, and we will share it with our customers and partners as soon as we can.
Second, we will take the recommendations from this RFO and make any changes necessary to improve our stability and resiliency.
Third, we will also improve the responsiveness and robustness of our customer notification tools and systems. Although we were successful in notifying many of our customers about the issues via alternate email addresses, text messages and HostPilot, not all customers were reached. Going forward, we will make more timely use of our website and social media—especially Twitter and Facebook.
I hope you accept my apology.
On behalf of all of us at Intermedia, I offer a profound apology. We understand how essential our services are to your business. We fully appreciate the breadth of our responsibility. We live and breathe it every day.
I’d appreciate any feedback you can share. Please email email@example.com. This mailbox is monitored by Intermedia’s senior management team.