Further answers to your questions
Dear Intermedia customers and partners,
We have received over 1,500 responses to the letters that were sent out earlier this week. (If you haven’t read the letters yet, you can find them on our blog. Here’s our CEO’s letter from Tuesday and my letter from Wednesday.)
As Intermedia’s COO, I understand why you may have further concerns. I fully recognize the importance of our services to your business, and I know how challenging Tuesday was for you.
That’s why I’m writing today. First, I want to assure you that your communications have been received, and that each one has been read by Intermedia’s team.
Second, while reading your responses, we have spotted a number of recurring questions. I want to provide answers for each of them.
1) Was this event due to an external attack? Was there a security breach?
No. This was not an attack, and no data was compromised.
2) How will you prevent this event from recurring?
Let me give you a little more information about what happened. There were actually two separate events: an initial routing issue caused by a failure in core network equipment in our Virginia datacenter, and a second routing issue that occurred across all our datacenters as we worked to resolve the initial one.
We’ve identified the cause of the initial issue and the issue is now resolved. We have also made a significant change in our routing policies that guarantees the second issue cannot occur again.
Taking this a step further, we’re working with our network hardware vendor to explore changes in our architecture that will further enhance our network’s robustness and availability.
We will have our Reason For Outage (RFO) report completed next week.
3) Was any mail lost? There is a big time gap in my inbox.
No mail was lost. All our inbound and outbound mail was queued, either in our system or by the sending system, and it has all since been delivered. We’ve monitored our queues closely, and we know that delivery is complete.
Many customers report that Outlook shows a gap in email timestamps that correspond to the duration of the network event. That does not mean that messages were lost. Rather, this just reflects the time when the queued messages were delivered.
4) How will you improve your methods of communications?
Our internal phone systems, email, and Customer Notification Tool (which utilizes SMS, HostPilot, phone outage messages, and alternate email addresses) all reside on the same production network that delivers our services. Therefore, this network issue affected our tools as well.
In the two days since my last letter, we have finished an engineering plan to move our communications tools on to a separate and isolated network. This means that we will never again have an issue that impacts our services and our ability to communicate at the same time. We are initiating this project immediately.
In the future, we will still rely on the channels listed above for regular communications with our customers and partners. We are also improving our use of secondary channels such as Twitter and Facebook.
In addition, we recommend that you update the contact information within your control panel to include an alternate email address, a phone number and cell phone numbers for receiving SMS messages.
5) Will I receive service credit as promised in your Service Level Agreement? Do I need to call support or open a ticket to do so?
You will receive your credit without needing to take any further action. We’ll apply credit proactively to your account.
We’re working through the details at this time. You can expect your account to be credited in the next billing cycle.
Thanks for your attention. More coming soon.
We value your feedback and your continued trust in Intermedia, and I can assure you that we’re taking seriously each of the 1,500 responses we’ve received.
Next week, you will receive one more email with information about your SLA credit and about our final RFO report.
As always, if you have immediate concerns, please don’t hesitate to call 800.379.7729 or email firstname.lastname@example.org.