Intermittent Connectivity Issues in One US Datacenter

Posted

Incident is resolved

Update 11:45AM CST: At this time our 3rd party datacenter has resolved this DDoS at their Atlanta datacenter. For the time being we’re marking this issue as resolved.

Update 2:00PM CST: Connectivity has become intermittent again and our 3rd party datacenter provider has reported they are investigating. We continue to restore customers to other locations during this time.

Update 10:20AM CST: Connectivity is now stable, and our 3rd party datacenter is reporting that all attack vectors that can lead to a DDoS taking down the entire Atlanta datacenter have been blocked. We are continuing to monitor and work with them to come to a final resolution.

Update 6:00AM CST: Connectivity continues to be intermittent but mostly stable, we are continuing to monitor and work with our provider.

Update 2:05AM CST: We are seeing the connectivity become unstable and are receiving alerts for our customers. We continue to restore customers to other locations during this time.

Update 11:00PM CST: We are seeing connectivity return to normal for the majority of customers, we continue to migrate customers to other locations during this time.

Update 9:00PM CST: We are seeing the connectivity become unstable and are receiving alerts for our customers. We continue to restore customers to other locations during this time.

Update 7:00PM CST: Connectivity continues to be intermittent but mostly stable, we are continuing to monitor and work with our provider.

Update 5:10 PM CST: Our 3rd Party Datacenter has implemented a dedicated transit link for the traffic through the upstream provider, which appears to be normalizing the traffic. At this time, most sites are reported as online, but we are still monitoring for any future issues or intermittent connectivity. As they continue the maintenance, intermittent connectivity may occur. We will continue to provide updates as more information becomes available.

Update 1:25 PM CST: Our 3rd party Datacenter’s Network Engineers are still actively working with their upstream provider to restore normal network connectivity. We will continue to provide further updates as we receive more information.

Update 7:30 AM CST:  Our 3rd party Datacenter is working to mitigate the attack, but for the moment, we are still experiencing downtime on the servers. We will update you shortly with further details.

Update 3:30 AM CST: Our 3rd party Datacenter is working to mitigate the attack, but for the moment, we are still experiencing downtime on the servers. We will update you shortly with further details.

Update 1:14 AM CST: The attacks have resumed on our 3rd party datacenter, resulting in downtime. Our 3rd party datacenter is continuing to work with their upstream network provider on mitigating the attacks, but as of now we do not have an ETA. As always, we will continue to provide updates whenever one is available.

Update Jan 02 12:42 AM CST: We are beginning to see connectivity return to affected sites. Our 3rd Party Datacenter is still reporting that there may be intermittent connectivity issues while they continue to work with their upstream provider to resolve this issue. We will continue to provide further updates as we receive more information.

Update 10:20pm Central: Our 3rd party datacenter provider is now restoring network connectivity in segments over time, this will see customers return to functionality over time. We will provide an update as this continue to be put in place.

Update 9:40 pm CST: Our 3rd party datacenter provider is currently working with their upstream network provider to put a plan in place to restore network connectivity. They are working to accelerate the process as much as possible, and we will provide further updates as more information becomes available.

Update 4:45pm Central: Our 3rd party datacenter provider has reported that this attack is happening to their upstream network provider, as such we are waiting for them to mitigate the attack. This is a new and separate upstream provider from recent attacks and as such they are having to mitigate their systems. We will provide an update as soon as one is available.

Incident – 11:55AM CST –We are still receiving reports of intermittent connectivity occurring on our Atlanta datacenter. Our 3rd Party datacenter provider is continuing efforts to resolve this issue with their upstream provider. We apologize for any inconvenience these issues are causing and we thank you for your patience.

Incident – 9:45AM CST – At this time we have reports of a large scale DDoS attack on one of our 3rd Party datacenters in the United States, which is causing downtime for a subset of our customers. Our datacenter provider is currently working to mitigate the attack. As more information becomes available we will update this post.