Degraded Performance for Global Edge Security resolved, 10/08/2019 04:31PM CDT Global Edge Security


Notification history

10/08/2019 04:31PM CDT
Incident Report

Summary
On October 8th, we received multiple reports of Global Edge Security (GES) enabled sites being down due to DNS not resolving. We introduced a bug into the domain deprovisioning process which led to the GES proxy CNAME being deprovisioned in our systems and therefore would not resolve. This code was reverted and the impacted domains were re-provisioned.


Customer Impact
Customers whose GES domains were deprovisioned would have experienced varied amounts of downtime between 10:33am CST and 2:15pm CST. 

Remediation
  • The code was reverted and impacted GES domains were re-provisioned. This revised code has been rolled out with additional testing and throttling in place in order to more closely monitor each impacted environment. 
  • For automation that can have immediate and far reaching impacts, we are reviewing testing and deployment procedures in order to reduce the likelihood of recurrence.

10/08/2019 04:25PM CDT
No further updates at this time
10/08/2019 03:38PM CDT
No further update at this time, but our team continues to monitor impact as we approach final confirmation of resolution.
10/08/2019 02:09PM CDT
The majority of issues pertaining to Global Edge Security have been resolved. We are currently monitoring and will be updating further upon confirmed resolution.
10/08/2019 01:27PM CDT
No further update at this time, but our team continues to work towards a full resolution of the issue.
10/08/2019 12:50PM CDT
There are no new updates at this time
10/08/2019 12:18PM CDT
The issue affecting a subset of customers utilizing the Global Edge Security Product has been identified and is in the process of being remedied
10/08/2019 11:42AM CDT
We've been made aware of an issue with {{Global Edge Security}}} for some of our customers. We are actively investigating the problem now.