As of 8:23PM on Oct 1st, the connectivity issues within our data center that affected our VDS customers has been resolved.
RCA
The data center we were hosted in was testing failover of the backup power systems when and unexpected situation occurred with the equipment and the power company. This incident caused half of the data center to lose power and caused a failure in one of our highly available networking switches. The data center team worked on replacing the failed switch, however the power failure caused a configuration setting to get reverted. This setting being reverted caused our VDS customer network to be non-routable between our firewalls and switches. This issue have now been resolved and proper procedures in place to mitigate this downtime in the future.
Tuesday, October 1, 2019