Hello folks, we want to take the time to reflect on the aftermath of Hurricane Idalia. For some background, Archous Networks currently has physical datacenter point of presence in Georgia. We host our secured remote VPN service, our monitoring service, and our configuration backup services along with a few custom solutions for customers out of this datacenter. We also have several clients located in the Southeast USA that depend on services from this datacenter. It’s important to note that the datacenter is owned and operated by one of these clients that we have a very good working relationship with.
As a preparation step for the hurricane, we asked the datacenter to ensure the UPS systems and backup generator were in working order and we were informed the UPS system was adequate and generator had passed automatic transfer testing with proper fuel for up to 48 hours.
At around 7:45am on Wednesday 8/30 we tracked the hurricane landfall off the Florida coast. Our Georgia datacenter is located about 100 miles inland from the point of landfall. The image below shows a trajectory map excerpt showing the location and revealed that the datacenter was directly within the path of the storm’s eyewall.
At 11:15am local time, the surrounding city and county lost general residential and commercial power due to storm damage. We received confirmation of a graceful transfer to generator power and verified through proactive monitoring that there was no impact to any services, including Internet connectivity, at this time.
At 1:38pm local time Archous started receiving alarms that all services were down at the datacenter. Remote access through both the primary Internet circuit and the backup/out of band were also down. The datacenter operator was informed and we were notified that the generator was no longer running. The root cause was determined to be lack of fuel and the generator’s fuel gauge was reportedly faulty. Also during this time, the local area was hit very hard with high winds and damage from debris which was affecting cellular phone coverage and network connectivity to local 911 call centers in the Southeast Georgia area.
A fuel truck was dispatched at around 2pm local time and at around 3:30pm local time power was confirmed restored on generator power. Once power was restored, Archous began supporting restoration of networking and server equipment located on site for both Archous and the datacenter customer-owned equipment. Due to the sudden nature of the power outage and age + uptime of some of the equipment that lost power, there was equipment that did not cleanly restore from the power loss. In parallel Archous was engaged to support restoration of emergency 911 Internet services well as Internet continuity for the local power company to continue to receive phone calls while the POTS services were down. Full restoration of all services was at 5:31pm local time. The datacenter continued to operate from 5:31pm event-free while on generator power through the next day.
The next day on 8/31 we were engaged to troubleshoot the local power company’s primary fiber optic Internet service and tracked down the fiber loss of service to multiple provider dependency power outages and equipment failures. Dispatch for the power company’s ISP was notified and work began to prioritize restoring power to the affected locations. Starlink backup Internet had been deployed the previous day so continuity remained in place for the power company’s phone and Internet dispatch service. The local power company was able to restore commercial power to the datacenter at 10:21am local time at which point we received confirmation the operations were no longer running on generator power.
At 5:20pm local time on 8/31 Archous received alarms that the primary fiber optic Internet service to the datacenter was down. The datacenter operator communicated to us that storm damage had caused aerial fiber cables to sag and that a semi truck had hit the cables and torn down the power, fiber mainline, and the associated power poles feeding the datacenter. The image below captures a snapshot of some of the damage.
The fiber provider was engaged and began work on restoring the fiber optic cables. The fiber provider assessed the damage at around 7pm and estimated 24-48 hours for the splicing work needed to restore services.
In the meantime, backup Internet services to the datacenter operated on Starlink. Besides obvious latency and speed related issues, Starlink has a downside that it does not support public IP address assignments to its services. 24-48 hours was not an acceptable downtime for some of the public IP services offered over the datacenter so Archous in partnership with Core Transit began building a plan to restore public IP connectivity via temporary Internet tunnels. At roughly 12am local time on 9/1 minimal public IP access was restored via Core Transit’s Internet tunnelling services which provided much-needed public IP address connectivity.
Internet services were restored ahead of the estimated time of repair by the fiber provider at around 6:20am.
Lessons Learned / Action Items
In summary, we will be working to reduce the customer impact for future situations by deploying a Seattle-based datacenter for west coast backup and continuity purposes. We estimate the Seattle datacenter to be online by 9/15. Prior to this event, we had a redundant fiber Internet service order already in-flight via a different provider to the datacenter location. We will be escalating this order as that would have prevented nearly 13 hours of outage/degraded service time. To bridge the redundancy gap in the meantime, the datacenter provider will be working with Core Transit to ensure a properly implemented solution is in place for quick recovery over the Starlink connection if this problem arises in the future. We also have communicated with the datacenter operator to ensure in the future that fuel quantities are physically confirmed beyond relying on gauge readings.
We appreciate all customers working with us during the week of 8/28-9/1 and appreciate everyone’s understanding of the situation. We recognize the nature of emergency response and natural disaster and how that can have a trickle affect to our customers. No one likes outages — especially a network consulting company. We hope the transparency of this situation and the action items we are taking reflects our commitment to our customers.