Thursday 3/23/23
At 10:00 PM SWK entered a maintenance window to process security updates and perform routine network maintenance on critical infrastructure. During this window our engineers were to connect new ports provided by our ISP to our firewalls. After testing it became clear that the ISP had not configured the ports properly so they were disconnected until a time that they would be able to correct the issue. Our primary and backup connections were moved back in to place and network connectivity was verified.
Friday 4/24/23
At 8:25 AM it was reported that a small subset of clients were not able to access the internet from their hosted machines in our east coast data center.
A number of issues were escalated to our Data Center engineers for evaluation at 8:30 AM. The team began troubleshooting the reports and classified this as a Priority 1 (Emergency) ticket.
At 9:20 AM the problem was identified to be an upstream ISP routing issue for clients that were on a small block of our leased public internet addresses. This routing issue was caused by the misconfigured redundant drop on the ISPs network equipment.
At approximately 9:50 AM the engineering team was able to reset the cached information in the ISPs switches by forcing a reset of the cached ARP table for those ports.
At 9:52 AM the impacted systems began to once again route traffic without issue.
Clients who purchased static public IP addresses in one particular block, could not successfully navigate outbound from their servers to the internet. This was a small subset of our overall data center residents, as it only affected 1 block of IP addresses.
A misconfiguration on an upstream ISP switch caused a subset of our public IP space to not route to the correct circuit.
Our provider has been engaged and briefed about this incident and assures us that steps are being taken to prevent these issues in the future.