Dear customers,
Equinix has announced a dark fiber maintenance between FR5 and FR8. Maintenance was scheduled for 8AM local time today, March 21st.

The impacted cross connects did not include the backbone links between FR5 and FR8 and FR5 and FR7.

At 08:10 am, we noticed that the backbone links between the sites went down. This only impacted customers at FR5 since some VLANs are still backhauled to FR7.

There was a small configuration mistake on RT-MX480-FKT3 which killed connectivity via Zayo(Only partial routes were imported)

This was fixed shortly after the initial downtime started.

We were able to restore connectivity for most customers by 09:30AM.

Each of the three PoPs(FR5, FR7, FR8) has at least one independent Transit and Peering link to keep any of the sites up in case backbone connectivity is cut. However, as previously mentioned, some VLANs were still backhauled to FR7 an we did not expected that the connectivity between three sites would fail.

Our management servers are still unreachable but we are already working on restoring connectivity.

Customers might have experienced packet loss during 04:32 PM and 04:52 PM CET today.

rt-mx240-fkt is the DDoS clean router where most of the cleaned traffic is aggregated and sent further down to the core network.

A recently onboarded customer received large carpet bombing DDoS attacks during the mentioned timeframe. During this incident, our IP prefix automation had not yet completed the DDoS detection config for the customer prefixes, especially carpet bombing detection. Another customer also received DDoS attacks in the same timeframe and this caused the congestion on rt-mx240-fkt since some traffic leaked due to the missing detection profile and another customer was already pushing around 30Gbps clean traffic through rt-mx240-fkt. The customer tunnel has now been moved to an edge router.

A packet generator running in our network was set up to send packets to a static mac address for testing. The server with that assigned mac address was rebooted during the testing. This caused the router RT-QFX10K-FKT2 to forward packets to all hosts in our mitigation cluster.

Certain rate limits on our routers triggered to prevent further damage. Customers might have experienced slow TCP performance 03:40 AM and 8AM.

Test1

Maintenance on rt-ptx10k-fkt3

RT-QFX10K-FKT Down