The sponsoring LIR for the prefixes 2a0e:97c0:470::/48 2a0e:97c0:472::/48 2a0e:97c0:473::/48
has removed the inet6num and route6 objects due to a human error while renewing the LIR service a month ago. This process was automatic.

Due to the removal of the route6 object, the three prefixes are currently unreachable from some destinations.
We are currently waiting for the LIR to restore the service.

According to a yet unverified report, at least one room at Equinix FR5 is currently experiencing a cooling outage, apparently caused by a water leak.

This seems to have caused various devices in this room to shut down, starting at around 5AM CEST, with the most impact seen at 05:20 AM CEST. There was some packet loss for around 30-45 minutes via GNM-IX since we were not immediately able to isolate the problem.

Some carriers are currently down at FR5, including RETN.

RS sessions to GNM-IX are currently set to shutdown. Traffic levels are back to normal.

Please note that our infrastructure is not directly affected by this incident since our racks are not located in the room that is affected.

RT-PTX10K-FKT3 went down at 11:54 CET due to an unknown reason.

As this router is primarily being used as edge router, traffic was automatically re routed via the other edge routers. However, there was a short traffic flap.

At 19:38 CEST, one of the linecards on RT-QFX10K-FKT2 rebooted unexpectedly due to an unknown reason.

There was a short traffic flap but it quickly recovered for most destinations. Some customers directly connected to this linecard were affected for roughly 12 minutes.

However, this reboot seems to have left some GRE tunnels on RT-QFX10K-FKT2 in a stale state. All GRE tunnels are up again.

Dear Customers,

we are aware about a downtime in FR5. We are currently checking the situation with Equinix.

Dear customers,
Equinix has announced a dark fiber maintenance between FR5 and FR8. Maintenance was scheduled for 8AM local time today, March 21st.

The impacted cross connects did not include the backbone links between FR5 and FR8 and FR5 and FR7.

At 08:10 am, we noticed that the backbone links between the sites went down. This only impacted customers at FR5 since some VLANs are still backhauled to FR7.

There was a small configuration mistake on RT-MX480-FKT3 which killed connectivity via Zayo(Only partial routes were imported)

This was fixed shortly after the initial downtime started.

We were able to restore connectivity for most customers by 09:30AM.

Each of the three PoPs(FR5, FR7, FR8) has at least one independent Transit and Peering link to keep any of the sites up in case backbone connectivity is cut. However, as previously mentioned, some VLANs were still backhauled to FR7 an we did not expected that the connectivity between three sites would fail.

Our management servers are still unreachable but we are already working on restoring connectivity.

Customers might have experienced packet loss during 04:32 PM and 04:52 PM CET today.

rt-mx240-fkt is the DDoS clean router where most of the cleaned traffic is aggregated and sent further down to the core network.

A recently onboarded customer received large carpet bombing DDoS attacks during the mentioned timeframe. During this incident, our IP prefix automation had not yet completed the DDoS detection config for the customer prefixes, especially carpet bombing detection. Another customer also received DDoS attacks in the same timeframe and this caused the congestion on rt-mx240-fkt since some traffic leaked due to the missing detection profile and another customer was already pushing around 30Gbps clean traffic through rt-mx240-fkt. The customer tunnel has now been moved to an edge router.

A packet generator running in our network was set up to send packets to a static mac address for testing. The server with that assigned mac address was rebooted during the testing. This caused the router RT-QFX10K-FKT2 to forward packets to all hosts in our mitigation cluster.

Certain rate limits on our routers triggered to prevent further damage. Customers might have experienced slow TCP performance 03:40 AM and 8AM.

Test1

Maintenance on rt-ptx10k-fkt3

RT-QFX10K-FKT Down