Customers might have experienced packet loss during 04:32 PM and 04:52 PM CET today.

rt-mx240-fkt is the DDoS clean router where most of the cleaned traffic is aggregated and sent further down to the core network.

A recently onboarded customer received large carpet bombing DDoS attacks during the mentioned timeframe. During this incident, our IP prefix automation had not yet completed the DDoS detection config for the customer prefixes, especially carpet bombing detection. Another customer also received DDoS attacks in the same timeframe and this caused the congestion on rt-mx240-fkt since some traffic leaked due to the missing detection profile and another customer was already pushing around 30Gbps clean traffic through rt-mx240-fkt. The customer tunnel has now been moved to an edge router.

A packet generator running in our network was set up to send packets to a static mac address for testing. The server with that assigned mac address was rebooted during the testing. This caused the router RT-QFX10K-FKT2 to forward packets to all hosts in our mitigation cluster.

Certain rate limits on our routers triggered to prevent further damage. Customers might have experienced slow TCP performance 03:40 AM and 8AM.

Test1

Maintenance on rt-ptx10k-fkt3

RT-QFX10K-FKT Down