FR5 Downtime

Updates

• 2026-04-18 09:06:05 UTC
  It seems like the MX480 router in FR5 is down, Equinix will check the router.
• 2026-04-18 09:22:01 UTC
  It seems like some devices in rack 1505 do not have power. An Equinix technician is currently checking the power circuits.
• 2026-04-18 09:46:23 UTC
  Power was restored for a brief moment. It seems like the technician is unable to follow instructions.
• 2026-04-18 10:24:04 UTC
  Power has been restored. It seems like a new server that was recently installed caused a breaker to trip either due to a faulty PSU or power overload.
  
  Our PDUs have multiple breaker groups split into 16 Amp circuits. We are currently planning to migrate a few servers from the affected rack 1505 to a different rack to reduce overall power load.
• 2026-04-24 03:30:55 UTC
  Post-mortem
  
  On April 18 2026 at 10:44 AM CET rack 1505 lost power for around one hour.
  
  We have identified the cause for this incident in the meantime:
  
  One port group of the PDUs in this rack was overloaded due to a short power spike and too many servers being connected to the same PDU port group. The PDUs have two port groups with 16A breakers each. The short power spike tripped the breaker of the port group where RT-MX480-FKT3 is connected. RT-MX480-FKT3 is the core router at Equinix FR5.
  
  We immediately sourced an additional PDU after the power outage was resolved and it was installed on Monday, April 27th. The PDU is now sharing the load with the other PDUs in this rack and no further outage is expected.
  
  This was an extremely unfortunate incident and it could have been prevented. The PDUs in this cabinet however are partially defective, the management module is not working even though the PDUs are brand new. Replacing the PDUs is not that easy though and we are still waiting for Equinix to offer us a replacement.
  
  We could have prevented this incident with a working management module.