Microsoft’s Azure’s East Australia region outage

Last week, Microsoft had a problem at its data center hosting Azure’s Australia East region. A voltage dip caused the chiller plant for two data halls to go offline, damaging some storage hardware. 

This resulted in major disruptions for customers like Bank of Queensland and Jetstar.

Microsoft has now released a preliminary report on the incident. 

The report explains that so many storage nodes were affected that data and all its copies went offline. Even after the storage nodes were recovered, a "tenant ring" with over 250,000 databases failed, impacting customers to varying degrees.

Despite their efforts, the onsite staff couldn't bring the chillers back online. It seems Microsoft wasn't prepared for the magnitude of the problem, with insufficient staff and emergency procedures that couldn't handle the situation.

According to the company, the size of the data center campus and the lack of staff at night were the main reasons for the delay in resolving the issue.

- CyberBeat