Google Explains & Apologizes for Gmail Outage

RIM could learn a thing or two from Google in terms of coping up to their outages. We mentioned the major outage that Gmail had yesterday which even threatened to take down Twitter with the barrage of users asking if it was just them. Google even has a specific page where they were posting updates throughout the whole outage detailing and confirming the outage.

This is what I call good communication. It sucks that Gmail was down due to what Google is now calling a “slightly underestimated load” during a routine server upgrade. You can read their explanation after the jump.

I just cannot help but compare how Google handled this compared to the many BlackBerry outages we have had over the years where RIM leaves us users in the dark wondering what the hell has happened. We have to resort to mailing lists like DataOutages to confirm these outages which RIM only reports to their big customers. It is just another case of how RIM caters to carriers and big customers and puts regular consumers second in line. What do you think?

Here’s what happened: This morning (Pacific Time) we took a small fraction of Gmail’s servers offline to perform routine upgrades. This isn’t in itself a problem — we do this all the time, and Gmail’s web interface runs in many locations and just sends traffic to other locations when one is offline.
However, as we now know, we had slightly underestimated the load which some recent changes (ironically, some designed to improve service availability) placed on the request routers — servers which direct web queries to the appropriate Gmail server for response. At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system "stop sending us traffic, we’re too slow!". This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded. As a result, people couldn’t access Gmail via the web interface because their requests couldn’t be routed to a Gmail server. IMAP/POP access and mail processing continued to work normally because these requests don’t use the same routers.
The Gmail engineering team was alerted to the failures within seconds (we take monitoring very seriously). After establishing that the core problem was insufficient available capacity, the team brought a LOT of additional request routers online (flexible capacity is one of the advantages of Google’s architecture), distributed the traffic across the request routers, and the Gmail web interface came back online.