GMail Outage Was a Chain Reaction

Google has published an explanation of the widespread GMail outage from September 1st. In this explanation, they trace the root cause to a layer of "request routers":

...a few of the request routers became overloaded and in effect told the rest of the system "stop sending us traffic, we're too slow!". This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

This perfectly describes the "Chain Reaction" stability antipattern from Release It!


Similar entries

  • dondodge: @fredwilson Outlook to Gmail -Save your Outlook contacts (export) to a CSV file. Gmail will import them. Enjoy the advantages of Gmail

  • The morning of March 2nd started out fairly normally. I moved quickly to get ready and jumped on my computer to check email, twitter, my feeds, etc. So I launched Outlook and happened to see an email from one of my good friends Matt Leonard (or @mjleonard if you are on Twitter). He's a great guy and a smart marketer so we're in touch often. The email was sent from Matt's gmail account and the subject line communicated the urgency of his message.Apparently, Matt flew to Nigeria the night before to attend the Tinapa Opening Ceremony. {???} He ended up staying at a hotel, which was attacked by armed robbers. {OK...} They took all of his money and his wallet. {???} His email explained that he needed money badly, to the tune of $1500 so he could settle his hotel bill. {LOL} He seemed very scared and he emphasized that he needed the money sent as soon as possible. As you can imagine, I was genuinely concerned for Matt... {sarcasm}Yes, Matt's gmail account had been hacked and I'm sure many people received the same email I did. Here was the original email:---------- Forwarded message ---------- Subject: *****URGENT REPLY NEEDED ********* Hello I am in a hurry writing this message, I had a trip to West Africa *NIGERIA* on visiting the TINAPA OPENING CEREMONY, Unfortunately for me all my money got stolen at the hotel where i lodged from the attack of some armed robbers.Right now,I'm without money and I'm even owing the hotel here, the hotel telephone service is disconnected,i have only access to emails,my mobile phone can't work here so i didn't bring it along, please can you lend me $1500 so i can return back and settle the hotel bills i would return it back to you as soon as i get home, I am so confused right now.You can have it sent through western union money transfer. My passport is with the Embassy here so i cant use my name to collect it now, But you can have it sent directly to a western union Manager here and i would get it through him as he helps the people here in the hotel to receive the money, I have already spoken to him, please let me hear from you so i can collect his full name and address where you can send the money tomorrow please,or if possible today. I am waiting for your reply.Thank you. I look forward to your positive response.Regards.-----------------OK, this was absolutely ridiculous, but being the good friend I am, I still wanted to reach out to Matt as soon as possible to let him know what was going on. I was pretty confident he didn't know yet. Now, I didn’t know if his actual account was taken over. I simply thought that someone was blasting out emails using his address as the sender. There’s a difference between the two. So, I created a new email (which was sent to his gmail account) informing him that his email address had been compromised. Unfortunately, that was the only email address I had for Matt...It was a brief email that looked like this:--------------------------Hi Matt.I just received a spam email from your gmail acct. Not sure if it'swidespread or not, but wanted to let you know.It was one of those Nigerian schemes.Let me know if you have any questions.Glenn-----------------------It was only a few minutes when I received an email back from Matt. My guess was that he wanted to quickly thank me for notifying him of the scam. I opened the email and was shocked to see that the original email from Matt wasn't a joke! Matt was in fact in Nigeria at a hotel and needed money. {sarcasm yet again}You see, “Matt” actually responded to my email!Holy smokes, the scammers who hacked Matt's gmail were answering emails right from his account! I'll admit it, that creeped me out. Read their response to my email below:---------- Forwarded message ----------Subject: Re: Email SpamTo: Glenn GabeHello,Its not a scam mail, i am really there now. i need you to help me with $1500, i shall pay you as soon as a i return. i hope to read from you asap. below is the info where the money will be sent to....------------------So, I direct messaged Matt via Twitter notifying him of the hack, this time using only 140 characters or less. :) Before long, I received a DM back from Matt that he was handling the situation. He thanked me, and got back to fixing the problem. So, after Matt recovered his gmail account (and after he got pummeled by jokes on Twitter), I asked him about the steps he took to recover his gmail account.After this happened to Matt, I asked myself if I would know what to do... I really didn't. I'm sure I would eventually figure it out, but I didn't know the exact steps. So with Matt's assistance, I decided to write this post to document the steps you should take to recover your gmail account after it has been compromised.So in Matt’s own words, here’s how to handle the situation:The process was pretty simple.First, from the Gmail login page, select "I can not access my account".Second, select the radio button "My account has been compromised". This will show a new prompt below "Please fill out our account recovery form to help us process your request as quickly as possible". Follow that link.From the Account Recovery page, select "I believe someone has taken over my account" and fill out as much information as possible.In my case, Google had contacted me pretty quickly via my alternate email with instructions to reset my password. {Glenn: Make sure your alternate email is active. You can check this from your Google account settings. You can also add additional email addresses just in case.}Some other useful notes from Matt:I'm not sure how someone hacked my gmail account. I did make a mistake by using the same password on way too many things. My main passwords are now individually unique. Passwords I set up for vendor accounts are no longer the same as I would use for other more sensitive services, like online banking.Matt’s Tips to Protect to Your Logins:1. Avoid unsecured networks2. Don't use the same password everywhere.3. Don't duplicate your email password with any login elsewhere (if you do, someone can use your email to login to your other accounts).4. Change passwords periodically.5. Don't share your passwords with anyone. Even if they're not going to abuse it, they may not store it properly.6. Know where you enter passwords. 3rd party Twitter apps that require passwords are a perfect example of sites that people will enter their password without much familiarity aside from a tweet referral.--------------------So there you have it. If you wake up one morning and everyone is asking how and why you checked into a Nigerian Hotel, forgot your wallet, ran up a $1500 bill, and now need money, you'll know how to handle it. :) And definitely feel free to connect with Matt on Twitter. It's ok to joke around with him about the situation. He's a nice guy and has handled the jokes with grace. But don't go too far, he can bench press 375 pounds and used to be a minor league hockey player. ;-)GG

  • The morning of March 2nd started out fairly normally. I moved quickly to get ready and jumped on my computer to check email, twitter, my feeds, etc. So I launched Outlook and happened to see an email from one of my good friends Matt Leonard (or @mjleonard if you are on Twitter). He's a great guy and a smart marketer so we're in touch often. The email was sent from Matt's gmail account and the subject line communicated the urgency of his message.Apparently, Matt flew to Nigeria the night before to attend the Tinapa Opening Ceremony. {???} He ended up staying at a hotel, which was attacked by armed robbers. {OK...} They took all of his money and his wallet. {???} His email explained that he needed money badly, to the tune of $1500 so he could settle his hotel bill. {LOL} He seemed very scared and he emphasized that he needed the money sent as soon as possible. As you can imagine, I was genuinely concerned for Matt... {sarcasm}Yes, Matt's gmail account had been hacked and I'm sure many people received the same email I did. Here was the original email:---------- Forwarded message ---------- Subject: *****URGENT REPLY NEEDED ********* Hello I am in a hurry writing this message, I had a trip to West Africa *NIGERIA* on visiting the TINAPA OPENING CEREMONY, Unfortunately for me all my money got stolen at the hotel where i lodged from the attack of some armed robbers.Right now,I'm without money and I'm even owing the hotel here, the hotel telephone service is disconnected,i have only access to emails,my mobile phone can't work here so i didn't bring it along, please can you lend me $1500 so i can return back and settle the hotel bills i would return it back to you as soon as i get home, I am so confused right now.You can have it sent through western union money transfer. My passport is with the Embassy here so i cant use my name to collect it now, But you can have it sent directly to a western union Manager here and i would get it through him as he helps the people here in the hotel to receive the money, I have already spoken to him, please let me hear from you so i can collect his full name and address where you can send the money tomorrow please,or if possible today. I am waiting for your reply.Thank you. I look forward to your positive response.Regards.-----------------OK, this was absolutely ridiculous, but being the good friend I am, I still wanted to reach out to Matt as soon as possible to let him know what was going on. I was pretty confident he didn't know yet. Now, I didn’t know if his actual account was taken over. I simply thought that someone was blasting out emails using his address as the sender. There’s a difference between the two. So, I created a new email (which was sent to his gmail account) informing him that his email address had been compromised. Unfortunately, that was the only email address I had for Matt...It was a brief email that looked like this:--------------------------Hi Matt.I just received a spam email from your gmail acct. Not sure if it'swidespread or not, but wanted to let you know.It was one of those Nigerian schemes.Let me know if you have any questions.Glenn-----------------------It was only a few minutes when I received an email back from Matt. My guess was that he wanted to quickly thank me for notifying him of the scam. I opened the email and was shocked to see that the original email from Matt wasn't a joke! Matt was in fact in Nigeria at a hotel and needed money. {sarcasm yet again}You see, “Matt” actually responded to my email!Holy smokes, the scammers who hacked Matt's gmail were answering emails right from his account! I'll admit it, that creeped me out. Read their response to my email below:---------- Forwarded message ----------Subject: Re: Email SpamTo: Glenn GabeHello,Its not a scam mail, i am really there now. i need you to help me with $1500, i shall pay you as soon as a i return. i hope to read from you asap. below is the info where the money will be sent to....------------------So, I direct messaged Matt via Twitter notifying him of the hack, this time using only 140 characters or less. :) Before long, I received a DM back from Matt that he was handling the situation. He thanked me, and got back to fixing the problem. So, after Matt recovered his gmail account (and after he got pummeled by jokes on Twitter), I asked him about the steps he took to recover his gmail account.After this happened to Matt, I asked myself if I would know what to do... I really didn't. I'm sure I would eventually figure it out, but I didn't know the exact steps. So with Matt's assistance, I decided to write this post to document the steps you should take to recover your gmail account after it has been compromised.So in Matt’s own words, here’s how to handle the situation:The process was pretty simple.First, from the Gmail login page, select "I can not access my account".Second, select the radio button "My account has been compromised". This will show a new prompt below "Please fill out our account recovery form to help us process your request as quickly as possible". Follow that link.From the Account Recovery page, select "I believe someone has taken over my account" and fill out as much information as possible.In my case, Google had contacted me pretty quickly via my alternate email with instructions to reset my password. {Glenn: Make sure your alternate email is active. You can check this from your Google account settings. You can also add additional email addresses just in case.}Some other useful notes from Matt:I'm not sure how someone hacked my gmail account. I did make a mistake by using the same password on way too many things. My main passwords are now individually unique. Passwords I set up for vendor accounts are no longer the same as I would use for other more sensitive services, like online banking.Matt’s Tips to Protect to Your Logins:1. Avoid unsecured networks2. Don't use the same password everywhere.3. Don't duplicate your email password with any login elsewhere (if you do, someone can use your email to login to your other accounts).4. Change passwords periodically.5. Don't share your passwords with anyone. Even if they're not going to abuse it, they may not store it properly.6. Know where you enter passwords. 3rd party Twitter apps that require passwords are a perfect example of sites that people will enter their password without much familiarity aside from a tweet referral.--------------------So there you have it. If you wake up one morning and everyone is asking how and why you checked into a Nigerian Hotel, forgot your wallet, ran up a $1500 bill, and now need money, you'll know how to handle it. :) And definitely feel free to connect with Matt on Twitter. It's ok to joke around with him about the situation. He's a nice guy and has handled the jokes with grace. But don't go too far, he can bench press 375 pounds and used to be a minor league hockey player. ;-)GG

  • En aquesta època nadalenca molts envien targetes de nadal o d'any nou impreses. Si ets dels que regalen aquestes targetes de nadal, Gmail ens presenta una sèrie de targetes molt originals de nadal per imprimir gratuïtes. En total són 6 dissenys diferents que podràs descarregar en format PDF. Cada targeta ve marcada amb punts per retallar i doblegar-la. Després d'imprimir les teves targetes de nadal, escriu el que vulguis, enganxa-li un segell i envia-la per correu tradicional o bé pots donar-se-la en persona als teus amics. :-)Via: WebadictosEnllaç: Gmail Christmas Card (Afegit al Blogroll Nadal - Postals)Actualitat

  • There is an easy workaround if you want to export your contacts to another GMail account, or any account that auto-adds your recipients to contacts. (Gmail does by default.)

    On the account that you want to export, click on "contacts", then click on "all contacts" then select "all". Click "compose" and all your contacts will be in the TO: field. Copy them to clipboard and sign out of your account. Sign into the account where you wish to upload your contacts. Click on "compose" and paste your contacts into the "bcc" field and send a "new addy " note to all your contacts. (Put your own addy in the TO;) They will be auto-added to the contacts in your new account. Be sure to BCC! :)

  • Always include:Device:Carrier:Country / Language:OS / Browser / build number (if applicable):
    Question from
    tammybattista1
    in
    Google Mobile
    -
    BlackBerry Devices

    No replies yet

  • Always include:Device:Carrier:Country / Language:OS / Browser / build number (if applicable):
    Question from
    tammybattista1
    in
    Google Mobile
    -
    BlackBerry Devices

    No replies yet

  • It’s really big problem to manage contacts in your address book,but thanks to Google because it has added a new feature in Gmail which was most wanted feature of Gmail users.
    Now you can manage your address book in your Gmail.You can merge all duplicate contacts in your Gmail address book with a single click.
    “We’ve added a single button that merges all your duplicate contacts at once. To clean up your contact list in one [...]

  • I am implementing the senior project and I have problems in login page and signup page.

    Because my project uses Google Calendar service and my login page uses gmail username and password to request the service. So in my signup page, I allow users to type their gmail username(but not their password) and keep in the database.

    The problem is that I would check or verify the gmail user name that they type in the signup page whether it exist in the gmail or not.

    Could anyone give me the solution please?

    Ps.I implemented this by using Visual studio 2008,C#, and ASP.NET

  • Firefox with Greasemonkey: Gmail recently made a few tweaks to its message count API, leaving tools like the favicon message indictors by Eric Bogs and Peter Wooley, along with Gina’s own Better Gmail 2 Firefox extension, broken. Now they’re fixed.
    Bogs and Wooley, whose clever user scripts cleverly show your unread message count in your Gmail [...]

  • Exclusive Lifehacker download for Windows: Gmail Keys adds Gmail-style keyboard shortcuts to Microsoft Outlook—saving you time when dealing with email overload.


    Once launched, you can start using the shortcut keys immediately—just start out with the J or K keys to move up and down in the list, use the O key to open the message or the / key to search. They should work almost exactly like the Gmail keyboard shortcuts. The utility should run completely in the background, and you can add it to your startup folder to keep it running after reboot.
    The one special case is enabling the archive key, which will require installing the included Outlook macro using our guide to adding a Gmail-like archive button to Outlook. When creating the button, add &. to the button text, which will enable the trigger. For instance, if your button is called Archive you would rename it to Archive&. (with the ampersand and period).
    Included Shortcut Keys:
    f: Forward message.
    r: Reply to the message.
    a: Reply to All.
    /: Focus the instant search box.
    c: Open a new message.
    j: Move down in the list to the next message.
    k: Move up in the list to the next message.
    o: Open the message.
    Shift+j: Move down in the list, selecting the next message as well.
    Shift+k: Move up in the list, selecting the next message above as well.
    Shift+i: Mark the message as read.
    Shift+u: Mark the message as unread.
    s: Mark the message as flagged.
    v: Open the Move message dialog.
    Period(.): Open the message context menu.
    y: Archives the Message (using a custom Outlook button)
    All credit goes to Lifehacker reader jayp, who not only came up with the idea and put together the original script but graciously released it as open-source so we could provide it for you here. Thanks!
    Gmail Keys for Outlook is a free download, Windows only. If you are an AHK user, you can grab the source code from the download folder and paste it into the script that you use daily. Otherwise, just launch the executable and optionally add it to your startup folder.
    Release History:

  • Official Gmail Blog: Offline Gmail graduates from LabsI have always been a fan of gmail offline since it was launched. Gmail offline has helped me a lot while travelling or when the internet connection was not avaliable.I am glad that now gmail offline has graduated from the gmail labs and become a regular gmail feature.Way to go Gmail..Thumbs up Google!!

  • Firefox: Gmail recently made a few tweaks to its message count API, leaving tools like the favicon message indicators by Eric Bogs and Peter Wooley, along with Gina's own Better Gmail 2 Firefox extension, broken. Now they're fixed.

    Bogs and Wooley, whose clever user scripts cleverly show your unread message count in your Gmail or Google Apps mail's browser tab icon, both updated their code to adapt to Gmail's API changes. Gina, whose Better Gmail 2 extension utilizes Bogs' and Wooley's scripts, has re-rolled those changes into her extension.
    Those with Better Gmail 2 already installed should see update notices soon. If you're seeing only installation errors instead, hit the link for a direct (but non-updating) link. If you're rocking just Greasemonkey and one of these two scripts, head to one of their UserScripts.org pages and click "Install" to update your browser.
    Better Gmail 2's Favicon Unread Count Fixed [Smarterware]
    Gmail Unread Message Count in Favicon for Greasemonkey
    Gmail Favicon Alerts 3 for Greasemonkey [UserScripts.org]

  • People sometimes ask me how to target Gmail as a placement in the Content Network.
    This video shows you how:

    Also, there's a premium placement in Gmail called the "Funbox". You can target your ads specifically for that spot. The last minute of this video tells you how to do that too.
    Google's "Funbox" is the blue area below the search box.
    Cool beans! Thanks to Austin Lau from Google for making this video. And thanks to Shelley Ellis for bringing it to my attention.
    Perry Marshall