According to the official Gmail blog, Ben Treynor, VP Engineering and Site Reliability Czar, claims that the reason Gmail was down for roughly 100 minutes yesterday was due to a portion of servers being taken offline for upgrades. The traffic, redirected to the remaining servers, were able to cope with the load but the routers responsible for directing the web queries to the servers couldn’t handle the additional responsibility.

From the blog post:

Here’s what happened: This morning (Pacific Time) we took a small fraction of Gmail’s servers offline to perform routine upgrades. This isn’t in itself a problem — we do this all the time, and Gmail’s web interface runs in many locations and just sends traffic to other locations when one is offline.

However, as we now know, we had slightly underestimated the load which some recent changes (ironically, some designed to improve service availability) placed on the request routers — servers which direct web queries to the appropriate Gmail server for response. At about 12:30 pm Pacific a few of the request routers became overloaded and in effect told the rest of the system “stop sending us traffic, we’re too slow!”. This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded. As a result, people couldn’t access Gmail via the web interface because their requests couldn’t be routed to a Gmail server. IMAP/POP access and mail processing continued to work normally because these requests don’t use the same routers.

Even though the service is free (to most – I have an account which I pay for) I can’t see this as being an acceptable answer. If you are the “VP Engineering and Site Reliability Czar” you, or the people under you, should have been able to predict what would happen in this exact scenario. This is what labs and load testing is for.

I think what bugs me most about this whole ordeal is that Treynor states that the service was unavailable for “about 100 minutes” but then concludes his post with “Gmail remains more than 99.9% available to all users, and we’re committed to keeping events like today’s notable for their rarity.”

Let’s do some basic math here folks. Google tells me that:

1 year = 525 948.766 minutes

If 99.99% is the availability figure then 00.01% must be the acceptable outage window right?

00.01% of 525,948.766 minutes = 52.5948 minutes per year.

Stated 100 minutes of outage – Allowed 52.5948 minutes per year = 47.4052 minutes of availability that we are all owed back from Google.

Based on my complicated calculations it appears as though Google has exceeded it’s %99.99 uptime mandate. I know the people at Google are smart but I don’t think they’re smart enough to rewrite the laws of basic mathematics. But maybe they are smarter than me as I was unable to find anything that stated that the 99.99% figure was a per year calculation. If the 99.99% uptime is spread over 100 years then, yes Google, you are still justified in your uptime calculations. If it is a yearly figure, however, I want to know how I go about recouping my 47.4052 minutes of availability that I am owed…I’m sure you’ll let me know.

UPDATE – Well I guess Google doesn’t owe me anything because they are only stating 99.9% uptime. My mistake. I have also found out that, based on the Google Apps SLA, they are stating 99.9% availability per month – http://www.google.com/apps/intl/en/terms/sla.html…very tricky 🙂

I had an idea early this morning that may or may not work and may or may not have been attempted before. Frankly, if it has been done before, it hasn’t been done in a while so it’s time to kick it off again. In an effort to get to know more about my peers and friends I’m going to start the ball rolling on the “5 Things You Might Not Know About…” project. The rules:

1. Create a blog post with the title “5 Things You Might Not Know About YOURNAME” (where YOURNAME is your first and last name).
2. List 5 things that people may or may not know about you (it can be anything really).
3. “Tag” 5 other people to do the same via the blog post, twitter, facebook, or all of the above.
4. See what happens.

So here are the 5 Things You Might Not Know About Andrew Hay:

1. At my 8th grade graduation ceremony the mother of one of my best friends thought that I was her sons teacher.
2. I was on the swim team in high school and made it to the city finals…whilst wearing a speedo.
3. My wife got rid of my high school speedo and it took me 2 years to realize it was gone. Note: This explains why the threat of wearing my speedo out in public no longer phased her.
4. I am an avid Rugby Union fan and have never broken any of MY bones playing Rugby.
5. I am a college drop out and do not have a degree or diploma.

Hopefully this gets the ball rolling. I’m going to tag the following people in the hopes that they join in on the insanity: Michael Santarcangelo, Justin Foster, Anton Chuvakin, Jennifer Jabbusch, and Erin Jacobs.

Here is a great, and scary, blog post from the folks over at Offensive Security. It details how easy it is to own a fully patched Windows 2000 SP4 server that has the Microsoft IIS FTP 5.0 remote system exploit. From the blog post:

A quick examination of the exploit showed some fancy manipulations in a highly restrictive environment that lead to a”useradd” type payload. The main issue was the relatively small payload size allowed by the SITE command, which was limited to around 500 bytes.

After a bit of tinkering around, we saw that the PASSWORD field would be most suitable to shove a larger payload (bindshell). A quick replacement of the original “user add” shellcode with a secondary encoded egghunter – and a bind shell was presented to us! I wonder how long this 0day has been around…As Rel1k would say to logan_WHD…”it’s OK, it’s OK…”.

The exploit can be downloaded from our exploit archive. To entertain the masses, we also made “Microsoft IIS 5.0 FTP 0 Day – The movie“

The movie can be found here: http://www.offensive-security.com/videos/microsoft-ftp-server-remote-exploit/msftp.html