Into The Weeds

If you're the least bit involved with the business-end of internet services (as I certainly am), you'll have already heard that a few weeks ago, Amazon Web Services (AWS)[aws.amazon.com] suffered a major outage of an east-coast region of their platform. This outage caused serious issues and downtime for many other internet systems (including, but not limited to: reddit, foursquare, heroku, quora, and my employer Linden Lab / Second Life) and services that have come to rely on AWS over the past few years as a reliable provider of what have become known as "cloud computing services." [wikipedia] This is their offical post-mortem of the incident.

What is particularly interesting and notable about this outage, in my opinion, is the set of lessons we in the industry can learn about putting our eggs in such a basket, what "high-availability" really means, the dangers of "sorcerer's apprentice synrome" and "auto-immune" vulnerabilities in redundancy engineering, and how to maintain a high level of service in this age of "cloud computing." People are still arguing and wanking about where to place the blame for all of the havok that this incident wreaked upon the internet, but the plain truth is that there's more than enough blame to go around for everyone -- the web sites and service providers, as well as Amazon itself. On one side of things, it's true that engineers and administrators should have spread deployments across multiple AWS regions (not just availability zones). On the other side of things, AWS has made it difficult to use multiple AWS regions, had indeed maintained that spreading deployments across availability zones would provide adequate insurance against an outage -- and it turned out that in this case they were very very wrong, and pushed a new cloud storage service (EBS) that proved to be even more unreliable and in many cases incompatible with the possibility of using multiple AWS sites.

Here's a quick rundown:

• Amazon outage and the auto-immune vulnerabilities of resiliency
• Amazon EC2 outage: summary and lessons learned
• Heroku status and post-mortem from the AWS outage -- an illustration of how NOT to do things if you want reliability, but an admirable case of owning up to and being honest about your faults.
• How SmugMug survived the Amazonpocalypse -- an illustration of how you SHOULD do things if you want reliability from a company that used AWS but was able to stay up.

No, in the end it turns out that it wasn't Skynet's fault after all. Just some over exuberance about new hottness, and a distinct deficit in reliability-engineering and availability paranoia.

Looks like they're assembling the new scoreboard at Fenway Park. It appears to be roughly the same size as the entire old structure, but consumed entirely by the digital board rather than surrounded by advertisements (for now at least). Nifty.

So I have some thoughts about the new CDMA iPhone (compatible with Verizon's network) released today that folks have been frothing about. Everybody knew that the AT&T exclusivity deal with Apple was ending this year, so it's inevitable that other carriers -- particularly Verizon, would eventually end up with an iPhone to sell to their customers.

The issue with Verizon, however, is that their CDMA network is incompatible with the electronics hardware in the iPhone for AT&T. This means that making an iPhone available to Verizon customers was never just a matter of flipping a switch somewhere, or installing some code in firmware. New electronics had to be designed to be integrated with and fit into the quite constraining iPhone form factor, and of course tested and approved by various regulatory officials.

CDMA, however, is on its way out the door. Verizon, AT&T, and other carriers, are already rolling out so-called 4G networks that will be wholly incompatible with already-odd-duck CDMA (the network, for example, lacks the ability to transmit voice and data at the same time -- no web surfing while talking).

My question is then, is it really worth it for Verizon (or Apple for that matter) to go through the effort and expense of designing and rolling out this new phone on an old, deprecated network? Apple COO Tim Cook reportedly said that building it as a 4G LTE phone would "force some design compromises" and "customers have told us they want the iPhone now." So for those still on Verizon's network, wanting an iPhone, but unwilling/unable to switch to AT&T, my advice would still be to wait. I'd hope that this little side detour into CDMA 3G-land doesn't delay the eventual release of a 4G LTE iPhone for both AT&T and Verizon in the near future.

The weather service has issued a Blizzard warning for Boston (among other places) this afternoon. Looks like things are starting to pick up out there. It's not officially a blizzard until the Citgo sign (<1/4 mile away) is no longer visible from my window though. (second picture added at 4pm)

• V - This Blog: With the advent of twitter (see notamateurhour) this blog has been lacking in content. Turns out it's much easier to just sit out a few lines of text regularly than it is to construct several paragraphs worth of content that's worth reading. We'll see if that changes.
• V - Hanging Out: With only very rare exceptions, I've been lacking in the hanging-out department for the past two or three months for various reasons (work, laziness, etc.). I think this is going to be my new years resolution: "hang out more." I don't remember the last time I made a run for the Border Cafe or saw some good pickin' at the Cantab lounge with my old homies.
• ^ - Work: I did some pretty neat things this year, and enjoyed doing them. A massive mysql upgrade and migration went off with minimal outage -- onto solid state disk hardware, which has worked out pretty sweet. I also did a whole bunch of fiddling and reworking with our DNS system for added speed and reliability, and built tools for making database reslaves an order of magnitude faster by using LVM snapshots. All in all, I'd say probably well worth the raise and titular promotion that I got this quarter. I'm not going to deny that there have been issues at Linden this year, and it hasn't been the happiest of times morale-wise, but from where I sit things seem to be looking up.
• V - Red Sox: A grim 2010 for the boys of summer. 'Nuff said. But I'm definitely looking forward to the Sox of '11.
• ^ - Green Lifestyle: With Kristy no longer needing the car to drive to rotations in Worcester or Cambridge, there was no logical reason remaining to own a car while living in the city. $250/month for parking plus $80/month for insurance plus the hundreds of dollars that the car was going to potentially start costing us in maintenance to get it fixed and keep running is far from worth it for something you only drive maybe once a week or so. Zipcar is more than affordable and convenient enough for those occasional jaunts to Costco or the mall, and getting back and forth between Boston and Cambridge is just a quick ride on the CT2 or 47 MBTA bus.
• ^ - Burning Man: This year I did Burning Man for my first, and definitely not last, time. It was an awesome experience pretty much beyond words. Hopefully I'll find the time to write a bit about it here and post some pictures, but you really have to go for yourself. Definitely one of the highlights of 2010 for me.
• = - Life in General: 2010 has been kind of a doozy of a year. Lots of stuff going on, things done, places gone to, lessons learned. Whew, I feel kind of tired just thinking about it all. I met a few awesome new friends, and said farewell to a few as well as they headed out of town and on with their lives. But I guess that's how the whole durned human comedy keeps perpetuatin' itself, down through the generations, westward the wagons, across the sands o time until -- aw, look at me I'm ramblin' again. Well, I hope you folks enjoyed yourselves this year, and here's to a happy and awesome 2011.

Yes. I want one. Right Now.

From the Chicago Tribune's review:

...a bacon and cheese sandwich where bread is replaced with fried chicken. Re-read that last sentence, and just try to grasp its cultural significance. The toothpaste is out of the proverbial tube.

...Double Down's larger implication is that KFC has broken through a barrier of culinary decency, besmirching the good name of sandwiches and all that is honorable... The absence of bread robs this “sandwich” of dignity.

All told, the ripples from this landmark product launch will be significant. Stand-up comedians will construct 10-minute riffs around this. Taco Bell will follow suit and introduce a taco called the Carne Go-Torta, where the tortilla shell is replaced with a meat sleeve. Freedom-loathing terrorists will hate us for this. I can see it now.


America, just because you can, it doesn't mean you have to.

After seeing black smoke outside the window, I snapped a couple of photos of what's going on in the Back Bay Fens now. Those flames look to be at least 50 feet high. Whatever it is, it seems mostly contained now, with some flareups. The fire department is referring to it as a "large brush fire" over their radios (I'm listening to the scanner).

Sox vs. Yankees tonight at Fenway. Here's the view outside my window right now:

So, I learned about this today. And was giddy like a schoolgirl after watching the trailer (below).

Made of awesome. Full of win. Nothing more needs to be said.

New A-Team Trailer Increases Nostalgia Factor by Roughly 300 percent