What is particularly interesting and notable about this outage, in my opinion, is the set of lessons we in the industry can learn about putting our eggs in such a basket, what "high-availability" really means, the dangers of "sorcerer's apprentice synrome" and "auto-immune" vulnerabilities in redundancy engineering, and how to maintain a high level of service in this age of "cloud computing." People are still arguing and wanking about where to place the blame for all of the havok that this incident wreaked upon the internet, but the plain truth is that there's more than enough blame to go around for everyone -- the web sites and service providers, as well as Amazon itself. On one side of things, it's true that engineers and administrators should have spread deployments across multiple AWS regions (not just availability zones). On the other side of things, AWS has made it difficult to use multiple AWS regions, had indeed maintained that spreading deployments across availability zones would provide adequate insurance against an outage -- and it turned out that in this case they were very very wrong, and pushed a new cloud storage service (EBS) that proved to be even more unreliable and in many cases incompatible with the possibility of using multiple AWS sites.

Here's a quick rundown:

• Amazon outage and the auto-immune vulnerabilities of resiliency
• Amazon EC2 outage: summary and lessons learned
• Heroku status and post-mortem from the AWS outage -- an illustration of how NOT to do things if you want reliability, but an admirable case of owning up to and being honest about your faults.
• How SmugMug survived the Amazonpocalypse -- an illustration of how you SHOULD do things if you want reliability from a company that used AWS but was able to stay up.

No, in the end it turns out that it wasn't Skynet's fault after all. Just some over exuberance about new hottness, and a distinct deficit in reliability-engineering and availability paranoia.

So I have some thoughts about the new CDMA iPhone (compatible with Verizon's network) released today that folks have been frothing about. Everybody knew that the AT&T exclusivity deal with Apple was ending this year, so it's inevitable that other carriers -- particularly Verizon, would eventually end up with an iPhone to sell to their customers.

The issue with Verizon, however, is that their CDMA network is incompatible with the electronics hardware in the iPhone for AT&T. This means that making an iPhone available to Verizon customers was never just a matter of flipping a switch somewhere, or installing some code in firmware. New electronics had to be designed to be integrated with and fit into the quite constraining iPhone form factor, and of course tested and approved by various regulatory officials.

CDMA, however, is on its way out the door. Verizon, AT&T, and other carriers, are already rolling out so-called 4G networks that will be wholly incompatible with already-odd-duck CDMA (the network, for example, lacks the ability to transmit voice and data at the same time -- no web surfing while talking).

My question is then, is it really worth it for Verizon (or Apple for that matter) to go through the effort and expense of designing and rolling out this new phone on an old, deprecated network? Apple COO Tim Cook reportedly said that building it as a 4G LTE phone would "force some design compromises" and "customers have told us they want the iPhone now." So for those still on Verizon's network, wanting an iPhone, but unwilling/unable to switch to AT&T, my advice would still be to wait. I'd hope that this little side detour into CDMA 3G-land doesn't delay the eventual release of a 4G LTE iPhone for both AT&T and Verizon in the near future.

• V - This Blog: With the advent of twitter (see notamateurhour) this blog has been lacking in content. Turns out it's much easier to just sit out a few lines of text regularly than it is to construct several paragraphs worth of content that's worth reading. We'll see if that changes.
• V - Hanging Out: With only very rare exceptions, I've been lacking in the hanging-out department for the past two or three months for various reasons (work, laziness, etc.). I think this is going to be my new years resolution: "hang out more." I don't remember the last time I made a run for the Border Cafe or saw some good pickin' at the Cantab lounge with my old homies.
• ^ - Work: I did some pretty neat things this year, and enjoyed doing them. A massive mysql upgrade and migration went off with minimal outage -- onto solid state disk hardware, which has worked out pretty sweet. I also did a whole bunch of fiddling and reworking with our DNS system for added speed and reliability, and built tools for making database reslaves an order of magnitude faster by using LVM snapshots. All in all, I'd say probably well worth the raise and titular promotion that I got this quarter. I'm not going to deny that there have been issues at Linden this year, and it hasn't been the happiest of times morale-wise, but from where I sit things seem to be looking up.
• V - Red Sox: A grim 2010 for the boys of summer. 'Nuff said. But I'm definitely looking forward to the Sox of '11.
• ^ - Green Lifestyle: With Kristy no longer needing the car to drive to rotations in Worcester or Cambridge, there was no logical reason remaining to own a car while living in the city. $250/month for parking plus $80/month for insurance plus the hundreds of dollars that the car was going to potentially start costing us in maintenance to get it fixed and keep running is far from worth it for something you only drive maybe once a week or so. Zipcar is more than affordable and convenient enough for those occasional jaunts to Costco or the mall, and getting back and forth between Boston and Cambridge is just a quick ride on the CT2 or 47 MBTA bus.
• ^ - Burning Man: This year I did Burning Man for my first, and definitely not last, time. It was an awesome experience pretty much beyond words. Hopefully I'll find the time to write a bit about it here and post some pictures, but you really have to go for yourself. Definitely one of the highlights of 2010 for me.
• = - Life in General: 2010 has been kind of a doozy of a year. Lots of stuff going on, things done, places gone to, lessons learned. Whew, I feel kind of tired just thinking about it all. I met a few awesome new friends, and said farewell to a few as well as they headed out of town and on with their lives. But I guess that's how the whole durned human comedy keeps perpetuatin' itself, down through the generations, westward the wagons, across the sands o time until -- aw, look at me I'm ramblin' again. Well, I hope you folks enjoyed yourselves this year, and here's to a happy and awesome 2011.

Yes. I want one. Right Now.
From the Chicago Tribune's review:

...a bacon and cheese sandwich where bread is replaced with fried chicken. Re-read that last sentence, and just try to grasp its cultural significance. The toothpaste is out of the proverbial tube.
...Double Down's larger implication is that KFC has broken through a barrier of culinary decency, besmirching the good name of sandwiches and all that is honorable... The absence of bread robs this “sandwich” of dignity.
All told, the ripples from this landmark product launch will be significant. Stand-up comedians will construct 10-minute riffs around this. Taco Bell will follow suit and introduce a taco called the Carne Go-Torta, where the tortilla shell is replaced with a meat sleeve. Freedom-loathing terrorists will hate us for this. I can see it now.

America, just because you can, it doesn't mean you have to.

Made of awesome. Full of win. Nothing more needs to be said.

New A-Team Trailer Increases Nostalgia Factor by Roughly 300 percent

MySQL 5 Upgrade

Up until this year, our mysql.agni master was running version 4.1. Shortly after I started at Linden Lab, there was a project to upgrade the entire tree to version 5.0. Our resources were limited back then, and the operation was a total failure. It turns out that MySQL 5.0 has a much different disk i/o usage profile for our query rate and load type. When the master was upgraded to version 5.0, it promptly collapsed under heavy disk i/o load and unacceptable response times. Worse still, we did not have a proper version 4.1 host slaving off of the new master, so there was extended downtime and some data loss as we failed back to the older version and had to replay queries from binary logs.

Fast forward to the end of 2009, and we were much smarter, and better equipped to upgrade. By this point, all of the slaves in the tree had been upgraded, and it was becoming difficult and confusing to maintain a mixed-version slaving tree, with some on version 4 and some on version 5. We were operating on upgraded hardware with more capacity and headroom -- particularly concerning disk i/o.

My former co-worker, Charity Majors, authored an extremely thorough public blog post concerning this upgrade, and our planning to make it come off without a hitch. We built a robust load-testing and analysis rig, and put the new version and hardware through its paces multiple times, in multiple ways, to make sure that we'd be OK this time. The major difference was our consultation with Percona and our decision to use one of their high-performance tuned builds of MySQL. In addition, we came up with an awesome plan for performing the upgrade, and a fallback mechanism if we needed it.

The trouble with fallback in this case is that normally, it's not possible to hook up a MySQL version 4.1 slave off of a 5.0 master. This means that once we upgraded to the new master and had writes and updates going to it, we'd have no version 4.1 server to fall back to without losing data. Charity was able to rig up a script that rotated the binary logs on the new 5.0 master every 5 minutes or so, and sent the contents of the log over the network to a fallback host, where we would feed the binary log queries (via mysqlbinlog piped into mysql and various filters to compensate for 4.1/5.0 incompatibilities) into the database and get it up to date and ready to fail over to if we had issues with version 5.0. Of course, performance wasn't our only concern, and there were millions of lines of code just waiting to bite us in the butt and potentially require falling back.

On January 5th, 2010, all went as well as could have possibly been imagined. We cut over to the version 5.0 master, and there were no issues. It's so nice to finally be on a relatively recent version of the software, and more importantly, to have our master and slaves all be on the same version. And so our march onwards continues towards ever-improved performance and stability!

Age 62, of Shorewood, passed away peacefully Friday, March 5, 2010 at University of Chicago Medical Center.

Born August 23, 1947 in Chicago, he enlisted in the U.S. Air Force and served during the Vietnam War. He worked as a Project Manager and Communications Instructor for Ameritech.

His family was his life.

Surviving are his wife, Kathy A. (nee Kupka) Raven; his children, Richard James (Jill) Raven, Cherie (Brian Chapin) Raven, and Kristin Ellen (Ben O'Connor) Raven; four grandchildren, Brook Elizabeth and Richard John Raven, and Audrey Katherine and Edward Michael Chapin; one brother, Daniel Joseph Raven; and several nieces and nephews.

Preceded by his parents, Stanley and Dorothy (nee Coakley) Raven; and three siblings, Raymond, Robert, and Dorothy Raven.

Funeral services for Richard M. Raven will be Wednesday, March 10, 2010 at 11:00 a.m. at the funeral home chapel. Internment will be in Resurrection Cemetery, Romeoville. In lieu of flowers, memorials to University of Chicago Cancer Research Center, 5841 S. Maryland Ave MC 1440, Chicago, IL 60637 would be appreciated. Visitation Tuesday, from 3-8 p.m. at:

Fred C. Dames Funeral Home 3200 Black at Essington Rds., Joliet (815) 741-5500 www.fredcdames.com

"The weird sense of duty really good sysadmins have can border on the sociopathic, but it's nice to know that it stands between the forces of darkness and your cat blog's servers."

Ingredients:

• ~3 strips bacon (I use precooked for convenience)
• 2 slices of bread
• Some peanut butter

Lightly toast two slices of bread. Spread peanut butter on the toasted bread. I like to put three half-slices of cold, pre-cooked bacon on each slice and then put it back in the toaster oven for a few minutes. If you're using regular bacon, or don't have a toaster oven, I suppose you could cook up the bacon in a skillet. Put the two pieces of the sandwich together, let it cool down for a bit, and then enjoy. Yum-tastic!