Into The Weeds

Interesting MySQL Projects at Linden Lab Part 1: MySQL Upgrade

2010-03-08T17:51:17Z

People have expressed interest in seeing some more details about our use of MySQL at Linden Lab, so I shall indulge in the next couple of posts. Over the past few months, I have been involved in two major projects involving our central MySQL database cluster. The Second Life central database cluster (known affectionately as mysql.agni) actually consists of a single read/write master database server, with a tree of somewhere around 15 read-only slaves hanging off of it that are split into groups of various purposes (some behind load balancers) and serve the bulk of the queries that make Second Life work.

MySQL 5 Upgrade

Up until this year, our mysql.agni master was running version 4.1. Shortly after I started at Linden Lab, there was a project to upgrade the entire tree to version 5.0. Our resources were limited back then, and the operation was a total failure. It turns out that MySQL 5.0 has a much different disk i/o usage profile for our query rate and load type. When the master was upgraded to version 5.0, it promptly collapsed under heavy disk i/o load and unacceptable response times. Worse still, we did not have a proper version 4.1 host slaving off of the new master, so there was extended downtime and some data loss as we failed back to the older version and had to replay queries from binary logs.

Fast forward to the end of 2009, and we were much smarter, and better equipped to upgrade. By this point, all of the slaves in the tree had been upgraded, and it was becoming difficult and confusing to maintain a mixed-version slaving tree, with some on version 4 and some on version 5. We were operating on upgraded hardware with more capacity and headroom -- particularly concerning disk i/o.

My former co-worker, Charity Majors, authored an extremely thorough public blog post concerning this upgrade, and our planning to make it come off without a hitch. We built a robust load-testing and analysis rig, and put the new version and hardware through its paces multiple times, in multiple ways, to make sure that we'd be OK this time. The major difference was our consultation with Percona and our decision to use one of their high-performance tuned builds of MySQL. In addition, we came up with an awesome plan for performing the upgrade, and a fallback mechanism if we needed it.

The trouble with fallback in this case is that normally, it's not possible to hook up a MySQL version 4.1 slave off of a 5.0 master. This means that once we upgraded to the new master and had writes and updates going to it, we'd have no version 4.1 server to fall back to without losing data. Charity was able to rig up a script that rotated the binary logs on the new 5.0 master every 5 minutes or so, and sent the contents of the log over the network to a fallback host, where we would feed the binary log queries (via mysqlbinlog piped into mysql and various filters to compensate for 4.1/5.0 incompatibilities) into the database and get it up to date and ready to fail over to if we had issues with version 5.0. Of course, performance wasn't our only concern, and there were millions of lines of code just waiting to bite us in the butt and potentially require falling back.

On January 5th, 2010, all went as well as could have possibly been imagined. We cut over to the version 5.0 master, and there were no issues. It's so nice to finally be on a relatively recent version of the software, and more importantly, to have our master and slaves all be on the same version. And so our march onwards continues towards ever-improved performance and stability!

Richard M. Raven 8/23/47 - 3/5/10

2010-03-07T00:24:03Z

Richard Michael Raven

Age 62, of Shorewood, passed away peacefully Friday, March 5, 2010 at University of Chicago Medical Center.

Born August 23, 1947 in Chicago, he enlisted in the U.S. Air Force and served during the Vietnam War. He worked as a Project Manager and Communications Instructor for Ameritech.

His family was his life.

Surviving are his wife, Kathy A. (nee Kupka) Raven; his children, Richard James (Jill) Raven, Cherie (Brian Chapin) Raven, and Kristin Ellen (Ben O'Connor) Raven; four grandchildren, Brook Elizabeth and Richard John Raven, and Audrey Katherine and Edward Michael Chapin; one brother, Daniel Joseph Raven; and several nieces and nephews.

Preceded by his parents, Stanley and Dorothy (nee Coakley) Raven; and three siblings, Raymond, Robert, and Dorothy Raven.

Funeral services for Richard M. Raven will be Wednesday, March 10, 2010 at 11:00 a.m. at the funeral home chapel. Internment will be in Resurrection Cemetery, Romeoville. In lieu of flowers, memorials to University of Chicago Cancer Research Center, 5841 S. Maryland Ave MC 1440, Chicago, IL 60637 would be appreciated. Visitation Tuesday, from 3-8 p.m. at:

Fred C. Dames Funeral Home 3200 Black at Essington Rds., Joliet (815) 741-5500 www.fredcdames.com

Sense of Duty

2010-02-23T04:09:52Z

Randall Munroe of XKCD hits the nail on the head with this one:

"The weird sense of duty really good sysadmins have can border on the sociopathic, but it's nice to know that it stands between the forces of darkness and your cat blog's servers."

Peanut Butter + Bacon = win

2010-02-10T18:53:09Z

You like peanut butter? You like bacon? Why not put them together! I've been making peanut butter and bacon sandwiches for a while now, so here is my technique -- a special gift for you and whomever you wish to spread the knowledge to.

Ingredients:

~3 strips bacon (I use precooked for convenience)
2 slices of bread
Some peanut butter

Lightly toast two slices of bread. Spread peanut butter on the toasted bread. I like to put three half-slices of cold, pre-cooked bacon on each slice and then put it back in the toaster oven for a few minutes. If you're using regular bacon, or don't have a toaster oven, I suppose you could cook up the bacon in a skillet. Put the two pieces of the sandwich together, let it cool down for a bit, and then enjoy. Yum-tastic!

Watch Frontline This Week (Second Life)

2010-02-01T00:29:24Z

It appears that Tuesday's episode of Frontline will deal, at least in part, with Second Life. The trailer (embedded below) has some good bits in there about the use of SL for collaboration and both business and social/pleasure stuff. So, check your local PBS listings and set yer tivos and stuff!

The Golden Age Of Video

2009-12-09T07:31:13Z

"We came, we saw, we kicked it's ass!"

Trust me, it's catchy. Watch it twice.

Sloth and Goonies and ugliesttatoos.com

2009-11-25T19:30:17Z

From an entry on ugliesttatoos.com(site is somewhat NSFW but this entry isn't):

That said, in a few years there will be a generation of adults for whom The Goonies is not the nostalgic icon that it is for us. And those people are going to ask you why you have a tattoo of a retarded pirate on your arm. If there are any measures you can take to prepare yourself for this outcome, you should probably do it.

...and that's sad.

DNS Slugtalk

2009-11-24T20:05:51Z

For those who are not aware, when I went to MIT I lived in the East Campus dormitory, on the hall known as Second West (a.k.a. Putz). These days, Putz has a pretty neat group of people and an informal tradition of holding "slug talks," or opportunities for someone to give a brief presentation and have a chance to share some knowledge with the group. Most of these so far have been computery in nature, and specifically computer-sciency. I hear there's going to be a good one next week on neural plasticity and long term potentiation (LTP -- it's how we learn and form memories in our brains pretty much).

Since my day-to-day life deals with computer science in a more practical and hands-on way, the topics that I have chosen are more practical in nature. Last semester, I gave a talk about MySQL in production, and specifically how we use it at Linden Lab to make Second Life work.

Just last week, however, I gave a slugtalk describing how DNS works out here in the real world. I think it went pretty well. Download my slide deck as a PDF here.

R.I.P. Geocities

2009-10-26T18:29:02Z

"Founded in 1994 as Beverly Hills Internet, what is now Yahoo GeoCities was one of the first services to offer an easy way for early Internet surfers to publish their own Web pages."

GeoCities, once one of the 10 most visited domains of the entire internet, and repository of around 10 terabytes of people's home pages and information dating back to 1994 is shutting down today. Ken Gagne says all that needs to be said about it in his excellent ComputerWorld blog post today. Soon we'll be losing such gems as my high school friend Josh's Phiggity-Phat Home Page from c. 1996, featuring amazing gems on Fred Couples, David Carradine, Peter North, and the A-Team. Such a loss, indeed. Thankfully, the archive team is working on saving it. Also, check out XKCD's tribute to geocities in today's comic.

OMG! Virus! (c. 1988)

2009-10-06T23:36:10Z

An awesome retro news report from 1988 about the super scary Morris Worm. Centered on MIT, it features none other than jis himself a few times. And an amish part-time virus hunter cum MIT student? Courtesy of The Scottographer .

Some good quotes: "the students were safe ... their computers were not." "the suspect, somewhere.... a dark genius."

Best Purchases Ever Made

2009-10-02T03:56:33Z

Take the following with a grain of salt. I'm just kind of bored and figured it's been a while since I've blogged. Here are some experiences of mine (in chronological order) that might be construed as product and/or brand endorsements:

HP Color LaserJet 2550n - September 2005: Paying $500 for a color laser printer turned out to be a financially and technically sound decision after all. In the 4 years that this printer has been around, the toner cartridges have been replaced once, and the printer is still in 100% working order. Contrast that to any of the ink-jet printers out there that break after a year, and need $50 ink cartridge replacements regularly (even when they're not being used and just dry out). These days, however, it looks like consumer-grade color laser printing is starting to go down the ink-jet road with small-quantity metered toner cartridges and shoddily built hardware.
Dell 2405FPW - August 2005: Sure, these days 24" LCDs are pretty standard, but back then it was a pretty big leap. It still holds up beautifully 4 years later and has put several CRT screens to rest.
TiVo Series 2 DVR - December 2004: Just see this entry from 2006 for some of the reasons why TiVo is a life-changing experience. And no, I still don't mean the crappy standard HD DVR that the cable/satellite companies foist upon us. These days, however, Comcast (in Boston at least) offers a DVR with the actual TiVo software installed. I find it quite serviceable, and a definite improvement over the craptastic regular DVR, but it's still not the whole experience. The 'ol Series 2 is still going strong, and I'm considering upgrading to a TiVo HD cablecard unit at some point.
Power Mac G4 - October 2003: It's hard to believe that it was 6 years ago that I made the switch (see this blog entry from '06 for some pontification on the subject). That G4 tower has been such a horse that it's still sitting here under my desk in use as a secondary system for bittorrent clients, file serving, and DVD/CD burning. Unfortunately, with Snow Leopard's removal of support for the PPC architecture, the end of the road may be coming soon.
Ikea "Poang" Chair - August 2002: Who knew that a $70 wood and fabric chair could be so comfortable, durable and not too bad looking either?
Sony 24" wega TV - December 2000: Almost 9 years later and it's just as good as the day it arrived as a christmas present. I know, not a "purchase," but it deserves to be on here anyway. It's not HD, but has a great picture and I can easily see it lasting another 9 years.

Stuff Magazine: Hickman Mania?

2009-09-23T15:58:50Z

On my way into the office this morning I did something I usually don't do. I grabbed a copy of Stuff Magazine because on the cover was none other than everybody's favorite bassist with the stylin' threads and handlebar mustache, Zachariah Hickman:

Turns out there's a section inside about 11 Bostonians with signature looks, and of course he's featured prominently in it. He calls it: "cowboy suits, seersucker, and general haberdashery." General haberdashery, indeed. Figured my pickin' partners would enjoy this.

Flood: Turkish Datacenter Into the Weeds

2009-09-20T00:17:04Z

This video is pretty awesome in a "glad that's not me" and "I hope they have offsite backups" kind of way. At around 5:21 watch the plate glass window fail, and then at 6:10 the video switches to a camera in the machine room. Servers and water don't mix.

Your junk email, illustrated.

2009-09-14T03:58:59Z

Came across this set of illustrations by Elliott Burford tonight, and I found them too awesome not to share.

Spam: an ongoing project illustrating the titles of emails found in your spam/junk box.

Linden Lab (The Makers of Second Life)

2009-08-28T04:52:03Z

Those of you who know me, know that I currently work as a Systems Engineer for Linden Lab, and we make and run the biggest and best virtual world in existence -- Second Life. Unlike the real-world economy, the economy in SL is still growing by leaps and bounds. That link is to the Q2 economy report where, among other stats, it's pointed out that we did $50 million per month in user-to-user transactions (that's in real US$ not L$). That's an increase of 94% from last year. Virtual land ownership increased 11% from last year, and both user hours (126 million) and peak concurrent users (88,000) were all time highs. About 750,000 unique members logged into Second Life at least twice last month.

With resident-owned land growing by 7% last quarter to 1.74 billion square meters of land, SL is larger than Bahrain, Singapore, or Tonga. That population of around 750,000 would make it #161 in the world, right between Guyana and Bhutan. The size of the economy could be estimated as being bigger than Timor or Samoa, with a $560 million GDP ($765 per resident). See this interesting blog post for some more comparisons to the real-world.

Second Life isn't just for furry sex, goreans, flying penises, and selling virtual hair and clothes. Check out the multiple case studies from corporations and government agencies using the virtual world successfully for meetings, conferences, trainings, recruitings, prototyping, simulations, marketing, and tons of other things. One of my favorites is one that I got a chance to directly work on -- the story of how IBM built out a private grid and is holding successful conferences in-world. They've saved hundreds of thousands of dollars, and other companies have been jumping on the bandwagon.

I work in Systems Engineering in the Boston office of Linden Lab (we have offices in San Francisco, Seattle, Mountain View, Davis, Boston, Brighton (UK), Virginia, and Singapore). Our group is responsible for maintaining and scaling the systems that keep the grid running, and deploying and designing additional systems and services to keep the grid running well into the future. One other statistic that hits me closer to home is that we remained at nearly 99.9% uptime this past quarter, and had total downtime of 0.24% as a percentage of user hours (downtime during peak times of the day hurts more than downtime in the middle of the night).

So why am I writing all of this right now? We're hiring! There is a variety of open positions , including at least one for a Systems Engineer. So, if you're interested, and you think it might be for you, and you might fit somewhere, or just want more information, let me know directly and I'll help point you in the right direction.