Into The Weeds

So here's a summary of the festivities around here over the past 4-5 days when one of our main transformers went "into the weeds." There will most likely be an article in the local school newspapers about it, and perhaps even the Boston Globe:

• Thursday 2/2/06 5:30pm: Datacenter operators for the evening notice a "burning odor" from one of the transformers feeding our machine room. It was clearly in the early stages of losing all of it's magic smoke. And it had a lot of magic smoke.


• 8:30pm: High-Voltage electrician arrives on site, with proper nomex protective gear, to work on the problem.


• 9:00pm: It is determined that the entire datacenter needs to get powered-off.


• 9:00pm - 10:00pm: All servers powered down as gracefully as possible


• 11:00pm: Power restored


• 11:00pm - 3:00am: Most of the crap that broke from being shut down for the first time in years is fixed and running. Just in time for us to learn that we're going to need to come in on Saturday for another forced outage!

So I spent about 13 hours at work on Saturday doing a planned SAN upgrade and switch migration. All-in-all things went fairly well, in spite of EMC's best efforts (as usual) to screw everything up. On the day before the work, they called and said something like, "Oh yeah, we just noticed there's this thing and blah blah blah and two of your hosts are going to completely lose all of their storage." Luckily we were able to rig up a link to another storage array and do a quick data backup and then a restore when the work was done. It was amusing to listen to all of the EMC people yelling at each other about this screw-up and trying to push the blame around appropriately. Our sales guy's words were something to the effect of "Do I need to fire somebody over this?" Alas, this is par for the course when we deal with them though. Something always gets messed up, important details get missed, and there is general strife.

They did provide us with a truckload of Dunkin Donuts (good), and Dunkin Donuts' Coffee (also good) as well as Dunkin Donuts' Bagels (somewhat grim). My boss provided lunch from Bertuccis, and we made it through the day in one piece.

I'm aware that during my 4-year-long "exile" in Champaign, Illinois, I did a lot of complaining and whining about being there. Some of the negatives that come to mind include the ever-present bizarre odor from the Kraft factory, the lack of really good grub (you couldn't drop $100 on a good, extravagant dinner for two even if you tried), and the overall grimness of the surroundings and local residents. I pretty much did everything but put up a big calendar and cross off the days with a big red magic marker until we finally moved back to Boston this past May. And, don't misunderstand me here, I'm wayyyy glad to be back. But, here are some things I miss about those four long years in the midwest:

• Homeownership: Sure, it's nice to not have to worry about mowing the lawn, dealing with a leaky basement, or worrying about equity, property taxes, etc. anymore. But it was also nice to have a yard, with my own trees, and a driveway to park my cars in. And of course there's the fact that the total monthly payment on that 800sq. ft. home was about 1/6th of what I currently pay for rent + parking.
• My Job: No, I'm not going to whine about my current job here -- it's really quite good. I will say, however, that the folks at UIUC don't know how good they have it. IT there is run like one heck of a tight ship (comparatively speaking of course). Significant investments in time, manpower, and political capital were made to put in place strict, but sound, IT policies that ensure that best practices are followed, and everyone's job is actually easier. I'm not going to get too specific here, but we especially excelled in the areas of configuration management, security, documentation, and project planning. Any folks from CITES reading this will probably be laughing, but really, you don't want to see what the alternative can be like.
• My Job (personnel): One thing about Champaign-Urbana is that there really aren't too many places for a good UNIX sysadmin to work. Sure, you can slave away at Wolfram Research and work for a total headcase psycho freak. Or you can go over to Amdocs and get laid off. So, UIUC was actually, believe it or not, where all of the good sysadmins came to work. I had the pleasure of working with, and learning from, some of the best. And here's where I'll give a shout out to Warner Brigham, who after many years of service, retired last week. He definitely falls into that category. Sorry I couldn't make the party!
• My Truck: I had a c.1991 Chevrolet Blazer 4wd that was pretty badass. It had tazmanian devil floor mats. It had two sets of fog lamps (alright, neither set worked). It had a thumping bass tube in the back. It had extra-large tires with aluminum wheels. It had a big crazy brush guard / cattle catcher in the front for mowing down pedestrians. It was cool. I ended up trading it to someone for a huge discount on my new TV. Unfortunately, they didn't appreciate it. You know who you are.
• Best Wok: Our favorite local chinese restaurant. Sure, just about all of the other chinese food in the area was mediocre at best, but these folks did it right. Their General Gao's (they just call it "General's" or "General Tso's" there) was second to none. I'm getting over this loss though, especially now that we're frequenting Chinatown Seafood here in Brookline.

Honorable mention: Big badass thunderstorms, closeness to the in-laws, the scenery on the UI campus, the free and always on time CUMTD bus, being able to have satellite TV instead of paying wayyy too much for digital cable, and Barack Obama.

I was going through some old files, cleaning up, the other day, when I came across the invoice for my computer. It's been roughly two and a half years since I "switched." I'm speaking of course, of buying a Mac to replace my windows/linux primary desktop. This makes me a "switcher," in the parlance of our times.

Looking back, I'm not entirely sure what it was that made me switch. It was probably a combination of things: having some extra money around, dealing with just one too many frustrations with my computer, getting a Playstation2 so windows gaming wasn't so important anymore, and hearing and experiencing good things about OSX. So, I took the leap, ordering a PowerMac G4 from apple.com with my educational discount. This wasn't my first experience with MacOS. I've owned a few MacSE's over time, and still have one in working order even. On the first day the iMac was released way back in 1998, I went to CompUSA and bought one. I enjoyed it for a few weeks, and then returned it. This was my plan all along, since there was no way I could spend $1500 on a new computer at that point. But, this time it was for real.

I'm happy to say that two and a half years later, my life is better for having made the move. The original vision of a "personal computer" was as a tool for our betterment. It was supposed to increase our productivity, open up new possibilities, and generally lead to the "betterment of civilization." Quite immediately upon my purchase, I found that I was swearing at my computer a lot less. I was pounding my desk and banging on my keyboard a lot less also. I was able to actually get things done with my computer -- surf the web, check my email, write and read documents, listen to music, watch videos -- without feeling like I was embroiled in a constant Man vs. Machine struggle.

The trend continues to this day. Whenever I turn on my Windows PC these days, I find that I'm constantly pressing F9. Mac users will recognize this as the key sequence for Expose -- It's a one of the most amazing things ever, a visual way to navigate through all of your open windows that's pretty hard to describe with words. Invariably, I'll end up cursing at the computer for not doing what I want it to, or forcing me to jump through stupid hoops in order to get a certain game to run. Maybe I'll need to upgrade my video driver, or downgrade it. Or I'll need to disable the antivirus software, or scan the system for spyware, or (my favorite) edit the registry to prevent all sorts of stupid stubs from starting up on reboot and filling my taskbar (e.g. RealPlayer, Quicktime, Java Updater, Macromedia Flash Updater, some crappy useless tool that came with my sound card, some other crappy useless tool that came with my video card). But, that's enough of that rant. Suffice it to say, I'm much happier when I'm back in Mac-land.

Don't get me wrong. I'm definitely up to the "challenge" of getting my Windows PC (or linux, with it's own counterpart listing of frustrations) working and doing what I want to do. Actually, it's not much of a challenge at all -- I'm no moron after all. When I need/want a challenge, I can boot into Linux and play around. My mail/web/fileserver at home still runs Linux, of course. But, you see, somewhere along the line, something changed in me. After working on computers for 8 hours a day and making a career out of it, I no longer got any thrill or satisfaction out of getting my computer to do something for me. I no longer felt the need to "play around" with my computer all the time. Now, I can relax, knowing that the struggle has already been won. The computer is not there to put obstacles between me and my objectives, or to force me to conform to its way of doing things. It is a tool, at my service. It increases my productivity and opens up new possibilities -- Finally!.

Any readers of this blog who are in my field, or in a similar field, understand what "Firefighting Mode" is. It's when your job consists almost entirely of fighting fires (i.e. handling emergencies, servicing interrupts), rather than dealing with things in a rational, prioritized manner, and devoting some time to medium-term and long-term infrastructure improvements. The problem is, that once you're in firefighting mode, the lack of medium and long-term planning, and the lack of infrastructure improvements just cause more fires to pop up down the road. Hence, it is a vicious cycle, and it can eventually wreak havoc on the morale of the overworked staff. I'm trying to put this into appropriate words here, and what follows is my first attempt. I'm sure the "powers that be" here are already aware of the situation, but it does feel good to get it off my chest and into words:

   The group is too bogged down with every-day tasks to do long-term or even medium-term development work, or to follow best practices for many of our operations. The security incident earlier this year was a direct result of this. SSH access lock-downs were supposed to be part of the planned system audit (over 1 year overdue) that was never accomplished because of lack of available, knowledgeable manpower in the group. Likewise, I have been far too busy to make progress on Linux knowledge exchange, or Solaris Jumpstart service installation. Both of these projects, as well as other possible infrastructure projects (tru64 infrastructure replacement, documentation wiki, deploying Solaris logging UFS, etc.) are essential to our ability to effectively manage a multitude of servers effectively and efficiently.
   Without proper manpower to attend to and improve systems administration infrastructure, organizations tend to fall into a vicious cycle of not realizing possible administration efficiency improvements and methodologies, and therefore not improving the important metric of “man-hours of administration work required per server”. This, combined with an ever-increasing number of servers, makes the problem worse, keeping the group in “firefighting mode,” and eventually affecting staff morale as well.

An interesting paper on the topic:
http://isic.ucsd.edu/papers/Firefighting.pdf
And I think I'll also throw in Mark Roth's "Sanity Through Organizational Evolution" paper here as well:
http://www.feep.net/LISA01-Service-Manager-paper/lisa01paper.ps

So I decided to "bite the bullet" today and get a new cell phone. Actually, it wasn't that much of a decision, since my employer is paying for it. I picked up my Treo 650 smartphone from my friendly local verizon dealer and have already gotten it syncing with my mac with minimal problems.

It's not a piece of cake, and palm's sync software really is a flaming pile of shit -- especially on Mac OS. Luckily, many others out there on the internet have worked out the kinks and I was able to get my contacts and calendar working. I've also set up "versamail" on it so I can check my email (IMAP) when I'm not with a computer. On the down side, it's not a real "cell phone," so all of Verizon's "get it now" apps aren't available for it. However, it's a Palm Pilot, so many apps are out there for it. It also has a web browser, so checking sports scores, etc. can probably be done that way, although I really liked ESPN's MLB GameCast on my old phone.

So we've had this position open for a while. I think that the HR people are just doing a crappy job of publicizing it. Honestly, our requirements are pretty low, and the pay is pretty good, especially for the academic world. All we need is a decent UNIX systems administrator. Someone with a few years of production experience in a somewhat decent environment. Someone with maybe a little bit of storage networking (preferably EMC storage, but if not that's okay) or enterprise backup (preferably veritas netbackup, but if not that's okay) experience. But, we're not too picky. Tell your friends. Spread the word. Apply!

Apologies for the lack of blogging lately. I've been off this week at the USENIX Annual Technical Conference (ATC). So far, there have been interesting sessions on VMware ESX Server, public key cryptography and nuclear weapons control, and I'm currently in a keynote speech about systems at Pixar.

Joining us this week at the conference has been CK. This is his first time in Boston; he has gotten to see several sights in boston this week, including restaurants, bars, music, visiting MIT, some friends, and a decent taste of driving around here.

I don't feel like writing much about the activities and notable events of the week thus far, so go ahead and check out CK's multiple blog entries, and what a grim dude had to say about this.

It is with a certain amount of remorse that I recently made the decision to accept a new job, and leave MIT IS&T after a year of employment. As many of you are probably aware, there are more than a few technical challenges and orginizational issues with the IT department here, which pretty much constitutes an IT "horror show" at times. I will spare you all the anecdotes until we come closer to my actual termination. Nevertheless, I always viewed it as a challenge and an opportunity to improve the shop. I'd like to think that I'm leaving them far better off than they were when I got here a year ago, but with a long way to go, and now with some serious staffing shortages.

Hopefully they can weather the storm and not succumb to some sort of horrible IT meltdown sinking them further into oblivion than they already are. To this aim, I have "volunteered" my services to remain on as a paid contractor helping out with some of my core competencies (Redhat RHN, Veritas Netbackup) to bridge the gap. Still, I can't help feeling a little bit guilty about jumping ship -- and yes, I do have some sort of allegiance to the "alma mater," I suppose.

As for my new job, it's at the Brigham and Women's Hospital, literally right down the street from where I live. The group I'm going to be working with is the Surgical Planning Laboratory (beware, the web page is out of date), and I'll be on a small team of systems administrators serving some pretty hardcore research. It sounds like there will be a lot of freedom, and that it's a good, well-run shop. More updates to come once I actually dive in on 6/26/06.

So, before I leave, they made me fill out an "Exit Survey." I figured it was a good time to air my concerns to the higher-ups, since everyone else down here in the trenches already knows what the issues are (heck, I practically spelled it out on my performance review). Some highlights of the exit survey:

Q: Why are you leaving IS&T/MIT?
A: Severe staffing shortage and lack of advanced technical knowlege in the group have contributed to a deteriorating IT infrastructure and overworked staff with low morale.

Q: Are there any other aspects of working in IS&T or at MIT that you would like to comment on?
A: My team is good. It's a wonder that they've been able to avoid an IT meltdown through this staff shortage and heavy workload.

Q: Please tell us three things that would help us create a better work environment for IS&T employees.
A: (I can only think of two)1. Better staffing and higher salaries to attract technical leaders. 2. Better project management (e.g. the power management debacle has required several nights of off-hour work with minimal days of notice and preparation time)

Today is July 28th, System Administrator Appreciation Day. It's supposedly the day when we, the sysadmins, are to be appreciated and lavished with gifts. Nobody ever seems to remember when exactly the day is, year to year, or have heard of it as well. Oh well, hopefully it will gain in popularity.

"A sysadmin makes sure your network connection is safe, secure, open, and working. A sysadmin makes sure your computer is working in a healthy way on a healthy network. A sysadmin takes backups to guard against disaster both human and otherwise, holds the gates against security threats and crackers, and keeps the printers going no matter how many copies of the tax code someone from Accounting prints out. A sysadmin worries about spam, viruses, spyware, but also power outages, fires and floods. When the email server goes down at 2 AM on a Sunday, your sysadmin is paged, wakes up, and goes to work. A sysadmin is a professional, who plans, worries, hacks, fixes, pushes, advocates, protects and creates good computer networks, to get you your data, to help you do work -- to bring the potential of computing ever closer to reality."

So, if you can read this, thank your sysadmin.

Seen while at UIUC:
Some memory from a small company called "Clearpoint" that Warner had around for Sun Ultras and 220rs. No, really, this isn't a joke.

So I'm very much aware of the power issues at MIT's W91 data center (which are still ongoing, AFAIK). But some pretty bad shit apparently went down at W92 last night as well. Unfortunately, the network and infrastructure services team over there aren't the best at "communicating," shall we say, but they did leave some gems on the 3down.mit.edu status site. Of course, as of noon today, there are still no incoming email messages for my mit.edu account, and up until about 11:00am, the web.mit.edu website was still unavailable (in spite of what the updates below state):

• Thu, Nov 2nd, 12:15am: W92's data center experienced a power failure when returning from generator power. Currently, web, email and related services are unavailable. A further update will be posted by 1am.
• Thu, Nov 2nd, 1:45am: Services are returning. MIT's Home Page and e-mail are restored. AFS servers are returning, as well. A further update by 3am.
• Thu, Nov 2nd, 3am: Most services are returned and operating normally. Two AFS fileservers continue to return to service. We will continue to monitor all services closely for the next few hours. Our apologies for this unplanned service interruption and thank-you for your patience and understanding.
• Thu, Nov 2nd, 10am: At approximately midnight, W92's data center experienced an unplanned outage as NStar power was restored. This loss of power impacted the following key services: email, web, central windows domain, AFS servers, etc. Services were restored between 1am and 4am and now, most services are available. We are aware that some websites remain unavailable as one AFS server is still experiencing difficulty. As we learn which websites are experiencing problems, we will post alternate web addresses here.
• Thu, Nov 2nd, 11am: We are currently experiencing delays in mail delivery from sites outside MIT due to backlogs from last night's power outage. We are working on clearing the backlog at this time.

It's nice that they can afford to buy everyone Aeron chairs, but apparently can't afford the materials or manpower for a working UPS and generator transfer. Of course, there were probably some mitigating factors.

When I left there a few months ago, they hadn't yet "discovered" log-based UFS for Solaris, so any power outage included at least a 3-4hour additional downtime for fsck and/or tape restores because of the corruption.

I get the feeling that their mail infrastructure was running pretty dangerously close to the "tipping-point" recently anyway, what with increases in spam and volume. They even had some unfortunate incident recently where somebody hacked a ton of accounts and used webmail to send out a boatload of spam through the authenticated servers. This had the effect of getting MIT's outgoing mail servers on some blacklists out there for a while.

<TECHY SPEAK> When a mail system is running without much overhead space, the deluge of queued mail from the internets will definitely send it into the weeds after even a brief outage. I don't envy the task of the folks trying to bring it back up, because it's a bit of a spiraling problem. A deluge of incoming messages overtakes the busy server, and the mail queues get long. As the queues get long, they take longer to process. This can be combated (at least in sendmail) by adding additional servers, multiplexing your queue directories into subdirectories for qf,df,xf files. On a busy mail system it also might become necessary to have up to a dozen separate sendmail queues with the properly tuned number of queue runners each (new in sendmail 8.12 or thereabouts), and with queue groups dedicated to a specific subset of your mail load either randomly, or sorted by destination, or both.</TECHY SPEAK>

I wonder what the chances are that the couple of folks who run the mail system over there have already done all of this, or are even using a relatively modern version of sendmail, or another MTA. Anyway, I remember back before we took these steps on the UIUC email system, an outage of an hour or so would back things up for several hours. Computers generally don't like doing anything in directories with 50,000+ files in them.

Maybe I should call the powers-that-be over there and ask if they could use some consulting help? (Just kidding. You guys are great. Well, most of you at least. Well, some of you at least.)

After a week hanging out at Gordon College up on the north shore (some of you readers may be familiar with that particular school) for a networking training class, it's nice to be back to the regular routine. No, I'm not going to become a network administrator, it's just some class that work paid for and made me take so that I could take over some of the networking responsibilities here.

The tour of the Gordon IT department was interesting though. It's nice to see that they're a much much better run shop than MIT -- even if one of their jobs is making sure all "objectionable" sites on the internet get filtered for their network, and finally getting high-speed networking and wireless to all of the residence halls, and their "server room" consists of about 5 racks in a room the size of my office. Their Christian/biblical naming scheme for all of their servers (faith, hope, charity, responsibility, gabriel, etc. were some that I noticed) was clever and appropriate. We also got to see the foundation being laid for their new science center, named after famous and rich alum and founder of Digital Equipment Corp., Ken Olsen.

I went to read my mit.edu webmail today and noticed that apparently all of mit.edu has dropped off of the internet. No connectivity, no DNS resolution, no nothing. Maybe they figured that today was a good day to secede from the internet? Or more likely, some wonky poorly maintained piece of equipment failed or someone tripped over something or dug up an important non-redundant network link.

EDIT: Apparently this was a planned outage. Seemed to last longer than 5 minutes though. Nice to see they're upgrading the backbone.

I figured this was as good a time as any to post some networking-related pictures that I had on file from my time at MIT IS&T. These are some of the finest cable mess nightmares that I've ever seen:

The first one is of the two network switches in the administrative server room. This is where the servers that hold your academic information and status, the admissions servers, and the institute's detailed financial records and important databases that can't go down live, among some other things. Note the nice rainbow of colors, and especially the strange number of orphan cables just hanging there. What goes where?

The second one is a close up of the previous. That makes for a really stable, mission-critical network, eh?

The third one is actually the first part of a success story. That's the back of the SAN switch cabinet and immediate underfloor area. The only way we could replace this mangled mess of fiber cabling (which stretched underneath the entire datacenter) was to rip out the old switches entirely and replace them with a new set of Cisco 9000 series directors. The fine folks at ASST (myself included) ran all new cables nicely bound in collars, routed cleanly and labeled. Unfortunately, we never had enough time to rip all of the old stuff out, so I never took an "after" picture.

Note: This one's only of interest to my fellow sysadmins.

It's time to revive an old parody from 1997 or so. Imagine this spoken to the same music and in the same voice as Baz Luhrmann's "Wear Sunscreen":

Wear a Leatherman.

If I could offer you only one tip for the future, having a Leatherman would be
it. The long-term benefits of a Leatherman have been proved by BOFHs, whereas
the rest of my advice has no basis more reliable than my own meandering
experience. I will dispense this advice now.

Enjoy the power and beauty of root. Oh, never mind. You will not understand the
power and beauty of your root access until it's taken away. But trust me, when
you need to kill a runaway process, you'll think back to the scripts you had
and recall in a way you can't grasp now how much possibility lay before you and
how much you could do. You are not as powerless as you imagine.

Don't worry about the Y2K bug. Or worry, but know that worrying is as effective
as trying to mount an old chain of Exabyte tape drives by chewing bubble gum.
The real troubles on your network are apt to be things that never crossed your
worried mind, the kind that get you called in at 4 a.m. on some weekend when
you were supposed to be recovering.

The lab that I work for, the SPL, bought a whole buttload of workstations from Dell earlier this year, as well as a compute cluster (which is still pretty much sitting there unused for various reasons). Harry (the dude in the video) was able to get a good deal on the stuff by agreeing to let them make a promotional video and interview some folks around here to gush about how great the Dell workstations are. So what follows is a pretty good peek at some of the stuff that we do here at the SPL:

Dell: Brigham and Women's Hospital Surgical Planning Lab (flash 8+ required)

I promised a while ago that there would be an upcoming post about Vista (the new version of Windows), given my early impressions running it on a few systems (or not running it as the case may be). It turns out that there is a bunch to say, so I'll split it up into a few entries. This is part 1 -- the intro. A note to all readers: I think I'm actually somewhat biased on this topic. I guess you could call me a "recovering Windows administrator" -- having given up on Windows a few years ago now (except where necessary) and become a Mac and Linux user and focused my career to UNIX-only systems administration. So Microsoft fans, zealots, and employees can take my ranting with a grain of salt perhaps.

It's hard to believe that it's been so long, but on October 25th, 2001, Windows XP was released to market. This means that it's been over 6 years now that desktop and workstation windows users have been using the same operating system and the same technology (with minor upgrades and service packs of course). What we now know as Vista started development in 2001 as a very ambitious next-version of windows. Vista was originally expected to ship sometime in 2003 as a middle step between XP and a future version with the codename of "Blackcomb." Sure enough, 2004 rolled around and feature creep and development headaches of various types made it clear that big changes needed to take place. Exciting plans for a new filesystem (winFS), among some other features, were abandoned, and development was started anew to be built off of the Windows 2003 server codebase.

Vista is arguably the largest and longest consumer software project of all time. There's a school of thought that it was doomed to failure from the start. The windows codebase is too complicated, old, huge and messy to maintain any longer. Developers state that these days there are more than 50 dependency layers in windows (the Broken Windows Theory). Some of the headaches of this are evident in the comments of this Mini-Microsoft blog entry , where the author bemoans the slipping of the schedule and hopes for leadership changes.

So here we are, in January of 2006, a few weeks from the retail launch of Vista. After billions and billions of development dollars, countless man-hours of work, PR nightmares and years of punditry, what is it that we'll get? The answer is really "not much." I got my hands on the final market version of Vista a few weeks back and installed it on a couple of systems. As a side note here I must add that the supposedly next-generation activation procedure and copy protection is, as expected from Microsoft, a pain in the rear end to normal users, but already hacked and easy to circumvent for people looking to pirate the OS. Overall, I liked the OS (with a few snags that I'll mention in my next entries on the topic), but the one, overriding feeling I had was: "We waited 5 years for this?" Stay tuned for future entries in this series.

Here's a nice quote from our good buddy Jim Allchin, back in 2004:

 I'm not sure how the company lost sight of what matters 
to our customers, both business and home, the most, but in my 
view we lost our way. I think our teams lost sight of what 
bug-free means, what resilience means, what full scenarios mean, 
what security means, what performance means, how important 
current applications are, and really understanding what the most 
important problems our customers face are. I see lots of random 
features and some great vision, but that does not translate into 
great products....I would buy a Mac today if I was not working at 
Microsoft.

Jim Allchin is, of course, none other than the windows development chief, and has been since the mid 1990s. Well, at least he's being honest, I suppose.

As mentioned in the previous entry in this series , I went through the experience of installing Vista on my work laptop. The laptop in question, is approximately 1.25 years old (it was purchased over the fall of 2005). It's a fairly nice Sony Vaio VGN-T350p. A handy little 10.7" wide mini-laptop with a 1.2ghz Pentium M processor, 512MB of RAM, 60GB hard drive, Intel 855GME graphics, and a DVR+-RW drive. Obviously not top-of-the-line with what's out there now, but still a very very nice system -- arguably even better than most laptops that are currently in operation out there.

Readers will probably notice that I have already spent a few entries complaining about the general ineptitude and organizational malignancy at MIT IS (information services). Fear not, however, as MIT is far from the only organization that I can pick on. The following is an entry from the UIUC status page:

Web Services will conduct a campus web server 
failover test today at noon. The following services will be 
down during this time: www.uiuc.edu, Web Services 
Toolbox, maps, and PH. The test should be completed by 
12:20.

Who the heck schedules a failover test, especially one involving definite downtime, for a major website and important service in the middle of a weekday? I think a disclaimer is in order here. The www.uiuc.edu website is no longer run by or affiliated with the campus' IT department (my former employer), CITES. At some point, it was decided that it would be run by the Office of Public Affairs (OPA). I mean, come on now, just look at how they run their own web site. This is the kind of crap that happens when you take critical IT infrastructure out of the hands of those most qualified and able to deal with it. There's lots lots more to this story. Believe it or not, a noontime planned outage of the main campus website is actually one of the least severe indicators in recent history that the UIUC main campus website may be run by a bunch of people who don't have a clue as to what they're doing. I won't go into some of the other clear indicators, but feel free to comment if you are more intimately familiar with the situation. The whole thing just seems ridiculous. It's as if the campus web server went from running in a well controlled datacenter, managed by qualified systems administrators, to running under some secretary's desk in an office somewhere.

Recent troubles concerning a mail server at MIT reminded me that I should get around to documenting one of my favorite tales of woe. It happened back in February 2002 while I was working at what was then CCSO, in what was then PSG. I had only been working at UIUC for 6 months or so at that point, and had only been administering the campus student/staff cluster "dataservers" for less than that.

I must warn. This is a long entry, and is probably uninteresting to most out there. However, I put it here for my own documentation (this is the story I tell at job interviews when asked about an arduous "troubleshooting experience") and for the possible enjoyment of a small population of my peers.

• ^ - Popeye's: Popeye's has opened their first branch (as far as I can tell) in Boston, and it's at Kenmore Square, just down the street from my apartment. Now, I'm sure some fair readers of this blog will have horrible, nasty things to say about Popeye's for all sorts of reasons, but nevertheless, I give them a solid up arrow. I remember as an undergrad, Kristy and I driving out to the service area on the Mass Pike that had a Popeye's, since that was, at the time, the only one anywhere near Boston.
• ^ - 38 pitches: Curt Schilling definitely hasn't been at a loss for words on his blog. Let's hope he keeps it up through the season -- there's really been some fascinating stuff on there. Supposedly the dude spends a large amount of time on the computer anyway, participating in forums and playing MMORPGs.
• V - The unsustainable american lifestyle: Man, I bet it must suck to have to use a car to get around all the time, doesn't it? I can't imagine being trapped in my car for an appreciable part of the day just to get to and from work -- and having to pay hundreds of dollars per month just for the privilege of filling the tank.
• ^ - Grand Chau Chau: This place belongs in the hall of fame of chinese restaurants. It is second to none. It's (arguably) within walking distance on a nice day, and that's what we did yesterday after catching a movie (Shooter). Damn tasty chinese food. It's the real deal.
• = - TOD: We need to do a better job of making the distinction between real, well-planned Transit Oriented Development, which I still believe can be a good thing, and doesn't have to simply be a sprawl-contributor, and Transit Proximate Development, which is a wolf in sheep's clothing. I had a nice conversation with someone about this last week, but have just now gotten around to actually organizing my thoughts.
• V - Qmail: Qmail sucks. It's complete disregard for any sort of standard as to how the rest of the world and all of the other MTAs work and are administrated is mind boggling. It's a shame too. I'm sure that DJB is a brilliant person, and the architecture of Qmail is very good and solid. But, dear lord it's a frustrating technology trap that hasn't been updated since sometime in 1998, and requires custom 3rd-party modules to even obtain some semblance of operability on today's internet. Please, Qmail fanboys (I know you're out there, and will come across this), save yourself the effort and refrain from arguing about it. Now, if you want to argue about Emacs vs. VI, that's another matter...

Well, first thing tomorrow I'm off on the 'ol iron bird to lovely Champaign, Illinois. A consulting customer out there is upgrading an old crusty Ultra10 running Sol 7 to a shiny new Dell RHEL 5 server. This should be a bit of an adventure though, since nobody's sure exactly what all is running on the old box, with various web sites, cgi scripts, content management systems, movable-type/mysql installations, etc. Loads of fun. Wish me luck.

Also, since it turns out that Steer Roast is tonight, I've got a chance to stop by and check it out, after missing it for all these years. Markus Feldmeirius is building some sort of lighted floor thing covered in sand (from what I could gather from talking to him about it), and the usual Senior House lunacy will be in full effect as well, of course. Lunacy abounds.

It had to happen at some point, I suppose. The end of an era...

walker-tray:~ benoc$ telnet relay1.cso.uiuc.edu 25
Trying 128.174.5.119...
Connected to relay1.cso.uiuc.edu.
Escape character is '^]'.
220 relay1.cso.uiuc.edu ESMTP Sendmail; ready at Wed, 30 May 2007 15:14:54 -0500  (CDT)
helo walker-tray
250 relay1.cso.uiuc.edu Hello [134.174.54.3], pleased to meet you
mail from: <benoc@alum.mit.edu>
250 2.1.0 <benoc@alum.mit.edu>... Sender ok
rcpt to: <benoc@uiuc.edu>
550 5.1.1 <benoc@uiuc.edu>... User unknown

Apologies to all for not blogging recently. It's not that I haven't had much to blog about, but rather there are certain exciting things going on that I probably shouldn't blog about until I know their ultimate outcome. Some of you readers probably already know what I'm alluding to. Hopefully I'll be able to spill the beans some point soon, otherwise I suppose it was just not "meant to be."

This status message was brought to my attention, posted up at the UIUC CITES Status web page today:

The hard drive on the gateway serving Wireless and 
QuickConnect is failing, and needs to be replaced. We will 
be taking it offline at 5pm today and replacing it.

I thought it was humorous. This is apparently someone's idea of a production wireless access system. Single hard drive, no mirroring, no hot swapping, let's take it down at 5pm to replace the part. What the heck, do they have these things running on Mac Minis or something?

See, it's not only MIT's IS&T that I poke fun at. CITES is not immune. This is all in good fun though, I'm sure there's a rational explanation or justification for everything.

Some things that I'm currently stoked about/for:

• 27 June 07 - Live Free or Die Hard - Here's a trailer for the next (last?) sequel in the most awesomely awesome action movie franchise evarrrr.
• 11 July 07 - San Francisco - I just heard the news today that I'll most likely be taking a trip to SF for a "visit" at this time. I remain cautiously optimistic.
• 23 Sep 07 - The War - Ken Burns' epic 14-hour-long documentary on World War II has apparently been 6+ years in the making. There's a trailer/teaser up on the website.
• 16 Oct 07 - GTA IV - Check out the first trailer on their website. Awesomeness. I'm such a GTA junkie that this will also be when I buy a next-gen console (although I haven't 100% decided which one). This game is so anticipated, that they've pre-announced that the next trailer will be available on the site on 6/28.

I came across an interesting article the other day in CIO magazine about a particular systems engineering problem at CERN. A particular experiment at the institute's Large Hadron Collider (LHC) is going to be generating a constant 1GB/sec data stream, and apparently they need an architecture to process and store the data.

Unfortunately, the article is thin on details. It does mention, however that the first stop for the data will be a group of about 200 PCs, and from there it gets processed and goes to another 50 PCs which will record it to disk. It mentions that they're using a standard 4Gbps Fiber Channel SAN with a clustered filesystem. I'm going to assume that they have the 50 PCs set up as a cluster to receive the data and write over the fiber network directly to shared LUNs on the SAN.

I've never heard of the storage clustering software they are using (Quantum's StorNext), but it sounds pretty interesting. Clustered storage is definitely becoming more popular, and can be a huge win from a performance, as well as management standpoint -- as these folks at CERN have no-doubt been thinking. We made use of clustered SAN storage for the netfiles system at UIUC, and I think it was a big win over any NFS solution. The StorNext software sounds particularly interesting, because it's supposedly vendor agnostic. The QFS software we used for netfiles was Solaris only, but quite stable and fast -- if not a bear to initially get set up.

It seems like they are using a relatively low-bandwidth SAN, unless they are talking about 4Gbps per link, and not 4Gbps total. But, then again, maybe the bottleneck is in all the processing done by the initial 200 pcs in the tree. And, of course, nothing is mentioned about backups. One would hope that they have a duplicate SAN somewhere else doing some sort of timed asynchronous backups in case something bad happens. Anyway, interesting stuff.

Yesterday, a number of internet services, including LiveJournal, Sun.com, Yelp, Technorati, Craigslist, and a good section of Second Life went belly up at the same time. The cause? A power outage in San Francisco, followed (more importantly) by some sort of failure of backup power at a big-time co-location facility there.

For those not familiar with the term, a co-location facility (sometimes called a co-lo), is a beefy building with all of the infrastructure required for putting your servers. They charge lots of money, and in exchange, generally provide you with lots of air conditioning, protected rooms, and uninterruptable power. This particular facility at 365 Main in San Francisco, touts itself as having 100% uptime in the recent past. As a matter of fact, just a couple of days ago, they put out a press release specifically bragging about their uptime. Of course, it's now been taken offline. Apparently a line (or "mob", if you will) of sysadmins gathered outside the facility when the thing went down.

According to their white paper and specs (and one would assume their service level agreements), this wasn't supposed to happen. Their "Ten 2.1-megawatt Hitec Continuous Power Systems", "three 20,000 gallon double-lined fuel tanks", and "extended protection time by means of the integrated diesel engine" aren't worth anything if they don't actually switch on in the case of a power failure. Some folks were saying that a disgruntled employee was to blame. I find this somewhat unlikely that that would happen in combination with a city-wide power outage.

But, on the other hand, Robert Goulet running around and slamming all of the red power-interrupt buttons would explain why the backup power didn't help any!

Anyway, the moral of the story for datacenter operators is, test, test, test. Unless of course you figure that the extra money you're bilking out of your customers and saving by not testing, will make up for the millions of dollars in lawsuits caused by the outage. And for the service operators and architects, putting all your eggs in one basket really still isn't a good idea -- even if it's a big fancy basket with supposed "100% uptime." But hey, those are the risks you've got to take. In some cases, it's not worth it to build out/rent a totally redundant data center for the few hours once-in-a-blue-moon co-lo facility outage.

Sorry for being re-miss with the blogging lately. I've been hardcore into Rocking the '80s lately.

As I mentioned a while ago, I moved the hosts www.hejweca.org and www.intotheweeds.com to a virtual server co-located somewhere at Andy's roadkill.com. And that's pretty awesome. But, I still use my server at home for webmail.hejweca.org and mail.hejweca.org, and some other things -- including a WebDAV server for sharing iCal calendar data. Unfortunately, a few months ago my self-signed SSL certificate expired, and I've been too lazy to regenerate one.

The problem is, of course, that the server only has one IP address to the outside (it's on a cable modem), and so uses name-based virtual hosting in apache. I'd have to say that one of the greatest annoyances in the world is that name-based virtual hosting does not work with SSL. That is to say, since webmail.hejweca.org, mail.hejweca.org and calendar.hejweca.org all share the same IP address and are separated into virtual hosts by name only, I can't have different SSL certificates for each vhost. This is a problem, because, as we all know, SSL certificates are tied to the name of the server you're going to. So, if you go to https://webmail.hejweca.org in your browser, you'd get a nasty warning that the certificate belongs to some other name. Anyway, it's a pisser because I'm trying to fix the SSL WebDAV server. Maybe I'll try to make it authenticated access internal-only, and not use SSL at all. Bah, that's nasty. Yes, I know why the problem exists; and that it really can't be fixed without re-jiggering the SSL protocol and the HTTP protocol in potentially dangerous and nasty ways, but maybe someone, somehow, somewhere is working on it.

And while I'm on the topic of general annoyances, why does Sun's patching system for Solaris have to suck so much? And, for that matter, why does Sun's package management system (if you can call it that) for Solaris suck so much too? And also, why are plane tickets so damn expensive these days? $350/person to go to Chicago? Give me a break!

There is a strange subset of information immutably lodged somewhere in the recesses of my memory. This stuff includes the date 8/21/96 (when our class first arrived at college 11 years ago next week). It also the following phone numbers. Now, I really have no reason to remember these, and wish I could use that space in my brain for something more useful that I have to look up all the time (like the common command line arguments to rsync, for example).

• 544-1410- This was the first phone number we ever had, I think it must have been between when I was 4 and 5 years old or so
• 323-2618- This was the next number. For some reason, the one above got randomly changed to this one
• 392-9428- When I was around 6, we moved to a new house
• 225-6243- Freshman year in college
• 225-6219- Sophomore year through the rest of school
• 945-2300- Apartment in Cambridge after graduation -- When I called MediaOne (which eventually became AT&T Broadband, and then Comcast) to set up phone service, the dude's "computer" was down for a while and as a reward for having to wait several minutes, he gave me what will probably be the coolest phone number I'll ever have.
• 356-5986- House in Champaign, IL

There are even some more in there, mostly of random friends in high school. These were just the first ones that came to mind. Also, here are some possible upcoming blog topics that I feel the need to bloviate upon. I'm not sure which one I'll get around to first -- but look for these to come up on this blog any day now:

• Julian Simon - I'm definitely not a libertarian. But he was, and some of his wacky theories and ideas are worth babbling about -- even if only for contradiction's sake. Incidentally, aside from being a libertarian, he's also the dude who gave the airline industry the idea of overbooking flights. Thanks, Julian.
• Second Life - I think it's time to talk about the phenomenon of SL and the "metaverse."
• Worse is Better - http://www.jwz.org/doc/worse-is-better.html
• Bootstrapping an Infrastructure - paper here -- This is pretty much the core IT infrastructure theory paper that I've followed since working early on in what was then PSG at the University of Illinois. And other sysadmins and systems architects/engineers really should take some of its lessons to heart as well. This is the way to approach the problems of large infrastructures effectively.

When re-installing my MIT "personal certificate" tonight, this awesome message popped up:

After seven years, I guess it might finally be time to change my athena password.

It's official!

I've given my notice here at the BWH Surgical Planning Lab. Towards the end of September, I'll be starting work as a Systems Engineer for Linden Lab (the makers of Second Life, if you didn't already know) in their Cambridge office.

If I've already talked to you about it, you know I'm getting pretty stoked about it. This should be quite an adventure.

After spending two weeks in San Francisco, visiting the headquarters of my new employer, Linden Lab, I am fully recovered from the jet lag, and back at home.

Unfortunately, the Red Sox are now down 1-2 in the ALCS vs. the Indians. The Rockies, on the other hand, are looking like a classic "Team of Destiny (TM)." When's the last time they lost a game? How good was that one-game NL wild-card playoff?

Anyway, come on down to my place, good 'ol apt. 1504 to help cheer on the Sox tonight as they battle the Indians (feather, not dot), for the privilege of meeting the Rockies in the 2007 fall classic. Refreshments and food will be provided. The fun starts at 6:30, when I get home from work. Game time is 8pm.

In honor and observance of the Writer's Strike currently going on, here is my version of a "clip show." This blog has over 175 entries now, some good, some bad, some awesome. I know there are several new readers here, and it's a pain to sift through the entire blog to find the good nuggets. Here, in my opinion, are the 15 most worthy entries if you're flipping through this place I like to call "Into the Weeds":

• yuck: Cold Stone Creamery Rant
• The Soul of the Commuter: Commuting sucks. In very many ways. Here I get back to my urban studies roots.
• UIUC Mail Server Disaster: A tale of sysadmin sorrow from 2002
• Trivia: See if you can identify who all of these footnotes to history are.
• The Cult of Scientology: Mini-rant and link to a freaking awesome article on scientolgy.
• Longwood Towers: Rant about why the Longwood Towers Condominiums are some place you should not be thinking about living in.
• Kozmo.com: : Classic tale of dot-bomb woe. Ever wonder what that "kozmo" box is that's sitting in the kitchen lounge on second west?
• Florence Co. ADMAX: Crazy terrorist prison in Colorado with all sorts of nasties inside.
• New Apartment: Check out the awesome view from my living room.
• Driving in Boston: A primer, with scary crazy maps of intersections.
• Captain Al Haynes: The amazing story of Capt. Haynes and United flight 232.
• Firefighting Mode: Some systems philosophy, and the condition of what things were like back when I worked at MIT IS&T.
• The Switch: Mac vs. PC
• Unofficial!!!!: The crazyness at UIUC known as "Unofficial St. Patrick's Day" and why it is awesome and awesomely stupid at the same time.
• Datacenter Transformer: A tale of woe from some crazy-ass electrical problem and superbad preparation and handling at MIT IS&T.

As of Philip Rosedale's blog post a couple of weeks ago, Linden Lab's mission statement is now:

To connect everyone to an online world that improves the human condition.

For some pretty powerful examples of what this is all about, check out a segment on the CBS Early Show that aired yesterday morning: New Life in Cyberspace. It features a woman with MS who has built a successful and profitable (but somewhat strange) virtual business in Second Life, and John Lester (Boston Operations Director for Linden Lab) assisting a group of disabled adults who have created new lives in the virtual world. No, really, check it out.

I think I'm beginning to understand why at least 5 of my former co-workers (that I actually know of) there in the systems management organization where I used to work are actively searching for new jobs. Apparently, among much other strife (more on that further below), they are looking for a new Executive Director over at CITES (the central IT department at uiuc.edu where I used to work). One of the candidates, Mike Smeltzer, gave a talk to the organization to pitch his candidacy and answer questions. Listen to some gems from this cheery talk:

"I think Pete [former CIO] and Stan [former executive director] have put up with an awful lot of just outright insubordination over my time at CITES among various people, various groups. If I'm in charge, then we're not going to have that."

"Once we define and lay out what CITES is going to be and what the organization is going to look like, you're going to have to decide if that's the organization that you want to be part of."

"And if you elect to stay, I expect you to be rowing all in the same direction. If you elect to stay, I expect you to avoid documenting your immaturity on listservs."

"Sally [current CIO] has promised the provost that, over time, we're going to reduce the headcount at CITES ... but when you have people that go out of their way looking for 'lifevests' it makes it kind of easier to do that."

"Guerrilla navigation is going to be strongly discouraged ... staff that are unable or unwilling to add value to their teams -- we're going to ask you to find a new boat."

Seriously: guerrilla navigation??? rowing all in the same direction??? insubordination???

Let's put this in a bit more context now. It seems (looking in from the outside) that the morale of at least everyone that I know of in CITES has been going way way way downhill continually over at least the past year. Here are a couple of specific things:

Salary Equity: One of the interesting things about working for the State of Illinois is that everyone's salary is published annually in a neat little thing known as the "gray book" this is a pretty powerful tool for transparency. On the other hand, it doesn't help things to know that (real example here) you've been working there for about 7 years, and are making roughly $51k and getting an annual "fuck you" raise of generally between 0 and 2 percent, while some new hires in your group start out making the same amount as you, or even slightly more, despite being green and, in some cases, incompetent. And new hires in other groups consistently are being hired at even more inflated salaries.

The Background Checks: Insubordination will not be tolerated! We all need to be rowing in the same direction! No voicing your opinions on mailing lists! And now, of course CITES instituting a policy of mandatory criminal background checks for everyone (new employees, as well as existing ones who are getting promoted).

"Anonymous" Survey:Coupling the above with Smeltzer's cheery "get off the boat" attitude made folks genuinely correctly suspicious when the supposedly "anonymous" web survey turned out to require username authentication -- thus making it, by definition, not anonymous. A hilarious argument then ensued and here are some replies from the powers that be: "... I assure you I'm not sniffing the network. If you'd like proof, come to my office and watch me generate a report, or you can just trust me...", and "...data is restricted to IT staff who need it ... individuals who have taken it on themselves to violate someone's privacy by leveraging the system and or log data they have at their disposal ... no longer work for the university. The survey ... is anonymous."

Activity Based Costing: A consulting company (WTC) has been engaged for an ABC (Activity Based Costing) study for CITES. This will result in a more accurate picture of activities and services, and related costs. It will also require participation on the part of everyone in the group. They'll need to begin tracking activites they perform, and the amount of time required for each, in as much granularity as possible. Estimate from WTC is that there will be somewhere between 400 and 800 activities required, to give you a sense of the level of detail. Doesn't that sound awesome? And here's the priceless response from a manager when someone complains that tracking everyone's time is a stupid idea: "...no one is asking you to report your time ... we will not be recording time, only tasks and how long it takes to perform them." Really? I mean, seriously? come on now...

Seriously folks, WTF? Is management trying to drive away all of the really smart and skilled people from the organization? Because, if that's their goal, they are doing a pretty damn good job.

Memo to CITES: wait a few months more like this without management sticking up for their employees, or paying attention to morale and issues like those mentioned above, and there will be nobody left to manage your systems and services. CITES systems management is the home to some of the most professional, skilled production (and non-production) systems administrators that I've ever worked with. Their development and use of automated systems administration tools and production methodologies serves the university extremely well, and should be appreciated -- not driven away.

So yesterday, I took a trip across the street, paid out around $180, and walked out with a single, small box capable of storing one terabyte of data.

For those of you keeping count, that's one thousand gigabytes, and that's pretty amazing. It's replacing a crusty and failing old LVM of three concatenated 250gb parallel ATA drives that have started suffering from some failed sectors of late. So, a terabyte should be enough storage for anyone for a long time, right? I remember saying the same thing in 1996 when the first one gigabyte drives were coming out. So, ten years from now, will we all have petabyte sized drives in our computers?

And now, some phrases that have crept into my vocabulary from either my own life experiences, my work, or my friends. Enjoy these shared bits of english-language awesomeness:

• Schtonky: In some circumstances, anything that is cool/awesome can be referred to as schtonky. More specifically, schtonk is a measure of some mixture of class, formality, masculinity, and old-timeyness. For example, a fine tobacco pipe carries very high schtonk value, as would any wood-paneled room with a fireplace where leather-bound books are. Snoop-doggy-dog dressed up as a pimp driving around in a '64 Impala is also quite schtonky, as is anything to do with Frank Sinatra, or the rat pack. Wal-Mart is not schtonky.




• Janky: Something that is constantly near the boundry of working and being broken is janky. But being janky is more than that. Pretty much every one of the crazy electronic / audio / mechanical projects we built in college was janky. Something that is janky but still working carries with it a certain amount of pride and charm. At MIT, on 2ndwest at east campus, the PTZ Audio Net