Geekery

Until I have time to get in and do some rather major work on my webserver, I’m afraid that comments and TrackBacks are turned off. I really don’t like doing this — I like the interaction aspect, both getting into discussions and just knowing that people stop by here from time to time — but the attacks on the server have been too severe and too regular, and I’m tired of battling them.

I’m pretty sure that there have been three major things causing my problems.

1. My server is just too old and slow to handle the attacks.

   Rather than paying for hosting space somewhere, I run my own webserver out of my apartment. This has quite a few advantages, in that I don’t have to worry about how much disk space I use, there are no bandwidth caps, and it’s allowed me to host websites for friends and family on the same server. However, the downside is that the server itself isn’t terribly powerful by today’s standards — only a single-processor 350Mhz G3.

   Now, really, that’s not that bad of a machine, and for general purposes — that is, serving static pages, which is what I started with years ago — it works wonderfully well. However, when I’m in the midst of getting hit by a spam attack, it just can’t handle the load, and it slows to the point of a virtual crawl. It’s never actually gone down — right now it’s showing a reported uptime of 197 days, 17 hours, and one minute — but there’s so much for it to process that it might as well go down.

   The issue is that comment attacks these days take the form of an automated script, or ‘bot’, that repeatedly and rapidly submits comments to the comment script on a weblog, sometimes hundreds of submissions per minute. While I have anti-spam measures such as MT-Blacklist installed, they still need to look at each submitted comment in order to determine whether it’s spam (and reject it), an actual user-submitted comment (and accept it), or something indeterminate (at which point it’s put into a moderation queue for me to look at).

   When I’m getting flooded with hundreds of comment submissions at a time, though, my server just can’t process the information fast enough to be able to respond, and my server essentially stops responding until it can work its way through everything.

2. Renaming the comment script is pointless.

   One of the accepted methods of combatting the spam attacks is to rename the script that MT uses to accept and process comments, on the theory that the ‘bots’ that the spammers use then won’t be able to submit anything. This used to work, but now it’s painfully obvious that the spammers have upgraded their bots to parse through the HTML code of a page to find the name of the comment script. At this point, I can rename my comment script, and the attacks start again within a minute or two after I rebuild my site. So much for that idea.

3. I made a mistake a while back that’s now biting me in the ass.

   The last time I set up my server, I made what in retrospect was obviously a mistake, though I didn’t think about it at the time. Each of the three primary accounts on my server — me, my dad, and Kirsten — use the same MySQL database for their MT data. Because of this, whenever a comment spam attack starts, it doesn’t matter which domain they’re aiming at — as the bot generally attacks by submitting a few comments to one entry ID number, then increments that by one and sends a few more comments, as it steps through entry IDs on the database it will end up hitting entries on every weblog in the database. A single comment attack on any single domain on my box can affect all three domains.

   Okay, yes, in retrospect, that was fairly amazingly dumb on my part. Of course, six months ago the comment spam attacks weren’t anywhere near the level that they are today, so it’s taken a while for this mistake to start showing the consequences. Things like this, however, are a big reason why I only provide hosting services for a few select friends and family, and I make sure they know that there may be occasional issues: as a sysadmin, I’m essentially learning as I go, which isn’t always the safest or most effective way to go about it. Kind of the webmaster’s version of driving by braille.

What I need to do now, then, is break everything down and start over. Luckily, I shouldn’t have to do a full nuke and pave on my server — just the MT systems. I need to do a complete export of all entries and comments for each weblog on the system, nuke the MySQL database that MT is using, then create three separate databases, reinstall MT, and re-import the weblogs. Not a fun process, but I think I should be able to do it fairly transparently, without losing all the various design tweaks and customizations we’ve made to the weblogs. It may result in anywhere from a few hours to a few days of downtime for the sites I host, but I’ll do my best to keep that to a minimum once I start.

Once I’ve done that, I’ll experiment with turning comments back on. I’m not entirely sure how that will go, as the spammers will still be able to attack, but at least at that point they’ll be limited to attacking one domain at a time instead of attacking one and getting two more in the process. This may or may not be enough to keep comments open…we’ll find out when I get to that point.

This has been a rough couple of days, and yesterday I skirted dangerously close to just pulling the plug on my server entirely. I started hosting my own websites back in 1995 because it was fun to do, and the project has grown over the years, always because I enjoyed it, and it’s fun to find all these neat new things that can be done. Installing MovableType, opening up comments to the world, hosting sites for Kirsten, Phil, and my dad — I love the fact that I can do this.

But these spam attacks have been taking all the fun out of it. Each time I see the server get hit and stop responding it gets more and more frustrating. Yesterday I was ready to just completely throw in the towel — at one point, even checking to see if it would be possible to import all my old entries into my LiveJournal account (it isn’t). Thankfully, after a couple hours of Prairie and Phil putting up with my whining and tossing ideas at me over IM, I just figured that even though I don’t like to do it, at this point simply turning off comments until I have a chance to rebuild the database and the MT installation was the best way to go.

So that’s where things stand at the moment. Feedback is still a good thing, so feel free to drop an e-mail my way if there’s something you’d like to toss my direction. Until I get the chance to spend a few hours/days doing maintenance on the box, though, this is how things stand.

“Sweet Home Chicago” by Blues Brothers, The from the album Blues Brothers, The (1980, 7:51).