Blog

Site updates and anything else related to the technology of blogging and my site(s).

Thanks, Six Apart

by

As might have been implied by my last post detailing an evening’s work tweaking templates and installing plugins, I’ve decided to stay with Movable Type for my weblog. There are a few reasons for this, but it boils down primarily to two things: familiarity and loyalty.

This isn’t at all a slight against WordPress (which I was actively poking at), Expression Engine, or any other weblogging system, for that matter. I’m actually quite impressed with WordPress, and if I were starting a project from the ground up, I’d definitely include it in the list of strong contenders to run the back end. For this site, though, I decided that it was better to stick with what I knew and spend some time tweaking things than to jump ship entirely.

Right now I have a little over three years worth of experience with Movable Type (I switched over to MT from a similar but far simpler package called NewsPro on Dec. 21, 2001). While I certainly wouldn’t rate myself terribly high in the pantheon of expert MT users out there, after this much time fiddling and tweaking, I don’t think I’m any slouch, either. While I’m sure I could learn the ins and outs of a new system easily enough, in this case I’d rather use and build upon the knowledge I have rather than starting over from scratch.

Besides, in the time I’ve been using MT, the software itself has worked quite well for me. My battles over the past weeks have been with the comment spammers and their abuse of the limited resources of my server, not MT. Moving to another system might have worked temporarily, but it would only be a matter of time (and likely not very much time, at that) before the attacks started hitting that system — and I’m still not convinced that a PHP solution is the best choice for my webserver. Better for me to make a few concessions (disabling comments after 30 days, for instance) than put my server through the effort of serving up an entirely dynamically-generated website.

There’s one more big reason why I wanted to stay with MT, though — and that’s Six Apart.

As I mentioned above, I started using MT back in its version 1.something days, back when there was no Six Apart, just Ben and Mena in their apartment. Back then, I was one of many people occasionally popping up on the Movable Type Support Forums, and as often as not, it would be either Ben or Mena personally answering the pleas for help when one stumbling block or another was found. It’s things like that that add a more personal touch to software — and one of the reasons I’m fond of shareware programs like NetNewsWire, ecto, or many other programs where the developers are still personally involved with their user base — there’s the feeling of a real, breathing person behind the software, rather than a faceless corporation.

Obviously, as Six Apart has grown, Ben and Mena aren’t always as personally involved with their user base as they used to be. However, in my experience, Six Apart has yet to lose that personal, “real person” feeling, and that’s in no small part due to the excellent people they’ve been hiring, many of whom have been loyal users of MT for longer than I have.

When I got Slashdotted after news of my departure from Microsoft broke across the ‘net, I was using Six Apart’s TypePad service. As it turns out, I had the unenviable position of being their first Slashdotting, and those next few days became something of an experience (for both myself and Six Apart, I believe) in how to handle such an event. I’d already spent much of the day waging a losing battle with my inbox as comments, TrackBack pings, and e-mail missives deluged me, when suddenly iChat popped up with a friendly hello from Mena herself. I was a bit taken aback — it’s not every day I get an IM from the President of a software company, after all — but again, it’s things like that that impress me. Rather than assigning my case to one of the tech support crew, she and I spent the next few minutes working out ways for me to tweak the code on my pages to ease the load on the TypePad servers.

A few weeks ago, I realized that due to my own absentmindedness, I’d accidentally paid for a year of TypePad that I wasn’t going to be using, as I’d moved back onto my own server. It was a little frustrating, but I had noone to blame but myself, and said as much when I grumbled about it here. Imagine my surprise, then, when I got an e-mail from Brad Choate, who’d come across my post, pointed it out to someone at Six Apart, and had made arrangements with Brenna to refund me that yearly fee. I hadn’t asked for this, and there was absolutely no reason for Six Apart to do this for me — but they decided that it would be a nice thing to do.

Then, just a few days ago, Anil Dash noticed that with my battles against the spammers I’d started looking at WordPress, and he sent me a friendly little note asking if there was anything they could do to help me with my MT installation. I let him know that my limitations weren’t with MT, but with my webserver (and was barely able to keep from mentioning how nice it would be to find an Xserve PowerMac Mac mini on my doorstep one day — it wouldn’t have been at all serious, but I don’t know if Anil stops by my page often enough to catch my sense of humor), and thanked him for his note. Again, this is the kind of thing that impresses me — sure, on the one hand, he’s “just another blogger”, but he’s also the Vice President of the Six Apart Professional Network.

What it boils down to is that over the years, time and time again, I’ve gotten incredibly friendly and personal service from the crew at Six Apart. I can’t think of a better way to build and maintain customer loyalty than that.

So, to Ben, Mena, Brad, Brenna, Anil, and all the rest of the crew at Six Apart — thanks, folks. Keep on rockin’. :)

Final comment tweaks

by

Some few final tweaks to the comments tonight, and I think I’m finally done tinkering. For now, at least. There’s always more projects coming down the line somewhere. :)

As I mentioned before, comments will now automatically turn off after 30 days. Most conversations only really continue for a day or two after a post goes up anyway, and this limits the number of entries on my site that can be targeted by comment spammers. I’ve decided to go ahead and leave TrackBacks open, however, for two reasons. Firstly, there are posts that will continue to be relevant as time goes by, so I don’t mind getting pings long after a post originally went up; and secondly, turning off TrackBack pings also removes them from the page entirely, and I’d prefer to keep them visible.

I’ve also re-installed Adam Kalsey‘s SimpleComments plugin, which integrates comments and trackbacks together. This way, rather than having all TrackBack pings listed together above the comments, there is one single chronological list that combines both.

Lastly, I’ve integrated Gravatar support, so those of you who have Gravatar icons will now see them displayed along with any comments you leave here.

iTunes “Space Food” by Tai-Fun from the album Essential Chillout (1999, 6:57).

Let’s try this again, shall we?

by

Allrighty then. I’ve done some restructuring and work on the server, and it’s time to bite the bullet and see how things go: comments are turned on again. Or, at least, they’re turned on for this entry and any going forward.

I have implemented Conversation Killer, so comments and TrackBacks will automatically close on any entry after one month. While I still wish that I could just leave comments on indefinitely, hopefully this will be an acceptable middle ground (and, really, it’s rare that a comment thread continues after a month anyway, so I’m okay with this approach).

There’s a little more tweaking to do, but we’re off to a good start. I’ll keep an eye on my server to see how things behave, but with any luck, this will put me back in business.

iTunes “Work It! Dance = Life (full mix)” by Various Artists from the album Work It! Dance = Life (full mix) (1996, 1:09:44).

That’s a lot of text

by

Fun fact for today regarding this weblog: a full export from MovableType of every post and every comment results in a 12.7 Mb text file containing 3,117 posts and 8,178 comments and trackback pings.

Wow.

I now have a WordPress site up and running that has all of my old entries (up to, but not including, this one) that I’m starting to poke around with to see what I think. So far, it’s been an interesting experience. The installation was dead simple. Importing my MovableType archives took some work — the PHP script kept timing out on the 12.7 Mb file, so I ended up having to break it into chunks and import six months at a time. That was likely more of a reflection on my server and the huge amount of data I was feeding it than WordPress itself, though.

I have the Staticize plugin installed, and it does seem to be making a difference: the initial load of any particular page takes a few seconds, but then any subsequent loads are pretty zippy as it can pull from the cached file (as a test, I even have all comments and trackback pings displaying on my posts about losing my position at Microsoft, and the load times are still bearable).

The WordPress interface takes some getting used to after years of familiarity with the MovableType interface. It’s certainly useable, and I like a lot of the options that are available to me, I’m just not quite as fond of the overall user interface (however, part of that might just be my fondness for sans-serif fonts, as the WP UI uses all serif fonts — to my eye, it feels more cluttered). Still, it’s the functionality that’s the key point, and it doesn’t look like I’ll have any worries there.

I’m not putting the WordPress blog live just yet (though, to be honest, if you’re really curious, it shouldn’t take too much guessing to figure out the URL), as I haven’t done anything in the way of customization just yet. At minimum, I want to make sure that my blogroll is set up and most (if not all) of the goodies in my sidebar are active. Diving into the design may end up taking a little longer — right now it’s using the default Kubrick theme, and while I’d prefer to move the current ‘distressed’ look and feel of this site over, I’m going to have a lot of relearning to do as years of familiarity with MovableType tags and design techniques battles with figuring out the WordPress tags and design techniques.

It’s a promising start, though I’ll freely admit that the real test won’t come until I bring the site live and find out just how my server reacts. I don’t know exactly when I’ll do that — part of me wants to just throw it in (sink or swim!), another part of me wants to make sure it’s perfect before I switch over, another part of me is still waffling over moving away from Movable Type, and yet another part of me thinks that I should concentrate on what I need to do to split the existing Movable Type database apart for Dad and Kirsten’s sites.

First steps are being taken, though. We’ll find out what path they lead me on.

iTunes “Wise Up! Sucker” by Pop Will Eat Itself from the album This is the Day…This is the Hour…This is This! (1989, 3:15).

What about [some other blogging tool]?

by

After reading my rant about comment spammers, Joel asked me if I’d thought about switching over to another weblogging system. Here’s a (somewhat expanded) copy of what I sent back.

I’ve enjoyed reading your site (and its comments) ever since TypePad… and I bring this up as an honest suggestion. Why not try out WordPress? It’s simple and while it’s not immune to comment spam there are a wealth of plug-ins and options that filter or destroy them quite nicely.

Switching systems is definitely one of the things on the “possible solutions” list (WordPress and ExpressionEngine being the two top contenders). One of the things that’s been keeping me from exploring that is a distinct lack of redirect-fu when it comes to making sure I don’t break my old permalinks. I’ve received one offer of possible assistance with that, though, so it may be less of a hassle than it’s looked in the past. In the best of all possible worlds I’d be able to keep my current permalink scheme, but I’m not sure if that’s possible with the other systems, so if I have to, I’d settle for working redirects.

Part of what keeps me on Movable Type, though, is simple customer loyalty and experience. I’ve been on MT/TypePad for years now, and it’s what I’m most familiar with. Plus, they’ve been very good to me — they even just refunded me the $120 I’d accidentally paid for a year of TypePad that I wouldn’t be using, purely out of the goodness of their heart (I didn’t even ask — they saw my post grumbling about my own absentmindedness and made the offer).

I’m also unsure about how much moving to a PHP-based system (as both WP and EE are) would impact my server. MT’s Perl codebase has high overhead when it’s working on something, but then very low overhead when it’s simply serving static pages. Thanks to that, until the spam attacks started getting this bad, it played very nicely on my system. Since PHP has to process every page as it goes out, that’s more overall processing, and the question becomes whether PHP is resource-friendly enough on my box to be worth the switch. I’d used MT’s new PHP integration to dynamically generate pages for a while (before I decided that I wanted to integrate plugins that didn’t play nicely with the PHP code), and there was a noticeable lag when first requesting a page. More info on this aspect from any current WP or EE users (or even developers) would certainly be appreciated.

No matter what, though, I’m not going to be up and disappearing. I’m frustrated and annoyed by the whole situation (though not as much as I was yesterday), sure…but I’m not that easy to shut up, either. ;)

Oh, one other thing: if I do move to another system, I want to be able to use tags instead of categories. I know that there’s a plugin for this for Expression Engine (John‘s using it), and it appears that there is a hack for WordPress also (though that’s from a few months ago). Something else for me to investigate while I’m deciding which direction to head.

Update: I’ve had one vote against going to a dynamic system such as WP or EE. Phil (who I host) has both a WP and an MT weblog set up on my server. To compare the two, click these links and compare how long they take to load: MT (serving static pages) and WP (serving dynamic pages). It’s a noticeable difference, the MT site pops right up, while you can watch the WP site build the page. Off of that example, at least, I’m thinking sticking with MT and static pages is a good idea.

Update: Whee — I’m still getting comments, they’re just “old-school” e-mail comments. :) This is good. Both indieb0i and Ryan (and Gregor) have let me know about the Staticize plugin for WordPress, which “is a highly advanced caching engine that dynamically and automatically caches pages on your site that need to be cached, when they need to be cached.” Essentially, only the parts of the page that really need to be dynamically generated are, and the rest of the page is static (at least, that’s how I’m reading it). Nice, and puts WP back in the possibilities list. Thanks!

The Spammers Have Won (for now)

by

Until I have time to get in and do some rather major work on my webserver, I’m afraid that comments and TrackBacks are turned off. I really don’t like doing this — I like the interaction aspect, both getting into discussions and just knowing that people stop by here from time to time — but the attacks on the server have been too severe and too regular, and I’m tired of battling them.

I’m pretty sure that there have been three major things causing my problems.

  1. My server is just too old and slow to handle the attacks.

    Rather than paying for hosting space somewhere, I run my own webserver out of my apartment. This has quite a few advantages, in that I don’t have to worry about how much disk space I use, there are no bandwidth caps, and it’s allowed me to host websites for friends and family on the same server. However, the downside is that the server itself isn’t terribly powerful by today’s standards — only a single-processor 350Mhz G3.

    Now, really, that’s not that bad of a machine, and for general purposes — that is, serving static pages, which is what I started with years ago — it works wonderfully well. However, when I’m in the midst of getting hit by a spam attack, it just can’t handle the load, and it slows to the point of a virtual crawl. It’s never actually gone down — right now it’s showing a reported uptime of 197 days, 17 hours, and one minute — but there’s so much for it to process that it might as well go down.

    The issue is that comment attacks these days take the form of an automated script, or ‘bot’, that repeatedly and rapidly submits comments to the comment script on a weblog, sometimes hundreds of submissions per minute. While I have anti-spam measures such as MT-Blacklist installed, they still need to look at each submitted comment in order to determine whether it’s spam (and reject it), an actual user-submitted comment (and accept it), or something indeterminate (at which point it’s put into a moderation queue for me to look at).

    When I’m getting flooded with hundreds of comment submissions at a time, though, my server just can’t process the information fast enough to be able to respond, and my server essentially stops responding until it can work its way through everything.

  2. Renaming the comment script is pointless.

    One of the accepted methods of combatting the spam attacks is to rename the script that MT uses to accept and process comments, on the theory that the ‘bots’ that the spammers use then won’t be able to submit anything. This used to work, but now it’s painfully obvious that the spammers have upgraded their bots to parse through the HTML code of a page to find the name of the comment script. At this point, I can rename my comment script, and the attacks start again within a minute or two after I rebuild my site. So much for that idea.

  3. I made a mistake a while back that’s now biting me in the ass.

    The last time I set up my server, I made what in retrospect was obviously a mistake, though I didn’t think about it at the time. Each of the three primary accounts on my server — me, my dad, and Kirsten — use the same MySQL database for their MT data. Because of this, whenever a comment spam attack starts, it doesn’t matter which domain they’re aiming at — as the bot generally attacks by submitting a few comments to one entry ID number, then increments that by one and sends a few more comments, as it steps through entry IDs on the database it will end up hitting entries on every weblog in the database. A single comment attack on any single domain on my box can affect all three domains.

    Okay, yes, in retrospect, that was fairly amazingly dumb on my part. Of course, six months ago the comment spam attacks weren’t anywhere near the level that they are today, so it’s taken a while for this mistake to start showing the consequences. Things like this, however, are a big reason why I only provide hosting services for a few select friends and family, and I make sure they know that there may be occasional issues: as a sysadmin, I’m essentially learning as I go, which isn’t always the safest or most effective way to go about it. Kind of the webmaster’s version of driving by braille.

What I need to do now, then, is break everything down and start over. Luckily, I shouldn’t have to do a full nuke and pave on my server — just the MT systems. I need to do a complete export of all entries and comments for each weblog on the system, nuke the MySQL database that MT is using, then create three separate databases, reinstall MT, and re-import the weblogs. Not a fun process, but I think I should be able to do it fairly transparently, without losing all the various design tweaks and customizations we’ve made to the weblogs. It may result in anywhere from a few hours to a few days of downtime for the sites I host, but I’ll do my best to keep that to a minimum once I start.

Once I’ve done that, I’ll experiment with turning comments back on. I’m not entirely sure how that will go, as the spammers will still be able to attack, but at least at that point they’ll be limited to attacking one domain at a time instead of attacking one and getting two more in the process. This may or may not be enough to keep comments open…we’ll find out when I get to that point.

This has been a rough couple of days, and yesterday I skirted dangerously close to just pulling the plug on my server entirely. I started hosting my own websites back in 1995 because it was fun to do, and the project has grown over the years, always because I enjoyed it, and it’s fun to find all these neat new things that can be done. Installing MovableType, opening up comments to the world, hosting sites for Kirsten, Phil, and my dad — I love the fact that I can do this.

But these spam attacks have been taking all the fun out of it. Each time I see the server get hit and stop responding it gets more and more frustrating. Yesterday I was ready to just completely throw in the towel — at one point, even checking to see if it would be possible to import all my old entries into my LiveJournal account (it isn’t). Thankfully, after a couple hours of Prairie and Phil putting up with my whining and tossing ideas at me over IM, I just figured that even though I don’t like to do it, at this point simply turning off comments until I have a chance to rebuild the database and the MT installation was the best way to go.

So that’s where things stand at the moment. Feedback is still a good thing, so feel free to drop an e-mail my way if there’s something you’d like to toss my direction. Until I get the chance to spend a few hours/days doing maintenance on the box, though, this is how things stand.

iTunes “Sweet Home Chicago” by Blues Brothers, The from the album Blues Brothers, The (1980, 7:51).

Comments/TrackBack down until further notice

by

Dammit.

Comments and TrackBack pings are currently disabled at the server level back online for all sites I host (www.michaelhanscom.com, www.hanscomfamily.com, www.geekmuffin.com*). As I’ve done this at the server level, this is not reflected in the sites themselves: they all still look like they accept comments, but they won’t work.

I hope to be able to get them turned back on soon.

This may or may not be realistic. Much as I’d hate to have to turn them off permanently, unless I can find an effective block against the attacks that continue to cripple my server, it’s starting to look like a definite possibility.

This sucks.

Update: Okay, it’s all back up and running. One new software tweak, and another rename to the scripts.

I think I need to figure out a shell script that will rename the comment and trackback scripts, update the mt.cfg file with the new info, and then rebuild the sites on a weekly basis. Which wouldn’t be fun, but I really am running out of ideas short of entirely disabling comments and trackbacks or moving to another weblogging system, neither of which are very high on my list of things to do.

* Actually, www.geekmuffin.com will be ‘broken’ until a full rebuild is done. Unfortunately, as I don’t have rebuild rights for Kirsten’s site, she’ll need to do that on her own when she gets a moment. :)

iTunes “Breathe” by Depeche Mode from the album Exciter (2001, 5:17).

No more combined feeds

by

While I’d been considering this for a little while, Dave’s ‘Information Aversion’ post prompted me to un-splice my Flickr photos from my RSS feeds. Having done that, I’ve updated my feeds page to list my current available syndication feeds, all broken out to allow readers to subscribe to as much or as little of my drivel as they please.

I now offer six different syndication feeds. The first three are various ways of getting actual weblog posts:

  • Excerpts Only: The lightest feed available, this will only deliver a short excerpt for each post. You’ll have to decide if you want to click through to my page to read the full post or not.

  • Full Posts: This is the default RSS feed for this site. The full front-page text of each post (extended entries are not included).

  • Full Posts with comments: This is the most information-rich feed. The full front-page text of each post is included (extended entries are not included), along with any comments made to that post. Entries will update in your RSS reader as new comments are added, until the post scrolls off the front page of my site.

The second three contain various extra information: comments to current active conversations on the weblog, interesting links I run across, and my photography.

All feeds are run through the Feedburner service in order to assure maximum compatibility and usability. Each feed will automatically optimize itself according to which aggregator requests it, and if anyone actually clicks on any of the feeds in a browser, rather than getting a page full of gobbledygook, they’ll get a nicely formatted page explaining what they’re seeing and providing them with a full complement of buttons to assist in subscribing them to whichever news aggregator they favor (try it out, it’s rather nifty — unless you use Safari, where this doesn’t seem to work…bummer).

(If you already subscribe to my del.icio.us or Flickr feeds directly through the respective services, there’s no real need to switch to using the Feedburner feed link — you’ll get the same information either way. Of course, if you do use the Feedburner feed link for those feeds, I’ll get more accurate statistics as to how many people are reading which RSS feeds, which makes me happy. Whatever works for you, though.)

iTunes “Lunatics Have Taken Over the Asylum, The” by Collide from the album Vortex (2004, 5:34).

I hate it when I’m stupid

by

A few months back, after having been a happy user of Six Apart‘s TypePad service for about a year and a half, I decided that I wanted to go back to having the finer level of control and options that Movable Type would offer, and moved my weblog back onto my own server. I made sure to make the move a couple months before my TypePad subscription ended, so that I would have time to move all my files over and off of their servers before I got locked out of my account.

Today I got a comment notification from my TypePad site. This was a bit surprising, as my account should have died in December. When I bounced over to the TypePad site, though, I was able to log in without a problem. Starting to wonder if I’d been stupid, I checked the Account status page.

Oh, dammit.

Sure enough, I’d made sure to save my credit card information and turn on auto-renewal at some point, and TypePad had very obligingly signed me up for another year of service in December. Me being my usual brilliant self with finances had never noticed this (in itself actually not a bad thing, though — I’m still far below where I’d like to be, but if I can take a $120 hit to my bank account without it causing major issues, things are definitely improving). I checked the TypePad FAQ, and sure enough, no refunds if you’ve paid in advance. Ah, well, that was what I expected anyway.

So now I’ve got a TypePad account that I’m not using that’s all paid up and useable until next December. What in the world do I do with that? I’ve discovered over time that I don’t do well trying to keep up with posting on multiple weblogs, so starting a second weblog just doesn’t seem like a realistic option. I don’t want to move back to TypePad — not only would that be a fairly major pain, but I don’t want to lose some of the extras I have with Movable Type that TypePad doesn’t offer. I use [Flickr][5] for my photos now, so a photoblog isn’t really necessary.

[5]: http://www.flickr.com/photos/djwudi/ Flickr: Photos from djwudi”

Prairie suggested I try selling it on Ebay, but there’s hassles with that. While I could probably delete most of my personal information, the URL for the site is djwudi.typepad.com, and I don’t want to hand the ‘djwudi’ name off to someone else — until I actually started using my full name regularly, ‘djwudi’ was my normal online ID, and I still use it in [quite a few places][7].

[7]: http://www.google.com/search?q=djwudi&ie=UTF-8&oe=UTF-8 Google for ‘djwudi'”

I guess I’ll just keep it around in case something really strikes my fancy and seems like a realistic project. No point in canceling it, as long as it’s paid up through the year, at least.