Over the past few days, I’ve noticed off and on that my webserver has been extremely slow to respond — less obviously when just browsing pages, but attempting to connect to the Movable Type interface was increasingly difficult, often resulting in nothing but timeouts and connection failures.
I had a hunch that I knew what was going on, but I wasn’t entirely sure at first. I logged in to the server locally — something I haven’t had to do in a while — and realized just how badly the machine was bogged down when the OS X user interface was almost as unresponsive as Movable Type. Not a good sign. Once I made it in and got a terminal window up, I ran top -u 15 to see what was going on.
Not surprisingly, every entry that top displayed was a perl process, with mysqld occasionally clawing its way to the top for a moment or two. Now I was almost entirely sure that one or more of the sites I host was under a major automated comment spam attack, as even with MT-Blacklist installed and refusing the majority of the submitted comments, it would require a certain amount of processing for each request, and while I’m not sure just how many a minute were being submitted, it was obviously enough to bring my server to its knees.
So, seeing if I could kill two birds with one stone, I renamed all the comment and trackback scripts on the webserver, figuring that this would kill any in-progress attack and in doing so, confirm that it was a spam attack. Sure enough, as the multitudes of perl processes slowly worked their way through to completion, top started running faster (it had been updating once every 6-10 seconds, rather than once a second) and other processes started to show up on the display. After about two minutes, there wasn’t a single perl process on top‘s list, top was updating at its standard once-per-second frequency, and the computer’s UI was responding as it should.
The downside to this technique is that it breaks comment and trackback ability. Easy enough to fix, though, with a quick change to MT’s config file and a rebuild of the sites. So, the comment scripts have been renamed, and I’m in the process of rebuilding the sites to reflect the new script locations.
And you know what?
Even in mid-rebuild, I’m already starting to watch the number of perl process climb. One or two I’d expect while rebuilding the site, but I’m currently seeing anywhere from two to ten at a time. I’ve got a really bad feeling that whatever spammer has me targeted has a script smart enough to scrape the pages to find the script locations, no matter what they are named.
This — in a word — sucks. Outside of turning comments off entirely for the targeted sites, which really doesn’t thrill me, I’m not sure where to go next.
Guess for now I’ll just have to keep an eye on things and see how they go.