This doesn’t affect this particular site at all, but in case Dad or Kirsten check in, there seem to be issues with the djwudi.com/hanscomfamily.com/geekmuffin.com/interalia.org server at the moment. Phil and I are trying to work on it and find out what the issue is — whatever it is, it popped up sometime during the night, when neither he nor I were doing anything with the machine. No current ETA on when we’ll be back up and running, though we’re trying to bring it back up as soon as possible.
Update: Still down, still no ETA, and I need to get out of the house for a bit. I’ll get more info up when I can.
This also means that any e-mail sent to my [\@michaelhanscom.com]{.citation cites=”michaelhanscom.com”} address will not get to me at the moment. Please use one of my other e-mail addresses if you have one of them and need to get ahold of me.
Just so I don’t lose track or forget about what we tried so far, a quick copy-and-paste of the work so far:
Phil: el problemo, senor
Michael: what’s up?
Phil: i think something has run awry in my crontab
Phil: and is chewing up way too many resources
Phil: on the webserver.
Michael: aah, is that what’s going on
Michael: i just had to restart it – it wasn’t responding to anything when I woke up, except pings
Phil: i suspect you’ll find tons of sa-learn processes
Michael: no ssh or httpd calls would answer
Michael: is it down again now?
Phil: i got ssh to work verrrrrry slowly.
Phil: No responses again now.
Michael: hrm
Michael: well, yeah, something’s borked – the UI is frozen
Michael: gonna restart it again
Phil: gah.
Phil: let me know when you restart it, i’ll ssh in and nuke my crontab before anything launches.
Michael: k
Michael: okay, login window is up
Michael: give it a shot
Phil: i’m in
Phil: gonna watch top and see what launches.
Michael: :nods
Phil: other than a lot of sendmail and procmail processes, nothing seems to be wigging out.
Michael: took out whatever it was that was having an issue? or it just doesn’t seem to be “issuing” right now?
Phil: i took out what i -think- was causing it. no idea if that was it or not.
Phil: but i recall something similar happening last time i tried to make a cron job for sa-learn.
Michael: okay
Michael: what’s sa-learn?
Phil: SpamAssassin’s Bayesian filter learning tool.
Phil: It’s the “learn this now, dammit” tool.
Michael: ah, okay
Phil: i’m still getting some false negatives that it should’ve caught.
Phil: um.
Phil: i don’t think that was it. i’m not getting any responses on ssh again.
Michael: yeah – the UI’s frozen again
Michael: wierd
Phil: goddamnit. what is going on?
Phil: i know i haven’t actually changed anything…
Michael: maybe it’s the machine? (though I hate to suggest that)
Michael: restarting again
Phil: well, we’ll see; i sincerely hope that’s not it.
Michael: you and me both
Michael: (and kirsten and dad too, I’d bet)
Phil: is it acting funny? fans any louder than usual?
Michael: doesn’t seem to be
Michael: i just sits in the corner and hums happily to itself (shrugs)
Michael: as opposed to Marvin, who sits in the corner and hums dismally to himself
Phil: no, i’d wager something is getting overloaded.
Phil: i don’t know what, though. hmmmm.
Michael: could I be getting hit by an attack of some sort, I wonder?
Phil: hmmm.
Michael: god_damn_ that thing takes a while to start up…lol
Phil: there’s a way to test that.
Michael: it’d have to be a pretty good attack to completely lock it up, though
Phil: during the brief time i can get on i’ll try to install snort.
Michael: it’s up
Phil: can’t get it to respond to ssh even now.
Michael: huh
Phil: n/m
Phil: there we go
Michael: okay
Phil: sshd probably wasn’t up yet
Michael: i’m logged in, have top -u 15 running in a term window
Phil: should see a wget process.
Michael: if it locks up, hopefully it’ll give me an idea of what the issue might be (maybe)
Phil: hopefully.
Michael: right now it’s mostly just top at the top of top (whee)
Michael: i’ve seen ssh, gzip, and tar pop up occasionally, but the drop right back down again
Phil: those were me.
Michael: i’ve got my eye on you, young man…lol
Michael: imapd just hit thetop – bouncing between 10 and 25%
Phil: probably mail trying to connect.
Michael: and…freeze
Michael: dammit
Michael: the last thing at the top was just imapd at only 10%
Phil: yeah, i noticed. hmm.
Phil: i’m going to nuke my entire crontab. nothing should be launching, but that doesn’t mean nothing is.
Michael: i’m pretty sure mine’s empty…very sure, in fact
Michael: okies, restarting
Phil: and /etc/crontab just has system stuff in it.
Michael: back up
Phil: yep. i’m in.
Phil: ah HA
Phil: apache2 is starting.
Michael: ?
Michael: i was about to run a permissions check/repair, just for grins and giggles…
Michael: oh, wait
Phil: if you still want to, go for it
Michael: apache2?
Michael: didn’t you kill that?
Phil: yes.
Phil: precisely my point. i thought i did too.
Michael: hmm
Michael: i wonder why it’s starting…and why it hasn’t been a problem until now
Michael: wow
Phil: OK, I need to go find the startup item for apache2 and slay it.
Michael: it’s fixing a lot of permissions
Phil: oh man, wait
Phil: that was NOT a good idea.
Michael: uhoh
Phil: any custom permissions I had set on my website are gonna be borked
Michael: oh shit
Phil: well, we’ll find out soon enough.
Michael: it seemed to be doing a lot in the /System/Library/Perl/ directory
Michael: trying to stop it now
Michael: spinning rainbow cursor at the moment
Phil: frozen again
Phil: from CL too
Phil: so apparently apache2 is not the problem, but it is a problem.
Michael: grrr
Michael: hmmm – the UI isn’t completely frozen
Michael: if I move the mouse, the cursor will jump to another point on the screen after a few seconds
Michael: still, it’s quite unuseable
Phil: probably the permissions check is taking a lot of cpu time.
Michael: gonna reboot again
Michael: christ, this is annoying…lol
Phil: tell me about it.
Phil: i want my email!
Michael: lol
Michael: if all else fails, there’s always the possibility of a reinstall
Michael: which, admittedly, could suck
Michael: system install, getting MT up and running again, making sure the websites don’t break, etc. etc., yadda yadda
Phil: Do an archive & install, then.
Phil: (If you have the disk space.)
Michael: :nods definitely
Michael: i do, there’s still gigs free
Michael: not the way I was planning on spending my saturday, though
Phil: nor I.
Michael: if I do have to do that…think it’s worth putting panther on, rather than jaguar?
Michael: i’d have to figure out the new mail system (postfix instead of sendmail, right?)
Michael: hm
Phil: if only for the fact that you’d have to configure postfix, i’m gonna say no
Michael: ah
Michael: okay
Michael: yeah – “starting apache 2 web server” is part of the startup window messages
Michael: and, we’re up again
Michael: holy shit – we’re locked up again
Michael: okay, the fact that that’s getting faster does not bode well
Phil: i noticed.
Michael: goddammit
Michael: y’know, if it was cloudy, i might not mind as much, but it’s a gorgeous day out there…lol
Michael: up
Michael: according to the apache2 control panel, it’s not running
Michael: even though the startup message was in the startup routine
Phil: ridiculous.
Michael: fuck – locked up
Michael: well, locking up
Michael: in the process
Michael: tried to load the sharing control panel
Michael: spinning rainbow cursor of death
Phil: this is bloody odd.
Phil: something has gone horribly wrong between yesterday afternoon and today.
Phil: i haven’t been around to do anything.
Michael: and i haven’t touched it
Phil: when it starts up, go to the apple menu, check about this mac, and see if anything on the amount of RAM or CPU speed looks unusual.
Michael: up
Michael: 640Mb ram…trying to remember if that’s right
Phil: yes.
Michael: yeah, that’s right
Phil: maxed out for this architecture
Michael: lol
Michael: brainless
Michael: hm
Michael: console log has some conflicts listed
Phil: do tell.
Michael: login window could not find image named ‘bang’
Phil: ….?
Michael: :shrugs
Michael: some startup items failed to launch due to conflicts
Michael: dammit…do I have ichat on that machine? lol
Phil: probably.
Michael: fuck, not that it matters
Michael: i’m losing the UI again
Michael: it just keeps slogging down, slower and slower
Michael: until it doesn’t respond
Phil: i noticed…. so weird.
Phil: incidentally, the apache2 item should now start apache
Michael: lol okay
Michael: it’s working, it’s just responding like XP installed on a 286
Michael: making glaciers look positively snappy, in other words
Michael: i don’t get it
Michael: okay – restarting it, but i’ve unplugged it from the network
Michael: going to see if that makes a difference at all (though I’m not sure why it would)
Phil: we’ll find out.
Phil: if that does it, you’re getting DOS’ed.
Michael: not thinking it’s a DOS or a hack or anything, simply because it’s behind my firewall, so huge amounts of traffic on my line should affect my connectivity/speed on this ‘puter, too
Phil: not if they specifically attacked the webserver by IP address or name.
Phil: well, some slowness, maybe.
Phil: but the webserver would see the majority of it.
Phil: …your windows box isn’t on the network, is it?
Michael: nope
Michael: well, yes, but it’s off
Michael: and on the few times I start it up, the first thing I do is apply any recent security updates
Michael: it’s as up to date as it can be without doing a daily security update check
Michael: okay, this isn’t good
Phil: eh, if it’s off, it doesn’t concern us.
Phil: what?
Michael: the box doesn’t seem to be restarting
Phil: …
Michael: i’m going to unplug it and let it sit for a few minutes
Phil: well, that’s not good.
Michael: given that this started happening when neither of us were doing anything on the box, and the fact that it seems to be a gradual thing – works fine for a bit, then gradually gets slower – and that it seems to be happening sooner on each reboot, and that it doesn’t want to reboot now…
Michael: i’m really starting to worry that it’s a hardware issue
Phil: that’s not good.
Michael: which, realistically, is a possibility – this little G3 was used when I got it, and it’s been running as a server 24/7 for, oh, probably around four or five years now
Michael: the three years since i’ve been in seattle, plus a year or two in anchorage
Michael: it would suck if that’s the case, though
Michael: i don’t have the funds for a replacement
Michael: and i could move things onto the G5, but i’ve liked having a server separate from my work machine
Phil: Yeah, I can definitely understand that… I was gonna ask about that.
Michael: it’s an option, though, should the g3 not come back
Phil: Do you still have the hardware diagnostic CD somewhere for the g3?
Michael: i don’t think I ever had one
Phil: must be a new thing.
Michael: :nods
Michael: either that, or I just didn’t get it with the machine
Phil: well, we have an hour and twelve minutes to figure something out.
Michael: 1:12? lol
Phil: I’m going out to the Irish festival at that point.
Michael: aah
Michael: yeah, I want to get out of the house at some point
Michael: shame to waste a perfectly good kilt day
Phil: amen. if I had a utilikilt i’d wear it today.
Michael: you should do it, man
Michael: not that you’d have it today, but still
Phil: yeah, i know. well, hey, my birthday’s coming up in a few months… we’ll see
Michael: excellent
Michael: okay, booting
Phil: OK, it freezes up, you need to restart in single-user mode and run fsck to see if there are disk errors.
Phil: http://www.macosxhints.com/article.php?story=2004011205473937&query=fsck
Michael: and I do that…how, again? lol
Phil: click the link
Michael: I did
Michael: The specific sequence of how to do this varies depending on what version of OS X you’re running. Go to Apple Support for specific instuctions.
Phil: it’s still detached from the network, right?
Michael: yeah
Phil: OK, let me find something that doesn’t suck i just noticed “check/fix the filesystem”
Michael: lol
Phil: i think you need to hold down S while booting.
Michael: gaaa…doesn’t seem to be booting
Michael: got the startup chime, nothing else
Phil: oh, maaaaan. that’s not good.
Phil: actually, though.
Phil: that might be a sign of a disk error caused by the forced reboots.
Phil: we had the exact same thing happen with a G4 at work.
Michael: :nods it got in last time, and I did a manual (correct) restart
Phil: iMovie froze, the finder crashed and wouldn’t restart, and everything rainbowed.
Phil: I’m going to suggest an archive and install.
Michael: :nods that’s what i’m thinking (grr)
Michael: i’ll deal with that later, though
Phil: just remember to hold down C as you turn it on with the jaguar disk in the driver (or panther; if you want, i can look up postfix guides while you wait)
Phil: k
Michael: right now…i’m a little annoyed, hungry, and want outside lol
Michael: i may do panther…it’s got enough under-the-hood upgrades that it might be a good idea
Michael: just take a little poking around to get postfix up
Phil: and really, from what i recall, setting up postfix wasn’t hard at all.
Phil: i did it at WWDC the day i installed panther.
Michael: as long as the websites survive, i’ll be satisfied
Phil: i think the config files made more sense to human beings
Michael: that’s my big worry, really
Michael: i’d hate to end up nuking everyone’s sites (again)
Phil: i do too. y’know, if all else fails, stick the HD into the G5. I assume it has an expansion slot for a second HD.
Michael: yeah, but I can’t just drop the drive in
Michael: different drive specs
Michael: serial ATA on the G5
Phil: oh, fuck, that’s right. dammit.
Michael: i’d need to drop the old drive(s) into ATA cases
Phil: if nothing else, even if the machine won’t boot, our data is still there.
Michael: yeah
Phil: you’re good, webmaster. go get food and sunshine; i’ll be doing the same myself shortly.
Michael: fuck – and i’m using michaelhanscom.com for most of my e-mail right now, too…lol
Michael: argh
Michael: ah, well
Michael: it’ll wait for a day
Michael: or a few hours, or whatever
Michael: anyway…i’m wandering – enjoy the irishfest
Phil: will do. later man
Update: Everything’s back up again. Phil tracked it down to a cron job that had piled a few thousand e-mail messages into his inbox, which caused everything to choke. Inbox is empty, cron is smacked back into submission, and all’s well that ends well.
