Server issues

This doesn’t affect this particular site at all, but in case Dad or Kirsten check in, there seem to be issues with the djwudi.com/hanscomfamily.com/geekmuffin.com/interalia.org server at the moment. Phil and I are trying to work on it and find out what the issue is — whatever it is, it popped up sometime during the night, when neither he nor I were doing anything with the machine. No current ETA on when we’ll be back up and running, though we’re trying to bring it back up as soon as possible.

Update: Still down, still no ETA, and I need to get out of the house for a bit. I’ll get more info up when I can.

This also means that any e-mail sent to my [\@michaelhanscom.com]{.citation cites=”michaelhanscom.com”} address will not get to me at the moment. Please use one of my other e-mail addresses if you have one of them and need to get ahold of me.

Just so I don’t lose track or forget about what we tried so far, a quick copy-and-paste of the work so far:

Phil: el problemo, senor

Michael: what’s up?

Phil: i think something has run awry in my crontab
Phil: and is chewing up way too many resources
Phil: on the webserver.

Michael: aah, is that what’s going on
Michael: i just had to restart it – it wasn’t responding to anything when I woke up, except pings

Phil: i suspect you’ll find tons of sa-learn processes

Michael: no ssh or httpd calls would answer
Michael: is it down again now?

Phil: i got ssh to work verrrrrry slowly.
Phil: No responses again now.

Michael: hrm
Michael: well, yeah, something’s borked – the UI is frozen
Michael: gonna restart it again

Phil: gah.
Phil: let me know when you restart it, i’ll ssh in and nuke my crontab before anything launches.

Michael: k
Michael: okay, login window is up
Michael: give it a shot

Phil: i’m in
Phil: gonna watch top and see what launches.

Michael: :nods

Phil: other than a lot of sendmail and procmail processes, nothing seems to be wigging out.

Michael: took out whatever it was that was having an issue? or it just doesn’t seem to be “issuing” right now?

Phil: i took out what i -think- was causing it. no idea if that was it or not.
Phil: but i recall something similar happening last time i tried to make a cron job for sa-learn.

Michael: okay
Michael: what’s sa-learn?

Phil: SpamAssassin’s Bayesian filter learning tool.
Phil: It’s the “learn this now, dammit” tool.

Michael: ah, okay

Phil: i’m still getting some false negatives that it should’ve caught.
Phil: um.
Phil: i don’t think that was it. i’m not getting any responses on ssh again.

Michael: yeah – the UI’s frozen again
Michael: wierd

Phil: goddamnit. what is going on?
Phil: i know i haven’t actually changed anything…

Michael: maybe it’s the machine? (though I hate to suggest that)
Michael: restarting again

Phil: well, we’ll see; i sincerely hope that’s not it.

Michael: you and me both
Michael: (and kirsten and dad too, I’d bet)

Phil: is it acting funny? fans any louder than usual?

Michael: doesn’t seem to be
Michael: i just sits in the corner and hums happily to itself (shrugs)
Michael: as opposed to Marvin, who sits in the corner and hums dismally to himself

Phil: no, i’d wager something is getting overloaded.
Phil: i don’t know what, though. hmmmm.

Michael: could I be getting hit by an attack of some sort, I wonder?

Phil: hmmm.

Michael: god_damn_ that thing takes a while to start up…lol

Phil: there’s a way to test that.

Michael: it’d have to be a pretty good attack to completely lock it up, though

Phil: during the brief time i can get on i’ll try to install snort.

Michael: it’s up

Phil: can’t get it to respond to ssh even now.

Michael: huh

Phil: n/m
Phil: there we go

Michael: okay

Phil: sshd probably wasn’t up yet

Michael: i’m logged in, have top -u 15 running in a term window

Phil: should see a wget process.

Michael: if it locks up, hopefully it’ll give me an idea of what the issue might be (maybe)

Phil: hopefully.

Michael: right now it’s mostly just top at the top of top (whee)
Michael: i’ve seen ssh, gzip, and tar pop up occasionally, but the drop right back down again

Phil: those were me.

Michael: i’ve got my eye on you, young man…lol
Michael: imapd just hit thetop – bouncing between 10 and 25%

Phil: probably mail trying to connect.

Michael: and…freeze
Michael: dammit
Michael: the last thing at the top was just imapd at only 10%

Phil: yeah, i noticed. hmm.
Phil: i’m going to nuke my entire crontab. nothing should be launching, but that doesn’t mean nothing is.

Michael: i’m pretty sure mine’s empty…very sure, in fact
Michael: okies, restarting

Phil: and /etc/crontab just has system stuff in it.

Michael: back up

Phil: yep. i’m in.
Phil: ah HA
Phil: apache2 is starting.

Michael: ?
Michael: i was about to run a permissions check/repair, just for grins and giggles…
Michael: oh, wait

Phil: if you still want to, go for it

Michael: apache2?
Michael: didn’t you kill that?

Phil: yes.
Phil: precisely my point. i thought i did too.

Michael: hmm
Michael: i wonder why it’s starting…and why it hasn’t been a problem until now
Michael: wow

Phil: OK, I need to go find the startup item for apache2 and slay it.

Michael: it’s fixing a lot of permissions

Phil: oh man, wait
Phil: that was NOT a good idea.

Michael: uhoh

Phil: any custom permissions I had set on my website are gonna be borked

Michael: oh shit

Phil: well, we’ll find out soon enough.

Michael: it seemed to be doing a lot in the /System/Library/Perl/ directory
Michael: trying to stop it now
Michael: spinning rainbow cursor at the moment

Phil: frozen again
Phil: from CL too
Phil: so apparently apache2 is not the problem, but it is a problem.

Michael: grrr
Michael: hmmm – the UI isn’t completely frozen
Michael: if I move the mouse, the cursor will jump to another point on the screen after a few seconds
Michael: still, it’s quite unuseable

Phil: probably the permissions check is taking a lot of cpu time.

Michael: gonna reboot again
Michael: christ, this is annoying…lol

Phil: tell me about it.
Phil: i want my email!

Michael: lol
Michael: if all else fails, there’s always the possibility of a reinstall
Michael: which, admittedly, could suck
Michael: system install, getting MT up and running again, making sure the websites don’t break, etc. etc., yadda yadda

Phil: Do an archive & install, then.
Phil: (If you have the disk space.)

Michael: :nods definitely
Michael: i do, there’s still gigs free
Michael: not the way I was planning on spending my saturday, though

Phil: nor I.

Michael: if I do have to do that…think it’s worth putting panther on, rather than jaguar?
Michael: i’d have to figure out the new mail system (postfix instead of sendmail, right?)
Michael: hm

Phil: if only for the fact that you’d have to configure postfix, i’m gonna say no

Michael: ah
Michael: okay
Michael: yeah – “starting apache 2 web server” is part of the startup window messages
Michael: and, we’re up again
Michael: holy shit – we’re locked up again
Michael: okay, the fact that that’s getting faster does not bode well

Phil: i noticed.

Michael: goddammit
Michael: y’know, if it was cloudy, i might not mind as much, but it’s a gorgeous day out there…lol
Michael: up
Michael: according to the apache2 control panel, it’s not running
Michael: even though the startup message was in the startup routine

Phil: ridiculous.

Michael: fuck – locked up
Michael: well, locking up
Michael: in the process
Michael: tried to load the sharing control panel
Michael: spinning rainbow cursor of death

Phil: this is bloody odd.
Phil: something has gone horribly wrong between yesterday afternoon and today.
Phil: i haven’t been around to do anything.

Michael: and i haven’t touched it

Phil: when it starts up, go to the apple menu, check about this mac, and see if anything on the amount of RAM or CPU speed looks unusual.

Michael: up
Michael: 640Mb ram…trying to remember if that’s right

Phil: yes.

Michael: yeah, that’s right

Phil: maxed out for this architecture

Michael: lol
Michael: brainless
Michael: hm
Michael: console log has some conflicts listed

Phil: do tell.

Michael: login window could not find image named ‘bang’

Phil: ….?

Michael: :shrugs
Michael: some startup items failed to launch due to conflicts
Michael: dammit…do I have ichat on that machine? lol

Phil: probably.

Michael: fuck, not that it matters
Michael: i’m losing the UI again
Michael: it just keeps slogging down, slower and slower
Michael: until it doesn’t respond

Phil: i noticed…. so weird.
Phil: incidentally, the apache2 item should now start apache

Michael: lol okay
Michael: it’s working, it’s just responding like XP installed on a 286
Michael: making glaciers look positively snappy, in other words
Michael: i don’t get it
Michael: okay – restarting it, but i’ve unplugged it from the network
Michael: going to see if that makes a difference at all (though I’m not sure why it would)

Phil: we’ll find out.
Phil: if that does it, you’re getting DOS’ed.

Michael: not thinking it’s a DOS or a hack or anything, simply because it’s behind my firewall, so huge amounts of traffic on my line should affect my connectivity/speed on this ‘puter, too

Phil: not if they specifically attacked the webserver by IP address or name.
Phil: well, some slowness, maybe.
Phil: but the webserver would see the majority of it.
Phil: …your windows box isn’t on the network, is it?

Michael: nope
Michael: well, yes, but it’s off
Michael: and on the few times I start it up, the first thing I do is apply any recent security updates
Michael: it’s as up to date as it can be without doing a daily security update check
Michael: okay, this isn’t good

Phil: eh, if it’s off, it doesn’t concern us.
Phil: what?

Michael: the box doesn’t seem to be restarting

Phil: …

Michael: i’m going to unplug it and let it sit for a few minutes

Phil: well, that’s not good.

Michael: given that this started happening when neither of us were doing anything on the box, and the fact that it seems to be a gradual thing – works fine for a bit, then gradually gets slower – and that it seems to be happening sooner on each reboot, and that it doesn’t want to reboot now…
Michael: i’m really starting to worry that it’s a hardware issue

Phil: that’s not good.

Michael: which, realistically, is a possibility – this little G3 was used when I got it, and it’s been running as a server 24/7 for, oh, probably around four or five years now
Michael: the three years since i’ve been in seattle, plus a year or two in anchorage
Michael: it would suck if that’s the case, though
Michael: i don’t have the funds for a replacement
Michael: and i could move things onto the G5, but i’ve liked having a server separate from my work machine

Phil: Yeah, I can definitely understand that… I was gonna ask about that.

Michael: it’s an option, though, should the g3 not come back

Phil: Do you still have the hardware diagnostic CD somewhere for the g3?

Michael: i don’t think I ever had one

Phil: must be a new thing.

Michael: :nods
Michael: either that, or I just didn’t get it with the machine

Phil: well, we have an hour and twelve minutes to figure something out.

Michael: 1:12? lol

Phil: I’m going out to the Irish festival at that point.

Michael: aah
Michael: yeah, I want to get out of the house at some point
Michael: shame to waste a perfectly good kilt day

Phil: amen. if I had a utilikilt i’d wear it today.

Michael: you should do it, man

Michael: not that you’d have it today, but still

Phil: yeah, i know. well, hey, my birthday’s coming up in a few months… we’ll see

Michael: excellent
Michael: okay, booting

Phil: OK, it freezes up, you need to restart in single-user mode and run fsck to see if there are disk errors.
Phil: http://www.macosxhints.com/article.php?story=2004011205473937&query=fsck

Michael: and I do that…how, again? lol

Phil: click the link

Michael: I did
Michael: The specific sequence of how to do this varies depending on what version of OS X you’re running. Go to Apple Support for specific instuctions.

Phil: it’s still detached from the network, right?

Michael: yeah

Phil: OK, let me find something that doesn’t suck i just noticed “check/fix the filesystem”

Michael: lol

Phil: i think you need to hold down S while booting.

Michael: gaaa…doesn’t seem to be booting
Michael: got the startup chime, nothing else

Phil: oh, maaaaan. that’s not good.
Phil: actually, though.
Phil: that might be a sign of a disk error caused by the forced reboots.
Phil: we had the exact same thing happen with a G4 at work.

Michael: :nods it got in last time, and I did a manual (correct) restart

Phil: iMovie froze, the finder crashed and wouldn’t restart, and everything rainbowed.
Phil: I’m going to suggest an archive and install.

Michael: :nods that’s what i’m thinking (grr)
Michael: i’ll deal with that later, though

Phil: just remember to hold down C as you turn it on with the jaguar disk in the driver (or panther; if you want, i can look up postfix guides while you wait)
Phil: k

Michael: right now…i’m a little annoyed, hungry, and want outside lol
Michael: i may do panther…it’s got enough under-the-hood upgrades that it might be a good idea
Michael: just take a little poking around to get postfix up

Phil: and really, from what i recall, setting up postfix wasn’t hard at all.
Phil: i did it at WWDC the day i installed panther.

Michael: as long as the websites survive, i’ll be satisfied

Phil: i think the config files made more sense to human beings

Michael: that’s my big worry, really
Michael: i’d hate to end up nuking everyone’s sites (again)

Phil: i do too. y’know, if all else fails, stick the HD into the G5. I assume it has an expansion slot for a second HD.

Michael: yeah, but I can’t just drop the drive in
Michael: different drive specs
Michael: serial ATA on the G5

Phil: oh, fuck, that’s right. dammit.

Michael: i’d need to drop the old drive(s) into ATA cases

Phil: if nothing else, even if the machine won’t boot, our data is still there.

Michael: yeah

Phil: you’re good, webmaster. go get food and sunshine; i’ll be doing the same myself shortly.

Michael: fuck – and i’m using michaelhanscom.com for most of my e-mail right now, too…lol
Michael: argh
Michael: ah, well
Michael: it’ll wait for a day
Michael: or a few hours, or whatever
Michael: anyway…i’m wandering – enjoy the irishfest

Phil: will do. later man

Update: Everything’s back up again. Phil tracked it down to a cron job that had piled a few thousand e-mail messages into his inbox, which caused everything to choke. Inbox is empty, cron is smacked back into submission, and all’s well that ends well.

iTunes: “Three Wishes” by Waters, Roger from the album Amused to Death (1992, 6:50).