Newmoon damaged by power spike, murkworks mail down

We just had a couple of power spikes at the Murk, like we do. This time though they appear to have taken out one of our UPSes–and it was the one that newmoon, our mail server, was plugged into. When I tried to bring newmoon back up, it worked for a while and then hung. I am not currently able to log into the system to restart its services.

So this means that as of this afternoon, mail services on murkworks.net are down. This impacts not only personal accounts but the various mailing lists we run. Please spread the word if you’re on any of our mailing lists, and if you have backup personal email accounts, let people know they should use them this weekend if they need to get in touch with you.

solarbird or I will post here when we have more updates. Watch here, either of our journals, or my Twitter or Facebook accounts.

Sorry about this, folks. :/

Mirrored from annathepiper.org.

Open Source is the future of PAIN

Current status of newmoon: fucked.

I had discovered that the reason the box wasn’t ssh’able upon bootup was that ssh was segfaulting. So were the imap and pop3 daemons, and all of them were bitching about libcrypto.so.0.9.8. Fine, I think, and as per solarbird‘s advice, I decide to pull /usr/lib out of newmoon’s last backup files.

Which went swimmingly until the part where I tried to cut into the extraction from the tar file, which must have been before the damn files were actually extracted properly. Because when I copied them in on top of /usr/lib on newmoon and rebooted, they bitched about being “too short”. And now I can’t even log in to the damn box, even from the console. I try to log in as root, and I never get a password prompt, and it tells me “Invalid login”.

So now I’m going to have to create a Debian Live CD, in the hopes that booting from THAT will let me fix the damn system, and if that doesn’t work I’m going to have to ctrl-alt-fuckit and see if I can just upgrade to the new Debian 6 release. And that’s going to take oh so very long given that the Murkworks’ bandwidth is CRAP. And this is assuming we even keep door and lodestone running, given that we have a Wind Advisory up until 9pm this evening, with the potential of 50mph gusts.

All of which, given that a) I am fighting off a cold and really wanted to curled up under a blanket for most of this afternoon, and b) I wanted to be editing my novel today rather than doing emergency system repair, does NOT have me in a pleasant mood right now. I swear, right here and right now, that the next person who tries to preach to me about how awesome Linux is and how much better off we’d all be running Open Source software is going to get a keyboard shoved down their throat. When it comes to Open Source vs. The Damn Software Works Without Me Having to Google How to Do Every Step of the Process, you know what? I’ll take The Damn Software Works.

Mirrored from annathepiper.org.

All clear? Maybe? This is Seattle so more like partly cloudy

Okay folks, newmoon is talking to the world again. Mail seems to be coming in and going out. Mailing lists are apparently working again, and it is now once more possible to ssh into the box.

It took solarbird getting home to help me beat the box into shape, though. And this is what we had to do:

• Dara discovered the cd drive in the box wasn’t actually properly plugged in. Plugged that back in properly, only to discover that it was old and crotchety enough that it was refusing to read the rescue disc I’d created
• Tried to burn a second rescue disc, but the old and crotchety drive wouldn’t read that either, so Dara swapped the old drive out for a different one we had
• Using our backups from earlier in the week, I grabbed a screwed up copy of a file and got us back to the point of being to log in
• Once we were able to log in, I had to reinstall a package called libssl, which I had to do by manually grabbing the .deb file since Debian’s actually had an OS update and apt-get wouldn’t let me get the version I wanted (for those of you for whom that actually makes sense)

Long story short, those of you who rely on newmoon, please, PLEASE check your stuff and make sure that everything looks okay. If you have files locally stored on the box please ssh in and check them. If you’re getting mail from newmoon, please doublecheck that your mail clients are behaving as you expect.

Please report any issues to me or Dara. Drop a comment on this post, or else send me email at either my murkworks or my gmail addresses.

Mirrored from annathepiper.org.