Ordinary Day

Crossposted from annathepiper.org. Original post: http://www.annathepiper.org/2020/06/21/mail-outage-on-murkworks-net/

Dara and I have been having to conduct critical unscheduled maintenance of our mail server today. This was precipitated by getting a voice mail from Comcast this morning asking us to call them, and noting a risk of possible cessation of our service. I went “WUT?” and called them, and was informed they’d noticed one of our machines carrying out DDoSes.

To which Dara and I both went “WUT?!” And after we thanked Comcast for the heads up, Dara moved forward with completely flattening the box so we could do a brand new clean install with the latest version of Debian Linux. (We were going to do some dist upgrades anyway over this forthcoming winter holiday, but this just escalated the priority.)

And we went down to Frys this afternoon to get a new hard drive and power supply too, since the box was down anyway. We’re both rather impressed that the hard drives previously in newmoon had dated back to 2004 (whoa) and 1998 (WHOA!). And we’ve brought home a new drive that’s acres more space than we had on newmoon before, not to mention much quieter. So the server room should have a significant reduction in noise now.

BUT. This also means we’ve had the server out all day, and it should be down probably for the rest of the evening. Hopefully it’ll be back up by tomorrow, but operating system upgrades can be dicey. So don’t be terribly surprised if we’re not up tomorrow morning.

ALSO: this impacts only our mail server. So the mailing lists we host (notably the LexFA list and the Filk list) are down, and users who have mail accounts on our system won’t be able to use those accounts till newmoon is back up. Web services are unaffected.

Please let Dara and me know if you have any questions, and watch her and my accounts for further updates!

Mirrored from angelahighland.com.

All,

Dara had to perform some maintenance on our web server last night, and it turned out to be an unexpectedly bumpy ride–thanks to unexpected changes in the process of updating to Apache 2.4. All this work kept her up until 4:30. AUGH.

Likewise, I have also updated all blogs we host to WordPress 4.2.1.

We THINK we’ve gotten everything working, but just in case, if anyone reading this has resources we host, doublecheck your stuff and let us know if anything looks screwy. And if anybody sees anything screwy with angelahighland.com in particular, let me know and I’ll check it out.

Mirrored from angelahighland.com.

We actually regained power overnight, but we didn’t get the servers back up and running until we got up this morning. However, we ARE now back up and running!

And Dara even took the opportunity to upgrade the network card in our webserver, so now hopefully our websites will be much more zippy for all you lovely visitors who come by to say hi. Let us know if you see any problems, won’t you?

Hope everybody made it through the windy action okay! And all hail the power company crews who’ve been working for the last several hours to get power back on for us all!

Mirrored from angelahighland.com.

Heads up, you guys, we have a Wind Event coming in.

The PI is reporting about the incoming wind fun tomorrow night, so here’s the obligatory MURKWORKS.NET MAY GO OFFLINE TOMORROW NIGHT advisory. Dara and I will post if our servers have to go down.

ALSO: the PI’s saying one of our local meteorologists is advising people to go home early if possible. And the current High Wind Watch is talking about the winds hitting us possibly during tomorrow night’s commute. Batten down the hatches, this could get messy.

Mirrored from angelahighland.com.

Attention everybody:

Dara needs to take down our web server this morning for security maintenance, so my website, hers, and all others we host will be temporarily inaccessible for a while this morning. Please stand by and we’ll post another update when web services are restored!

ETA: Web maintenance is now complete. Please let Dara or me know if any of our web resources look b0rked!

ETA #2: Second round of maintenance will be happening TONIGHT. Dara has to do our other server, door. So we’ll have another round of inaccessibility this evening as she works on that. Watch this space for further details.

ETA #3: Murkworks.net updates are confirmed finished for the evening, so far as we know. Dara finished before I got home, but we should be good. Please let Dara or me know if you see any weirdness visiting any of the sites we host.

Mirrored from angelahighland.com.

As many of you know, Dara and I host our own teeny Internet site, including Web and mail support. As part of this, we host several mailing lists, using MailMan as our mailing list system of choice.

This weekend one of our users on the LexFA list (that’s the mailing list for the Lexington Fantasy Association) reported a weird problem to me. He was subscribed to the list with a Yahoo email address, but not receiving email from the list. I logged into the list’s administration website, checked the member settings, and determined that he was indeed correctly subscribed to the list, and not set Nomail or blocked for too many bounces or anything of that nature. As far as the list was concerned, he should have been getting mail and wasn’t.

So, since software testing is in fact what I do for my day job, I immediately went, “Hey, I have a Yahoo account myself. Let’s see if I can reproduce this problem.”

I COULD. I was able to subscribe my Yahoo account to the list. I was able to post to it–which I confirmed by monitoring the list’s archives, where the message showed up. Likewise, Dara confirmed by monitoring our system logs that the message got to our server.

Where it fell over, however, was that message trying to get back to Yahoo so that my Inbox could actually see it.

And further investigation finally got me to this Computerworld article. The tl;dr version of this, in case you aren’t a techie, is that basically Yahoo instituted an anti-spam tactic that sounds good in theory: i.e., it’s trying to prevent spammers from sending mail that pretends to be from legitimate Yahoo users. Yahoo has a setting in place that basically now says “If you get a mail that claims to be from a Yahoo user, and it didn’t actually come from our servers, you should bounce it because it’s probably spam”.

The problem with this, though, is that it breaks mailing list behavior. Because what happens now is this:

• Yahoo user sends a mail to a mailing list she’s on.
• The mailing list goes “ah, I have a mail from a subscriber! I shall send it to all the other subscribers!”
• Then the mail tries to come back to the Yahoo user. Except now Yahoo’s own servers see this mail come in, which is claiming to be from a Yahoo user. (Because it IS, because it’s the user’s own mail to her mailing list that she’s on.) BUT, Yahoo also sees that this mail didn’t come from Yahoo’s servers. (Which it didn’t, at least in our case, because our mailing list is not hosted with Yahoo.)
• Yahoo’s servers go “MUST BE SPAM” and promptly ditch the mail before the user ever sees it.
• End result: the user wonders where the hell her mail is, and whether something is broken about the list, or whether she got unsubscribed by mistake. When all this time, nothing is wrong with the list at all.

Yahoo, as per this Help link on yahoo.com and this link on the Yahoomail tumblr, is aware of the problem. However, their suggestion for how mailing lists should handle this is suboptimal–i.e., that we should set our mailing lists to have the list be the sending address. This would result in not being able to see who sends what messages.

So for now Dara and I are moving forward with an attempt to do a distribution upgrade on our mail server, for starters. If this is successful, this should let us upgrade our MailMan system to a version that’ll handle Yahoo’s more stringent settings.

In the meantime, though, if anyone reading this is trying to get mailing list mails at a Yahoo address and you’re having trouble with it, chances are good that this is why. You may need to consider getting your mail at an alternate address.

Apologies to folks directly impacted by this on our mailing lists–hopefully we can get a more recent version of MailMan running, and fix this problem! more as I know it!

ETA Dara and I were up till 1am last night trying to fix this, and now we do at least have a fix in place. We updated our MailMan install to the latest available version, 2.1.18-1, which has settings to talk to what Yahoo did and let mail come on through. However, THAT required Dara to do local tweaking of the source code so that we could actually have emails to the list still have identifiable senders, about which she is displeased. She posts about it here.

Mirrored from angelahighland.com.

FYI to all, our mail server at Murkworks.net remains down. Dara tried to swap out the power supply last night, but that didn’t work, so now she’s working on swapping out the motherboard in the system in question.

Anyone who usually gets mail on our system, please tell folks to contact you via alternate email addresses today if they need to. If you need to ping me by email, my annathepiper address on gmail will work.

Mailing lists that we host remain also out of commission (which is particularly irritating to me, given that we just took over the filk list)!

Web services however are UP, and all web sites that we host should be working correctly, since we keep web sites on one of the other servers.

More bulletins as events warrant.

ETA: The attempt to install the motherboard was UNSUCCESSFUL. It’s apparently dead. So I’ll be going out to get us a fresh one for Dara to install. Watch this post for further updates!

Mirrored from angelahighland.com.