annathepiper: (Wet Strongbow)
Power has gone down at the Murkworks, and as of this writing, Puget Sound Energy's outage app claims we will be back by 4:30pm. But given that the count of outages is about 43K and rising, I am dubious about that repair time. :(

More importantly, we are less worried about the outage per se and more about power surges on the lines. :( Dara has measured surges of about 14 volts coming over our lines. So we have unplugged ALL THE THINGS. If you live in the area and you are in PSE's service area in particular, UNPLUG YOUR ELECTRONICS NOW.

As I have warned before, storm outages like this are exactly why I have my backup site on Wordpress.com now. So if you happen to need to look up anything about my books during this weekend, please look on angelahighland.wordpress.com. I will post there as well as here when our weather situation has stabilized and our power is back on.

All other resources hosted by our site are also down. This includes our mailing lists as well as web sites.

Stay safe and dry, fellow Cascadians!
annathepiper: (Hard Day)
For everybody who has resources hosted on lodestone, the Murkworks.net web server: Dara and I discovered this morning that the root password on that box is no longer working. We have to therefore assume that the system has been compromised, and all data on it is now suspect. We will be doing a complete system reset on the box, and we will have to ask all users with lodestone passwords to change passwords--not only on lodestone, but on newmoon and door as well if you have accounts on those boxes.

Which means that we will need to reach out to individual users by phone and work with you to get your passwords safely updated. We will not, repeat, NOT send password data by email. If we don't already have current contact data for you, please email me ASAP at my gmail account, annathepiper, and let me know how we can get in touch with you.

Note as well that this means that all web pages we host will be down until this is resolved. Particularly the webpages that are hosted on Wordpress installs.

We apologize for this inconvenience. And you may assume we are swearing vigorously about having to do ANOTHER server reset. :(
annathepiper: (Hard Day)

Dara and I have been having to conduct critical unscheduled maintenance of our mail server today. This was precipitated by getting a voice mail from Comcast this morning asking us to call them, and noting a risk of possible cessation of our service. I went “WUT?” and called them, and was informed they’d noticed one of our machines carrying out DDoSes.

To which Dara and I both went “WUT?!” And after we thanked Comcast for the heads up, Dara moved forward with completely flattening the box so we could do a brand new clean install with the latest version of Debian Linux. (We were going to do some dist upgrades anyway over this forthcoming winter holiday, but this just escalated the priority.)

And we went down to Frys this afternoon to get a new hard drive and power supply too, since the box was down anyway. We’re both rather impressed that the hard drives previously in newmoon had dated back to 2004 (whoa) and 1998 (WHOA!). And we’ve brought home a new drive that’s acres more space than we had on newmoon before, not to mention much quieter. So the server room should have a significant reduction in noise now.

BUT. This also means we’ve had the server out all day, and it should be down probably for the rest of the evening. Hopefully it’ll be back up by tomorrow, but operating system upgrades can be dicey. So don’t be terribly surprised if we’re not up tomorrow morning.

ALSO: this impacts only our mail server. So the mailing lists we host (notably the LexFA list and the Filk list) are down, and users who have mail accounts on our system won’t be able to use those accounts till newmoon is back up. Web services are unaffected.

Please let Dara and me know if you have any questions, and watch her and my accounts for further updates!

Mirrored from angelahighland.com.

annathepiper: (Default)
All,

Please be advised that we're having a big ol' windstorm in the Puget Sound metro area today. As of this writing, Puget Sound Energy has over 138,000 outages and Seattle City Light has over 35,000. Our power took a couple of hard hits, although we came back up while I was in the process of taking down our servers.

We've continued to have hard power hits, so I have not yet brought our servers back up. If we stay stable long enough, I will bring our servers back up by 2pm Pacific time, but as we're continuing to have power hits, I ain't betting on this happening.

I will post further updates to my social media accounts, as well as here, since my phone will continue to talk to the world even if our power goes out.

More bulletins as events warrant!
annathepiper: (Little Help?)

All,

Dara had to perform some maintenance on our web server last night, and it turned out to be an unexpectedly bumpy ride–thanks to unexpected changes in the process of updating to Apache 2.4. All this work kept her up until 4:30. AUGH.

Likewise, I have also updated all blogs we host to WordPress 4.2.1.

We THINK we’ve gotten everything working, but just in case, if anyone reading this has resources we host, doublecheck your stuff and let us know if anything looks screwy. And if anybody sees anything screwy with angelahighland.com in particular, let me know and I’ll check it out.

Mirrored from angelahighland.com.

annathepiper: (Alan and Sean Ordinary Day)

ATTENTION anybody who has a mail account on Murkworks.net:

Right now we’re having issues with any of our accounts sending mail out to Gmail addresses. Gmail is incorrectly dinging us as a spam sender, and either shunting mail from us into Spam folders or else outright bouncing it. Needless to say, this is massively annoying.

I’ve just been informed on Twitter that Google is already investigating similar reports, so this may or may not be clearing itself up soon. Dara’s been trying to investigate as well.

In the meantime, if you have alternate email addresses to use, you should do so. We apologize for the inconvenience. :(

Mirrored from angelahighland.com.

annathepiper: (Alan YES!)

We actually regained power overnight, but we didn’t get the servers back up and running until we got up this morning. However, we ARE now back up and running!

And Dara even took the opportunity to upgrade the network card in our webserver, so now hopefully our websites will be much more zippy for all you lovely visitors who come by to say hi. Let us know if you see any problems, won’t you?

Hope everybody made it through the windy action okay! And all hail the power company crews who’ve been working for the last several hours to get power back on for us all!

Mirrored from angelahighland.com.

annathepiper: (Default)
We just lost power in Kenmore, and the UPSes are complaining, so we're shutting down the servers. 65,000 people out and rising, as per Puget Sound Energy.

This may be a while, and we may not be back until tomorrow. Will post to all the various social networks to let folks know when our servers are back up.
annathepiper: (Wet Strongbow)

Heads up, you guys, we have a Wind Event coming in.

The PI is reporting about the incoming wind fun tomorrow night, so here’s the obligatory MURKWORKS.NET MAY GO OFFLINE TOMORROW NIGHT advisory. Dara and I will post if our servers have to go down.

ALSO: the PI’s saying one of our local meteorologists is advising people to go home early if possible. And the current High Wind Watch is talking about the winds hitting us possibly during tomorrow night’s commute. Batten down the hatches, this could get messy.

Mirrored from angelahighland.com.

annathepiper: (Alan YES!)
So local friends, if any of you are still without power and you need somewhere to charge your devices and/or laptops, contact us.

Dara has brought the servers back up, but door is being wonky, so some of our hosted sites still aren't working quite correctly. And I haven't been able to restart the MUSH yet. Will continue to poke at our system functionality until we're fully running again. More bulletins as events warrant.

Meanwhile, much relieved, as this means I can continue work on Chapter 20. Not to mention do my laundry so I will have something to wear to work tomorrow!
annathepiper: (Hard Day)
For the benefit of anyone who didn't see me post to the social networks, the power at the Murkworks has gone down in tonight's windstorm. I've reported the outage to Puget Sound Energy, but no estimate yet on when we'll be back online. Given that they're up to over 70,000 customers with reported outages, I think they're in for a busy night. So we may be dark for a while.

We still have outgoing connectivity, but to be on on the safe side Dara has taken down our servers. So our web and mail connectivity is out, as well as the MUSH. Apologies to all who have resources hosted with us. We'll bring stuff back up as soon as we have power restored.

Meanwhile, Dreamwidth/Livejournal people, keep an eye on my posts on these two places. People who follow me on the social networks, I'll be posting updates to Facebook, Google+, and Twitter as well as long as we have backup power for our laptops and devices. And I'm reachable at my gmail address, annathepiper, if you need to send me email.

Writing Chapter 20 via lamplight, woo!
annathepiper: (Hard Day)

Attention everybody:

Dara needs to take down our web server this morning for security maintenance, so my website, hers, and all others we host will be temporarily inaccessible for a while this morning. Please stand by and we’ll post another update when web services are restored!

ETA: Web maintenance is now complete. Please let Dara or me know if any of our web resources look b0rked!

ETA #2: Second round of maintenance will be happening TONIGHT. Dara has to do our other server, door. So we’ll have another round of inaccessibility this evening as she works on that. Watch this space for further details.

ETA #3: Murkworks.net updates are confirmed finished for the evening, so far as we know. Dara finished before I got home, but we should be good. Please let Dara or me know if you see any weirdness visiting any of the sites we host.

Mirrored from angelahighland.com.

annathepiper: (Hard Day)

Attention everybody:

Dara needs to take down our web server this morning for security maintenance, so my website, hers, and all others we host will be temporarily inaccessible for a while this morning. Please stand by and we’ll post another update when web services are restored!

ETA: Web maintenance is now complete. Please let Dara or me know if any of our web resources look b0rked!

ETA #2: Second round of maintenance will be happening TONIGHT. Dara has to do our other server, door. So we’ll have another round of inaccessibility this evening as she works on that. Watch this space for further details.

ETA #3: Murkworks.net updates are confirmed finished for the evening, so far as we know. Dara finished before I got home, but we should be good. Please let Dara or me know if you see any weirdness visiting any of the sites we host.

Mirrored from angelahighland.com.

annathepiper: (Final Test)

As many of you know, Dara and I host our own teeny Internet site, including Web and mail support. As part of this, we host several mailing lists, using MailMan as our mailing list system of choice.

This weekend one of our users on the LexFA list (that’s the mailing list for the Lexington Fantasy Association) reported a weird problem to me. He was subscribed to the list with a Yahoo email address, but not receiving email from the list. I logged into the list’s administration website, checked the member settings, and determined that he was indeed correctly subscribed to the list, and not set Nomail or blocked for too many bounces or anything of that nature. As far as the list was concerned, he should have been getting mail and wasn’t.

So, since software testing is in fact what I do for my day job, I immediately went, “Hey, I have a Yahoo account myself. Let’s see if I can reproduce this problem.”

I COULD. I was able to subscribe my Yahoo account to the list. I was able to post to it–which I confirmed by monitoring the list’s archives, where the message showed up. Likewise, Dara confirmed by monitoring our system logs that the message got to our server.

Where it fell over, however, was that message trying to get back to Yahoo so that my Inbox could actually see it.

And further investigation finally got me to this Computerworld article. The tl;dr version of this, in case you aren’t a techie, is that basically Yahoo instituted an anti-spam tactic that sounds good in theory: i.e., it’s trying to prevent spammers from sending mail that pretends to be from legitimate Yahoo users. Yahoo has a setting in place that basically now says “If you get a mail that claims to be from a Yahoo user, and it didn’t actually come from our servers, you should bounce it because it’s probably spam”.

The problem with this, though, is that it breaks mailing list behavior. Because what happens now is this:

  • Yahoo user sends a mail to a mailing list she’s on.
  • The mailing list goes “ah, I have a mail from a subscriber! I shall send it to all the other subscribers!”
  • Then the mail tries to come back to the Yahoo user. Except now Yahoo’s own servers see this mail come in, which is claiming to be from a Yahoo user. (Because it IS, because it’s the user’s own mail to her mailing list that she’s on.) BUT, Yahoo also sees that this mail didn’t come from Yahoo’s servers. (Which it didn’t, at least in our case, because our mailing list is not hosted with Yahoo.)
  • Yahoo’s servers go “MUST BE SPAM” and promptly ditches the mail before the user ever sees it.
  • End result: the user wonders where the hell her mail is, and whether something is broken about the list, or whether she got unsubscribed by mistake. When all this time, nothing is wrong with the list at all.

Yahoo, as per this Help link on yahoo.com and this link on the Yahoomail tumblr, is aware of the problem. However, their suggestion for how mailing lists should handle this is suboptimal–i.e., that we should set our mailing lists to have the list be the sending address. This would result in not being able to see who sends what messages.

So for now Dara and I are moving forward with an attempt to do a distribution upgrade on our mail server, for starters. If this is successful, this should let us upgrade our MailMan system to a version that’ll handle Yahoo’s more stringent settings.

In the meantime, though, if anyone reading this is trying to get mailing list mails at a Yahoo address and you’re having trouble with it, chances are good that this is why. You may need to consider getting your mail at an alternate address.

Apologies to folks directly impacted by this on our mailing lists–hopefully we can get a more recent version of MailMan running, and fix this problem! more as I know it!

ETA Dara and I were up till 1am last night trying to fix this, and now we do at least have a fix in place. We updated our MailMan install to the latest available version, 2.1.18-1, which has settings to talk to what Yahoo did and let mail come on through. However, THAT required Dara to do local tweaking of the source code so that we could actually have emails to the list still have identifiable senders, about which she is displeased. She posts about it here.

Mirrored from angelahighland.com.

annathepiper: (Final Test)

As many of you know, Dara and I host our own teeny Internet site, including Web and mail support. As part of this, we host several mailing lists, using MailMan as our mailing list system of choice.

This weekend one of our users on the LexFA list (that’s the mailing list for the Lexington Fantasy Association) reported a weird problem to me. He was subscribed to the list with a Yahoo email address, but not receiving email from the list. I logged into the list’s administration website, checked the member settings, and determined that he was indeed correctly subscribed to the list, and not set Nomail or blocked for too many bounces or anything of that nature. As far as the list was concerned, he should have been getting mail and wasn’t.

So, since software testing is in fact what I do for my day job, I immediately went, “Hey, I have a Yahoo account myself. Let’s see if I can reproduce this problem.”

I COULD. I was able to subscribe my Yahoo account to the list. I was able to post to it–which I confirmed by monitoring the list’s archives, where the message showed up. Likewise, Dara confirmed by monitoring our system logs that the message got to our server.

Where it fell over, however, was that message trying to get back to Yahoo so that my Inbox could actually see it.

And further investigation finally got me to this Computerworld article. The tl;dr version of this, in case you aren’t a techie, is that basically Yahoo instituted an anti-spam tactic that sounds good in theory: i.e., it’s trying to prevent spammers from sending mail that pretends to be from legitimate Yahoo users. Yahoo has a setting in place that basically now says “If you get a mail that claims to be from a Yahoo user, and it didn’t actually come from our servers, you should bounce it because it’s probably spam”.

The problem with this, though, is that it breaks mailing list behavior. Because what happens now is this:

  • Yahoo user sends a mail to a mailing list she’s on.
  • The mailing list goes “ah, I have a mail from a subscriber! I shall send it to all the other subscribers!”
  • Then the mail tries to come back to the Yahoo user. Except now Yahoo’s own servers see this mail come in, which is claiming to be from a Yahoo user. (Because it IS, because it’s the user’s own mail to her mailing list that she’s on.) BUT, Yahoo also sees that this mail didn’t come from Yahoo’s servers. (Which it didn’t, at least in our case, because our mailing list is not hosted with Yahoo.)
  • Yahoo’s servers go “MUST BE SPAM” and promptly ditch the mail before the user ever sees it.
  • End result: the user wonders where the hell her mail is, and whether something is broken about the list, or whether she got unsubscribed by mistake. When all this time, nothing is wrong with the list at all.

Yahoo, as per this Help link on yahoo.com and this link on the Yahoomail tumblr, is aware of the problem. However, their suggestion for how mailing lists should handle this is suboptimal–i.e., that we should set our mailing lists to have the list be the sending address. This would result in not being able to see who sends what messages.

So for now Dara and I are moving forward with an attempt to do a distribution upgrade on our mail server, for starters. If this is successful, this should let us upgrade our MailMan system to a version that’ll handle Yahoo’s more stringent settings.

In the meantime, though, if anyone reading this is trying to get mailing list mails at a Yahoo address and you’re having trouble with it, chances are good that this is why. You may need to consider getting your mail at an alternate address.

Apologies to folks directly impacted by this on our mailing lists–hopefully we can get a more recent version of MailMan running, and fix this problem! more as I know it!

ETA Dara and I were up till 1am last night trying to fix this, and now we do at least have a fix in place. We updated our MailMan install to the latest available version, 2.1.18-1, which has settings to talk to what Yahoo did and let mail come on through. However, THAT required Dara to do local tweaking of the source code so that we could actually have emails to the list still have identifiable senders, about which she is displeased. She posts about it here.

Mirrored from angelahighland.com.

annathepiper: (Hard Day)

FYI to all, our mail server at Murkworks.net remains down. Dara tried to swap out the power supply last night, but that didn’t work, so now she’s working on swapping out the motherboard in the system in question.

Anyone who usually gets mail on our system, please tell folks to contact you via alternate email addresses today if they need to. If you need to ping me by email, my annathepiper address on gmail will work.

Mailing lists that we host remain also out of commission (which is particularly irritating to me, given that we just took over the filk list)!

Web services however are UP, and all web sites that we host should be working correctly, since we keep web sites on one of the other servers.

More bulletins as events warrant.

ETA: The attempt to install the motherboard was UNSUCCESSFUL. It’s apparently dead. So I’ll be going out to get us a fresh one for Dara to install. Watch this post for further updates!

Mirrored from angelahighland.com.

annathepiper: (Hard Day)

FYI to all, our mail server at Murkworks.net remains down. Dara tried to swap out the power supply last night, but that didn’t work, so now she’s working on swapping out the motherboard in the system in question.

Anyone who usually gets mail on our system, please tell folks to contact you via alternate email addresses today if they need to. If you need to ping me by email, my annathepiper address on gmail will work.

Mailing lists that we host remain also out of commission (which is particularly irritating to me, given that we just took over the filk list)!

Web services however are UP, and all web sites that we host should be working correctly, since we keep web sites on one of the other servers.

More bulletins as events warrant.

ETA: The attempt to install the motherboard was UNSUCCESSFUL. It’s apparently dead. So I’ll be going out to get us a fresh one for Dara to install. Watch this post for further updates!

Mirrored from angelahighland.com.

annathepiper: (Final Test)
FYI all, as of this writing Murkworks.net is down for the count for unknown reasons. Dara fought off a DDOS attack earlier today, but we've continued to have weirdness that may or may not be related to that.

As of this post all resources hosted on our site, including my and Dara's official pages at angelahighland.com and crimeandtheforcesofevil.com respectively, are unavailable. Also impacted will be the Murkworks MUSH, other sites we host, mail, and domain names for which we provide DNS support.

Those of you who have resources hosted with us, apologies for this, we'll let everybody know as soon as we're up and running again. I will continue to be reachable on the various social networks, as well as my gmail address. Talk to me if you have any questions.

ETA: Dara is talking to Comcast, who says apparently our modem lost its static IP addresses. We're PRETTY sure that isn't supposed to do that. We may be dealing with fallout from yesterday's amusing weather here, people. More as we know it.
annathepiper: (Hard Day)
The Murknet just went down, folks, and I've confirmed via a call to Comcast that there's a reported service outage in our area. So as always whenever this happens, all resources hosted on our servers are down for the duration. Comcast's automated system says they hope to have the problem resolved in the next four hours!

Apologies to all for any inconvenience. Watch this space, or my various social network accounts, for more bulletins as events warrant! And if you need to ping me, do so at my gmail addresses, or otherwise on the aforementioned social networks.

ETA: Yay, we're back! Please now resume enjoying Murkworks.net-ish goodness! \0/
annathepiper: (Default)
Seattle had some exciting weather this morning, and much to my surprise, it's only taken until NOW for us to lose power at the Murkworks. [personal profile] solarbird has successfully shut down our servers, but yeah, all our resources are offline for the time being. Watch this space for further updates!

Hopefully we won't be down long.

ETA: And we weren't, yay! Came up around 4:30, so all resources should be back online.

Profile

annathepiper: (Default)
Anna the Piper

March 2017

S M T W T F S
   1234
5678 9 1011
121314151617 18
19 202122232425
262728293031 

Syndicate

RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Mar. 28th, 2017 07:45 pm
Powered by Dreamwidth Studios