Mechanipus downtime

Chime · May 30, 2015, 1:44 AM

Still trying to identify details, but both Mechanipus-Fremont servers are unreachable at this point. Suspected routing problem within HE-Fremont. Games affected:

crossover medieval nocturne statuvariabilis tgg whrofh goetia belleauwood oldandnew thedrift fnt masseffect julesletters muffinmu magicplays dystopia cofab cmuffin wildcard nolabigeasy chaoticcosmos clockwork hermworld battlemux pony rifts spellbound suicidechilde cotl hpb dcfallen eldritch reno arrakis darkspires darkwater stromm nevermore uumux republic thereach reach2 spacething

No ETA available at this time.

Chime · May 30, 2015, 1:51 AM

See http://status.linode.com/incidents/2rm9ty3q8h3x for the most up-to-date info.

Chime · May 30, 2015, 1:54 AM

Incidently, this probably means other large MUs hosted in that datacenter (not by me!) are down also. (Tapestries, et al...)

Chime · May 30, 2015, 2:30 AM

Update! Part (maybe a large part) of Fremont lost power.

Been a few years now since HE-FREMONT was having these problems, but it's somewhat infamous for having bad luck with electrical issues. (Getting struck by lightning and frying servers, etc.) Fun times.

Still no ETA. I'll keep watching and let everyone know once it's back in my hands.

Chime · May 30, 2015, 3:41 AM

Update: HE-Fremont lost utility power; PG&E reports 500-4999 properties affected in Fremont.

Like most modern datacenters, HE has diesel generators that come on line right away to prevent this. One of them suffered some sort of mechanical failure. Guess which one we were on!

Sponge · May 30, 2015, 4:08 AM

@Chime said:

Like most modern datacenters, HE has diesel generators that come on line right away to prevent this. One of them suffered some sort of mechanical failure. Guess which one we were on!

I'm going to go out on a limb and guess the host didn't have the benefit of independent power supplies.

Do you need host in the meantime? I got much capacity.

Chime · May 30, 2015, 4:42 AM

@Sponge said:

I'm going to go out on a limb and guess the host didn't have the benefit of independent power supplies.

Do you need host in the meantime? I got much capacity.

I suspect you're right, and I cannot imagine why. Multiply-redundant power generation facilities and regular testing/maintenance are a pretty critical part of the industry now.

That said, this particular datacenter has a long history of trouble (lightning strike, truck crashed into the building, etc).

I'll likely be moving both KUU and KURAMORI out to Texas. bleh.

For my day-job, we're in a local datacenter that has a much better track-record, and yes-- redundant power generation and supply all the way to each physical machine.

Linode is a hosting provider though, so the idea is they should ensure those things are taken care of. It's looking like HE.net is just a bad place to go.

As for hosting... well. Thank you for the offer! I suspect that as I gradually transition out of mush-hosting, lots of people will want to take you up on that.

Derp · May 30, 2015, 6:55 AM

The Reach is up. Eldritch is not. I don't know what other games are up or down because I don't play there.

Chime · May 30, 2015, 6:55 AM

We're back up and it looks stable.

Chime · May 30, 2015, 6:56 AM

@Derp That's likely a function of which games are correctly configured to autostart. Yell at @thenomain, maybe...

He'll probably ask me to fix it tho.

Derp · May 30, 2015, 6:58 AM

@Chime said:

@Derp That's likely a function of which games are correctly configured to autostart. Yell at @thenomain, maybe...

He'll probably ask me to fix it tho.

Internet cookies to whoever fixes it first.

Chime · May 30, 2015, 7:00 AM

@Derp Sigh. Try now.

Derp · May 30, 2015, 7:01 AM

@Chime

ALL THE INTERNET COOKIES FOR YOU!

Coin · May 30, 2015, 12:08 PM

Thank you, @Chime.

Thenomain · May 30, 2015, 4:27 PM

Hey, I was getting my ass kicked at Ora et Labora. Priorities.