Server outage

Question

My apologies to everyone who experienced the server outage yesterday. We were down for a horrendous 15+ hours, starting at 1pm on Monday and lasting until about 5am this morning. Our server admin Brian was up all night.

The problem wasn't isolated to WikiTree. Sites including Reddit, AirBnB, Instagram, Foursquare, Pinterest, etc. were also affected by the problem at our host (Amazon Web Services), but I don't know if anyone was hit quite as hard as us.

We're working on solutions to help make sure this doesn't happen again.

Comments

Thanks for the update! I was wondering this morning, when I got an internal error, then some site lag later. At least it's back up and running eh? :)

Thanks for fixing it, Brian!

---

Thanks for the report; I was trying to work on pages while flying across country yesterday, and just figured it was Southwest's crummy wi-fi.  FYI, I still can't get on to wikitree this morning, and even the g2g is pretty sluggish.

Best of luck getting back up and running smoothly. It always is a bummer when this happens. Could have been worse: could have happened while you were on vacation, Chris!

---

This ended up taking almost a full day -- 24 hours -- before operations on WikiTree returned to normal.

I've been working on websites since 1995 and I think this is the worst web service outage I've ever experienced.

---

Thank you and your team so much for the hard work of getting us back running.   Frustration uses up energy twice as fast as hard physical labor and I think everyone appreciates how terribly frustrating the last 24 hours has been.  You have all been amazing.

---

Ditto what Gail said. I hope Brian has slept in really late today.

Answer 1

Still experiencing problems this morning. Might it not pay to wait until word is given that all is fixed?

Comments

And the outage continues! I'm surprised you were able to get through to post your message, Mike.

We've now taken the WikiTree database offline completely, in an attempt to let the database catch up. (Our web server on Amazon is functioning fairly normally, but not the database servers. They can't keep up with the traffic.)

---

I think the server is back up and operating normally again. Finally!

---

Should I stop greeting guests until it's running better ?

Activity is chuggling sometimes.  Maggie N