The site was down for about four hours, from about 04:40 BST (20:40 PDT, 23:40 EDT) to 08:50 BST (00:50 PDT, 03:50 EDT). I'm sorry for any inconvenience. This was thanks to the personnel at the data center where we keep our servers. I had an open "ticket" with them about the problems I had with the new hardware yesterday (see this message), and they decided to run an aggressive system test -- against my instructions not to disrupt the site without talking with me first! The aggressive test used nearly all of our processing power and so caused Pocket-Monkey to go mostly non-responsive.
I've shifted everything away from the affected hardware so we can do these thorough diagnostics on the new hardware without affecting the main site.
Ironically, this is all part of making sure that things are fast and reliable. I'd've rathered not have an unscheduled four-hour outage en route, but... :-)
Sorry again for any inconvenience. And again, don't worry, we have multiple redundant backups in multiple locations, etc. -- T.J. Crowder First Primate Pocket-Monkey.com
I just got confirmation from the data center about what the problem is with the new hardware, and it's a bit of a relief: It seemed to me that after the first few days, we weren't getting the speed benefit we should have been getting from the new hardware. The fault they think they've found (a flaky network card) would account for that (and for the problems I had yesterday). The network card would have an error and retry the operation (successfully, 99.9% of the time), which tends to slow things down.
Once we get that fixed and I switch back over to it, we should see the benefit of the new hardware again.
Thanks for your patience, and sorry again for any trouble, -- T.J. Crowder
Indeed, that was the problem, we've replaced the network card and everything's testing out "good" now. Sometime today or tomorrow I'll rebalance the site to start making use of that hardware again (it involves taking the site down for about 90 minutes, too many people around at the moment to do that...).
Okay, we're back to full power again. It was a couple of days before I had a chance to rebalance the site to bring the new hardware back online, but we're there now. :-)
Forum
software by
Crowder Software Pocket-Monkey and the Pocket-Monkey logo are trademarks of T.J. Crowder and Jock Murphy. All other trademarks are the property of their respective owners.
