A few weeks ago, no fewer than three completely separate events converged to bring BZPower to its knees. None of them were bad, per se, but they all occured about the same time so it was difficult to determine just what the bloody heck was going on.

I think we've got it all figured and fixed (mostly).

Regular readers of this blog will remember just how desperate I was to get things stabilized. After much angst and many emails and pathetic whining, the server software was stabilized and upgraded. So now we're able to handle many more visitors at once, which is good. Because we're now handling many more visitors at once. And now I think I know why.

First things first. Around May 28th, I noticed things weren't going so well on BZP (technically speaking). So I performed my usual optimization tasks, which take a few hours, and it helped but not much. It turns out the management software on the server automatically updated itself to the next version... which broke a few things, like the email server. In trying to fix things, I managed to upgrade the database server, but not the core issue. It took another week to get the mail server and everything else back up and running, after updating the web software and PHP software (which in turn broke the Premier application form, which I still haven't been able to fix).

Still no love. So more whining and begging, and our host installed some caching software, which was exactly what we needed to be able to stay online without crashing.

Just as soon as we were stable, I noticed we were having about double the normal number of visitors online. A-ha! A clue. Rather than peaking at around 350 concurrent visitors, we were regularly at 500 to 600, sometimes spiking about 700. That's a lot, and it had been straining the software enough to crash the servers repeatedly. Visitor logs didn't show where all the new people were coming from, unfortunately.

So I began looking for reasons why we might be having so many visitors. One big thing that I think brought in many visitors was "The Many Deaths of Toa Tuyet" by Greg Farshtey, which we published on BZP on June 12th. If word of that got out, it's very possible new people are visiting and poking around. But that's still a lot of people. I teased Greg about crashing the server, knowing he wasn't responsible - he took it well

Then, a couple of days ago, I was helping one of my kids go through the LEGO Kids Inner Circle survey process (that's a new program LEGO is running to get feedback from kids), and took a survey about Bionicle. It was very thorough. About halfway through, one of the questions made me stop and blink: BZP was one of the answers. A-ha! A clue.

I took a screenshot for posterity.

It looks like BZP is being well-recognized by LEGO for what we do for the Bionicle community, which I appreciate very much. That's good. But they also swamped us, which caused the site to have problems. That's bad. But it forced us to address and fix those issues, which means we're in better shape than we've been in a while. That's good.

You'll notice there's more good than bad, and it ends on a good note. That's good.

So, while there may be other factors at work, I think this explains a good portion of the last months' worth of activity here on BZPland.