Return to the Analytics Campaign.

Where’s the fail whale?

Posted by Justin Kistner on Wednesday, September 9th, 2009

fail_by_numbers It has been a lot of fun so far working on this campaign. We have had the chance to share behind the scenes stories and numbers. Other marketers have related to our experiences and that brings us all a little closer together. This past week we experienced a server meltdown and we didn’t have a fail whale or a monster that had escaped. We looked more like Gmail with a busted looking page that generated conversation about the being broken. And we learned from that.

When you trip, take a bow

Optimizing a marketing campaign isn’t only about tools/technologies, but also about dealing with the unexpected. A server meltdown was definitely unexpected. Having a sense of humor when one makes a mistake is important. The CTO of CoTweet Mr. Bill put it like this, “The [fail] whale’s popularity comes from the idea that failures are worth celebrating and learning from.” Well said, Mr. Bill.

An older story comes to mind as well. When Thomas Edison was interviewed by a young reporter who boldly asked Mr. Edison if he felt like a failure, a perplexed, Edison replied, “Young man, why would I feel like a failure? I now know definitively over 9,000 ways that an electric light bulb will not work. Success is almost in my grasp.” Later after about 10,000 attempts, Edison invented the light bulb.

FWIW, we weren’t running a single, wimpy server. We had a load balancer driving traffic to two redundant servers that had a load balanced database as well. Still killed it. We’ve since had a chance to identify the failure culprit and have corrected it. We’ve beefed up our set up though. Ran some load testing on it. Should be good to go. If not, you’ll see our sweet new fail whale. We decided to make a series of awesome error pages too. :)

Server meltdowns and error pages happen to all of us. Well. Not all of us. Our On Demand team is poking fun at us. They run our product from a staffed 24/7/365 Network Operations Center (NOC) that handles a gazillion page views a day and never fails. But we, the marketing team, don’t get access to the redundant servers in the NOC. I’m sure most marketing departments don’t have a NOC behind their websites. So our web guy, Diggles, is writing up a post for marketers about how to avoid server meltdowns and what are the first steps you need to do if it happens to you.

noc

Mmmm…look at all those screens monitoring server activity in the NOC *wipes drool off keyboard*

Tags:

Read more on Analytics. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

One Response to “Where’s the fail whale?”

  • CG prime Says:
    September 9th, 2009 at 10:22 am

    Ah, good times, good memories.

    We already knew the superhuman On Demand team couldn’t possibly be involved. That outfit is rock solid, unlike, as you said, things that we marketeers touch.

    If I had one of those numerical WebTrends business cards, I’d probably choose 404 as my personal number, though on most days I’m mainly in 302 mode.

    • Leave a Reply