Your Site Testing Isn’t Working? Here’s Why

(image by Peter Alfred Hess)

Site testing is a hot topic in digital marketing. Whether you call it conversion rate optimization, CRO, UX testing or a 4-letter expletive, the goal of site testing is the same: you want to get more conversions out of your traffic.

More conversions means more sales opportunities, which means more sales, so it’s no wonder that everyone gets excited about the potential of conversion rate optimization!

You can find literally hundreds of case studies where companies improved clicks, conversions or any number of other KPIs by testing changes to their site.

With numbers like 41% increase in sales, 400% more conversions or 3,600% increase in social shares floating around out there, it’s easy to think, “Hey, all I have to do is run a few tests and I’ll have to hire an extra sales team just to manage all the new leads!”

Unfortunately, for each of those runaway successes, there are dozens, if not hundreds of site tests that end up more like this…

…which is why site testing is sometimes described with those 4-letter expletives I mentioned earlier…

At Disruptive, we’ve run thousands of site tests using various testing platforms like VWO, Optimizely and Unbounce, so we’ve seen both amazing and not so amazing results. However, after all of those tests, we’ve discovered 4 very important reasons why tests fail.

In our experience, avoiding these 4 problems not only reduces your risk of a site testing faceplant, it also dramatically increases your likelihood of achieving the killer results your looking for from your testing.

So, ready to find out how to actually boost your conversions? Read on.

1. Lack of a Game Plan

Just like any other aspect of digital marketing, if you want to succeed at site testing, you need a great strategy. According to VWO, only 1 in 7 A/B tests produce a winning result.

That means 86% of them fail.

Fortunately, the right testing strategy can dramatically improve your success rate. For example, using the tactics outlined in this blog post, the majority of our tests (5/7) substantially improve our clients’ conversion rates.

What makes our tests so effective? Yes, we’ve got years of experience under our belt; but, more importantly, we start every set of tests with a thorough testing strategy that maximizes our chances for success and sets us up to learn from everything—even failure.

With that in mind, here are some of the questions we ask while planning each series of tests:

A. What is the overall site testing objective?

Depending on your overall objective, there are two ways to approach testing.

1. Big-to-little

If your objective is to get big results fast, you’ll want to start by testing sweeping changes to your page(s) or site. What you’re doing right now probably isn’t working, so you need to try something very different to get the ball rolling.

While this approach can certainly be effective, it does have downsides.

First off, you don’t really know why your test did or didn’t work. There are simply too many variables at work to really assess the individual impact of all the new elements on your conversion rate.

As a result, you’re kind of left where you started. Things might be working now, but all you really know is that what you’re doing now is better than what you were doing before. However, once you’ve got things working properly, you can strike from a position of relative strength and really optimize your new design.

If your site traffic is less than 6,000 visits per month, this is probably the best approach to take. With limited traffic, each test will take a significant period of time, so trying to optimize each site element upfront can take months-to-years.

2. Little-to-big

On the other hand, if your objective is to really understand your target demographic and how specific changes to your site affect potential clients, it’s best to start small and work your way up. Your site is working well enough, what you want to know is how to maximize its potential.

This approach requires a lot more time and traffic to generate big results, but it also equips you with the knowledge you need to make informed future decisions. Over time, the sum of your small changes will add up to big design and content improvements that markedly improve the efficiency of your website.

This approach is fantastic for websites with 6,000+ website visits per month and is how giant websites like Amazon produce millions in added revenue.

Defining your overall objective allows you to structure your actual testing strategy to produce the results you are looking for. From there, you can identify the actual test parameters that will allow you to achieve your goals.

B. What am I trying to test?

This question might seem obvious, but you wouldn’t believe how many people think they are testing one thing and when they are actually testing something else.

If your test isn’t set up to test what you think you’re testing, you won’t learn anything. For example, there’s nothing wrong with testing to see if a new call-to-action, hero shot and page layout will improve your conversion rate; but, if your new page content/design performs better, you can’t assume that it was because of the new CTA.

It might have been the CTA, but it also might have been the new hero shot, page layout or the combination of all three!

That being said, the more dramatic your changes, the more likely they are to produce dramatic results. If you want to know how a new landing page will affect conversion rates, go for it! Just don’t use the results of your test to draw conclusions about your new CTA.

C. How much traffic am I willing to risk on this test?

If you’re new to site testing, that 86% failure rate can be daunting. After all, there’s a very real chance that your tests are going to perform about the same or worse (possibly much worse) than your current iteration. The more traffic you test, the more you put your bottom line at risk if something goes south.

The problem is, unless your website has a lot of traffic that’s already converting at an acceptable rate, a conservative testing strategy will slow down your time to results. If your site isn’t working well (which is probably why you’re testing), your test probably isn’t going to make things that much worse.

The more traffic you push into your test, the faster you can figure out whether or not your new idea is working, learn from the test and press on to bigger and better things.

For this reason, I typically recommend testing around 50% of your traffic so that you don’t put your bottom line at serious risk. Once your site is performing at an acceptable level, you can dial back to 10-20% to avoid rocking the boat too much.

D. How soon do I need results?

As mentioned in the previous section, the more traffic you put towards a test, the faster you’ll get results. For example, if your online marketing isn’t profitable and you’ll have to shut things down in 6-months if you don’t boost your conversion rate by 200%, you might commit most or all of your traffic to your test to get things figured out as quickly as possible.

Marketers that want real results know that testing is an ongoing effort that will continue to pay big dividends. Fortunately, with a great strategy in place, you don’t have to test 7 times to produce a win. You’ll always have tests that don’t perform the way you expect them to, but the right strategy will allow you to use those tests as learning experiences that set your next test up for even greater success.

2. Poor Preparation

For a supposedly “data-driven” process, a surprisingly large number of of optimization tests are run on “gut instinct.” Now, that’s not to say that marketing intuition isn’t an important component of successful testing, but use your instincts to come up with ideas…and then do the research you need to make sure that those ideas merit testing (and are being tested properly).

Instead of just shooting from the hip, define your prospective buyer persona, their pain points and how your test will meet their unmet needs before you put your test together. Remember, the better defined your test is, the more you will learn from it.

Additionally, while you may have a killer idea for a specific test, you’ll want to check your analytics to make sure that what you’re testing is worth the time and effort.

Focus on the areas of your site that really contribute to your conversion rate and get those optimized first. Your bank account will thank you later.

3. Jumping the Gun

There’s a lot of emotion involved in site testing. If one arm of a test starts to perform better, it’s easy to start seeing dollar signs. If your variant starts to do worse, it’s hard not to turn off the poor performer. It’s losing you money, after all.

The problem is, you have to wait to get meaningful data. It’s a statistical fact that the more traffic you push through your test, the more accurate your results will be. If you jump the gun and call a test early, you’ll be basing your decisions on emotion rather than data.

If that’s how you’re going to to run your website, you might as well skip the whole testing thing.

Now, this is considered heresy amongst the statistical community, but you don’t always have to hit 95% confidence before accepting the results of a test. If you’ve got low traffic, 75-80% confidence may be all that you can get in a reasonable time frame. Sure, there’s a higher probability that your results might be off, but it’s the best you can do.

For marketing, it’s more important to decide what your acceptable confidence level is and then get enough data to reach that level of confidence. If you decide you’re okay with a 25% chance that your test data is wrong, then call the test at 75% confidence. Just don’t say you’re going to wait until you hit 75% confidence and then pull the plug at 50%.

If you want to improve your conversion rate in meaningful ways, you need to wait for meaningful data. As exciting or devastating as the initial results might look, you have to wait for the data to sort itself out.

4. Ignoring Your Traffic Sources

Finally, one of the biggest (and most overlooked) reasons why site testing fails to produce the desired results is that your website is only part of the equation. No matter how optimized your site is, if you are sending the wrong traffic to your site, you won’t get the conversions you need.

Traffic optimization is where many CRO platforms like Unbounce, VWO or Optimizely fall short. Sure, they do a great job of helping you optimize your website; but, if you’re sending the wrong people to your site in the first place, you’ve got a fundamental problem that no amount of site testing will fix.

At Disruptive, we’ve conducted 2,000+ AdWords account audits and discovered that 76% of PPC budgets are spent on search terms that never convert. Paid search advertising is the best way to get high-intent, high-converting traffic to your website; but, if 76% of your traffic has no chance of converting, it doesn’t matter how optimized your site is.

To really get the most out of your website, you need to push the right traffic to your site and present them with the kind of site experience that will make them want to convert. It’s as simple as that.

Conclusion

Site testing may be difficult, but—when done right—it’s well worth the effort. After running thousands of tests, it’s become clear that without a great game plan, preparation, patience and traffic optimization, getting your CRO efforts to produce is an uphill battle.

Fortunately, site testing doesn’t have to be a 1/7 roll of the dice. Put the right strategy and traffic in place, stick to your plan and you’ll soon be writing a case study of your own.

By the way, if you’d like to skip the learning curve and jump straight to getting results, let us know here or in the comments below! I’d be happy to take a look at your site testing efforts and give you some recommendations.

You’ve heard my 2 cents, now I want to hear yours! Which of these problems have you run into or seen others struggle with? Are there other challenges you would add to this list?