A/B Testing Research Tips

One way to increase your sales is to improve your conversion rate, right? Although it’s true, conversion rate optimization isn’t easy.

If you just base your tests on your gut feelings, you may find a few winning variations, but chances are, most of them will lose.

So, how you do improve your conversions?

You have to analyze data before you run A/B tests. In order to help you improve your conversion rate, I decided to enlist the help of Peep from ConversionXL to come up with an infographic that breaks down what you need to do.

How to Come Up With Winning A/B Tests Using Data

To add to the infographic above let me tell you even more.

I run hundreds of A/B tests, and the variations that typically have the most impact are those with drastic changes.

To come up with those drastic changes, you need to analyze your data first to see what you need to change.

You can get the data by looking at your Google Analytics reports as well as a heatmap of your site through Crazy Egg.

In addition, consider running surveys.

The rest of this post will explain how to organize your data and how to draw out helpful customer insights from surveys and user tests that will assist with your conversion rate optimization.

Evaluating Customer Surveys

Customer surveys are valuable because they allow you to talk visitors who have already become customers. By asking the right questions, you can learn what almost prevented them from making a purchase and can also learn what ultimately convinced them to trade their money for your product. This information identifies hurdles that you can address with copy or other features and also highlights key selling points that convince customers to buy.

You can use a form builder like Wufoo to create custom, easy-to-use customer surveys.

When evaluating the responses, you’re looking for three main things:

Patterns
The first thing you’re looking for are patterns. Is there a key benefit that gets repeatedly emphasized? Make note of that. Are there certain words that get used over and over again? Make note of that as well.
Objections
The next thing to look for is objections. What specific hurdles do customers mention that almost prevented them from making a purchase? These are key points that need to be addressed on your website and landing pages.
Language
As you’re evaluating, make note of the words your customers use. How do they describe your business? What do they say about your product? Which specific words do they use to talk about what you do? The words used by your customers are a great source for phrases and headlines you can later use in your copy.

For example, suppose you run an eCommerce store that sells cowboy boots. You’re interested in learning more about your customers so you can optimize your site and improve your copy.

As part of a 5-question survey, you ask the following question:

What were the most important factors that convinced you to place an order with BootOutfitters.com?

In response, you may get the following answers:

Price was the most important factor to me. I looked at other websites, but you guys had the best price for these boots.
Definitely selection. I couldn’t find these boots anywhere else.
I decided to buy because of the convenience. It’s so much easier to buy boots online than to go to a store.
Free shipping on returns is what did it for me. I’m worried that the boots won’t fit but feel safe with free return shipping.
It’s so much easier to buy from you guys. I hate going into a store and getting bothered by a salesman.
I didn’t have time to go to the store. Since I knew which boots and size I wanted already, it was much easier to buy from you guys.
I love the fact that you have free shipping. That made my decision so much easier.
You guys have so many boots to choose from. That helped me find the best pair of boots.
Large selection and free return shipping, in that order.
I couldn’t find these boots anywhere else. Plus I had a $15 off coupon.

So what did BootOutfitters.com learn?(Yes, this is a small sample size, but it’s large enough to provide an example of how to evaluate survey results.) They learned that the following points mattered to the given number of customers.

Specifically, they learned that selection, convenience, price, free shipping, and free return shipping are all factors that are important to customers. Of those factors, selection and convenience were mentioned the most with price and free return shipping following behind in the next tier.

This type of information is valuable because it lets you know exactly what’s going through your customers’ heads. You may think that low prices are what’s the most important to your customers and emphasize that on your website. Then, after conducting a survey like this, you realize that a large selection is more important than low prices. You can then adjust your site accordingly to emphasize this important factor. It doesn’t mean you don’t mention price or ever offer discounts, but it shouldn’t be the main feature you focus on.

Instead of using a headline like “Lowest Priced Boots Online. Guaranteed.”, you could use a headline like “The Best Selection of Boots Online with the Lowest Prices and Completely Free Shipping.” This second headline includes three of the top five factors that are important to your customers and provides a new headline you can test against your current one to see which one performs better. The test is based on survey results, so you have more confidence it will turn into a winning version than if you just based it off of a guess from your gut.

So what about objections? Did we learn about any objections from these responses?

Yes, as a matter of fact we did. In response number four the customer mentions he was worried about buying the right size but he felt confident in making a purchase once he knew return shipping was free. This response lets you know that being able to return boots that don’t fit is something that matters to your customers.

Now that you know this, you can evaluate your website to see how well you do at emphasizing this point. Maybe you don’t do a great job, and even though this customer noticed your policy, another ten customers didn’t find it because it’s hidden on your homepage and landing page. Qualaroo surveys will help you to figure out if this is a problem for more people, but at the very least, you now know that it’s a hurdle for some customers and you can evaluate and test your website accordingly.

Qualaroo surveys help you to identify hurdles that are preventing visitors from making a purchase.

The final point to review is language. What language did your customers use that you may be able to incorporate into your copy.

Here are some key sentences from our example results:

It’s so much easier to buy boots online than to go to a store.
I’m worried that the boots won’t fit but feel safe with free return shipping.
I hate going into a store and getting bothered by a saleman.
You guys have so many boots to choose from. That helped me find the best pair of boots.
I couldn’t find these boots anywhere else.

These sentences & phrases can then be worked into your copy with lines like:

Buying from BootOutfitters.com is easier than finding a local store with the same selection.
Free return shipping means you can rest assured your boots will fit or you can return them for free. No questions asked. Guaranteed.
Avoid getting hrassed by a boot salesman. Buy a new pair of boots from the comfort of your home.
Find the perfect pair of boots on BootOutfitters.com
You’ll find boots on BootOutfitters.com you can’t find anywhere else.

Since this is a guide on conversion rate optimization, and not on copywriting, you’ll want to make sure you test any and all of these changes. You can come up with new versions that test this copy against your current copy and then run a test to see the results, but you never want to make changes without testing. These survey results provide a good starting point for conversion tests and give you an idea of what matters to your customers, what objections they have, and what words they use to talk about your business.

Taking A Look At On-Site Survey Results

Next up are on-site survey results. On-site surveys like those available from Qualaroo allow you to survey your customers to learn about their experience on your site, especially to learn what’s preventing them from making a purchase. You’ll learn whether price, shipping costs, or product misunderstandings are preventing customers from making a purchase.

This is an example of what a live Qualaroo survey looks like.

Here’s a review of the types of questions you can and should ask:

Is there anything you can’t find on this page?
Is there anything confusing on this page?
Do you have any questions at this point?
What’s your biggest concern about purchasing [insert product name here]?
What’s the number one reason that’s stopping you from making a purchase?
What else would you like to see on this page?
What can we help you with?
Why didn’t you complete your purchase today?
What could we have done to convince you to complete the purchase?
What’s the biggest problem we can help you solve?
What are you looking for in your ideal solution?
What else can we place on this page to convince you to buy?

As mentioned, these questions will help you to learn more about the misunderstandings customers have when they’re on your site and what’s preventing them from making a purchase. So how do you evaluate the results?

You basically do the same thing you did with regular surveys. You look for patterns, identify objections, and pay attention to the language used.

For example, if you sell a heatmap software service and ask the following question, “What’s the number one reason stopping you from making a purchase today?” visitors may respond in the following ways.

I don’t understand how heatmaps work. Can you explain more about them?
Your service is too expensive. I can’t afford.
Why don’t you offer a free trial? I’d really like to sign up for a free trial.
What exactly are heatmaps? I’m not entirely sure.
How will heatmaps help me to improve my site? Do you have any case studies that show this?
I don’t really understand how heatmaps work, Need to do more research before buying.
Too expensive. I only want to pay $10 per month.
I really would like to try a free trial first before making a commitment.
Are heatmaps useful for blogs or just for SaaS sites?
Not sure how much heatmaps can help on an eCommerce site.

When you evaluate the results, you’ll notice that:

6 out of the 10 responses mention that they don’t understand how heatmaps work.
2 out of the 10 responses refer to price as being too expensive.
2 out of the 10 responses request a free trial before making a purchase.

This again is a very small sample size but also shows how even a small number of responses can help you to optimize your site.

For example, you could respond in the following ways:

You could add a case study section to your site to help people understand how heatmaps can be used on different types of sites, including blogs and eCommerce stores.
You could add more information to the homepage to educate customers on the different uses for heatmaps and how they can be used to improve a site.
You could consider adding a less expensive plan for bargain shoppers if that’s something that works with your business model.
You could experiment with offering a free trial to see if it increases sign ups and paid subscriptions.

All of these are great examples of how you can use the results from Qualaroo surveys to optimize your site, and once again, you need to use the results to come up with hypotheses for testing on your site. It’s never guaranteed that one of these changes will improve your conversions, but at least you have data to back up the tests you eventually run which means you’re more likely to be moving in the right direction.

Analyzing User Tests

The final piece of data to consider are the results from the user tests you conduct. The nice thing about user tests is that you only need a small number to learn more about your site. Usually five or so tests is enough to get all of the information you need for the current version of your website.

You also should know that user tests help you to catch the big, glaring, easy-to-find mistakes on your site. You may think that your checkout process is simple and easy to complete, but visitors can’t figure out how to enter their phone number and zip code, get frustrated, and eventually leave. Small problems like this can kill conversion rates, and conducting user tests is a quick and easy way to find out what frustrations or problems people have with your site, as well as what they like and what’s easy to use.

This is a screenshot of a user test for WalMart.com from UserTesting.com.

When evaluating user tests, you want to pay attention to things visitors like and things that don’t make sense for them. Maybe there’s something that seems intuitive to you but isn’t intuitive to your visitors. You want to make note of that. Your customers may also mention they want to know how much something costs but can’t find a price. That’s something else you want to make note of.

For example, a visitor may search for pricing information on a page that doesn’t include pricing info. You may think it’s not important, but visitors use valuable time searching for a price that otherwise could be spent learning more about your product. Without conducting a user test, you’ll never know visitors are experiencing problems similar to this. They may have trouble finding a shopping cart, or they may not see that free shipping is offered on orders over $50. User tests help you to find out what people can’t find and what they’re having trouble with on your site.

Each and every observation visitors make on your site through a user test will teach you more and more about your site and the experience visitors are having with it. You’ll find out what annoys them and what delights them, a process that’s both enlightening and frustrating. But no matter how frustrating the process is and no matter what you discover, watching the videos will provide insights about your site that you can’t learn any other way.