Success has a lot to do with your sense of being Find out how to train yourself to make money and stay in tune with your mind, body and soul
Hey there,
I like to get straight-to-the-point, wasting nobody's time.
1.1 Explanation
A/B testing is exactly what it sounds like: you have two versions of an element (A and B) and a metric that defines success. To determine which version is better, you subject both versions to experimentation simultaneously. In the end, you measure which version was more successful and select that version for real-world use.
A/B Testing Example: http://ift.tt/1Wf7u0Y
1.2. Framework
1.2.1. Step 1: Hypothesis
If you’re testing a button color, why do you think green will be better than blue? Are you randomly testing colors or do you think a certain contrast between the button and background colors will make the button more noticeable to customers?
Creating a good hypothesis and planning the test(s) to prove the hypothesis will give your tests direction and yield actionable insights that are less likely to be due just to chance. Likewise, A/B testing should be skipped in situations where you know that an idea almost certainly will improve your app and the risks associated with blindly implementing the idea are low.
For example, Robot Invader, consistently asks beta users for feedback. After playing the beta version of their newest game, Wind-up Knight 2, several players thought there wasn’t enough congratulatory “glitter” after completing achievements.
The recommendation from users was that more pomp and circumstance be added so that players would feel rewarded after accomplishing certain tasks and be more aware of the new features they just unlocked. The downsides of implementing something like this are close to zero, and the likely impact is positive.
There is no reason to spend time and resources to test something that probably is good and has low risk. Jumping to implementation is perfectly advisable.
1.2.2. Step 2: Test Impactful Elements First
After you have a clear idea of what you are testing and why, it’s important to choose the elements you decide to test on each variant with careful consideration. Although there likely won’t be a technical limit to how many variants you can test with most modern AB testing solutions, it can be a waste of time and resources if you over-do it. Smaller, sequential, AB tests may seem like they have less power, but they make the data collection and evaluation much more straight forward, typically leading to fewer mistakes.
You will want to be sure to test the elements of your campaign that will make the largest difference in gaining higher conversions. While every little design element can have affect your conversions, it is better to start with running tests on larger elements that seem like they will impact the user’s decision process the most. For example, instead of first running tests for every button’s color, shape, and size, consider elements like the wording in a call-to-action or the amount of a discount in an virtual goods promotion.
1.2.3. Step 3: Run Your Test For The Full Duration
It can be tempting to set up an AB test and monitor its performance so closely that you are collecting inaccurate data. Drawing conclusions too soon in the testing process can cause you to want to end the test early. This usually manifests when you see an early “clear winner” or decide out of intuition the test ran long enough based on the amount of data collected.
Evaluating the variants’ performance and choosing a winner before the planned test is fully complete will drastically increase the chance of choosing the incorrect variant. By not allowing your AB test to run the entire pre-chosen duration can, and will, lead you to incorrect and incomplete conclusions.
Run tests for at least a month. The longer you run the test, the more confidence you can have in the accuracy of the results.
1.2.4. Step 4: Evaluate
You’ve done your test and it was thoroughly exciting. But it doesn’t stop here. Now you need to evaluate and take the next steps. For example, here are the results from an A/B test:
- A ‘Stay Stylish for Autumn’ Landing page bounce rate of 76.8%
- B ‘Get Stylish this Autumn’ Landing page bounce rate of 72.5%
The B version of the landing page showed an encouraging reduction in bounce rate. This is excellent news and means that you could switch your copy to the latter, knowing with confidence that it would yield better results.
But why did you see this positive change? A/B testing is not just about random experiments, it’s about learning what your audience wants and why. Mindless testing is pointless. So look to the data at your disposal to discover why your users preferred one landing page over another.
1.3 Best Practices
-
Don’t run your test during seasonal periods where the results may be skewed by natural changes in consumer behavior.
-
Your choice of what to test will obviously depend on your goals. For example, if your goal is to increase the number of sign-ups, then you might test the following: length of the sign-up form, types of fields in the form, display of privacy policy, “social proof,” etc. The goal of A/B testing in this case is to figure out what prevents visitors from signing up. Is the form’s length intimidating?
-
When doing A/B testing, never ever wait to test the variation until after you’ve tested the control. Always test both versions simultaneously. If you test one version one week and the second the next, you’re doing it wrong. It’s possible that version B was actually worse but you just happened to have better sales while testing it. Always split traffic between two versions.
-
If you are testing a core part of your website, include only new visitors in the test. You want to avoid shocking regular visitors, especially because the variations may not ultimately be implemented.
-
Your A/B-Testing tool should have a mechanism for remembering which variation a visitor has seen. This prevents blunders, such as showing a user a different price or a different promotional offer.
-
If you are testing a sign-up button that appears in multiple locations, then a visitor should see the same variation everywhere. Showing one variation on page 1 and another variation on page 2 will skew the results.
-
Run tests one step at a time. It’s a simple guideline. But an easy mistake to make. If you’re testing the color of your call-to-action button, that’s all you should test. Testing anything else simultaneously, even changing the copy on the button, makes your findings less precise and certain. Did the color change increase conversions, or the new copy?
-
As with any scientific experiment, you need to have enough data points to gather statistically significant results. This means you need to have a minimum number of users participating in each test. Depending on how you structure the test (how many variants) and what your expected results are (a small improvement off of an already high conversion rate or a large improvement off of a low conversion rate), you might need thousands of users to get statistically significant results.
1.4. Creating A Good Hypothesis
A hypothesis is essentially a change and effect statement that often follows a simple established syntax:
- Changing (element tested) from _____ to _____ will increase/decrease (a conversion metric).
This statement is only a theory that can be proved or disproved. It mainly documents how you expect a change made on a website/web page to increase/decrease a conversion metric. Remember that it’s important that the impact of your change must be measured in quantifiable terms. Here’s one good hypothesis statement, for example:
- Changing the headline from ‘Grab your tickets now!’ to ‘Tickets filling out soon – only last 50 left!’ will increase ticket sales online.
This was it, it was a long read and perhaps a bit too long for Reddit. If you'd like to give feedback, feel free to do so. I'm more than happy to talk or discuss any of the topics mentioned above :)
No comments:
Write comments