Today, let’s get really specific and look at a concrete example of how quantitative research can impact revenue.
Last month I launched a new online form for a client. If it wound up bringing in more revenue than the old form, we would roll out versions of it to various parts of the website.
The form went live on August 17th; ten days later I was looking at usage patterns in Google Analytics and found something interesting: Most of the site’s visitors were using iPhones, but submissions for iPhone users were 43% below the form’s average.
This meant that the people using the most popular device for this site were among the least likely to actually complete the form.
When the average form submission is worth over $100, that’s a problem that adds up.
A note on deciding when to intervene
If you’re wondering, “Why wait 10 days?”, the answer is: You need a decent sample to have any meaningful data to act on. It’s foolish to try to fix a problem until you know it really is a problem, and you need a decent number of users to establish a pattern.
If you’re wondering, “I can’t believe he only waited 10 days”, then you’re probably a fellow conversion rate optimization nerd who likes to follow a tight protocol in testing. Yes, ideally everything would happen in 4-week intervals, but when there’s money on the line — especially someone else’s money — I prioritize practical action over the purity of practice.
Run this diagnostic on your own site
Here’s how I discovered the problem. You can follow along by opening up your Google Analytics dashboard:
- In the sidebar navigation, go to Audience —> Technology —> Browser & OS
- Set the view to Comparison (It’s the button on the right with the zigzag horizontal bars; See screenshot)
- The primary dimensions I find most useful are Browser (the default) and Operating System.
- In the right-hand column, select Goal Conversion Rate.
You’ll see this:
(This image doesn’t show client data — it’s from a dummy analytics account. View a higher resolution version here.)
Setting the right-hand column to “Goal Conversion Rate” will look at all your goals and show you how conversion rates on each different browser are performing against the average for all goals.
If you’re tracking very different types of goals on your site — like ticket sales and newsletter sign-ups — this view might not be as useful since the process, content, and technology used to acquire newsletter sign-ups is very different from that of online checkout.
The more diverse your goals, the more you’ll want to choose the conversion rate for a single goal in that column’s drop-down menu— ticket sales for example — rather than look at averages across all goals.
You may find, as I did this week, that people using a particular type of browser or operating system are showing an average conversion rate that’s suspiciously lower than others. You’ll notice that the column is sorted by the total number of users during the designated period of time. Pay the closest attention to those near the top — at least at first. You don’t want to be optimizing for results near the bottom because you may be looking at a sample so small that it’s either statistically insignificant or not worth optimizing for or both.
The fix is in
Turns out, the new form was being a bit buggy on iOS touch screens — sometimes user input wasn't registering. I worked with the form developer to push an overnight update to the system that has since fixed the problem — As of this writing, we've gone from 43% below average to 15% above average on iOS.
How would I have ever known that this was a problem without studying usage patterns? Customers certainly aren’t going to report that your online form is buggy on their iPhone. They’ll either just call to complete the transaction or they’ll go somewhere else or they’ll put off the decision entirely. (Doing nothing is often your biggest competitor.)
If all you’re doing with your analytics is counting “hits”, you’re missing out on potential revenue. Dig deeper, and you may find the data you need to warrant worthwhile investment in improving your website.
Thanks for reading,
PS. What's your organization's experience with Google Analytics? Do you have it installed on your site? Is anyone monitoring it? In what way? Do you think GA insights could help your company improve its website? What's preventing your team from doing more with GA? Hit reply and let me know.