Should we always “let the data decide”?
Can we please stop saying "let the data decide"? It's a familiar mantra that comes from the right place, but often it ends up being an excuse for more guessing.
Now before you storm off in a huff, let me explain. While I agree in principle, it's important to define what we mean by "data". Most people making this argument mean quantitative data coming out of their analytics platform; page views, clicks, acquisitions, etc. That's certainly important, but is only half the picture.
Bringing the balance
Quantitative data tells you what your users are doing. Users are hitting your product page. They're staying there for an average of X seconds. X% of them are clicking the buy button. Understanding the what is huge!
But quantitative data can't answer an important question: why are users doing what they're doing? If we're going to improve the performance of a specific tactic, we need to uncover what's driving users' behavior.
You shouldn't just guess
Often, the rationale given for the "show me the data" approach is to alleviate guessing. Is this page working well? Is it well designed? Should we use the photo or the infographic? Without quantitative data, we would have to guess at the answers. Analytics help us objectively evaluate success.
Let's look at an example. We just launched a landing page and Facebook ad campaign for an ultra-high tech stroller with wifi/bluetooth connectivity, an iPad docking station, etc. At $300, the price point is high, but we feel confident based on market research that affluent first-time parents will be on board.
We craft a page talking about the ultra-cool factor. We have a picture of a mother jogging behind the stroller with her child watching the iPad Pro docked in the station. Our CTA is solid: "Upgrade Your Baby Now!" Brilliant!
Reviewing the analytics after launch, you notice that while 5000 people visited the landing page, you sold only 1 stroller.
Without the why, you're still guessing
Objectively speaking, we'll all agree that a 5000/1 conversion rate is a failure. We obviously need to do something to improve it. But what? There are lots of possibilities:
Maybe we're not reaching the correct audience
Maybe the audience we are targeting doesn't want this kind of product
Maybe the button is not obvious enough
Maybe the content on the landing page isn't clear
Maybe they don't like the picture we used on the landing page
Maybe they don't have $300 for a high-tech stroller
Maybe, even though they like the product, they're not comfortable buying from us because they're unfamiliar with our brand
Maybe this audience would prefer a simpler stroller solution
Etc, etc, etc
The piece of data we're missing here is the why. Knowing the right tweak to improve performance gets a lot easier when you understand why users aren't responding. Without that important piece of information, we're just shooting in the dark at how to improve it.
When optimizing tactics, having an analyst guessing isn't any better than having a designer guessing.
The solution - look at numbers AND listen to users
Before optimizing a troubled tactic, take time to gather and analyze qualitative data. If you have a department that handles that for you, reach out and get their point of view on why users are behaving like they are. Test the existing flow through usertesting.com or a similar service. If you can, interview some end users who dropped out of the flow. Ask them why.
In our example, let's pretend that after testing our concept through usertesting.com, most of our participants tell us that while like the product and are OK with the price point, Christmas is right around the corner and the timing is just bad.
Now that we know they why, we can really craft a good fix. Maybe we change the content to how great a Christmas gift this will be for your child. Or we could do a social media campaign where users can send this as a "wish list" item to family or friends. Or we allow users to break their payment in half - first payment now and second payment on February 1st after the holidays have passed.
Without having that qualitative data, we might have tried lots of fixes and wasted lots of time and money with no return. More importantly, we would probably never think of the solution that actually addresses the users' concerns.
Remember the yin and yang of data
So the next time you encourage someone to "let the data decide", remember that data is more than just the numbers your analytics tool spits out. Leveraging both quantitative and qualitative data will give you the insight you need to optimize pages without guessing. It's about balance, Grasshopper.
