Earlier this year I accepted a position at a dating site with millions of users. I was hired to design user experiments, which became a battle between data and my intuition as a designer.
Eventually, I found the right balance of combining what the data indicated, and what my intuition was telling me, to hit my goals.
Following my intuition
My first project was to improve revenue by refining an upgrade page. Two experiments were built to test against the control, both of which broke the site's overall template.
Experiment A included a credit card form within the page, and Experiment B included a refreshed design of the control. The experiment designs were more aesthetically pleasing, but still under-performed the control upgrade page by .5-1 per cent.
The control page was fairly skeletal: it included a list of features on one side of the page and a subscription selection form with a call to action. Analysing the data revealed that Experiment A performed the worst of all three, and breaking the template did nothing for subscription rates.
While refining things, the control's template and the purchase flow (which included the credit card form popping up in a new window) remained intact within the experiment. Though not pretty, staying close to the control's design kept the user's trust intact.
Applying the data
After restarting the experiment, there still weren't any measurable performance indicators. It felt like hitting a wall. I started making one change at a time, but it soon became clear that this process yielded very small productive results – and the devs started to get annoyed with all the tests.
Then I had an epiphany: I was chasing the local maximum – I had hit the testing limit. I wanted to innovate, but most of all I wanted users to feel enough passion about the product that they would want to pay to upgrade to the premium service. My new goal was to figure out why these experiments kept failing.
To solve this issue, the right questions needed to be asked. The problem had to be more than aesthetics, right? That's when the combination of data and my gut feelings worked together, and solid hypotheses were developed to test.
Getting it just right
One thing the product had going for it was a brand that is valued by its users. I started creating emails that were sent out to a small percentage of the user base, and saw results quickly. My idea was this: if even one of our users felt some kind of emotion from this email and they engaged with it, something new could be learned.
Stepping out of the current style guide let me focus on the brand itself. I created fun and playful emails that performed favourably for both the company and users, but not without criticism from my teammates for pushing the limits of the style guide.
But of course, more tests had to continue to prove this method was working.
A version of the email was tested that had been adjusted based on what the data was saying, resulting in a robotic message that under-performed the control – leaving me to assume that the data couldn't tell me, or anyone else for that matter, how to design anything related to these issues.
Emotion had to be applied in the designs, through tone or imagery, in order to get results that impacted the user, as well as our metrics.
The greatest risk
In the end, taking a very risky, humanistic approach made the user feel not just wanted, but needed as well. And what user isn't needed? The goal wasn't just monetisation, it was also for the user to invest in the product. Making assumptions that you know what the user wants and that the data will tell you everything, well, ‘makes an ass out of u and me'.
Designers become obsessed with innovating and forget the real reason why we do what we do, especially in the context of designing with data. We have to consider data and our own intuition, applying both to what we build.
Failures are inevitable – until we apply what we learn from how we fail we will continue to chase the local maximum instead of innovating. My opinion is this: don't hold data above common sense. Take chances. Follow your intuition. Let data support your design, not define it.
Words: Natasha Irizarry
Natasha Irizarry is a self-proclaimed UX evangelist. She works as a consultant for companies who have user experience and design related problems. Follow her on Twitter at @natashairizarry. This article appeared first in issue 261 of net magazine.
Like this? Read this!
