Insight, Analytics and the New Marketing
Death by the Sword of Data
My earlier post talked briefly about the power of testing and the emergence of a different breed of creatives who embrace the feedback of results they can get from the web and make it part of the way they work. But can the role of data in the creative process be taken too far?
According to Doug Bowman it can. 2 weeks ago he resigned as Visual Design Lead at Google and joined Twitter as their Creative Director. Doug posted the following on his blog :
“I’m thankful for the opportunity I had to work at Google. I learned more than I thought I would. I’ll miss the free food. I’ll miss the occasional massage. I’ll miss the authors, politicians, and celebrities that come to speak or perform. I’ll miss early chances to play with cool toys before they’re released to the public. Most of all, I’ll miss working with the incredibly smart and talented people I got to know there. But I won’t miss a design philosophy that lives or dies strictly by the sword of data.”
Kevin Fox, who worked with Doug as at Google as Senior User Experience Design Lead, replied to this on his blog. Though doesn’t agree that Google’s design philosophy “lives or dies strictly by the sword of data” he does have a word of caution for the Google engineers :
“Data-driven design is a vital tool for hill-climbing iteration of a site, but you should take great care not to use it as an appeals process whenever you and your designer reach an impasse. It sidelines the designer into being no more than a brainstormer, devoid of design ownership. I realize this is not the usual case, so just treat it as a cautionary tale. Also please keep in mind that everyone has opinions on design, and that your UX professional has devoted years of their life to learning to separate their subjective opinions from their objective understanding about how the larger audience will interpret an interface. It’s not as demonstrable as code that passes unit-tests, but trust in it anyhow.”
I agree testing (or any form of data driven design) can be counterproductive if you don’t learn from the past. Obsessive testing can slow down the creative process. As mentioned in a previous post, there are design and communications principles that have been around for a long time. They don’t necessarily need to be tested over and over again. Experts familiar with them should be given the authority to overrule over-eager data analysts. A knowledge and test management system can help here.
Test management systems usually include the following basic components :
Testing History : findings from previous tests should be documented consistently so that they can be stored in an insight repository. At the very minimum this can be a simple spreadsheet that lists all tests that have run in the past, their objective, hypothesis and the outcome.
Testing Pipeline : the testing pipeline keeps track of all the tests that are being planned. Depending on the number of tests you run and the complexity of your organization this can be anything from a simple spreadsheet that lists all planned tests with a short description, timeline and status to a system that is managed through sophisticated campaign management platforms.
Testing Briefing : Every test should have a briefing document that outlines the testing hypothesis, the test design, the timings, the anticipated benefit, cost and ROI and the owner. This detailed briefing document will standardize the inputs required to build the testing history and pipeline.
Guidelines : Storing the results of past tests in an insights repository is not enough. The findings need to be aggregated and written up in guidelines that are communicated throughout the organization. Google’s Ten principles that contribute to a Googley user experience is a great example of how to do this. Again, tried and tested principles do not need to be tested over and over again.
Testing Prioritization : New tests can be prioritized based on what we know already (through testing history and guidelines), what we have in the testing pipeline and what the potential ROI could be from the rollout. Prioritization of tests will prevent excessive testing.
Now Google is not your average company of course. I don’t know many companies who test too much. The opposite is usually the case. Most companies’ creative decisions don’t live or die by the sword of data, they are based on often very subjective expert opinions or even worse, on what web analytics author Avinash Kaushik calls the Hippo – the Highest Paid Person’s Opinion … .
This means companies tend to leave millions of dollars on the table. They don’t test new ideas and fail to build up a body of knowledge on what works and what doesn’t. A testing management system will help these companies as well. It can identify knowledge gaps and areas where mandatory tests are required. And it can institutionalize testing by making it an integral part of the creative process.
Reading your post, I dare say that measurement and analytics of any kind ONLY serve as kind of lighthouse in the dark. The greatest danger is when data is interpreted and used as the Final Word on the subjective aspects of communication.
Data on some levels kill Art.
For how can you quantify a feeling or perception?