Lindsay Matthews
Data-driven product development - A/B testing in the enterprise

Video URL: https://vimeo.com/152729425

Discussion: Issue #13

Come learn from Yammer’s long history with data-driven development, packed with real examples. Hear about our experiences using a/b testing, including everything from small design changes to big new features. We’ll cover how to pick what to test (and what not to), running a multivariate test, identifying metrics and analysing the results. productmanagementfestival.com

Our notes:

• measure what’s important: it’s easy to get people to click on a big flashing red button, but that’s not the goal. always track & measure how an improvement contributes to the main goal (and for that, you need to know what that is)
• write down the goal ahead of time, before you’d design and start the test
• “data doesn’t tell you why something’s happening, it only tells you what’s happening” – you need to come up with theories and validate them by data, interviews or further tests
• you will be wrong often – that’s just how A/B testing works
• learning something is the goal with testing, not shipping software! (create a culture where it’s OK to have bold ideas and make mistakes)
• have an opinion! you can’t A/B test everything (but validate with data when it’s important or you’re uncertain)
• product vision helps you get over local maxima: things might test badly and make things worse in the short term, but help us achieve what we want to do in the long term
• main summary of the talk:
  1. celebrate learning, not shipping (fckk, we are horrible at this )
  2. find metrics that work for your product – but your entire product, not a specific feature
  3. create short, testable assumptions

Join us any time and get to be a better product person!