Businesses across sectors and around the world have witnessed rapid and dramatic changes in customer and business behavior during COVID. Historical data on sales, product, marketing or other operational performance metrics are suddenly an imperfect guide. Even in the best of times, attributing business results to the actions of your teams is challenging. So, how do you make sense of data during an unprecedented global pandemic?

Don’t Rely on History

Companies often compare results – sales, bookings, churn or other metrics – before and after a product update, a new customer success initiative, a marketing campaign or another operational change to evaluate the success of that change. However, history often offers an unreliable comparison, and in 2020, we’d venture to say it’s virtually useless. Even under normal circumstances, external events beyond your control – unexpected press coverage, changing customer demographics, seasonality, or any number of other factors – can throw a wrench into your analysis. During COVID, so much about market dynamics and buyer behavior has dramatically shifted; comparing results before and after won’t make sense for most businesses.

Promote a Culture of Experimentation

Any data-driven culture is grounded in experimentation: testing changes and letting the data determine which option is best. You may have heard about – or even tried! – A/B testing on website button colors or email headlines. But the power of experimentation is not confined solely to marketing, where we see this approach applied more regularly. It matters in product development, in customer support, in operations and in countless areas across your business. With experimentation, some customers receive a new experience while others receive the existing “control” experience. Results between the two groups can be compared because they occur under the exact same external conditions – allowing you to see the true effect. As regions and businesses reopen, the ability to understand what is working and why will help your team find signal through the noise so they can invest their time and resources more efficiently.

Understand – and Invest Behind – What Works

In these uncertain times, the value of experimentation in understanding change has become even more important. Resources, whether time or budget dollars, are scarce even in the best of times. During COVID, most businesses need to be highly conscious of how they allocate resources. Here too, experimentation is key. While teams often consider experimentation in the context of optimization – picking the option customers like best – A/B testing can also be put to use to quantify the impact of product, process and other changes that can otherwise be challenging to assess. When was the last time someone in your product organization told you, “we changed something, we think customers like it, but we don’t know how much sales have changed”? Or, “it was only a back-end change, customers won’t even see it!” With experimentation, the difference in conversion, sales or any other success metric can be measured between those receiving the new experience and those who receive the status quo. Armed with this information, your team can invest talent and budget behind ideas and experiences that work, and drop the ones that don’t.

Innovation has always been important, but it is particularly critical today. COVID has introduced significant secular changes in how people work, learn, shop, play and interact with one another.

And while many companies worked quickly to adjust, offering new products, services and delivery methods to meet customers’ needs, others are just beginning to understand the value of experimentation and how crucial it is to building a durable business. Importantly, experimentation done right not only fuels innovation and growth, it also serves to reduce risk – allowing teams to try new ideas, measure outcomes and make informed decisions. Experimentation can unleash the creativity of employees in every part of your business. Even the ideas that don’t work out often inspire your team to think harder about customers and what they need. And every once in a while, you might stumble on a winner.

ABOUT THE AUTHOR

Cathy Tanimura is Vice President of Analytics & Data Science at Summit Partners. She works closely with the Summit team and our portfolio companies to create and execute strategies that effectively use data and analytics to drive better decisions and build better products. Prior to Summit, Cathy worked at Strava, where she built an analytics and data science team focused on product, marketing and business development. Previously, she led analytics teams at Okta and Zynga.

Planning for 2021

Additional frameworks and perspective as you plan for the year ahead

Stay in Touch

Stay on top of the latest insights, interests and updates from Summit Partners.