Businesses across sectors and around the world witnessed rapid and dramatic changes in customer and business behavior during COVID. Historical data on sales, product, marketing or other operational performance metrics suddenly became an imperfect guide. Even in the best of times, attributing business results to the actions of your teams is challenging. So, how do you make sense of data as we continue to emerge from an unprecedented global pandemic?

Don’t Rely on History Alone

Companies often compare results – sales, bookings, churn or other metrics – before and after a product update, a new customer success initiative, a marketing campaign or another operational change to evaluate the success of that change. However, history often offers an unreliable comparison, and in 2020 and 2021, the data has proven to be particularly challenging. Even under normal circumstances, external events beyond your control – unexpected press coverage, changing customer demographics, seasonality, or any number of other factors – can throw a wrench into your analysis. During COVID, so much about market dynamics and buyer behavior dramatically shifted; comparing results before and after doesn’t make sense for most businesses.

Promote a Culture of Experimentation

Any data-driven culture is grounded in experimentation: testing changes and letting the data determine which option is best. You may have heard about – or even tried! – A/B testing on website button colors or email headlines. But the power of experimentation is not confined solely to marketing, where we see this approach applied more regularly. It matters in product development, in customer support, in operations and in countless areas across your business. With experimentation, some customers receive a new experience while others receive the existing “control” experience. Results between the two groups can be compared because they occur under the exact same external conditions – allowing you to see the true effect. As regions and businesses continue to reopen, the ability to understand what is working and why will help your team find signal through the noise so they can invest their time and resources more efficiently.

Understand – and Invest Behind – What Works

In these uncertain times, the value of experimentation in understanding change has become even more important. Resources, whether time or budget dollars, are scarce even in the best of times. As we continue to navigate COVID, most businesses need to be highly conscious of how they allocate resources. Here too, experimentation is key. While teams often consider experimentation in the context of optimization – picking the option customers like best – A/B testing can also be put to use to quantify the impact of product, process and other changes that can otherwise be challenging to assess. When was the last time someone in your product organization told you, “we changed something, we think customers like it, but we don’t know how much sales have changed”? Or, “it was only a back-end change, customers won’t even see it!” With experimentation, the difference in conversion, sales or any other success metric can be measured between those receiving the new experience and those who receive the status quo. Armed with this information, your team can invest talent and budget behind ideas and experiences that work, and drop the ones that don’t.

Innovation has always been important, but it is particularly critical today. COVID introduced significant secular changes in how people work, learn, shop, play and interact with one another.

And while many companies worked quickly to adjust, offering new products, services and delivery methods to meet customers’ needs, others are just beginning to understand the value of experimentation and how crucial it is to building a durable business. Importantly, experimentation done right not only fuels innovation and growth, it also serves to reduce risk – allowing teams to try new ideas, measure outcomes and make informed decisions. Experimentation can unleash the creativity of employees in every part of your business. Even the ideas that don’t work out often inspire your team to think harder about customers and what they need. And every once in a while, you might stumble on a winner.


Cathy Tanimura is Vice President of Analytics & Data Science at Summit Partners. She works closely with the Summit team and our portfolio companies to create and execute strategies that effectively use data and analytics to drive better decisions and build better products. She is the author of SQL for Data Analysis: Advanced Techniques for Transforming Data into Insights. Prior to Summit, Cathy worked at Strava, where she built an analytics and data science team focused on product, marketing and business development. Previously, she led analytics teams at Okta and Zynga.

Related Content

Additional frameworks and perspective from Summit Partners

Subscribe to The Ascent

The Ascent Newsletter


The Ascent is our quarterly newsletter offering practical perspectives to help leaders navigate the challenges and opportunities of building a growth company. Sign-up to receive the latest issue and stay up to date on the latest perspectives, stories and activity from across the Summit Partners Network.