What is the added value of experimentation?
5 questions about experimenting
- Article
- Customer Experience
Most organisations that offer digital products, experiences, and services conduct experiments, typically to increase conversions. In practice, alongside many other applications, we see the risks you face if you don’t experiment. In this article, you’ll find answers to the most frequently asked questions about experimenting.
What is experimentation?
By experimenting, you establish a relationship between the changes you make and the value they generate. The most well-known type of experiment is an A/B test, where you compare at least two groups or versions. You test a hypothesis that specifies what you're changing and which metric you expect to see the change in.
Previously, experimentation was mainly about conversion rate optimisation (CRO), primarily conducted in e-commerce environments with the aim of improving conversion rates.
How can you use experimentation in your organisation?
For example, service environments and offline contact centres are optimised through experimentation. This allows customers to increasingly solve their own issues within a service environment without needing to contact customer service. Customers find this convenient, while the organisation saves costs.
In apps designed to engage users with content, we also see organisations experimenting to increase usage and retention. Are you looking for higher retention? Higher adoption? More profit? Or perhaps greater customer satisfaction? There are many different metrics that can guide your experimentation programme.
What is the added value of experimentation?
Experimentation helps you discover whether changes are actual improvements. In practice, we often see changes made to digital products, experiences, or services based on gut feelings. Clear metrics are not defined in advance to measure whether they have the desired impact. This can present risks. The experiments may not add any value, fail to be prioritised for implementation, or even have a negative impact on the bottom line.
For example, running an experiment that doesn’t contribute to achieving your business goals, or worse, implementing a decision based on gut instinct and later realising it has damaged your business objectives.
Experimentation helps mitigate these risks and provides transparency about the impact of changes. It enables you to learn more about your customers and the relationship between the changes you make and their impact on the organisation.
Thus, by experimenting, you commit to establishing a link between the solution you devise and the metric you aim to influence. This shifts the focus from the solution itself to its effect. Experimentation has a significant impact on the way an organisation operates. The desired effect becomes central. Additionally, you learn what does and doesn’t work, ensuring that only changes with the desired impact are implemented.
3 practical examples of experimentation
Higher online conversion
It was expected that website visitors wanted a discount promotion on a telecom provider’s landing page. We decided to display the promotion at the top of the page. The A/B test showed that this adjustment led to more conversions than the page without the promotion.
Cost savings through more self-service
The contact centre of a telecom provider received many questions about renewing contracts. We added information about this to the website so customers could look it up themselves. The number of contacts with the centre decreased, and we were able to calculate the exact cost savings based on the average cost per contact.
Preventing revenue loss
A key stakeholder wanted to add a banner to the homepage, giving information about product bundles. We tested this through an A/B test and found that the proposed changes resulted in fewer sales. So, the old version remained live, preventing the company from losing revenue.
How does experimentation work in practice?
When you start experimenting, you follow a continuous process that involves several steps. This includes conducting preliminary research, formulating a hypothesis, and analysing experiment results. The scientific method of experimentation serves as the main guide, consisting of six steps:
- Observation: You examine various signals in the data, or from your expertise, to assess the situation on the web.
- Research Question: Based on the insights from the observation, you formulate a research question.
- Hypothesis: From the research question, you formulate a testable hypothesis.Step
- Experiment: You set up and execute the experiment, ensuring it aligns with your
- Analysis: Once the experimental data is collected, you analyse it to determine whether the hypothesis was correct.
- Conclusion: You interpret the insights from the analysis and decide your next step.
Where are most organisations now when it comes to experimenting?
As a data consultancy, we see significant differences in organisations' maturity regarding experimentation. In organisations where customers complete a large part of their digital journey themselves, experimenting is a key part of the strategy. Think of hotel websites, online shops, banks, insurers, and digital governments.
With the large numbers of customers, interactions, and conversions, experiments make a big difference. The ability to make mistakes and learn from them is a crucial part of the organisational culture.
On the other hand, there are plenty of companies that rarely experiment. They rely on gut feelings and continue doing what they’ve always done. Innovation and change are much slower here, as a lot is invested in changes that don’t work or even have a negative impact on organisational goals.
Improve your decision-making, start experimenting
Do you want to learn in a reliable, data-driven way whether something actually works? Then you need to start experimenting. This increases your focus, helps you make better decisions, reduces risks, and ensures continuous learning. You’ll know what does and doesn’t work, avoiding unnecessary “optimisations” that don’t bring you any closer to your goal.
Contact us here if you want to discuss the opportunities of experimentation for your organisation.
This is an article by Sebastiaan van Buitenen
Sebastiaan is an experimentation expert passionate about creating valuable digital experiences for both users and organisations. His hands-on experience in experimentation enables him to elevate organisations to the next level, both in the quality and quantity of their experiments. With Sebastiaan on your team, experimentation will get a significant boost. His approach, informed by his background in applied cognitive psychology and a focus on research-driven optimisation and psychological heuristics, makes him a reliable partner for driving impact.
Receive data insights, use cases and behind-the-scenes peeks once a month?
Sign up for our email list and stay 'up to data':