The technical deep dive of scaling experimentation effectively

If the technical setup of an experiment isn’t sound, the results can be unreliable. You may end up drawing conclusions based on flawed data, leading to poor decision-making. In addition, technical issues such as flickering or slow loading times can negatively impact the user experience. This undermines the very purpose of experimentation: improving the user experience based on reliable data. It is therefore essential to ensure your technical setup is properly in place.

It’s wise to check your setup if:

• You don’t fully trust the test results, for example due to concerns about data quality.
• You know you haven’t implemented a specific solution to identify users on your website.
• You’re aware of recurring issues such as flickering or performance problems during tests.
• You’re not sure whether your technical setup has been properly configured in the first place.

During a check, we examine several aspects of your setup based on a number of research questions.

1. We check the data quality in the analytics tool:

• Are important conversions being measured correctly?
• Have events related to the experiments themselves been implemented, and is everything working properly?

2. We also look at the testing tool:

• Are there issues with Sample Ratio Mismatch (SRM)?
• Do you get the expected outcomes when running A/A tests?
• Is the tool loading correctly, and is there proper integration with the data layer?
• How are tests triggered and coded?
• What are important considerations for the organisation when it comes to client-side versus server-side testing tools?

3. We look at topics that are important for both the testing tool and the analytics tool:

• Is there a solution in place to recognise users as accurately as possible (including those who don’t log in)?
• How are the tools integrated with users’ consent choices?

If you're looking for more technical support, feel free to get in touch with us. We can help improve your data quality, set up a visitor identity service, eliminate or reduce flickering, build test code with minimal impact on performance, and advise you on choosing a new tool.

We also look beyond this specific focus. This technical check is ultimately part of scaling experimentation effectively: a solution to take experimentation within your organisation to the next level of maturity. Read more about the full solution here.

We have two phases we can assist you in. Phase 1 is a basic setup for your experimentation, while phase 2 offers an improved maturity, higher experiment quality and higher winrate. The final cost may depend on how far along your existing setup is.

You can only run a good experiment when you have good data. With our broad knowledge in the fields of Customer Experience, Data Analytics, Technical Web Analytics, Data Engineering, AI & Data Science, Data Strategy and Product- and Project Management, we can support you with every data challenge.

Is your data quality not sufficient to run high-quality experiments? Don’t worry, we are here to help!