But what are those other changes then, if not experiments?

Often these are optimisations. This is the process of adapting what you have to be more effective or get new results.

What do optimisations look like? 

• Changing a targeting setting in a running Meta ads campaign, e.g. excluding a certain audience or broadening the age range

• Trying a different type of subject line in the weekly newsletter 

• Adding negative keywords to improve a Google Ads Search Campaign 

• Shortening some copy on the website

• Added a new FAQ question to the website

This doesn’t mean we don’t measure the performance of those areas, but rather that we don’t track it in our weekly experiments sheet and don’t define a hypothesis or measure of success. We don’t compare it to a benchmark or a variant to decide if we should continue with that change.

This might seem like a tomato tom-ay-to distinction. I agree, the line is blurry between optimisations and experiments. Sometimes just having a larger website or bigger risk situation would cause some of these to be classified or run as experiments - e.g. A/B testing the subject line format against another. It’s all about the risk vs reward.

Example of how it may differ per company

Let’s give it a go with that first example: changing a targeting setting in a running Meta ads campaign.

Scenario 1: This is a huge campaign with a £20,000 monthly ad budget, and improving ad performance is the key focus of the quarter. So you would classify the change as an experiment, monitoring the performance the week before and after the change, or using the Meta ads experiment feature to run two variants.

Scenario 2: This is a small campaign with a £600 monthly ad budget. It is running fine, and your focus right now is on retention. You’ve noticed some overlap with another campaign, so you’ve decided to classify it as an optimisation and just change it.

Questions to ask yourself

This might still seem like definition picking, but this is one of the challenging aspects of running experiments.

If you aren’t sure whether it should be an experiment or optimisation, ask yourself the following questions:

1. Is this a key focus area for this quarter?

2. Is there a big risk or reward involved in this change?

3. Can we measure the impact of this change?

If the answer is yes to all three of these questions, you’ll want to define it as an experiment. If not, it’s an optimisation.