Setting up Trials for Success: Why Sample Size Matters
Fri Oct 04 2024
Author
Alok Joshi
One may not be the voice of many, and many may not be the voice of one. When setting up trials to test out a hypothesis, it is important to find the sweet spot of a sample size that will accurately represent the whole cohort. If you are going to be basing decisions from the results of your trial, getting your sample size right matters as it gives more consistency between tests, reliability for confidence and avoids bias ensuring more accuracy.
What is sample size?
The sample size refers to the number of individual observations or participants included in a trial.
When conducting a trial, you generally take a subset of the overall population and split them into two groups: the control group (the group that remains status quo) and the treatment group (the group that has the intervention of the hypothesis you are testing). This is so you can help isolate the impact of the intervention. How to select these groups is another important consideration for trials all together, but we’ll cover this another article.
Let’s say for example, we were looking to trial a new dynamic pricing algorithm for a retailer and assume we have 1000 stores.
It may not make practical sense to utilise the entire fleet of stores for the trial. We effectively want to assess the effect whilst using the least amount of stores in the treatment group whilst maintaining statistically valid results. Some reasoning behind this:
- It can be prohibitively expensive to roll out changes to all stores just to test the new pricing
- We do not understand if the change is a good one - if we inadvertently roll out these to half of the stores (500), and the results were unfavourable, then there would be a huge cost to the business
- There can be diminishing returns to gathering more samples over a certain threshold which means the insights you gain become incrementally smaller
- You may be constrained by technology enablement e.g. only 80 stores have electronic shelf labels that are needed for showing a dynamic price, but what if the ideal sample size is 150?
- If time is one your side, increasing the length of your trial can be a suitable method of minimising the number of stores impacted
On the other hand, if we chose too few stores - say we chose a single store for the treatment and a single store for the control group. It is unrealistic to generalise the insights to all the 1000 stores just from these two samples. In this case, we cannot rely on the results as the difference in performance for these stores is likely due to noise.
As with all things - there is a trade-off between using too few and too many samples. Hence, being able to efficiently determine the right sample size is an important step to creating and evaluating our trial and its results.
So what can influence sample size?
This is where the statistical methodology you use and experience come into play. Let’s look at some factors to consider in relation to our dynamic pricing trial example. Along with the length of the trial you are running, you would need to consider things like:
- Effect Size:This is the size of the difference you’re trying to detect. If the effect size is large, you can use a smaller sample size. If it’s small, you’ll need a larger sample size to reliably detect the difference. Using our dynamic pricing example, you would need to determine the magnitude of change to the current pricing and its subsequent impact to sales (price elasticity modelling). If we are anticipating a big impact from a price change, you may not need as many samples to detect the effect.
- Statistical Significance:This is the likelihood that your results are not due to chance. This is where "p-value" comes in. The p-value is a statistical measurement used to validate a hypothesis against observed differences between the control and treatment groups. The lower the p-value, the greater the statistical significance of the observed difference. There needs to be a strong correlation between the intervention you are making and the anticipated impact. With pricing it would be safe to say that lowering pricing will bring more sales. But how much lower will bring this effect? And if the price was not low enough, and sales still increase, was it because of another factor e.g. an event or marketing campaign? Eliminate as much of “the chance” by finding this threshold.
- Statistical Power:This measures the probability of correctly identifying an effect if there is one. Higher power requires a larger sample size. Think about how many stores you would need to be confident in measuring the sales effect of changing pricing? To get higher confidence you will naturally need more stores. But how many is the question.
The above is a lot to take in - but luckily jahan.ai have simplified the whole process with jahanTestLab. We can help navigate through the complexities of setting up a trial in an easy to use and intuitive UI to guide you through the setup, automating and optimising configuration decisions for you such as the actual products or locations to run a trial in, the control/treatment group participants, sample sizes and key metrics to track and monitor.
Contact us if you would like a demo of jahanTestLab in action.
Now, if you are wondering how you determine the sample sizes, stay tuned for our technical write up, where we’ll go into the next level of detail on some of the data science and calculations needed to come up with actual sample size!
#businessexeperimentation #trials #abtesting #samplesize #testandlearn #jahanai #retail #ai #machinelearning
At jahan.ai, we build end-to-end AI twins for retail and supply chain businesses. We take the concept of a digital twin, a virtual replica of business processes, further with advanced AI, which not only mirrors these processes but also automatically optimises them. In this way, we support businesses in driving efficiency, going beyond their potential.