Time period of research and sampling
The time period from which you select respondents is crucial to getting a balanced view of how successful advice has been. For example, if you were to interview respondents the day after advice, this would be too soon for a client to digest everything that needs to be done and is unlikely to have done anything significant as a positive action. However, they would have a fresh recollection of the experience of the actual advice process.
On the other extreme (i.e. advice taken a year ago), you would have the opposite issue, in that actions may have been completed, but recollection of the actual advice experience will be very low. Our recommendation would be to select customers who have had advice between 2 and 6 months ago.
- There is some flexibility on the time periods depending on the total number of customers – in some cases, you may need to do a longer period (e.g. up to 9 months) if it is required to achieve a good base.
- When analysing, ensure the time periods are comparable. For example, there may some seasonal or operational reasons why certain weeks may expect to have higher demand or other factors that could influence higher or lower scores.
Sampling and sample sizes
When conducting any research, it is important that you have a robust approach to sampling to ensure you have a high degree of confidence within the data. The approach will vary depending on the total number of customers in your ‘sample universe’ (i.e. the total number of people that can be possibly surveyed).
A ‘sample universe’ is the total number of people within the market you want to represent. For example, if you want to represent adults in the UK, your sample universe is 49m. If you have 1000 clients and wish to represent all of them, your universe is also 1000. However, if for example you have a 50/50 split of males and females in your client base and wish to look at the differences between them, your universe size reduces to 500 for each. Similarly, if you only wanted to interview clients from one quarter (assuming an even split in the year), your universe size reduces to 250 (1000/4).
If you have a small universe (for example 100 or 200 clients), it would be advisable to select all customers to approach for research, and then try to achieve as many returns as possible from them. This could be a challenge if you need the majority of a population to respond, so engagement will be critical (see Methodology - Client engagement). For a larger universe, you will need to select a ‘sample’ of customers who will represent the entire base. You will note from the below table that ideal sample sizes diminish regardless of base size, so those with a very large client base will not need to achieve proportionally larger sample sizes to get the same robustness. For larger bases, we would still recommend trying to include the entire sample universe where possible to ensure an accurate spread of responses.
There may be some instances where it is not be appropriate to survey them given their circumstances in which case they should be omitted or approached at a different time. However, clients shouldn’t be selected on whether they are likely to give good or bad answers. Where you choose not to go to the entire base for research, clients should be randomly selected to avoid any potential bias.
50 to 75 people should be your minimum target sample size in all cases as anything below this will not be considered robust. You can use the table below as a rough guide to the ideal sample sizes needed.
The below table shows the variation of confidence intervals depending on the sample size used. As a general rule, the higher the sample size in relation to the sample universe, the higher the reliability will be. For example, a sample size of 80 on a universe of 100 will give a reliability of 5% either side of the result. So a 50% answer will actually be between 45% and 55%.
|Base size of ‘universe’
|Sample size required to be 95% confident of +/- 4%
|Sample size required to be 95% confident of +/- 5%
|Sample size required to be 95% confident of +/- 6%
|Sample size required to be 95% confident of +/- 7%
Example: Agency Z has a client base of 1000 clients and wish to review their clients seen in the last 6 months. They saw 500 clients in that period (which is their sample universe). Ideally, a sample size of 170 would be robust, meaning you would need responses from 1 in 3 people. At this level, we can be 95% certain that a respondent’s answer is within +/- 5%, so if the combined answers to a question is 80%, the true answer will lie somewhere between 75% and 85%. However, they then actually achieve a slightly lower response rate of around 1 in 4, with 130 responses. This is still robust data to work with, but the data has a wider variance of +/-6%.
Sampling golden rules
- Try to include your entire universe of clients wherever possible
- If taking a sample of your base, randomly select them, and do not cherry pick
- Aim for a minimum of 50 to 75 completions out of any universe (although sample sizes of this size may still have a large variance)
- Ideally, aim for a confidence interval of +/-5% or better
- If customer numbers permit, aim for 200 completions which gives a robust view for most situations
More in 'Debt advice evaluation toolkit'