This section explains the most effective ways to carry out an evaluation of your debt advice service and focus on some important issues, such as client engagement, how to select clients and ensuring validity of your findings.
Our recommendations have been drawn from the extensive research used in developing this evaluation framework and in its testing with clients and frontline advisers.
There are potentially three ways that you could use the evaluation framework (these can be used in combination with each other):
- A one off annual evaluation eg. a snapshot of clients seen within that period
- Regular on-going evaluation eg. every 3, 6 months. Interviewing clients who have had recent advice
- Longitudinal surveys eg. re-visiting the same clients and tracking progress towards outcomes
The survey has been designed to be used in multiple contact channels (eg. face to face, telephone and online debt advice interventions), so will be adaptable depending on both your client preferences and your internal resources. For ease, we suggest you select one preferred methodology where possible. However, we understand that many client bases will be different and will require different or mixed approaches. The main options are as follows:
Online is the suggested methodology of survey completion if it is possible to do so. It represents the most accurate, quickest, cost effective and least labour intensive method of completion.
Additional advantages include the fact that clients can complete the survey in their own time, and that it is easy to send out reminders. Project completion is quick, with the vast majority of responses likely within a week after launch and a reminder may be needed to maximise the response.
However, challenges will include those who are digitally excluded, i.e. they do not have access to a computer or the internet. Even if access is present, it was apparent during our testing that there were a proportion of clients that do not possess an email address. For these customers, an offline channel will need to be used if you wish to survey them. Additionally, if literacy is an issue, self-completion could be a challenge.
We have signed up with SmartSurvey, a UK based online survey provider to create an online version of the client survey. A copy of the survey can be made available for your service to use, subject to a subscription charge. Charges vary depending upon the length of your required subscription and whether your service is not-for-profit (discount available) or commercial. Charging details and information about SmartSurvey can be found on their website. If signing up with SmartSurvey, a copy of our client survey can be found within your account. This is one of many online survey providers so you may wish to use one you are more familiar with to build an online version of the client survey.
Once you have your results from using any online survey you will need to input figures of the data into one of the excel based tools as detailed in the Analysis - Analysing online completed questionnaires section.
Paper self-completion by post
Similar to online, this methodology enables the client to complete the survey in their own time. No technology is required as clients would be posted a copy of the questionnaire with a covering letter asking them to return it.
There are some costs associated with postal research such as printing, stationary, postage and free-reply envelopes. If a reminder letter is needed, this will increase the costs. Postal research is also slower. Customers may sit on the questionnaire for two or three weeks before returning, which coupled with postage lead times, can mean that a meaningful response can take as much as a month. If reminders are needed, then this time is increased.
This methodology relies on good levels of literacy. If this is relevant for your service there will be a risk that they will not return, potentially biasing the survey. In these cases, a different methodology should be considered to capture their thoughts.
See Analysis - Analysing manually completed (paper) questionnaires, we have illustrated how best to take the responses you receive from the surveys and turn them into results and measure the outcomes achieved.
Face to face (via paper survey), client self-completion or assisted completion
This methodology is suitable for clients who are unable or unwilling to complete a self-completion survey. This would involve a paper version of the questionnaire being asked in an advice setting, either by self-completion on site, or by an advisor administering the survey in conjunction with the client.
There may be a risk of bias here, potentially if the client feels uneasy to provide a negative answer to the advisor. One solution is for a different independent person to carry out the survey, such as a support worker, where possible. See Analysisfor guidance on the best way to analyse the data.
Telephone is the most challenging way to get client feedback using the framework. It will require significant resource allocated to calling customers and then spending time completing the survey with them. In our pilot we found that it can take between 5 to 10 calls before someone agrees (or has time) to complete the survey. We also know from research that in certain situations, some clients prefer to avoid answering telephone calls. Given this, subject to available funding, the most effective way to undertake telephone research would be by outsourcing to a research organisation that would use a software system (Computer Aided Telephone Interviewing, or CATI) to conduct surveys on screen, with results automatically ready for analysis.
If outsourcing is not possible then you can undertake this in house with the interviewer completing the paper survey based on the respondent’s answers. However, this method will require additional internal resources and quality control measures.
Ensuring that customers are engaged with the survey process is vital in getting good response rates and useable data to analyse. Suggestions to help improve levels of response include:
- Avoid ‘cold’ contact with customers. If they do not know a survey is likely, engagement may be low and refusal rates can be high. If possible, send out a letter or email contextualising the advice session that the client received in advance of the survey going out to get early client buy in. If possible, at the end of an advice session, we would encourage seeking consent to contact for evaluation and mention that a survey might be possible.
- In your correspondence with your clients, it is helpful if the respondent has a ‘hook’ to increase engagement. Some surveys offer an incentive or prize draw (although we are not recommending that you need to do this), and for many it is enough to know that their input will help improve the services offered. Additionally, some may find it engaging if there is promise to share back some of the key data after research along with “what we are going to do to improve it” or similar.