Local surveys
Existing Survey Evidence:
Existing data sources may not provide the information you need at a local level. In geographically focused regeneration programmes, neighbourhood satisfaction is a key issue, as are fear of and experience of crime and antisocial behaviour, and lifestyle choices which affect health such as smoking, exercise and the consumption of fruit and vegetables. There is no existing data source which will provide this type of information at small area level. There is, however, national and local authority level data on these types of issues in the Scottish Household Survey, the Scottish Crime Survey and the Scottish Health Survey which provides a useful benchmark.
It is worth checking with other partners whether they have conducted any recent surveys, such as NHS Trust surveys on perceptions and behaviours.
Why Commission a Local Survey:
Outcome agreements emphasise measurability, and surveys of the local population are the best way of providing this for issues which are not covered by existing standard data sources. They can be a very useful tool for identifying issues of concern to the local community. And repeat surveys can be used for measuring changes in people’s perceptions, experiences and behaviour for outcome indicators.
What to Include in a Survey?
It is important to make sure that all the information you need is being gathered through a locally commissioned survey. This requires significant forward-planning.
Here are some tips on how to make sure the information is as useful as possible:
• Use questions asked in national surveys such as the Scottish Household Survey, so that local results can be compared to local authority and national figures.
• Before designing a questionnaire think carefully about how the results will be analysed. This makes it easier to input the results at the end of the process, and may help to structure the questions – what coding should be used for the responses, and which issues need an open question? In general it is best to try to keep open questions to a minimum, as they are generally difficult to analyse and aggregate responses across a large number of survey respondents.
• Use questions that will provide information which could be used for outcome indicators. This can be difficult to predict, particularly if the survey is being used as a means of identifying suitable priorities. One way to deal with this is to choose questions that relate to experiences rather than attitudes. These are more likely to show change over the short periods involved in most regeneration projects and programmes.
• Be careful with the language used to make sure local people can read and understand it easily. It is always best to pilot a survey, even if this just means sending it out to one or two of the community representatives to see whether they find it easy to complete. If there are optional questions for different groups, it is important to make sure these routings are clear and easy to use. Bear in mind that for some people English may not be their first language, and some may have literacy difficulties.
• People will generally be more likely to complete short questionnaires which are clearly relevant to them. This includes making as much use of tick-boxes rather than lots of open ended questions.
• Consider carefully the area to be covered by the questionnaire, the population in this area, and the number of surveys to be conducted. The number of responses is crucial for determining the validity of your results. Sample sizes of under 350 are likely to have poor statistical validity. If the sample is to be divided at all, for example, to consider the experiences or concerns of particular groups, then you should consider this at the design stage. You may need to ‘boost’ the sample for particular groups, to make sure there are enough respondents from each group to ensure statistically valid results.
• You should also consider how to distribute questionnaires to ensure that the sample is as representative as possible. This is best achieved through random door-to-door surveys, with repeat calls when there is no answer the first time. However, this can be costly. Another cheaper option is a quota survey, with interviewers being required to meet quotas of the different types of people who live in the area, so that the results are obtained from a representative sample of the local population. This can only be done if you have up to date information about the makeup of the area’s population, for example from a recent Census. If the Census was a long time ago or if you have reason to believe that there has been a lot of change recently in the area’s population, a quota sample may be less appropriate.
If a postal survey is used, it is worth remembering that this may not reflect all groups in the population. Particular groups – for example those with more time on their hands or those who have a pressing concern about a particular issue – may be more likely to respond. This can give you a biased result. This form of survey won’t provide statistically valid samples, but it may still provide useful information on some key issues within the local community.
• If the survey is likely to be repeated, it may be worth considering a panel survey, where you return to the same person at the same address each time . Where the person has moved on, the original sample will shrink and this will mean that you need a larger sample. Some local authorities or CPPs may have set up a panel of this type at local authority level. This type of panel survey can be useful if you want to track changes in perceptions, experiences and life styles among people who have lived in the area over the period of change, as well as finding out how much the population of the neighbourhood has changed.
The design of a panel survey is quite specialised, and you may need advice, particularly if you decide to refresh the sample with new replacements of a similar type and to track people who move away. It may be particularly useful in an area where major change or a major intervention is planned.
The design of surveys and questionnaires needs a lot of thought, and you should think ahead and seek further advice to make sure all the relevant information is being gathered. This includes consulting partners who may have ideas for other questions or indicators which they may be working towards.
Another reason to consult all locally active partners is that some may also be planning a local survey, or a local authority wide survey. By co-commissioning a survey, with boosted samples in the regeneration areas where necessary, you may reduce your costs significantly. Partners may also have conducted other surveys recently which may hold additional useful information.
There are many sources of information on the internet that provide information and advice about how to design surveys and questionnaires. Two useful books are: Real World Research by Colin Robson (Blackwell Publishers, 2002) and Questionnaire Design, Interviewing and Attitude Measurement by AN Oppenheim (Continuum International Publishing Group, 2000). More detailed information about sampling is available from the Civil Service Policy Hub website, in the Magenta book on policy evaluation. This has a chapter with very useful information on survey sampling.
You should ensure that repeat surveys use the same methods and, where necessary, identical questions as the initial survey, to ensure comparable results. This is why it is so important to think it through carefully from the outset. Monitoring change over time is difficult if different sampling methods and questions have been used.
Possible Alternatives to Surveys:
There is a significant cost implication with surveys. If costs appear disproportionate to the scale of the intervention being contemplated, other options may have to be considered. For example, community representatives or local contacts may indicate that neighbour disputes are an issue in a community but survey work is felt to be disproportionately costly. The outcome goal might be a decrease in the level of neighbour disputes with the indicator being the change in the number of complaints to housing or environmental services departments.
It may also be useful to consider what information is held by other agencies, including those that are not directly involved in a regeneration initiative. For example, housing associations or the Council’s housing department may hold information about their tenants that could be useful. This may include information about particular equalities groups, or household types. They may also have undertaken their own tenant satisfaction surveys which may have useful information, particularly in relation to neighbourhood satisfaction.
The Importance of Survey Information:
Household surveys add considerably to the evidence base for regeneration outcomes. The cost of a household survey need not be excessive and there is an expectation that monitoring and evaluation should be built into a programme of any scale. Policy is increasingly built on an evidence based approach and surveys can play a very important role in providing evidence of impact. If a survey is undertaken it needs to be robust, with a representative sample of the population in the relevant community.
