6.4 Testing your form in real-life conditions
16-Sept-2022
2 mins
The most important component of testing your survey is to do so in real-world conditions (pilot the survey). This stage allows to review If the pilot data corresponds to your expectations and is also usable in your analysis tools and compatible with your analysis plan. The pilot should be as close to real conditions as possible: same mobile devices, same context, same type of respondants, etc.
It will also help you to understand what might bias your answers and conclusion. On this point you should see this Technical note: errors and bias in surveys made by Terre des Hommes that can help you identify bias and how to mitigate them. It is important to remember that the mode of data collection used (traditional face-to-face interviews (F2F) or computer-assisted telephone interviews (CATI) on mobile phones) or the gender of the enumerator can create biases as highlighted in this article. For more information on bias, please refer to 5.2 Understanding potential biases.
TABLE OF CONTENTS
Role-playing in real situation
A good way to test your form is to role-play it during enumerator training. This not only allows you to get a feel for the content of the form in a real-life situation, but also to identify aspects of the questionnaire that may be misunderstood.
A good way to do this is to have one person answer the questions with a little deliberate deviation from the expected answers and see how the ‘enumerators’ react, how they catch up, identifying potentially missing guidance from the form and/or wording that needs to be revised. One way of working is to have each enumerator take turns with a question, while all enumerators fill in the answers accordingly.
It is often very useful for the trainer to plan a debriefing session after the role-playing, both so that each person can indicate the misunderstood aspects that were identified during the exercise, and for the trainer to review the results of the data collection to see if there were any discrepancies between what the enumerators have filled in compared to what they should have filled in.
Field Pilot
One of the best methods of testing a form which is complete is a pilot survey. Pilot surveys are exactly the same as the real data collection and take place in one of the locations which is part of the real data collection. Pilot surveys should take place at least a few days before the real thing in order to allow for adjustments and corrections to be made to the form before final deployment.
For surveys that will be conducted on a large scale, particularly where they will be conducted across multiple sites and organisations, it is necessary as a final form testing step to conduct a pilot data collection with a group of enumerators. This will test not only the form of the questionnaire, but also the quality and contextual relevance of any data or indicators calculated using the resulting data.